# Insulator-to-Metal Crossover near the Edge of the Superconducting Dome in
Nd1-xSrxNiO2

Yu-Te Hsu<EMAIL_ADDRESS>High Field Magnet Laboratory (HFML-EMFL) and
Institute for Molecules and Materials, Radboud University, Toernooiveld 7,
6525 ED Nijmegen, Netherlands  Bai Yang Wang  Department of Physics, Stanford
University, Stanford, CA 94305, United States  Stanford Institute for
Materials and Energy Sciences, SLAC National Accelerator Laboratory, Menlo
Park, Stanford, CA 94025, United States  Maarten Berben  High Field Magnet
Laboratory (HFML-EMFL) and Institute for Molecules and Materials, Radboud
University, Toernooiveld 7, 6525 ED Nijmegen, Netherlands  Danfeng Li
Stanford Institute for Materials and Energy Sciences, SLAC National
Accelerator Laboratory, Menlo Park, Stanford, CA 94025, United States
Department of Applied Physics, Stanford University, Stanford, CA 94305, United
States  Department of Physics, City University of Hong Kong, Kowloon, Hong
Kong, China  Kyuho Lee  Department of Physics, Stanford University, Stanford,
CA 94305, United States  Stanford Institute for Materials and Energy Sciences,
SLAC National Accelerator Laboratory, Menlo Park, Stanford, CA 94025, United
States  Caitlin Duffy  High Field Magnet Laboratory (HFML-EMFL) and Institute
for Molecules and Materials, Radboud University, Toernooiveld 7, 6525 ED
Nijmegen, Netherlands  Thom Ottenbros  High Field Magnet Laboratory (HFML-
EMFL) and Institute for Molecules and Materials, Radboud University,
Toernooiveld 7, 6525 ED Nijmegen, Netherlands  Woo Jin Kim  Stanford Institute
for Materials and Energy Sciences, SLAC National Accelerator Laboratory, Menlo
Park, Stanford, CA 94025, United States  Department of Applied Physics,
Stanford University, Stanford, CA 94305, United States  Motoki Osada  Stanford
Institute for Materials and Energy Sciences, SLAC National Accelerator
Laboratory, Menlo Park, Stanford, CA 94025, United States  Department of
Materials Science and Engineering, Stanford University, Stanford, CA 94305,
United States  Steffen Wiedmann  High Field Magnet Laboratory (HFML-EMFL) and
Institute for Molecules and Materials, Radboud University, Toernooiveld 7,
6525 ED Nijmegen, Netherlands  Harold Y. Hwang<EMAIL_ADDRESS>Stanford
Institute for Materials and Energy Sciences, SLAC National Accelerator
Laboratory, Menlo Park, Stanford, CA 94025, United States  Department of
Applied Physics, Stanford University, Stanford, CA 94305, United States  Nigel
E. Hussey<EMAIL_ADDRESS>High Field Magnet Laboratory (HFML-EMFL) and
Institute for Molecules and Materials, Radboud University, Toernooiveld 7,
6525 ED Nijmegen, Netherlands  H. H. Wills Physics Laboratory, University of
Bristol, Tyndall Avenue, Bristol BS8 1TL, United Kingdom

###### Abstract

We report a systematic magnetotransport study of superconducting infinite-
layer nickelate thin films Nd1-xSrxNiO2 with $0.15\leq x\leq 0.225$. By
suppressing superconductivity with out-of-plane magnetic fields up to 37.5 T,
we find that the normal state resistivity of Nd1-xSrxNiO2 is characterized by
a crossover from a metallic $T^{2}$-behavior to an insulating
log(1/$T$)-behavior for all $x$ except $x=0.225$, at which the resistivity is
predominantly metallic. The log(1/$T$)-behavior is found to be robust against
magnetic fields, inconsistent with scenarios involving localization or Kondo
scattering, and points to an anomalous insulating state possibly driven by
strong correlations. In the metallic state, we find no evidence for non-Fermi-
liquid behavior arising from proximity to a putative quantum critical point
located inside the superconducting dome.

The realization of superconductivity in infinite-layer nickelates [1] has
created a new model system for studying unconventional superconductivity [2,
3, 4, 5, 6, 7, 8, 9, 10, 11]. While the defining framework of the high-$T_{\rm
c}$ cuprates, namely a 3$d^{9}$ electronic configuration in square planar
geometry, is reproduced in the infinite-layer nickelates, the extent to which
the two oxide families can be considered analogous remains an outstanding open
question.

The relative strength of the electron correlations, for one, is currently
unclear. Undoped cuprates are antiferromagnetic Mott insulators [12]. Undoped
bulk NdNiO2, by contrast, does not order down to 1.7 K [13] while thin film
NdNiO2 displays only weakly insulating behavior below $\approx$ 70 K [1]. The
re-entrant insulating behavior found in nickelates at high doping [14, 15, 16,
17, 18] further contrasts with the correlated Fermi-liquid ground state found
in highly overdoped cuprates [19]. Thirdly, various theoretical calculations
[20, 21, 3, 4, 5, 22, among others] have thus far found that the low-energy
electronic structure of infinite-layer nickelates contains contributions from
both the Ni 3$d_{x^{2}-y^{2}}$ and Nd 5$d$-orbitals – rather than the single-
band picture established for the cuprates [12] – though the role or relevance
of the Nd 5$d$-orbitals remains controversial, particularly in superconducting
(SC) samples [23]. X-ray [21], electron energy loss spectroscopy [24] and Hall
coefficient [14, 15] studies, however, do seem to indicate the presence of
multiple bands. Lastly, the peak critical temperature $T_{\rm c}\sim$ 10 K in
nickelates is comparatively low, raising the question whether the two families
even share the same pairing mechanism. Evidently, further investigation of the
normal state from which nickelate superconductivity emerges is required to
ascertain what commonalities, if any, they share with the cuprates.

Typically, the phase diagram of unconventional superconductors harbours a dome
of superconductivity bisected by a transition line to some other order, which
competes with superconductivity for the dominant ground state [25]. One
strategy for determining the evolution of this transition line inside the
superconducting (SC) dome and the influence of its termination (at a putative
quantum critical point) on the transport properties is to suppress the
superconductivity in a high magnetic field and to study the form of the
resistivity down to the lowest possible temperatures [26, 27, 28]. Here, we
present a high-field magnetotransport study on a series of Nd1-xSrxNiO2 (NSNO)
thin films with $0.15\leq x\leq 0.225$, spanning the entire SC dome. The
moderate upper critical field [1] $\mu_{0}H_{\rm c2}$ $\lesssim$ 20 T in NSNO
ensures that the normal state can be accessed in all SC films down to
$T\approx$ 0.5 K under continuous magnetic fields up to 37.5 T. At the lowest
temperatures, the resistivity undergoes a logarithmic-divergence whose
magnitude is systematically reduced with increasing doping. The resilience of
the diverging resistivity to strong magnetic fields excludes localization or
Kondo physics as its origin. Its resemblance to the logarithmically-divergent
resistivity observed in underdoped cuprates [29, 30, 31] is noted, however,
and suggests that the electronic ground state of infinite-layer nickelates is
itself strongly correlated, though possibly not quantum critical. At
$x=0.225$, this insulating behavior cedes to a $T^{2}$ metallic resistivity
characteristic of a correlated Fermi-liquid. Notably, no evidence of non-
Fermi-liquid behavior associated with a putative quantum critical point was
observed.

Nd1-xSrxNiO2 thin films were grown by pulse laser deposition technique using
the high-fluence optimized growth conditions described in Ref. [32].
Electrical resistivity was measured using a four-point method with an
alternating current $I$ = 10 $\mu$A at a frequency between 13 and 30 Hz, with
the magnetic fields applied parallel to the crystalline $c$-axis. High-field
measurements up to 37.5 T were performed using a Bitter magnet coupled to a
3He refrigerator at the High Field Magnet Laboratory in Nijmegen, the
Netherlands.

Figure 1: Semi-log plots of the $T$-dependence of the in-plane resistivity
$\rho_{ab}$ for (a) $x$ = 0.15 (b) $x$ = 0.175 (c) $x$ = 0.20 and (d) $x$ =
0.225 measured at zero magnetic field (solid line), at $\mu_{0}H$ = 35 T
(filled squares), and extrapolations from the normal-state magnetoresistance
to zero field (open squares). Error bars for $\rho_{ab}(H\rightarrow 0)$ are
smaller than the symbols. The inset to panel (d) shows the $T-x$ phase diagram
for Nd1-xSrxNiO2, with $T_{\rm c}$ defined as the mid-point of superconducting
transition in the samples shown here. The blue shaded regions represent the
weakly insulating (WI) behavior found on both sides of the superconducting
(SC) dome [14]. Note that the resistivity ranges in each panel are equivalent.

Figure 1(a-d) shows the normal-state in-plane resistivity $\rho_{ab}(T)$ of
four superconducting NSNO thin films with $x$ = 0.15, 0.175, 0.20 and 0.225,
respectively. The zero-field normal-state resistivity is extrapolated by
fitting the magnetoresistivity isotherms $\rho_{ab}(H)$ shown in Fig. 2(a)
using the form of magnetoresistance (MR $\equiv\rho_{ab}(H)-\rho_{ab}(0)$)
inferred from the field-derivatives (see details in the Supplemental Material
[33]). In short, we used a second-order polynomial function to fit the normal
state resistivity curve in the high-field range where SC fluctuations are
suppressed. At $T>T_{\rm c}$, the magnitude of the MR is small ($<$ 4% at 35
T). Below $T_{\rm c}$, the MR remains small and, crucially, has a negligible
$T$-dependence, thus the $T$-dependence of $\rho_{ab}(H\rightarrow 0)$ and
normal-state resistivity measured at sufficiently high fields are found to
show the same qualitative behavior, independent of the strength of the
magnetic field and/or assumed MR functional form (see Figs. S2 and S3 in
[33]). Overall, our conclusions are thus independent of the details of the
extrapolation methods.

For $x\neq$ 0.225, $\rho_{ab}(T)$ exhibits metallic behavior down to
$T\approx$ 10 K before crossing over to insulating behavior with a log(1/$T$)
divergence down to 0.5 K. The magnitude of the resistivity divergence
diminishes systematically with increasing $x$ and is minimized at $x=0.225$. A
similar insulator-to-metal crossover was also observed in the field-induced
normal state in various underdoped (UD) cuprates [29, 30, 34, 31, 35]. Indeed,
the magnitude of the upturn in NSNO ($\sim$ 10%) is comparable to that
observed in the electron-doped cuprates Pr2-xCexCuO4 [34] and Nd2-xCexCuO4
[35], though it is around one order of magnitude smaller than the
corresponding upturns in hole-doped La2-xSrxCuO4 [29, 30] and
Bi2Sr2-xLaxCuO6+δ [31]. The origin of the log($1/T$) resistivity in UD
cuprates remains unclear. Nevertheless, it has been noted that low-$T$ upturns
in $\rho_{ab}(T)$ are not universal in UD cuprates [36, 37], and irradiation
studies have indicated that disorder, perhaps in combination with strong
correlations, might play the dominant role [38]. The resilience of the
resistive upturn in NSNO against magnetic field as well as the positive MR
(except for $x$ = 0.20) appears to rule out weak localization, Anderson
localization and Kondo scattering [8] as its origin. The systematic evolution
of the upturn in NSNO with doping, coupled with the fact that for 0.15 $\leq
x\leq$ 0.20, the resistive upturns begin either at or slightly below $T_{c}$,
suggests some form of electronic order that competes with the
superconductivity for spectral weight, such as the recently reported short-
range antiferromagnetic correlations in NSNO [39]. While this conjecture also
offers a viable explanation for the evolution of the metallic response (see
below), it does not explain the re-entrant weakly insulating state beyond $x$
= 0.225.

Figure 2: (a) Resistivity isotherms as a function of magnetic field across the
temperature range 0.5 K $\leq T\leq$ 30 K. (b) Upper critical field $H_{\rm
c2}$ extracted using the 50% $\rho_{\rm n}$ criterion (see main text for
details). Open and filled symbols correspond to the extractions made using the
extrapolated zero-field resistivity and the measured 35 T resistivity,
respectively. Dashed lines are fits to the expression $\mu_{0}H_{\rm
c2}(T)=\mu_{0}H_{\rm c2}(0)(1-(T/T_{\rm c})^{2}))$ down to 2 K, below which an
anomalous upturn in $H_{\rm c2}$ is observed.

To gain further insight into the low-$T$ electronic ground state of NSNO, we
turn to examine the doping dependence of the resistively-determined upper
critical field $H_{\rm c2}$. Here we use the criterion previously adopted [40]
for NSNO, namely the field scale at which $\rho_{ab}(H)$ reaches 50% of the
normal-state resistivity $\rho_{\rm n}$. This choice is motivated by the
reduced impact of SC fluctuations or vortex creep to its functional form [41],
despite the fact that the SC order parameter remains finite at this field.
Therefore, $H_{c2}$ shown in Fig. 2 should be considered as the mean-field
estimate and a lower bound. As can be seen, $H_{\rm c2}$ closely tracks
$T_{\rm c}$ in NSNO, peaking at $\mu_{0}H_{\rm c2}\approx$ 20 T for $x=0.175$.
For each value of $x$, $H_{\rm c2}$ also exhibits an anomalous upturn below
$\approx$ 2 K, consistent with a previous finding [40] for $x=0.225$, possibly
indicating an unconventional or multi-gap SC ground state in the infinite-
layer nickelates. We note that, while the choice of a different criterion e.g.
$\rho_{\rm n,90\%}$ or $\rho_{\rm n,95\%}$ changes the magnitudes of $H_{\rm
c2}$, the qualitative behavior of $H_{\rm c2}(T)$ and its evolution with
doping remains the same (see Fig. S5 in the Supplemental Material [33]).
Curiously, $H_{c2}$ is found to be the lowest for $x=0.225$. This finding
essentially excludes the possibility that the low-$T$ resistivity divergence
becomes vanishingly small due to an incomplete suppression of the
superconductivity. Rather, the doping-induced insulator-to-metal crossover is
characteristic of the field-induced normal-state of the superconducting NSNO
investigated here.

Figure 3: (a,b) Zero-field resistivity as measured (black lines) and
extrapolated (open squares) plotted as a function of $T^{2}$ for (a) $x$ =
0.20 and (b) $x$ = 0.225. Resistivity measured at 35 T down to 4.2 K is also
shown for $x$ = 0.225 (red line). Blue and pink lines are fits made to zero-
field and 35 T resistivity, respectively, using $\rho=\rho_{0}+AT^{2}$. (c)
Semi-log plot of the resistive upturn $\Delta\rho_{\rm ab}=\rho_{\rm
ab}-\rho_{\rm min}$, after the resistivity value at its minimum $\rho_{\rm
min}$ is subtracted from zero-field resistivity. Data of non-SC films are
reproduced from [14].

The suppression of the resistive upturn as $x\rightarrow 0.225$ provides an
opportunity to examine more closely the form of low-$T$ metallic resistivity
in NSNO. As shown in Fig. 3, $\rho_{ab}$ for $x$ = 0.20 can be well-fitted to
the expression $\rho=\rho_{0}+AT^{2}$ for 300 K${}^{2}\lesssim T^{2}\lesssim
2000$ K2 (corresponding to 17 K $\lesssim T\lesssim$ 45 K) – the lower bound
being set by the onset of the resistive upturn. For $x$ = 0.225,
$T^{2}$-resistivity is observed only below 7 K ($T^{2}\lesssim$ 50 K). A pure
$T^{2}$-resistivity as $T\rightarrow 0$ is one of the defining signatures of a
Fermi-liquid ground state and its extension up to 45 K in Nd0.8Sr0.2NiO2
suggests a predominance of electron-electron over electron-phonon scattering
and the presence of strong (unscreened) electron correlations. The same
analysis is applied to the other SC concentrations and summarized in Fig. S6
and S7 in the Supplemental Material [33]. While the $T^{2}$-regime is rather
narrow for $x<$ 0.20, due to the growth of the resistivity upturn, we
nevertheless find a systematic doping evolution of its onset temperature. To
date, there is no doping level at which the low-$T$ resistivity is found to
have a lower power-law exponent than 2 that might indicate proximity to a
quantum critical point. Hence, while there is some evidence from the transport
properties of a competing order, it does not, to date, exhibit clear
signatures of quantum criticality.

The magnitude of the resistivity upturn $\Delta\rho_{ab}$, defined the
difference between $\rho_{ab}(T)$ and the resistivity minimum $\rho_{\rm
min}$, also exhibits a systematic doping evolution across the doping series.
As shown in Fig. 3(c), $\Delta\rho_{ab}$ is monotonically suppressed with
increasing $x$ until $x=0.225$, at which point, it becomes very small. For
$x>$ 0.225, $\Delta\rho_{ab}$ again becomes sizeable. For lower-doped non-SC
samples, the resistive upturn is closely correlated with an increase in the
Hall coefficient $R_{\rm H}$ (compare Fig. 1(b) and Fig. 3(a) in Ref. [14]),
indicating that the upturn may be due to a drop in carrier density. In SC
samples, the small upturns of order 5% make it difficult to discern any change
in $R_{\rm H}(T)$, particularly if $R_{\rm H}$ itself has an underlying
$T$-dependence in the metallic state. In tandem with the systematic evolution
of $\Delta\rho_{ab}(x)$, the drop in carrier density suggests the emergence of
a second order parameter that competes with superconductivity for spectral
weight, though this is not expected to impose a log$(1/T)$ dependence on the
resistivity. We note that if the resistive upturn is indeed a signature of
charge gapping, the gap, though small in magnitude, must be remarkably robust
in the presence of an applied magnetic field. An alternative scenario for the
upturn is the development of a secondary scattering mechanism. In this case,
the upturn would be an additive contribution to $\rho_{ab}(T)$. In Fig. S7 and
Fig. S8 in the Supplemental Material [33], we plot the remnant (insulating)
contribution to $\rho_{ab}(T)$ after subtracting off a $T^{2}$-component, i.e.
$\rho_{\rm rem}=\rho-\rho_{T^{2}}$ and a $T$-linear component, i.e. $\rho_{\rm
rem}=\rho-\rho_{T}$, respectively. It is clear from Figs. S7 and S8 that the
systematic evolution of the resistive upturn is not qualitatively changed by
our choice of fitting routine.

Curiously, while the metallicity is optimized at $x=0.225$, the strength of
the superconductivity in NSNO, as reflected in the absolute magnitude of
$H_{\rm c2}$, is not. In fact, the $T_{\rm c}$ and $H_{\rm c2}$ are highest at
$x=0.175$, showing no clear correlation to the normal state properties in the
series of samples investigated here. Evidently, the relation between normal
state and superconducting state properties in unconventional superconductors
is a complex one, with multiple instabilities vying for spectral weight and
frustrating the outcome for the eventual dominant ground state.

In summary, we find a number of salient features of the normal electronic
ground state of superconducting nickelates. Firstly, the weakly insulating
behavior observed outside the SC dome is found to persist in the field-induced
normal state within the SC regime. Secondly, at all dopings except $x=0.225$,
the low-$T$ resistivity follows closely a log(1/$T$) divergence, behavior
reminiscent of that found in various underdoped cuprates [29, 30, 34, 31, 35].
Thirdly, prior to the onset of the upturn, the $T$-dependence of the normal-
state resistivity in the metallic regime undergoes a crossover from $T$-linear
at high-$T$ to $T$-quadratic at lower temperatures. Lastly, at $x$ = 0.225
where the resistivity divergence is the smallest, the resistivity below 10 K
can be well-described by the expression $\rho=\rho_{0}+AT^{2}$, suggestive of
the recovery of a correlated Fermi-liquid state. The resilience of the
resistivity upturn against magnetic fields appears to preclude localization or
Kondo scattering as the origin of the non-metallic behavior. Rather, it
implies the presence of strong unscreened electron correlations, possibly
associated with fluctuations of a competing order parameter.

We acknowledge the support of the HFML-RU/NWO, a member of the European
Magnetic Field Laboratory (EMFL). This work was supported by the Netherlands
Organisation for Scientific Research (NWO) grant No. 16METL01 ‘Strange Metals’
and the European Research Council (ERC) under the European Union’s Horizon
2020 research and innovation programme (Grant Agreement No. 835279-Catch-22).
The work at SLAC/Stanford is supported by the Gordon and Betty Moore
Foundations Emergent Phenomena in Quantum Systems Initiative through grant
number GBMF9072 (synthesis equipment) and the US Department of Energy, Office
of Basic Energy Sciences, Division of Materials Sciences and Engineering,
under contract number DE-AC02-76SF00515.

## References

  * Li  _et al._ [2019] D. Li, K. Lee, B. Y. Wang, M. Osada, S. Crossley, H. R. Lee, Y. Cui, Y. Hikita, and H. Y. Hwang, Nature 572, 624 (2019).
  * Hirsch and Marsiglio [2019] J. E. Hirsch and F. Marsiglio, Physica C 566, 1353534 (2019).
  * Botana and Norman [2020] A. S. Botana and M. R. Norman, Phys. Rev. X 10, 011024 (2020).
  * Wu  _et al._ [2020] X. Wu, D. Di Sante, T. Schwemmer, W. Hanke, H. Y. Hwang, S. Raghu, and R. Thomale, Phys. Rev. B 101, 060504(R) (2020).
  * Been  _et al._ [2021] E. Been, W.-S. Lee, H. Y. Hwang, Y. Cui, J. Zaanen, T. Devereaux, B. Moritz, and C. Jia, Phys. Rev. X 11, 011050 (2021).
  * Lechermann [2020a] F. Lechermann, Phys. Rev. B 101, 081110(R) (2020a).
  * Jiang  _et al._ [2020] M. Jiang, M. Berciu, and G. A. Sawatzky, Phys. Rev. Lett. 124, 207004 (2020).
  * Zhang and Vishwanath [2020] Y.-H. Zhang and A. Vishwanath, Phys. Rev. Res. 2, 023112 (2020).
  * Sakakibara  _et al._ [2020] H. Sakakibara, H. Usui, K. Suzuki, T. Kotani, H. Aoki, and K. Kuroki, Phys. Rev. Lett. 125, 077003 (2020).
  * Choi  _et al._ [2020] M.-Y. Choi, W. E. Pickett, and K.-W. Lee, Phys. Rev. Res. 2, 033445 (2020).
  * Kang  _et al._ [2020] B. Kang, C. Melnick, P. Semon, G. Kotliar, and S. Choi, preprints , arXiv:2007.14610 (2020).
  * Lee  _et al._ [2006] P. A. Lee, N. Nagaosa, and X.-G. Wen, Rev. Mod. Phys. 78, 17 (2006).
  * Hayward and Resseinsky [2003] M. A. Hayward and M. J. Resseinsky, Sol. State Sci. 5, 839 (2003).
  * Li  _et al._ [2020] D. Li, B. Y. Wang, K. Lee, S. P. Harvey, M. Osada, B. H. Goodge, , L. F. Kourkoutis, and H. Hwang, Phys. Rev. Lett. 125, 027001 (2020).
  * Zeng  _et al._ [2020] S. Zeng, C. S. Tang, X. Yin, C. Li, M. Li, Z. Huang, J. Hu, W. Liu, G. J. Omar, H. Jani, Z. S. Lim, K. Han, D. Wan, P. Yang, S. J. Pennocook, A. T. S. Wee, and A. Ariando, Phys. Rev. Lett. 125, 147003 (2020).
  * Osada  _et al._ [2020] M. Osada, B. Y. Wang, K. Lee, D. Li, and H. Hwang, Phys. Rev. Mater. 4, 121801(R) (2020).
  * Osada  _et al._ [2021] M. Osada, B. Y. Wang, B. H. Goodge, S. P. Harvey, K. Lee, D. Li, L. F. Kourkoutis, and H. Y. Hwang, Advanced Materials , 2104083 (2021).
  * Zeng  _et al._ [2021] S. W. Zeng, C. J. Li, L. E. Chow, Y. Cao, Z. T. Zhang, C. S. Tang, X. M. Yin, Z. S. Lim, J. X. Hu, P. Yang, and A. Ariando, , arXiv:2105.13492 (2021).
  * Nakamae  _et al._ [2003] S. Nakamae, S. Behnia, N. Mangkorntong, S. Nohara, H. Takagi, S. J. C. Yates, and N. E. Hussey, Phys. Rev. B 68, 100502(R) (2003).
  * Lee and Pickett [2004] K.-W. Lee and W. E. Pickett, Phys. Rev. B 70, 165109 (2004).
  * Hepting  _et al._ [2020] M. Hepting, D. Li, J. Jia, H. Lu, E. Paris, Y. Tseng, X. Feng, M. Osada, E. Been, Y. Hikita, Y.-D. Chuang, Z. Hussain, K. J. Zhou, A. Nag, M. Garcia-Fernandez, M. Rossi, H. Y. Huang, D. J. Huang, Z. X. Shen, T. Schmitt, H. Y. Hwang, B. Moritz, J. Zaanen, T. P. Deveraux, and W. S. Lee, Nat. Mater. 4, 381 (2020).
  * Lechermann [2020b] F. Lechermann, Phys. Rev. X 10, 041002 (2020b).
  * Kitakani  _et al._ [2020] M. Kitakani, L. Si, O. Janson, R. Arita, Z. Zhong, and K. Held, npj Quant. Mater. 5, 59 (2020).
  * Goodge  _et al._ [2021] B. H. Goodge, D. Li, M. Osada, B. Y. Wang, K. Lee, G. A. Sawatzky, H. Y. Hwang, and L. F. Kourkoutis, Proc. Natl. Acad. Sci.(USA) 118, e2007683118 (2021).
  * Shibauchi  _et al._ [2014] T. Shibauchi, A. Carrington, and Y. Matsuda, Ann. Rev. Condens. Matter. Phys. 5, 113 (2014).
  * Jin  _et al._ [2011] K. Jin, N. P. Butch, K. Kirshenbaum, J. Paglione, and R. L. Greene, Nature 476, 73 (2011).
  * Analytis  _et al._ [2014] J. G. Analytis, H.-H. Kuo, R. D. McDonald, M. Wartenbe, P. M. C. Rourke, N. E. Hussey, and I. R. Fisher, Nat. Phys. 10, 194 (2014).
  * Licciardello  _et al._ [2019] S. Licciardello, J. Buhot, J. Lu, J. Ayres, S. Kasahara, Y. Matsuda, T. Shibauchi, and N. E. Hussey, Nature 567, 213 (2019).
  * Ando  _et al._ [2020] Y. Ando, G. S. Boebinger, A. Passner, T. Kimura, and K. Kishio, Phys. Rev. Lett. 75, 4662 (2020).
  * Boebinger  _et al._ [1996] G. S. Boebinger, Y. Ando, A. Passner, T. Kimura, M. Okuya, J. Shimoyama, K. Kishio, K. Tamasaku, N. Ichikawa, and S. Uchida, Phys. Rev. Lett. 77, 5417 (1996).
  * Ono  _et al._ [2000] S. Ono, Y. Ando, T. Murayama, F. F. Balakirev, J. B. Betts, and G. S. Boebinger, Phys. Rev. Lett. 85, 638 (2000).
  * Lee  _et al._ [2020] K. Lee, B. H. Goodge, D. Li, M. Osada, B. Y. Wang, Y. Cui, L. F. Kourkoutis, and H. Hwang, APL Mater. 8, 041107 (2020).
  * SM [1] See Supplemental Material at [URL will be inserted by publisher] for additional information on extraction of zero-field resistivity and upper critical fields using different methods, and doping evolution of normal-state resistivity in non-superconducting films .
  * Fournier  _et al._ [1998] P. Fournier, P. Mohanty, E. Maiser, S. Darzens, T. Venkatesan, C. J. Lobb, G. Czjzek, R. A. Webb, and R. L. Greene, Phys. Rev. Lett. 81, 4720 (1998).
  * Li  _et al._ [2002] S. Y. Li, W. Q. Mo, Y. M. Xiong, C. H. Wang, X. G. Luo, and Z. Sun, Phys. Rev. B 65, 224515 (2002).
  * Rullier-Albenque  _et al._ [2007] F. Rullier-Albenque, H. Alloul, C. Proust, P. Lejay, A. Forget, and D. Colson, Phys. Rev. Lett. 99, 027003 (2007).
  * Proust  _et al._ [2016] C. Proust, B. Vignolle, J. Levallois, S. Adachi, and N. E. Hussey, Proc. Natl. Acad. Sci.(USA) 113, 13654 (2016).
  * Rullier-Albenque  _et al._ [2008] F. Rullier-Albenque, H. Alloul, F. Balakirev, and C. Proust, Europhys. Lett. 81, 37008 (2008).
  * Lu  _et al._ [2021] H. Lu, M. Rossi, A. Nag, M. Osada, D. F. Li, K. Lee, B. Y. Wang, M. Garcia-Fernandez, S. Agrestini, Z. X. Shen, E. M. Been, B. Moritz, T. P. Devereaux, J. Zaanen, H. Y. Hwang, K.-J. Zhou, and W. S. Lee, Science 373, 213 (2021).
  * Wang  _et al._ [2021] B. Y. Wang, D. Li, B. H. Goodge, K. Lee, M. Osada, S. P. Harvey, L. F. Kourkoutis, M. R. Beasley, and H. Y. Huang, Nat. Phys. 17, 473 (2021).
  * Brandow [1998] B. Brandow, Phys. Rep. 296, 1 (1998).


# CycleSAM: One-Shot Surgical Scene Segmentation using Cycle-Consistent
Feature Matching to Prompt SAM

Aditya Murali Pietro Mascagni Didier Mutter Nicolas Padoy University of
Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France  IHU Strasbourg,
Strasbourg, France  Fondazione Policlinico Universitario A. Gemelli IRCCS,
Rome, Italy

###### Abstract

The recently introduced Segment-Anything Model (SAM) has the potential to
greatly accelerate the development of segmentation models. However, directly
applying SAM to surgical images has key limitations including (1) the
requirement of image-specific prompts at test-time, thereby preventing fully
automated segmentation, and (2) ineffectiveness due to substantial domain gap
between natural and surgical images. In this work, we propose CycleSAM, an
approach for one-shot surgical scene segmentation that uses the training
image-mask pair at test-time to automatically identify points in the test
images that correspond to each object class, which can then be used to prompt
SAM to produce object masks. To produce high-fidelity matches, we introduce a
novel spatial cycle-consistency constraint that enforces point proposals in
the test image to rematch to points within the object foreground region in the
training image. Then, to address the domain gap, rather than directly using
the visual features from SAM, we employ a ResNet50 encoder pretrained on
surgical images in a self-supervised fashion, thereby maintaining high label-
efficiency. We evaluate CycleSAM for one-shot segmentation on two diverse
surgical semantic segmentation datasets, comprehensively outperforming
baseline approaches and reaching up to 50% of fully-supervised performance.

##  1 Introduction

SAM [8] is a powerful general-purpose segmentation model that achieves
excellent results in the natural image domain. Several works have successfully
applied or adapted SAM for medical image segmentation [19, 11, 18, 3, 5] as
well as surgical tool/scene segmentation [17, 13, 14, 21]. Yet, standard
application of SAM requires user-provided prompts in the form of points,
boxes, or masks at test-time, preventing fully-automated segmentation.

A few SAM-adaptation approaches have been proposed to bypass user prompts at
test time [22, 23, 14]. [14] and [22] augment SAM with a custom prompt
encoder, with the former using ground truth class names as prompts at test-
time while the latter builds class prototype prompts by extracting foreground
features for each object from the training images. However, these approaches
are generally applied in a fully-supervised fashion and therefore still
require a substantial amount segmentation annotations, which are extremely
expensive in the surgical domain; moreover, they only achieve modest
performance gains over traditional semantic segmentation models such as
DeepLabv3+ and UNet.

In contrast, we are primarily interested in leveraging SAM for label-efficient
segmentation. Our key insight, supported by [17, 13], is that despite the
domain shift between natural and surgical images, SAM can still produce
accurate masks if provided with good prompts. This decoupling of prompt and
mask generation can enable SAM to achieve large gains over traditional
segmentation approaches in label-efficient settings, as even partial accuracy
in prompt generation can translate to accurate masks. To this end, in this
work we focus on the problem of one-shot surgical scene segmentation,
addressing both complex anatomical structures and surgical tools in contrast
to [22], which only considers the latter.

Fig. 1: An overview of CycleSAM. A ResNet50 encoder SSL-pretrained on surgical
data is used to extract multi-scale features
$\\{\mathcal{H}_{\mathcal{R}}^{s}\ |\ s\in[1..S]\\}$,
$\\{\mathcal{H}_{\mathcal{T}}^{s}\ |\ s\in[1..S]\\}$ for the reference image
$\mathcal{I}_{\mathcal{R}}$ and test image $\mathcal{I}_{\mathcal{T}}$
respectively. The multi-scale features are passed to the trainable CycleSelect
module which generates both positive and negative point prompts for object $c$
in the test image using cycle-consistent feature matching. Finally, these
point prompts are passed to a frozen SAM model along with the original test
image to produce a segmentation mask for the object.

We build our approach, CycleSAM, as an extension of PerSAM [23], which uses
feature matching to identify points in the test image that match each object
class in the training image. To obtain higher-fidelity matches, we propose
CycleSelect, a module for improved feature point matching consisting of (1) a
spatial cycle-consistency masking function suppressing points that do not
rematch to the object foreground region in the training image, (2) multiscale
feature matching, and (3) a host of improvements to the point prompt sampling
process including multiple foreground point selection and relevant negative
point selection. Then, recognizing that feature-based point matching is highly
dependent on a discriminative underlying feature space, we propose to augment
the ViT feature extractor from SAM with MoCov2 [2] and DINO [1] self-
supervised feature extractors pretrained on in-domain surgical images as in
[15]. Finally, to tune the point features, we train linear feature projectors
by enforcing an embedding-level contrastive loss between foreground and
background point features.

We evaluate our approach on two challenging surgical scene segmentation
datasets, Endoscapes-Seg50 [12] and CaDIS [4], selected specifically for their
inclusion of anatomical structures as well as tools. In our experiments, we
show that CycleSAM outperforms both traditional specialist and SAM-based
segmentation approaches in the one-shot segmentation setting. Moreover, with
just one labeled training image, CycleSAM reaches nearly 50% of fully-
supervised performance, thus taking a large stride towards closing the
performance gap.

In summary, our contributions are as follows:

  1. 1.

We propose CycleSAM, a feature-matching based approach for 1-shot adaptation
of SAM to complex surgical scene segmentation.

  2. 2.

We introduce a novel feature matching module, CycleSelect, that uses spatial
cycle consistency to greatly improve point correspondence across images.

  3. 3.

We comprehensively outperform state-of-the-art SAM adaptation methods for
label-efficient surgical scene segmentation.

##  2 Methods

In this section, we provide a detailed description of CycleSAM, which takes a
reference (training) image $\mathcal{I}_{\mathcal{R}}$, corresponding
reference mask $\mathcal{M}_{\mathcal{R}}$, test image
$\mathcal{I}_{\mathcal{T}}$, image encoder $\phi_{\text{img}}$, and SAM model
$\phi_{\text{SAM}}$ and outputs a binary mask $\mathcal{M}_{c}$ for each
object class $c\in[1...C]$. We first describe CycleSelect, a module that
produces a set of point prompts $\mathcal{P}^{c}_{\mathcal{T}}$ in
$\mathcal{I}_{\mathcal{T}}$ for each object class $c$ based on dense feature
matching; each of the $\mathcal{P}^{c}_{\mathcal{T}}$ can then be passed to
the SAM mask decoder $\phi_{\text{SAM}}^{\mathcal{D}}$ $C$ different times to
produce a mask for each class. Then, we describe the architecture and training
process of CycleSAM consisting of (1) a feature similarity warm-up stage to
refine the matching feature space and (2) the finetuning stage of PerSAM [23]
that learns class-specific reweightings of the three masks $\hat{M}_{1}^{c}$,
$\hat{M}_{2}^{c}$, and $\hat{M}_{3}^{c}$ predicted by
$\phi_{\text{SAM}}^{\mathcal{D}}$.

###  2.1 CycleSelect

CycleSelect, detailed in Algorithm 1, is based on the key assumption that
points belonging to an object class $c$ should have similar feature
representations across different images. It follows that given an image-mask
pair as a reference, we can identify object points
$\mathcal{P}^{c}_{\mathcal{T}}$ in a test image that match the reference
foreground points $\mathcal{P}^{c}_{\mathcal{R}}$ by comparing their
underlying feature representations.

The first step of CycleSelect is therefore to extract features for the
reference image $\mathcal{I}_{\mathcal{R}}$ and test image
$\mathcal{I}_{\mathcal{T}}$ with an image encoder $\phi_{\text{img}}$,
yielding feature maps $\mathcal{H}_{\mathcal{R}}$ and
$\mathcal{H}_{\mathcal{T}}$; we then spatially resize these feature maps and
the reference mask $\mathcal{M}_{\mathcal{R}}$ to $d_{\text{match}}\times
d_{\text{match}}$. Next, to build a target feature $h^{c}_{\text{target}}$ for
each class $c$, we first evenly sample $\tilde{X}$ points from the foreground
region and use these points to index the reference feature map
$\mathcal{H}_{\mathcal{R}}$, yielding $\tilde{X}$ point features
$h^{c}_{\text{points}}$; then, we average the features of all the foreground
points, yielding a foreground summary feature $h^{c}_{\text{fg}}$, which we
concatenate with $h^{c}_{\text{points}}$ to yield
$h^{c}_{\text{target}}\in\mathbb{R}^{X\times d_{\text{channel}}}$, where
$X=\tilde{X}+1$ and $d_{\text{channel}}$ is the feature channel dimension.

Algorithm 1 CycleSelect Feature Similarity

Input: Image Encoder $\phi_{\text{img}}$, Ref Img $\mathcal{I}_{\mathcal{R}}$,
Test Img $\mathcal{I}_{\mathcal{T}}$, Ref Mask $\mathcal{M}_{\mathcal{R}}$,
Obj Class $c$

$\mathcal{H}_{\mathcal{R}}\leftarrow$
Downsample($\phi_{\text{img}}(\mathcal{I}_{\mathcal{R}}),d_{\text{match}}$);
$\mathcal{H}_{\mathcal{T}}\leftarrow$
downsample($\phi_{\text{img}}(\mathcal{I}_{\mathcal{T}}),d_{\text{match}}$)

$\mathcal{M}_{\mathcal{R}}^{c}\leftarrow$
Downsample($\mathcal{M}_{\mathcal{R}}$, $d_{\text{match}}$);
$\mathcal{M}_{\mathcal{R}}^{c}[\mathcal{M}_{\mathcal{R}}^{c}\neq c]=0$;
$\mathcal{M}_{\mathcal{R}}^{c}[\mathcal{M}_{\mathcal{R}}^{c}=c]=1$;
$P^{c}_{\text{ref}}\leftarrow$ Argwhere($\mathcal{M}^{c}_{\mathcal{R}}$)

$P_{\text{sample}}\leftarrow$ PointSample($P^{c}_{\text{ref}}$);
$h^{c}_{\text{points}}\leftarrow\mathcal{H}_{\mathcal{R}}[P_{\text{sample}}]$

$h^{c}_{\text{fg}}\leftarrow$
Mean($(\mathcal{H}_{\mathcal{R}}[\mathcal{M}^{c}_{\mathcal{R}}=1]$);
$h^{c}_{\text{target}}\leftarrow$ Concat($h^{c}_{\text{points}}$,
$h^{c}_{\text{fg}}$); $\mathcal{S}_{\text{all}}\leftarrow
h^{c}_{\text{target}}\cdot\mathcal{H}_{\mathcal{T}}^{\text{T}}$

$s\leftarrow$ Argmax(Abs($\mathcal{S}$), 0); $\lambda_{\text{agg}}\leftarrow$
Sign($\mathcal{S}_{\text{all}}[s]$)

$\mathcal{S}\leftarrow\lambda_{\text{agg}}\cdot$
Max($\mathcal{S}_{\text{all}}$, 0) + $(1-\lambda_{\text{agg}})\cdot$
Min($\mathcal{S}_{\text{all}}$, 0)

$\mathcal{S}_{\text{rematch}}\leftarrow$ Flatten($\mathcal{H}_{\mathcal{T}}$,
0, 2) $\cdot$ Flatten($\mathcal{H}_{\mathcal{R}}$, 0, 2)${}^{\text{T}}$

$\mathcal{M}_{\text{SCC}}^{c}\leftarrow\mathbf{0}^{d_{\text{match}}^{2}}$;
$\mathcal{M}_{\text{SCC}}^{c}\leftarrow\mathcal{M}_{R}$[Argmax($\mathcal{S}_{\text{rematch}}$,
1)]

$\mathcal{M}_{\text{SCC}}^{c}\leftarrow$
Reshape($\mathcal{M}_{\text{SCC}}^{c},d_{\text{match}}\times
d_{\text{match}}$)

$\mathcal{S}_{\text{SCC}}^{c}\leftarrow\mathcal{S}$;
$\mathcal{S}_{\text{SCC}}^{c}[\mathcal{M}_{\text{SCC}}^{c}=0]\leftarrow\mathcal{S}[\mathcal{M}_{\text{SCC}}^{c}=0]-\lambda_{\text{SCC}}$

Algorithm 2 CycleSelect Multiscale Matching

$\mathcal{H}_{\mathcal{R}}\leftarrow\\{\mathcal{H}_{\mathcal{R}}^{s}\in\mathbb{R}^{d_{\text{match}}^{s}\times
d_{\text{match}}^{s}}\ |\ s\in[1..S]\\}$;
$\mathcal{H}_{\mathcal{T}}\leftarrow\\{\mathcal{H}_{\mathcal{T}}^{s}\in\mathbb{R}^{d_{\text{match}}^{s}\times
d_{\text{match}}^{s}}\ |\ s\in[1..S]\\}$

$\mathcal{S}_{\text{SCC}}^{\text{all}}\leftarrow\\{$Resize(CycSelect($\mathcal{H}_{\mathcal{R}}^{s}$,
$\mathcal{H}_{\mathcal{T}}^{s}$, Resize($\mathcal{M}_{\mathcal{R}}^{s}$,
$d_{\text{match}}^{s}$))$\ |\ s\in[1..S]$, $d_{\text{match}}^{1}$)$\\}$

$\mathcal{S}_{\text{SCC}}\leftarrow\sum_{s\in[1..S]}{W_{\text{scale}}^{s}*\mathcal{S}_{\text{SCC}}^{s}}$

Once we have computed $h^{c}_{\text{target}}$, we proceed to identify similar
point features $h^{c}_{i}\in\mathcal{H}_{\mathcal{T}}$. To do so, we compute a
cosine similarity matrix
$\mathcal{S}_{\text{all}}=h^{c}_{\text{target}}\cdot\mathcal{H}_{\mathcal{T}}^{\text{T}}$
between the $X$ target features and test feature map. Finally, we aggregate
the similarities to each of the $X$ target features into a global similarity
map $\mathcal{S}$, where the indices of the highest values correspond to the
points in the test image $\mathcal{P}^{c}_{\mathcal{T}}$ most likely to belong
to the target class $c$.

Spatial Cycle-Consistency Masking. While we could directly use $\mathcal{S}$
to sample point prompts, this often leads to spurious matches because the
feature maps produced by $\phi_{\text{img}}$ are not sufficiently
discriminative, particularly at the point granularity. To obtain a more robust
similarity matrix, inspired by several prior works [6, 7, 9, 16], we introduce
a spatial cycle consistency constraint that enforces matched points
$p\in\mathcal{P}^{c}_{\mathcal{T}}$ to rematch to points belonging within
reference object. Intuitively, such a constraint can help filter out poor
matches that result from noisy feature maps and local ambiguities, and is
particularly important for dealing with test images without the target object.

To integrate this cycle-consistency constraint into our approach, rather than
simply removing points from $\mathcal{P}^{c}_{\mathcal{T}}$, we compute a
cycle-consistency mask $\mathcal{M}_{\text{SCC}}^{c}$ in three steps: (1) we
densely compute the cosine similarity
$\mathcal{S}_{\text{rematch}}\in\mathbb{R}^{d_{\text{match}}^{2}\times
d_{\text{match}}^{2}}$ between all test and reference point features, (2) we
find the index of the reference feature $h_{\mathcal{R}}^{j}$ that is most
similar to each test feature $h_{\mathcal{T}}^{i}$, and (3) if the index of
the rematched feature $i$ is not in the reference mask
$\mathcal{M}_{\mathcal{R}}$, we set the corresponding value in
$\mathcal{M}_{\text{SCC}}^{c}$ to $1$. We can then combine
$\mathcal{M}_{\text{SCC}}^{c}$ and the original similarity matrix
$\mathcal{S}^{c}$ into a final cycle-consistent similarity matrix
$\mathcal{S}_{\text{SCC}}^{c}$; to do so, we simply subtract a constant
$\lambda_{\text{SCC}}$ from the elements of $\mathcal{S}$ masked by
$\mathcal{M}_{\text{SCC}}^{c}$.

Multiscale Feature Matching To help deal with differing object sizes, we
propose to incorporate multiscale modeling by repeating CycleSelect at
different scales $s\in[1..S]$, yielding a set of similarity matrices
$\\{\mathcal{S}_{\text{SCC}}^{s}\ |\ s\in[1..S]\\}$. We can then aggregate
these similarities into a single similarity matrix $\mathcal{S}_{\text{SCC}}$
after resizing each matrix to $d_{\text{match}}\times d_{\text{match}}$, as we
do for the multiple target features. Algorithm 2 details the multiscale
matching.

Point Prompt Sampling PerSAM [23] uses only a single point prompt per image;
we instead select the top $k$ foreground and background points from the final
similarity matrix $\mathcal{S}_{\text{SCC}}^{c}$, resulting in a more
informative prompt. Then, to boost the utility of the background points, we
cluster the similarity scores into 3 bins, and pick the top $k$ points from
the bin with the lowest scores. Intuitively, this clustering process separates
the points into strong positive matches, ambiguous points, and strong negative
matches. By picking the points with highest similarity from the strong
negative matches, we can obtain informative exclusion point prompts while
remaining confident that the selected points are indeed negatives. Finally, we
threshold the positive matches with a per-class learnable threshold $t_{c}$;
we compute these thresholds during train-time as a weighted average of the
similarity matrix $\mathcal{S}_{\text{SCC}}^{c}$ as follows:

$\centering\begin{gathered}t_{c}^{+}=\sum_{i,j}{(\mathcal{S}_{\text{SCC}}^{c})_{i,j}*(\mathcal{M}_{R^{\prime}}^{c})_{i,j}};\
\
t_{c}^{-}=\sum_{i,j}{(\mathcal{S}_{\text{SCC}}^{c})_{i,j}*(1-\mathcal{M}_{R^{\prime}}^{c})_{i,j}}\\\
t_{c}=0.5*(t_{c}^{+}+t_{c}^{-})\end{gathered}\@add@centering$ (1)

Following this process, we obtain $2k$ point prompts
$\mathcal{P}_{\mathcal{T}}^{c}=\\{\mathcal{P}_{\mathcal{T}}^{c,+};\mathcal{P}_{\mathcal{T}}^{c,-}\\}$
that we pass to the SAM mask decoder $\phi_{\text{SAM}}^{\mathcal{D}}$ to
obtain masks $M_{1}^{c}$, $M_{2}^{c}$ and $M_{3}^{c}$.

Table 1: Results on Endoscapes-Seg50 (ES50) and CaDIS. We split the baselines
into 4 categories: (1) traditional segmentation approaches, for which we use
DeepLabV3+ (DLV3P) as a representative baseline and experiment with various
weight initializations and linear probing/finetuning, (2) SAM-based baseline
approaches, including finetuning, adapter, and automatic prompt generation
approaches, (3) our proposed CycleSAM and variants (backbone, initialization),
and (4) a fully supervised DeepLabV3+ model to represent the ceiling
performance. All performances from groups 1-3 are directly comparable.

Method |  Finetune Backbone | ES50 | CaDIS
---|---|---|---
mIoU | mDice | mIoU | mDice
DLV3P-R50-SurgMoCov2 | No | 15.07 | 24.83 | 36.53 | 49.85
DLV3P-R50-SurgDINO | No | 16.94 | 26.51 | 38.19 | 50.63
DLV3P-R50 | Yes | 14.65 | 22.62 | 43.20 | 54.76
DLV3P-R50-SurgMoCov2 | Yes | 15.70 | 24.05 | 41.61 | 54.39
DLV3P-R50-SurgDINO | Yes | 14.05 | 21.79 | 43.30 | 55.93
SelfPromptSAM [20] | No | 2.6 | 3.53 | 7.47 | 13.94
SurgicalSAM [22] | No | 6.90 | 10.50 | 29.34 | 37.61
PerSAM [23] | No | 4.42 | 7.97 | 25.01 | 34.56
PerSAM-F [23] | No | 4.61 | 8.33 | 25.41 | 35.95
PerSAM-F-MedSAM [11, 23] | No | 3.38 | 6.81 | 8.33 | 14.63
CycleSAM-ViT | No | 8.17 | 14.26 | 31.82 | 42.89
CycleSAM-SurgDINO | No | 15.52 | 22.13 | 35.91 | 47.85
CycleSAM-SurgMoCov2 | No | 16.40 | 24.05 | 33.76 | 44.49
CycleSAM-DLV3P-SurgDINO | No | 20.56 | 28.48 | 47.37 | 61.78
CycleSAM-DLV3P-SurgMoCov2 | No | 22.34 | 30.86 | 44.21 | 59.40
DLV3P-R50-FullSup | Yes | 45.02 | 57.60 | 83.18 | 90.61

###  2.2 CycleSAM Architecture and Training

Figure 1 illustrates the overall architecture of CycleSAM, the main component
of which is the trainable CycleSelect module. CycleSelect contains four
different trainable components: (1) a Feature Projection block
$\phi_{\text{proj}}$ comprising two linear feature projectors for each object
class and feature scale ($2CS$ in total), (2) a feature pyramid network (FPN)
[10] $\phi_{\text{FPN}}$ to fuse multiscale image features, (3) per-class
trainable scale weights $W_{\text{scale}}\in\mathbb{R}^{C\times S}$ to
reweight the similarity matrices $\\{\mathcal{S}_{\text{SCC}}^{s}\ |\
s\in[1..S]\\}$ produced by the multiscale matching in CycleSelect, and (4)
per-class trainable mask weights $W_{\text{mask}}\in\mathbb{R}^{C\times 3}$ as
in [23] to reweight the three masks predicted by
$\phi_{\text{SAM}}^{\mathcal{D}}$. To train CycleSAM in a one-shot fashion, we
simply treat a copy of the reference image $\mathcal{I}_{\mathcal{R}}$ as the
test image, referring to this copy as
$\mathcal{I}_{\mathcal{R}^{{}^{\prime}}}$. We train in two stages: the first
is a feature similarity warmup stage where we train $\phi_{\text{proj}}$,
$\phi_{\text{FPN}}$ and $W_{\text{scale}}$, and the second is the mask
refinement training process of PerSAM [23] where we only train
$W_{\text{mask}}$.

Feature Similarity Warmup PerSAM [23] directly uses feature maps
$\mathcal{H}_{\mathcal{R}}$ and $\mathcal{H}_{\mathcal{T}}$ from
$\phi_{\text{img}}$, which in their case, is the ViT image encoder from
$\phi_{\text{SAM}}$, to construct a target feature $h_{\text{target}}$ and
compute a similarity matrix $\mathcal{S}$. However, when dealing with surgical
images, the extracted feature space is far less discriminative due to
substantial domain shift. As a result, we propose to instead use frozen SSL-
pretrained ResNet50 encoders [15] for $\phi_{\text{img}}$. Then, to improve
the quality of the multiscale matching in CycleSelect, we further process the
SSL features with a trainable FPN $\phi_{\text{FPN}}$; thus, the updated image
encoder becomes
$\phi_{\text{img}}=\phi_{\text{fpn}}\circ\phi_{\text{backbone}}$. Finally,
because such SSL-pretrained encoders can lack the fine-grained precision of
the SAM ViT encoder, which is specifically trained for pixel-level
segmentation, we additionally apply asymmetric linear feature projectors to
the test feature map $\mathcal{H}_{\mathcal{T}}$ and target feature
$h_{\text{target}}$ prior to computing $\mathcal{S}$:
$W_{\text{feat\\_map}},W_{\text{feat}}\in\mathbb{R}^{d_{\text{proj}}\times
d_{\text{enc}}}$, where $d_{\text{enc}}$ is the number of output channels of
$\phi_{\text{img}}$ and $d_{\text{proj}}$ is the final projected feature
dimension. We include these projectors for each class $c$ and feature scale
$s$, yielding a total of $2CS$ projectors
$\phi_{\text{proj}}:\\{W_{\text{feat\\_map}}^{c,s},W_{\text{feat}}^{c,s}\ |\
c\in[1..C],s\in[1..S]\\}$.

To train these layers, we introduce a feature similarity contrastive loss
$\mathcal{L}_{\text{sim}}$ (see Algorithm 3) that maximizes the similarity of
the target feature and foreground test features and minimizes that of the
target feature and background features. Note that because we test image is
simply the reference image during training, we also have a test mask
$\mathcal{M}_{\mathcal{R}}^{{}^{\prime}}$ that we use to separate foreground
and background features. Finally, to train the multiscale weights
$W_{\text{scale}}^{c}$, we simply treat the predicted similarity matrix
$\mathcal{S}_{\text{SCC}}^{c}$ as a segmentation mask and compute an L1 loss
against the ground truth mask $\mathcal{M}_{\mathcal{R}^{\prime}}^{c}$.

Algorithm 3 Feature Similarity Contrastive Loss

$h_{\text{test}}^{c,\text{fg}}\leftarrow\mathcal{H}_{\mathcal{R}^{\prime}}[\mathcal{M}_{\mathcal{R}^{\prime}}^{c}=1]$;
$h_{\text{test}}^{c,\text{bg}}\leftarrow\mathcal{H}_{\mathcal{R}^{\prime}}[\mathcal{M}_{\mathcal{R}^{\prime}}^{c}=0]$

$\mathcal{L}_{\text{sim}}\leftarrow 0.5*(1-$
cos($h_{\text{test}}^{c,\text{fg}}$, $h_{\text{target}}^{c}$) + $0.5*$
max(cos($h_{\text{test}}^{c,\text{bg}}$, $h_{\text{target}}^{c}$))

Mask Weighting After pre-training the feature similarity components
($\phi_{fpn}$, $\phi_{\text{proj}}$, $W_{\text{scale}}$), we freeze them and
proceed to train $W_{\text{mask}}$. We follow the same process as [23],
predicting a mask $\hat{\mathcal{M}}_{\mathcal{R}^{\prime}}$ for
$\mathcal{I}_{\mathcal{R}^{\prime}}$ using CycleSAM and computing losses (Dice
and Binary Cross-Entropy) against the ground-truth mask
$\mathcal{M}_{\mathcal{R}^{\prime}}$. We continue to asymmetrically augment
$\mathcal{I}_{\mathcal{R}}$ here to improve generalization.

##  3 Experiments and Results

We begin Section 3.1 by describing our evaluation datasets and metrics, then
proceed to the main results. Then, in Section 3.2, we present an ablation
study investigating the impact of CycleSAM’s improved prompt selection
mechanisms.

Table 2: Point Sampling Ablation Study on Endoscapes-Seg50.

Multiple Points ($k=10$) | Negative Points | Learnable Threshold | mIoU
---|---|---|---
✗ | ✗ | ✗ | 13.68
✗ | ✓ | ✗ | 14.39
✓ | ✗ | ✗ | 16.58
✓ | ✓ | ✗ | 22.22
✗ | ✗ | ✓ | 14.24
✓ | ✓ | ✓ | 22.34

###  3.1 Main Results

We evaluate all methods on two different surgical scene segmentation datasets,
Endoscapes-Seg50 and CaDIS, and report $\text{mIoU}^{\text{NB}}$ (excluding
the background) and mean Dice score (mDice). Endoscapes-Seg50 [12] is a
segmentation dataset for safe dissection in laparoscopic cholecystectomy that
consists of 6 classes (5 anatomical structures and 1 tool class); it presents
a unique challenge due to the inclusion of fine-grained anatomical
structures/windows (e.g. cystic plate, calot triangle). Meanwhile, CaDIS [4]
is a dataset for scene segmentation in cataract surgery that, like Endoscapes-
Seg50, also contains both anatomical structures and tools. CaDIS includes
three different segmentation tasks of increasing granularity: in this work,
since we already adopt the challenging 1-shot setup, we focus on the first
task, an 8-class segmentation problem (4 anatomical structures, 1 tool class,
3 other). For training, we randomly select one image, ensuring that it
contains all of the object classes, and using it for all of the methods.

Table 1 shows the results of all methods on both datasets, separated into four
categories: (1) specialist models, for which we consider DeepLabV3Plus-
ResNet50 (DLV3P-R50) as the base segmentation model and experiment with
different backbone initializations (default, SurgMoCov2, and SurgDINO), (2)
SAM-based approaches, including adaptation approaches [22, 11] and automatic
prompt generation approaches [23, 20], (3) CycleSAM, also tested with various
initializations for $\phi_{\text{img}}$, and (4) a fully-supervised
DeepLabV3Plus model to represent the ceiling performance. For SurgMoCov2 and
SurgDINO, we use the weights provided by [15], and use Cholec80-pretrained
models for the Endoscapes-Seg50 experiments and CaDIS-pretrained models for
the CaDIS experiments. We also test CyclSAM variants using the original SAM
ViT backbone as well as the 1-shot trained DLV3P-R50 models from group (1) to
initialize $\phi_{\text{img}}$.

Our best approach is CycleSAM initialized with the trained best-performing
group 1 model, DLV3P-SurgMoCov2. CycleSAM-DLV3P-SurgMoCov2 not only
outperforms all baseline methods but also achieves $\sim$50% of fully
supervised mIoU and $>$50% mDice on both datasets. Interestingly, we find that
the best baseline methods are the group 1 specialist DeepLabV3Plus models,
with the best models coming close to CycleSAM (difference of $5.40$ mIoU for
Endoscapes-Seg50, $4.07$ mIoU for CaDIS). On the other hand, we
comprehensively outperform existing SAM-based baseline methods, which are
quite ineffective for one-shot surgical scene segmentation due to their
reliance on the off-the-shelf feature representations. This is reiterated by
studying CycleSAM-ViT, which directly uses the SAM ViT feature extractor as
$\phi_{\text{img}}$ and performs similarly to the other SAM-based models.
Furthermore, PerSAM-F-MedSAM, which uses MedSAM [11] as the underlying model
rather than SAM, performs worse than the original PerSAM, highlighting the
limitations of current off-the-shelf adaptations of SAM to medical domains. By
flexibly taking advantage of any domain-specific feature extractor, CycleSAM
achieves massive performance gains over existing SAM-based approaches
($+15.44$ mIoU for Endoscapes-Seg50, $+18.03$ mIoU for CaDIS).

###  3.2 Prompt Selection Ablation Study

Table 2 shows the impact of our improved point sampling over the default
approach of PerSAM [23]; by including 5 different points as prompts and
including relevant negative points, we achieve an improvement of $8.54$ mIoU
on Endoscapes-Seg50, nearly doubling the performance.

##  4 Conclusion

We introduce CycleSAM, a method for one-shot adaptation of SAM for surgical
scene segmentation using cycle-consistent feature matching. Our approach goes
beyond the well-studied problem of tool segmentation and tackles two
challenging surgical anatomy segmentation tasks. By leveraging surgical domain
SSL-pretrained image encoders, CycleSAM effectively addresses the domain shift
problem, vastly outperforming existing methods in the one-shot setting.

##  5 Acknowledgement

This work was supported by French state funds managed by the ANR within the
National AI Chair program under Grant ANR-20-CHIA-0029-01 (Chair AI4ORSafety)
and within the Investments for the future program under Grants
ANR-10-IDEX-0002-02 (IdEx Unistra) and ANR-10-IAHU-02 (IHU Strasbourg). This
work was granted access to the HPC resources of IDRIS under the allocation
2021-AD011011640R1 made by GENCI.

## References

  * [1] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)
  * [2] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)
  * [3] Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
  * [4] Grammatikopoulou, M., Flouty, E., Kadkhodamohammadi, A., Quellec, G., Chow, A., Nehme, J., Luengo, I., Stoyanov, D.: Cadis: Cataract dataset for surgical rgb-image segmentation. Medical Image Analysis 71, 102053 (2021)
  * [5] Hu, X., Xu, X., Shi, Y.: How to efficiently adapt large segmentation model (sam) to medical images. arXiv preprint arXiv:2306.13731 (2023)
  * [6] Hundt, A., Murali, A., Hubli, P., Liu, R., Gopalan, N., Gombolay, M., Hager, G.D.: ” good robot! now watch this!”: Repurposing reinforcement learning for task-to-task transfer. In: 5th Annual Conference on Robot Learning (2021)
  * [7] Jabri, A., Owens, A., Efros, A.A.: Space-time correspondence as a contrastive random walk. Advances in Neural Information Processing Systems (2020)
  * [8] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [9] Lebailly, T., Stegmüller, T., Bozorgtabar, B., Thiran, J.P., Tuytelaars, T.: Cribo: Self-supervised learning via cross-image object-level bootstrapping. In: The Twelfth International Conference on Learning Representations (2023)
  * [10] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
  * [11] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
  * [12] Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N., Costamagna, G., Mutter, D., Marescaux, J., Dallemagne, B., et al.: The endoscapes dataset for surgical scene segmentation, object detection, and critical view of safety assessment: Official splits and benchmark. arXiv preprint arXiv:2312.12429 (2023)
  * [13] Oguine, K.J., Soberanis-Mukul, R.D., Drenkow, N., Unberath, M.: From generalization to precision: Exploring sam for tool segmentation in surgical environments. arXiv preprint arXiv:2402.17972 (2024)
  * [14] Paranjape, J.N., Nair, N.G., Sikder, S., Vedula, S.S., Patel, V.M.: Adaptivesam: Towards efficient tuning of sam for surgical scene segmentation (2023)
  * [15] Ramesh, S., Srivastav, V., Alapatt, D., Yu, T., Murali, A., Sestini, L., Nwoye, C.I., Hamoud, I., Sharma, S., Fleurentin, A., et al.: Dissecting self-supervised learning methods for surgical computer vision. Medical Image Analysis 88, 102844 (2023)
  * [16] Son, J.: Contrastive learning for space-time correspondence via self-cycle consistency. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14679–14688 (June 2022)
  * [17] Wang, A., Islam, M., Xu, M., Zhang, Y., Ren, H.: Sam meets robotic surgery: An empirical study in robustness perspective. arXiv preprint arXiv:2304.14674 (2023)
  * [18] Wang, C., Li, D., Wang, S., Zhang, C., Wang, Y., Liu, Y., Yang, G.: Sam-med: A medical image annotation framework based on large vision model. arXiv preprint arXiv:2307.05617 (2023)
  * [19] Wu, J., Fu, R., Fang, H., Liu, Y., Wang, Z., Xu, Y., Jin, Y., Arbel, T.: Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
  * [20] Wu, Q., Zhang, Y., Elbatel, M.: Self-prompting large vision models for few-shot medical image segmentation. In: MICCAI Workshop on Domain Adaptation and Representation Transfer. pp. 156–167. Springer (2023)
  * [21] Xie, W., Willems, N., Patil, S., Li, Y., Kumar, M.: Sam fewshot finetuning for anatomical segmentation in medical images. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 3253–3261 (January 2024)
  * [22] Yue, W., Zhang, J., Hu, K., Xia, Y., Luo, J., Wang, Z.: Surgicalsam: Efficient class promptable surgical instrument segmentation (2024)
  * [23] Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Gao, P., Li, H.: Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048 (2023)