BELLE2-CONF-PH-2021-037

The Belle II Collaboration

#
Measurement of the $B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell}$ branching ratio
and $|V_{cb}|$ with a fully reconstructed accompanying $B$ meson in 2019-2021
Belle II data

F. Abudinén  I. Adachi  K. Adamczyk  L. Aggarwal  P. Ahlburg  H. Ahmed  J. K.
Ahn  H. Aihara  N. Akopov  A. Aloisio  F. Ameli  L. Andricek  N. Anh Ky  D. M.
Asner  H. Atmacan  V. Aulchenko  T. Aushev  V. Aushev  T. Aziz  V. Babu  H.
Bae  S. Baehr  S. Bahinipati  A. M. Bakich  P. Bambade  Sw. Banerjee  S.
Bansal  M. Barrett  G. Batignani  J. Baudot  M. Bauer  A. Baur  A. Beaubien
A. Beaulieu  J. Becker  P. K. Behera  J. V. Bennett  E. Bernieri  F. U.
Bernlochner  V. Bertacchi  M. Bertemes  E. Bertholet  M. Bessner  S. Bettarini
V. Bhardwaj  B. Bhuyan  F. Bianchi  T. Bilka  S. Bilokin  D. Biswas  A. Bobrov
D. Bodrov  A. Bolz  A. Bondar  G. Bonvicini  M. Bračko  P. Branchini  N. Braun
R. A. Briere  T. E. Browder  D. N. Brown  A. Budano  L. Burmistrov  S. Bussino
M. Campajola  L. Cao  G. Casarosa  C. Cecchi  D. Červenkov  M.-C. Chang  P.
Chang  R. Cheaib  P. Cheema  V. Chekelian  C. Chen  Y. Q. Chen  Y. Q. Chen
Y.-T. Chen  B. G. Cheon  K. Chilikin  K. Chirapatpimol  H.-E. Cho  K. Cho
S.-J. Cho  S.-K. Choi  S. Choudhury  D. Cinabro  L. Corona  L. M. Cremaldi  S.
Cunliffe  T. Czank  S. Das  N. Dash  F. Dattola  E. De La Cruz-Burelo  S. A.
De La Motte  G. de Marino  G. De Nardo  M. De Nuccio  G. De Pietro  R. de
Sangro  B. Deschamps  M. Destefanis  S. Dey  A. De Yta-Hernandez  R. Dhamija
A. Di Canto  F. Di Capua  S. Di Carlo  J. Dingfelder  Z. Doležal  I. Domínguez
Jiménez  T. V. Dong  M. Dorigo  K. Dort  D. Dossett  S. Dreyer  S. Dubey  S.
Duell  G. Dujany  P. Ecker  S. Eidelman  M. Eliachevitch  D. Epifanov  P.
Feichtinger  T. Ferber  D. Ferlewicz  T. Fillinger  C. Finck  G. Finocchiaro
P. Fischer  K. Flood  A. Fodor  F. Forti  A. Frey  M. Friedl  B. G. Fulsom  M.
Gabriel  A. Gabrielli  N. Gabyshev  E. Ganiev  M. Garcia-Hernandez  R. Garg
A. Garmash  V. Gaur  A. Gaz  U. Gebauer  A. Gellrich  J. Gemmler  T. Geßler
G. Ghevondyan  G. Giakoustidis  R. Giordano  A. Giri  A. Glazov  B. Gobbo  R.
Godang  P. Goldenzweig  B. Golob  P. Gomis  G. Gong  P. Grace  W. Gradl  S.
Granderath  E. Graziani  D. Greenwald  T. Gu  Y. Guan  K. Gudkova  J.
Guilliams  C. Hadjivasiliou  S. Halder  K. Hara  T. Hara  O. Hartbrich  K.
Hayasaka  H. Hayashii  S. Hazra  C. Hearty  M. T. Hedges  I. Heredia de la
Cruz  M. Hernández Villanueva  A. Hershenhorn  T. Higuchi  E. C. Hill  H.
Hirata  M. Hoek  M. Hohmann  S. Hollitt  T. Hotta  C.-L. Hsu  K. Huang  T.
Humair  T. Iijima  K. Inami  G. Inguglia  N. Ipsita  J. Irakkathil Jabbar  A.
Ishikawa  S. Ito  R. Itoh  M. Iwasaki  Y. Iwasaki  S. Iwata  P. Jackson  W. W.
Jacobs  D. E. Jaffe  E.-J. Jang  M. Jeandron  H. B. Jeon  Q. P. Ji  S. Jia  Y.
Jin  C. Joo  K. K. Joo  H. Junkerkalefeld  I. Kadenko  J. Kahn  H. Kakuno  A.
B. Kaliyar  J. Kandra  K. H. Kang  S. Kang  R. Karl  G. Karyan  Y. Kato  H.
Kawai  T. Kawasaki  C. Ketter  H. Kichimi  C. Kiesling  C.-H. Kim  D. Y. Kim
H. J. Kim  K.-H. Kim  K. Kim  S.-H. Kim  Y.-K. Kim  Y. Kim  T. D. Kimmel  H.
Kindo  K. Kinoshita  C. Kleinwort  B. Knysh  P. Kodyš  T. Koga  S. Kohani  K.
Kojima  I. Komarov  T. Konno  A. Korobov  S. Korpar  N. Kovalchuk  E.
Kovalenko  R. Kowalewski  T. M. G. Kraetzschmar  F. Krinner  P. Križan  R.
Kroeger  J. F. Krohn  P. Krokovny  H. Krüger  W. Kuehn  T. Kuhr  J. Kumar  M.
Kumar  R. Kumar  K. Kumara  T. Kumita  T. Kunigo  M. Künzel  S. Kurz  A.
Kuzmin  P. Kvasnička  Y.-J. Kwon  S. Lacaprara  Y.-T. Lai  C. La Licata  K.
Lalwani  T. Lam  L. Lanceri  J. S. Lange  M. Laurenza  K. Lautenbach  P. J.
Laycock  R. Leboucher  F. R. Le Diberder  I.-S. Lee  S. C. Lee  P. Leitl  D.
Levit  P. M. Lewis  C. Li  L. K. Li  S. X. Li  Y. B. Li  J. Libby  K. Lieret
J. Lin  Z. Liptak  Q. Y. Liu  Z. A. Liu  D. Liventsev  S. Longo  A. Loos  A.
Lozar  P. Lu  T. Lueck  F. Luetticke  T. Luo  C. Lyu  C. MacQueen  M. Maggiora
R. Maiti  S. Maity  R. Manfredi  E. Manoni  A. Manthei  S. Marcello  C.
Marinas  L. Martel  A. Martini  L. Massaccesi  M. Masuda  T. Matsuda  K.
Matsuoka  D. Matvienko  J. A. McKenna  J. McNeil  F. Meggendorfer  F. Meier
M. Merola  F. Metzner  M. Milesi  C. Miller  K. Miyabayashi  H. Miyake  H.
Miyata  R. Mizuk  K. Azmi  G. B. Mohanty  N. Molina-Gonzalez  S. Moneta  H.
Moon  T. Moon  J. A. Mora Grimaldo  T. Morii  H.-G. Moser  M. Mrvar  F. J.
Müller  Th. Muller  G. Muroyama  C. Murphy  R. Mussa  I. Nakamura  K. R.
Nakamura  E. Nakano  M. Nakao  H. Nakayama  H. Nakazawa  A. Narimani Charan
M. Naruki  A. Natochii  L. Nayak  M. Nayak  G. Nazaryan  D. Neverov  C.
Niebuhr  M. Niiyama  J. Ninkovic  N. K. Nisar  S. Nishida  K. Nishimura  M. H.
A. Nouxman  K. Ogawa  S. Ogawa  S. L. Olsen  Y. Onishchuk  H. Ono  Y. Onuki
P. Oskin  F. Otani  E. R. Oxford  H. Ozaki  P. Pakhlov  G. Pakhlova  A.
Paladino  T. Pang  A. Panta  E. Paoloni  S. Pardi  K. Parham  H. Park  S.-H.
Park  B. Paschen  A. Passeri  A. Pathak  S. Patra  S. Paul  T. K. Pedlar  I.
Peruzzi  R. Peschke  R. Pestotnik  F. Pham  M. Piccolo  L. E. Piilonen  G.
Pinna Angioni  P. L. M. Podesta-Lerma  T. Podobnik  S. Pokharel  L. Polat  V.
Popov  C. Praz  S. Prell  E. Prencipe  M. T. Prim  M. V. Purohit  H. Purwar
N. Rad  P. Rados  S. Raiz  A. Ramirez Morales  R. Rasheed  N. Rauls  M. Reif
S. Reiter  M. Remnev  I. Ripp-Baudot  M. Ritter  M. Ritzert  G. Rizzo  L. B.
Rizzuto  S. H. Robertson  D. Rodríguez Pérez  J. M. Roney  C. Rosenfeld  A.
Rostomyan  N. Rout  G. Russo  D. Sahoo  Y. Sakai  D. A. Sanders  S. Sandilya
A. Sangal  L. Santelj  P. Sartori  Y. Sato  V. Savinov  B. Scavino  M. Schnepf
M. Schram  H. Schreeck  J. Schueler  C. Schwanda  A. J. Schwartz  B. Schwenker
M. Schwickardi  Y. Seino  A. Selce  K. Senyo  I. S. Seong  J. Serrano  M. E.
Sevior  C. Sfienti  V. Shebalin  C. P. Shen  H. Shibuya  T. Shillington  T.
Shimasaki  J.-G. Shiu  B. Shwartz  A. Sibidanov  F. Simon  J. B. Singh  S.
Skambraks  J. Skorupa  K. Smith  R. J. Sobie  A. Soffer  A. Sokolov  Y.
Soloviev  E. Solovieva  S. Spataro  B. Spruck  M. Starič  S. Stefkova  Z. S.
Stottler  R. Stroili  J. Strube  Y. Sue  R. Sugiura  M. Sumihama  K. Sumisawa
T. Sumiyoshi  W. Sutcliffe  S. Y. Suzuki  H. Svidras  M. Tabata  M. Takahashi
M. Takizawa  U. Tamponi  S. Tanaka  K. Tanida  H. Tanigawa  N. Taniguchi  Y.
Tao  P. Taras  F. Tenchini  R. Tiwary  D. Tonelli  E. Torassa  N. Toutounji
K. Trabelsi  I. Tsaklidis  T. Tsuboyama  N. Tsuzuki  M. Uchida  I. Ueda  S.
Uehara  Y. Uematsu  T. Ueno  T. Uglov  K. Unger  Y. Unno  K. Uno  S. Uno  P.
Urquijo  Y. Ushiroda  Y. V. Usov  S. E. Vahsen  R. van Tonder  G. S. Varner
K. E. Varvell  A. Vinokurova  L. Vitale  V. Vobbilisetti  V. Vorobyev  A.
Vossen  B. Wach  E. Waheed  H. M. Wakeling  K. Wan  W. Wan Abdullah  B. Wang
C. H. Wang  E. Wang  M.-Z. Wang  X. L. Wang  A. Warburton  M. Watanabe  S.
Watanuki  J. Webb  S. Wehle  M. Welsch  C. Wessel  P. Wieduwilt  H. Windel  E.
Won  L. J. Wu  X. P. Xu  B. D. Yabsley  S. Yamada  W. Yan  S. B. Yang  H. Ye
J. Yelton  J. H. Yin  M. Yonenaga  Y. M. Yook  K. Yoshihara  T. Yoshinobu  C.
Z. Yuan  Y. Yusa  L. Zani  Y. Zhai  J. Z. Zhang  Y. Zhang  Y. Zhang  Z. Zhang
V. Zhilich  J. Zhou  Q. D. Zhou  X. Y. Zhou  V. I. Zhukova  V. Zhulanov  R.
Žlebčík

###### Abstract

We present a measurement of the $B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell}$
($\ell=e,\mu$) branching ratio and of the CKM parameter $|V_{cb}|$ using
signal decays accompanied by a fully reconstructed $B$ meson. The Belle II
data set of electron-positron collisions at the $\mathchar 28935\relax{(4S)}$
resonance, corresponding to 189.3 fb-1 of integrated luminosity, is analyzed.
With the Caprini-Lellouch-Neubert form factor parameterization, the parameters
$\eta_{\rm EW}F(1)|V_{cb}|$ and $\rho^{2}$ are extracted, where $\eta_{\rm
EW}$ is an electroweak correction, $F(1)$ is a normalization factor and
$\rho^{2}$ is a form factor shape parameter. We reconstruct 516 signal decays
and thereby obtain $\mathcal{B}(B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell})=\left(5.27\pm 0.22~{}\rm{\left(stat\right)}\pm
0.38~{}\rm{\left(syst\right)}\right)\%$ , $\eta_{EW}F(1)|V_{cb}|\times
10^{3}=34.6\pm 1.8~{}\rm{\left(stat\right)}\pm 1.7~{}\rm{\left(syst\right)}$,
and $\rho^{2}=0.94\pm 0.18~{}\rm{\left(stat\right)}\pm
0.11~{}\rm{\left(syst\right)}$.

Belle II, …

##  1 Introduction

A precise understanding of $B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell}$ decays
is important for future measurements of $R(D^{*})=\mathcal{B}(B\rightarrow
D^{*}\tau\nu)/\mathcal{B}(B\rightarrow D^{*}\ell\nu)$ [1, 2] and of the
magnitude of the Cabibbo-Kobayashi-Maskawa matrix element $V_{cb}$ [3, 4],
where persistent tensions exist between inclusive $B\rightarrow X_{c}\ell\nu$
and exclusive $B\rightarrow D^{*}\ell\nu$ measurements [1]. We study
$e^{+}e^{-}\rightarrow\mathchar 28935\relax{(4S)}\rightarrow$
$B^{0}{\kern-1.60004pt\kern 1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$
events, where the decay of the accompanying $B^{0}$ or $\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}$ is reconstructed in a hadronic
final state using the full event interpretation algorithm (FEI) [5] and the
signal bottom meson of opposite flavor is then reconstructed in the
$D^{*\pm}\ell^{\pm}\nu_{\ell}$ final state.

##  2 Belle II experiment

Belle II [6] is an experiment at the SuperKEKB super $B$ factory [7], an
energy-asymmetric $e^{+}$ (4 GeV) $e^{-}$ (7 GeV) collider in Tsukuba, Japan.
Collision data with an integrated luminosity corresponding to 189.3 fb-1 were
collected from March 2019 to July 2021 at a center-of-mass (c.m.) energy of
10.58 GeV, corresponding to the mass of the $\mathchar 28935\relax{(4S)}$
resonance, as well as 18.0 fb-1 at 60 MeV below the nominal c.m. energy.

The Belle II detector consists of several nested detector subsystems arranged
around the beam pipe in a cylindrical geometry. The innermost subsystem is the
vertex detector, which includes one or two layers of silicon pixels and four
outer layers of silicon strips. Outside the silicon, the central drift-chamber
reconstructs charged-particles trajectories (tracks). Outside the chamber, a
Cherenkov light-imaging and time-of-propagation detectors provide charged
particle identification. Further out is an electromagnatic calorimeter with
CsI(Tl) crystals. A uniform 1.5 T magnetic field aligned with the beam axis is
provided by a superconducting solenoid. Multiple layers of scintillators and
resistive plate chambers, located between the magnetic flux-return iron
plates, detect $K^{0}_{L}$ and muons.

The analysis uses simulated Monte Carlo (MC) samples to determine the signal
efficiency and background yields. These samples are generated using EvtGen [8]
and consist of $e^{+}e^{-}\rightarrow\mathchar 28935\relax{(4S)}\rightarrow
B\kern 1.79993pt\overline{\kern-1.79993ptB}{}$ (generic) and
$e^{+}e^{-}\rightarrow q\overline{q}$ processes, where $B$ indicates a $B^{0}$
or a $B^{+}$ meson and $q$ indicates an $u$, $d$, $c$, or $s$ quark
(continuum). The latter is simulated with KKMC [9] and PYTHIA [10]. The
luminosity of the generic and continuum samples is 1 ab-1. The signal is
modeled using the CLN form factor parameterization [11], and the time-
integrated $B^{0}{\kern-1.60004pt\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$-mixing parameter [12]. All
samples are analyzed with the basf2 framework [13, 14]. In this paper, the
natural system of units with $c=\hbar=1$ is used. The inclusion of charge-
conjugated decay modes is implied unless otherwise stated.

##  3 Event selection

The reconstruction begins by fully reconstructing a $B^{0}$ or $\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}$ ($B_{\rm tag}$) in hadronic decay
modes with the FEI algorithm [5]. The algorithm starts by selecting candidates
for stable particles, which include muons, electrons, pions, protons, kaons,
and photons, from tracks and electromagnetic energy deposits in each event.
Subsequently, the algorithm carries out several stages of reconstruction of
intermediate particles such as $\pi^{0}$, $K^{0}_{S}$, $J/\psi$, $D$ and
$D^{*}$ mesons, $\mathchar 28934\relax$, $\mathchar 28931\relax$, and
$\mathchar 28931\relax_{c}$ baryons. Intermediate particles are reconstructed
in specific decay modes from combinations of stable and other intermediate
particle candidates. The final stage of the algorithm reconstructs the $B^{0}$
mesons in 31 hadronic modes, using boosted decision trees (BDTs). The
$B_{\rm{tag}}$ candidates are required to have a BDT classifier output greater
than 0.001, a beam constrained mass
$M_{\rm{bc}}=\sqrt{E_{\rm{beam}}^{2}-|\vec{p}_{B_{\rm{tag}}}|^{2}}>5.27\,$GeV
, and an energy difference $\Delta E=E_{B_{\rm{tag}}}-E_{\rm{beam}}$ in the
interval [-0.15, 0.1] GeV, where $E_{\rm{beam}}$ is half of the collision
energy, and $\vec{p}_{B_{\rm{tag}}}$ and $E_{B_{\rm{tag}}}$ are the momentum
and energy of the $B_{\rm{tag}}$ candidate, all in the center-of-mass frame.
The efficiency of the FEI is calibrated with $B\rightarrow X\ell\nu$ decays
[5].

An event-level selection requires more than three tracks, between 2 and 7 GeV
of total energy in the electromagnetic calorimeter, and a ratio of the second
to zeroth order Fox Wolfram moments, R2 [15], to be smaller than 0.4. The
remaining $B^{0}$ or $\kern 1.79993pt\overline{\kern-1.79993ptB}{}^{0}$ meson
(the signal side $B_{\rm{sig}}$) is reconstructed in its decay of
$B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell}$ ( $D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$, $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-}$). Charged
particles are required to originate from the interaction point and have a
transverse momentum greater than 0.2 GeV. To identify electrons and muons, a
likelihood-ratio like quantity for each particle hypothesis is calculated,
which combines information from several detector subsystems. The likelihood
performance is calibrated with well-known physics processes. In addition,
electron and muon momenta are required to be greater than 1.0 GeV in the
center-of-mass frame to reject continuum background. Kaon and pion candidates
are combined to reconstruct $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-}$ candidates
whose invariant mass $\left(m\left(K\pi\right)\right)$ is required to be
within the interval [1.85, 1.88] GeV. The $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ candidates are combined with an
additional low-momentum pion to reconstruct $D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ candidates restricted to the
$D^{*-}$\- $\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ mass difference
$\Delta m$ in the range [0.143, 0.149] GeV. Subsequently, $B_{\rm{sig}}$
candidates are reconstructed by combining $D^{*-}$ candidates with either
$e^{+}$ or $\mu^{+}$ candidates. At least one combination of $B_{\rm{tag}}$
and $B_{\rm{sig}}$ candidates is required with no remaining tracks. The
missing neutrino mass squared
($m^{2}_{\rm{miss}}=(P_{\rm{beam}}-P_{B_{\rm{tag}}}-P_{D^{*}}-P_{\ell})^{2}$,
where $P$ denotes a four vector) is required to be in the range [-0.5, 0.5]
GeV2. If multiple combinations of $B_{\rm{tag}}$ and $B_{\rm{sig}}$ candidates
are found in an event, the candidate with the largest BDT output for the
$B_{\rm{tag}}$ and the best $\Delta m$ for the $B_{\rm{sig}}$ is selected.
Figure 1 shows the $m\left(K\pi\right)$, $\Delta m$, and $m_{\rm{miss}}^{2}$
distributions. The figures include data points with statistical uncertainties
and histograms for simulated signal and background candidates scaled to the
equivalent data luminosity. The signal yield is estimated by counting the
number of selected events on data from which simulated background is
subtracted. Checks based on data in the $\Delta m$ sidebands show good
agreement between simulated and experimental background distributions.

Figure 1:  Distributions of $K\pi$ invariant mass (left), $D^{*}$\- $D^{0}$
mass difference (middle), and missing neutrino mass squared (right) of
$B^{0}\rightarrow D^{*-}e^{+}$ $\nu_{e}$ (top) and $B^{0}\rightarrow
D^{*-}\mu^{+}\nu_{\mu}$ (bottom) candidates in data (points) and simulation
(histograms). Vertical lines enclose the regions of the selected events.

##  4 Measurement of branching ratio

The branching ratio for the decay $B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell}$
is estimated as

$\displaystyle\mathcal{B}\left(B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell}\right)$
$\displaystyle=\frac{\left(N^{\rm{rec}}-N^{\rm{bg}}\right)\epsilon^{-1}}{4N_{B\kern
1.25995pt\overline{\kern-1.25995ptB}{}}\left(1+f_{+0}\right)^{-1}\mathcal{B}\left(D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\right)\mathcal{B}\left(\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-}\right)},$
(1)

where $N^{\rm{rec}}$ is the number of reconstructed events in data,
$N^{\rm{bg}}$ is the number of reconstructed background events, $\epsilon$ is
the signal reconstruction efficiency, $N_{B\kern
1.25995pt\overline{\kern-1.25995ptB}{}}$ is the number of produced $B\kern
1.79993pt\overline{\kern-1.79993ptB}{}$ pairs, and $f_{+0}$ is the ratio of
the number of produced $B^{+}{\kern-1.60004ptB^{-}}$ and
$B^{0}{\kern-1.60004pt\kern 1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$
pairs. In Eq. 1, the values corresponding to electron and muon modes are
averaged. The values of $N^{\rm{bg}}$ and $\epsilon$ are estimated from the
background and signal simulation. The value of $N_{B\kern
1.25995pt\overline{\kern-1.25995ptB}{}}$ is determined using the R2
distribution after a subtraction of the continuum background using off-
resonance data. The values for $f_{+0}$,
$\mathcal{B}(D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0})$, $\mathcal{B}(\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-})$, and
fraction of mixed and unmixed $B^{0}{\kern-1.60004pt\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$ are taken from Ref. [12]. The
input values for the branching fraction measurement are summarized in Table 1.

Table 1:  Input values for the measurement of branching ratio with the systematic uncertainties, described in Sec. 6.  Variables | Values
---|---
$N^{\rm{rec}}$ |  $545$ (data)
$N^{\rm{bg}}$ | $29.4\pm 11.2$
$\epsilon$ | $(9.55\pm 0.67)\times 10^{-4}$
$N_{B\kern 1.25995pt\overline{\kern-1.25995ptB}{}}$ | $(197.17\pm 5.72)\times 10^{6}$
$f_{+0}$ | $1.058\pm 0.024$
$\chi_{d}$ | $0.1875\pm 0.0017$
$\mathcal{B}(D^{*-}\rightarrow\pi^{-}\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0})$ | $(67.7\pm 0.5)\%$
$\mathcal{B}(\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-})$ | $(3.950\pm 0.031)\%$

##  5 Measurement of $|V_{cb}|$

Figure 2 shows distribution of the recoil variable

$\displaystyle w=\frac{P_{B}\cdot
P_{D^{*}}}{m_{B}m_{D^{*}}}=\frac{m^{2}_{B}+m^{2}_{D^{*}}-q^{2}}{2m_{B}m_{D^{*}}},$
(2)

where $q^{2}=(P_{\ell}+P_{\nu_{\ell}})^{2}$ and $m_{B,D^{*}}$ are the known
masses of the indicated particles. The $B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell}$ decay-width differential in $w$ is as follows [4,
16]:

$\displaystyle\frac{d\Gamma}{dw}=\frac{\eta_{\rm
EW}^{2}\color[rgb]{0,0,0}G^{2}_{F}}{48\pi^{3}}m^{3}_{D^{*}}(m_{B}-m_{D^{*}})^{2}g(w)F^{2}(w)|V_{cb}|^{2}.$
(3)

Here, $\eta_{\rm EW}$ is an electroweak correction (calculated to be
$1.00662\pm 0.00016$ in Ref. [17]). From lattice QCD, $F(1)$ is calculated as
$0.906\pm 0.004~{}\rm{(stat)}\pm 0.012~{}\rm{(syst)}$ [17]. The product
$g(w)F^{2}(w)$ describes the phase-space factor and the form factor, which is
parameterized with $R_{1}(1),R_{2}(1),\rho^{2}$ in the CLN approach [11]. The
CKM matrix element $|V_{cb}|$ is determined by fitting the
$\Delta\Gamma/\Delta w$ distribution with the form factor parameters. However,
here the product $\eta_{\rm EW}F(1)|V_{cb}|$ is measured instead, in order to
separate theory uncertainty of $\eta_{\rm EW}$ and $F(1)$. In this paper, the
$R_{1}(1)$ and $R_{2}(1)$ values are taken from external measurements [1], as
shown in Table 2.

Figure 2:  Distributions of $w$ for $B^{0}\rightarrow D^{*-}e^{+}\nu_{e}$ (left) and $B^{0}\rightarrow D^{*-}\mu^{+}\nu_{\mu}$ (right) candidates in data (points) and simulation (histogram) after the event selection. The shaded band shows the systematic uncertainty of the simulation, which is summarized in Sec. 6.  Table 2:  Input values for $R_{1}$(1) and $R_{2}$(1) [1].  $R_{1}(1)$ | 1.270 $\pm$ 0.026
---|---
$R_{2}(1)$ | 0.852 $\pm$ 0.018
Correlation coefficient of $R_{1}(1)$ and $R_{2}(1)$ | -0.715

###  5.1 Unfolding method

In order to estimate the true $w$ distribution from the observed $w$ values,
an iterative unfolding method is used [18]. The number of signal events
populating $w$ bin, $N_{i}$, is estimated from the reconstructed variables as
follows:

$\displaystyle N_{i}=\sum_{j}U_{ij}(N^{\rm{rec}}_{j}-N^{\rm{bg}}_{j}),$ (4)

where $i$ is the $w$ bin number. We define 10 bins in the range $[1.0,1.5]$
each with a width of 0.05. The matrix
$U_{ij}=P(w^{\rm{true}}_{i}|w^{\rm{rec}}_{j})$ models the probability that
events reconstructed in the $w$ bin $j$ are in the true-$w$ bin $i$, which is
calculated by Bayes’ theorem according to

$\displaystyle U_{ij}$ $\displaystyle=P(w^{\rm true}_{i}|w^{\rm rec}_{j})$
$\displaystyle=P(w^{\rm rec}_{j}|w^{\rm true}_{i})\times P(w^{\rm
true}_{i})/P(w^{\rm rec}_{j})$ $\displaystyle=P(w^{\rm rec}_{j}|w^{\rm
true}_{i})\times P(w^{\rm true}_{i})/\sum_{k}P(w^{\rm rec}_{j}|w^{\rm
true}_{k})P(w^{\rm true}_{k}).$ (5)

Here, $P(w^{\rm{rec}}_{j}|w^{\rm{true}}_{i})$ is estimated with simulation. To
avoid bias from the simulated signal, $P(w^{\rm true})$ is calculated using
the reconstructed $w$ distribution on data as follows.

  1. 1.

$P(w^{\rm true}_{i})$ is assumed uniform ($P(w^{\rm true}_{i})=0.1$ for all
bins).

  2. 2.

$U_{ij}$ is calculated by using Eq.(5).

  3. 3.

$P(w^{\rm true}_{i})$ is set to $\sum_{j}U_{ij}(N^{\rm rec}_{j}-N^{\rm
bg}_{j})/\sum_{ij}U_{ij}(N^{\rm rec}_{j}-N^{\rm bg}_{j})$.

  4. 4.

Steps 2. and 3. are repeated 10 times, until $U_{ij}$ converges.

The unfolding performance is validated with the simulation.

###  5.2 Fitting method

To determine $|V_{cb}|$, a binned maximum likelihood fit is performed using

$\displaystyle\Delta\chi^{2}=-2\ln\left(L\right)=2\sum_{i}\left(N^{\rm{exp}}_{i}-N_{i}+N_{i}\ln\left(N_{i}/N^{\rm{exp}}_{i}\right)\right)+\prod_{i}\prod_{j}\Delta
x_{i}W^{-1}_{ij}\Delta x_{j},$ (6)

where $i$ denotes the $w$ bin, $N_{i}$ is the number of observed events in the
$i$th bin, $x_{i}$ is a systematic parameter defined as the normalization
uncertainty in the $i$th reconstructed-$w$ bin, $\Delta x_{i}$ is the
deviation of the systematic parameters from the nominal value, and $W_{ij}$ is
the covariance of the systematic parameters, modeled by multivariate Gaussians
functions. Finally, $N^{\rm{exp}}_{i}$ is the expected yield in the $i$th bin,
which is written as follows:

$\displaystyle N^{\rm{exp}}_{i}\left(B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell}\right)$ $\displaystyle=4\epsilon_{i}N_{B\kern
1.25995pt\overline{\kern-1.25995ptB}{}}\left(1+f_{+0}\right)^{-1}\tau\left(B^{0}\right)\mathcal{B}\left(D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\right)\mathcal{B}\left(\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-}\right)$
$\displaystyle\left(1+\sum_{j}U_{ij}\Delta
x_{j}\right)\int^{w^{max}_{i}}_{w^{min}_{i}}dw\frac{d\Gamma}{dw}\left(B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell}\right),$ (7)

where $\epsilon_{i}$ is the signal reconstruction efficiency in the $i$th bin.
The differential distribution is obtained using Eq.(3). In the fit there are
two free parameters, $\eta_{\rm EW}F(1)|V_{cb}|$ and $\rho$, and ten nuisance
parameters $\Delta x_{i}$. The two-dimensional contour of
$\eta_{EW}F(1)|V_{cb}|$ and $\rho$ is estimated by using a marginalized
likelihood [19],

$\displaystyle
L_{\rm{marg}}=\frac{1}{J}\sum_{j=1}^{J}\exp\left(-\sum_{i}\left(N^{\rm{exp}}_{ij}-N_{i}+N_{i}\ln\left(N_{i}/N^{\rm{exp}}_{ij}\right)\right)\right),$
(8)

where $J=10000$ and $N^{\rm{exp}}_{ij}$ is the expected yield in the $i$th bin
with the $j$th set of nuisance parameters, which is generated following the
covariance matrix. The fitter performance is validated with simplified
simulated experiments.

##  6 Systematic uncertainties

Systematic uncertainties are evaluated for several sources associated with the
detector response, MC modeling, and physics inputs. For the branching ratio
measurement, the systematic uncertainty of each source is propagated to the
result based on Eq.(1) and summarized in Table 3. The $B_{\rm tag}$
reconstruction efficiency with the FEI algorithm is studied using
$B\rightarrow X\ell\nu$ decays and a systematic uncertainty of 3.9% is
assigned [5]. The tracking efficiency is studied with $\tau$ decays and the
maximum data-simulation difference of 0.3% is taken as systematic uncertainty
for each track in the final state. The reconstruction efficiency of the low
momentum $\pi^{-}$ is studied by using
$B^{0}\rightarrow\pi^{+}D^{*-}(D^{*-}\rightarrow\pi^{-}\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0})$ decays. The data-MC ratio of the
$\pi$ momentum distribution is evaluated relative to the high momentum
distribution; a 3–4% systematic uncertainty is assigned in each momentum bin,
which is dominated by the statistical uncertainty of the control samples.
Electron and muon identification efficiencies and misidentification rates are
studied by using $e^{+}e^{-}\rightarrow e^{+}e^{-}\ell^{+}\ell^{-}$,
$e^{+}e^{-}\rightarrow e^{+}e^{-}(\gamma)$,
$e^{+}e^{-}\rightarrow\mu^{+}\mu^{-}\gamma$, decays of $J/\psi$, $D^{*}$,
$\tau$, and $K_{s}^{0}$. The lepton identification and misidentification
uncertainties associated with the size of the control samples, background
contamination, modeling of the fitting function, trigger, and the difference
of the results across samples are evaluated as a functions of each lepton
angle and the absolute value of the lepton momentum. These uncertainties are
propagated to the branching fraction measurement resulting in a total 2.0%
systematic error. The potential variations in the amount of background from
$B\rightarrow D^{**}\ell\nu$ decays, hadronic $B$ decays and misreconstructed
$D^{*}$ mesons are evaluated to propagate the uncertainty of the branching
fraction of the background processes and of beam backgrounds resulting in a
1.2% systematic uncertainty. The number of produced $B\kern
1.79993pt\overline{\kern-1.79993ptB}{}$ pairs is estimated from the R2
distribution after a subtraction of the continuum background using off-
resonance data. A systematic uncertainty of 2.9% is assigned to account for
the limited statistics of off-resonance data, operation conditions of the
detector and accelerator including beam energy, and selection efficiencies. A
systematic uncertainty for the event-level selection is estimated to be 1.0%,
to cover the maximum data-simulation difference of the total energy in the
electromagnetic calorimeter. The uncertainty from the limited size of
simulated samples is estimated to be 1.8%. The following sources of systematic
uncertainty are from external measurements: the ratio of the number of
produced $B^{+}{\kern-1.60004ptB^{-}}$ and $B^{0}{\kern-1.60004pt\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$ pairs (1.2%), the ratio of the
number of mixed and unmixed $B^{0}{\kern-1.60004pt\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}}$ (0.9%), the branching fractions
of $D^{*-}\rightarrow\pi^{-}\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}$
(0.7%) and $\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow
K^{+}\pi^{-}$ (0.8%), and form factors (0.1%) [12]. The uncertainties from the
various sources are assumed to be independent and the quadratic sum is taken
as a total systematic uncertainty. For the measurement of
$\eta_{EW}F(1)|V_{cb}|$ and $\rho^{2}$, the effect of the systematic
uncertainty is included in the likelihood calculation (the second term in
Eq.(6)) with the covariance matrix

$\displaystyle
W_{ij}=\sum_{k}\frac{\left(N_{i}^{k}-\mu_{i}\right)\left(N_{j}^{k}-\mu_{j}\right)}{\mu_{i}\mu_{j}}.$
(9)

Here, $k$ runs over the sources of uncertainties, $\mu_{i}$ is the mean of the
expected yield in the $i$th $w$ bin, $N_{i}^{k}$ is the variation of the
expected yield in the $i$th bin for the $k$th source of uncertainties. Figure
3 shows the estimated covariance matrix.

Table 3:  Summary of fractional systematic uncertainties on the branching ratio.  Systematic sources | Relative uncertainty (%)
---|---
FEI efficiency | 3.9
Low momentum $\pi$ efficiency | 4.1
Tracking efficiency | 0.9
Lepton particle identification | 2.0
Background | 1.2
$N_{B\kern 1.25995pt\overline{\kern-1.25995ptB}{}}$ | 2.9
$f_{+0}$ | 1.2
Number of mixed $B\kern 1.79993pt\overline{\kern-1.79993ptB}{}$ | 0.9
$\mathcal{B}\left(D^{*-}\rightarrow\pi^{-}\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\right)$ | 0.7
$\mathcal{B}\left(\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0}\rightarrow K^{+}\pi^{-}\right)$ | 0.8
ECL energy | 1.0
Form factor | 0.1
MC sample size | 1.8
Total | 7.3
Figure 3:  Total covariance matrix for the $\eta_{EW}F(1)|V_{cb}|$ and
$\rho^{2}$ measurement. The axes denote the $w$ bin intervals.

##  7 Results and conclusion

The result for the branching fraction is

$\displaystyle\mathcal{B}\left(B^{0}\rightarrow
D^{*-}\ell^{+}\nu_{\ell}\right)$ $\displaystyle=\left(5.27\pm
0.22~{}\left(\rm{stat}\right)\pm 0.38~{}\left(\rm{syst}\right)\right)\%$ (10)

while the results for $|V_{cb}|$ are

$\displaystyle\eta_{EW}F(1)|V_{cb}|\times 10^{3}$ $\displaystyle=34.6\pm
1.8~{}\left(\rm{stat}\right)\pm 1.7~{}\left(\rm{syst}\right)$ (11)
$\displaystyle\rho^{2}$ $\displaystyle=0.94\pm
0.18~{}\left(\rm{stat}\right)\pm 0.11~{}\left(\rm{syst}\right).$ (12)

The two-dimensional probability contours for $\eta_{EW}F(1)|V_{cb}|$ and
$\rho^{2}$ are shown in Fig. 4. The observed $\ \Delta\Gamma/\Delta w$ values
are shown in Fig. 5 with the best fit function overlaid. The reduced
$\chi^{2}$ of the fit is 1.6 with p-value of 40.7$\,\%$, which is estimated by
simulation. Under the assumption that $\eta_{\rm EW}=1.00662\pm 0.00016$ and
$F(1)=0.906\pm 0.004~{}\rm{(stat)}\pm 0.012~{}\rm{(syst)}$ [17], we obtain
$|V_{cb}|\times 10^{3}=37.9\pm 2.7$. The results are consistent with the world
averages of $\mathcal{B}(B^{0}\rightarrow D^{*-}\ell^{+}\nu_{\ell})=(5.06\pm
0.12)\%$ and $\eta_{EW}F(1)|V_{cb}|\times 10^{3}=35.27\pm 0.38$ based on
exclusive $B\rightarrow D^{*}\ell\nu_{\ell}$ decays within one standard
deviation [1].

Figure 4:  Two dimensional probability contours for $\eta_{EW}F(1)|V_{cb}|$
and $\rho^{2}$ at the 68% (solid) and 90% (dashed) confidence level. The best
fit point is also shown.  Figure 5:  Observed $d\Gamma(B^{0}\rightarrow
D^{*}\ell\nu)/dw$ distribution with the best fit function and one and two
standard-deviation bands overlaid.

##  8 Acknowledgement

These acknowledgements are not to be interpreted as an endorsement of any
statement made by any of our institutes, funding agencies, governments, or
their representatives.

We thank the SuperKEKB team for delivering high-luminosity collisions; the KEK
cryogenics group for the efficient operation of the detector solenoid magnet;
the KEK computer group and the NII for on-site computing support and SINET6
network support; and the raw-data centers at BNL, DESY, GridKa, IN2P3, INFN,
and the University of Victoria for offsite computing support.

## References

  * [1] Y. S. Amhis et al., HFLAV, Averages of b-hadron, c-hadron, and $\tau$-lepton properties as of 2018, Eur. Phys. J. C 81 (2021) no. 3, 226, arXiv:1909.12524 [hep-ex].
  * [2] F. U. Bernlochner, M. F. Sevilla, D. J. Robinson, and G. Wormser, Semitauonic b-hadron decays: A lepton flavor universality laboratory, Rev. Mod. Phys. 94 (2022) no. 1, 015003, arXiv:2101.08326 [hep-ex].
  * [3] N. Cabibbo, Unitary Symmetry and Leptonic Decays, Phys. Rev. Lett. 10 (Jun, 1963) 531–533.
  * [4] M. Kobayashi and T. Maskawa, CP Violation in the Renormalizable Theory of Weak Interaction, Prog. Theor. Phys. 49 (1973) 652–657.
  * [5] F. Abudinén et al., Belle-II, A calibration of the Belle II hadronic tag-side reconstruction algorithm with $B\rightarrow X\ell\nu$ decays, arXiv:2008.06096 [hep-ex].
  * [6] T. Abe et al., Belle-II, Belle II Technical Design Report, arXiv:1011.0352 [physics.ins-det].
  * [7] K. Akai, K. Furukawa, and H. Koiso, SuperKEKB, SuperKEKB Collider, Nucl. Instrum. Meth. A 907 (2018) 188–199, arXiv:1809.01958 [physics.acc-ph].
  * [8] D. J. Lange, The EvtGen particle decay simulation package, Nucl. Instrum. Meth. A 462 (2001) 152–155.
  * [9] B. F. L. Ward, S. Jadach, and Z. Was, Precision calculation for e+ e- —$>$ 2f: The KK MC project, Nucl. Phys. B Proc. Suppl. 116 (2003) 73–77, arXiv:hep-ph/0211132.
  * [10] T. Sjostrand, S. Mrenna, and P. Z. Skands, A Brief Introduction to PYTHIA 8.1, Comput. Phys. Commun. 178 (2008) 852–867, arXiv:0710.3820 [hep-ph].
  * [11] I. Caprini, L. Lellouch, and M. Neubert, Dispersive bounds on the shape of form factors, .
  * [12] P. Zyla et al., Particle Data Group, Review of Particle Physics, PTEP 2020 (2020) no. 8, 083C01.
  * [13] T. Kuhr, C. Pulvermacher, M. Ritter, T. Hauth, and N. Braun, Belle-II Framework Software Group, The Belle II Core Software, Comput. Softw. Big Sci. 3 (2019) no. 1, 1, arXiv:1809.04299 [physics.comp-ph].
  * [14] The Belle II Collaboration, Belle II Analysis Software Framework (basf2), https://doi.org/10.1007/s41781-018-0017-9.
  * [15] G. C. Fox and S. Wolfram, Observables for the Analysis of Event Shapes in e+ e- Annihilation and Other Processes, Phys. Rev. Lett. 41 (1978) 1581.
  * [16] M. Neubert, Heavy quark symmetry, Phys. Rept. 245 (1994) 259–396, arXiv:hep-ph/9306320.
  * [17] J. A. Bailey et al., Fermilab Lattice, MILC, Update of $|V_{cb}|$ from the $\bar{B}\rightarrow D^{*}\ell\bar{\nu}$ form factor at zero recoil with three-flavor lattice QCD, Phys. Rev. D 89 (2014) no. 11, 114504, arXiv:1403.0635 [hep-lat].
  * [18] G. Cowan, A survey of unfolding methods for particle physics, Conf. Proc. C 0203181 (2002) 248–257.
  * [19] K. A. Olive et al., Particle Data Group, Review of Particle Physics, Chin. Phys. C 38 (2014) 090001.


11institutetext: Panasonic Holdings Corporation 11email:
<EMAIL_ADDRESS>
22institutetext: Chubu University 22email<EMAIL_ADDRESS>
# Few-shot Adaptive Object Detection
with Cross-Domain CutMix

Yuzuru Nakamura equal contribution11 Yasunori Ishii 1⋆1⋆ Yuki Maruyama 11
Takayoshi Yamashita 22

###### Abstract

In object detection, data amount and cost are a trade-off, and collecting a
large amount of data in a specific domain is labor-intensive. Therefore,
existing large-scale datasets are used for pre-training. However, conventional
transfer learning and domain adaptation cannot bridge the domain gap when the
target domain differs significantly from the source domain. We propose a data
synthesis method that can solve the large domain gap problem. In this method,
a part of the target image is pasted onto the source image, and the position
of the pasted region is aligned by utilizing the information of the object
bounding box. In addition, we introduce adversarial learning to discriminate
whether the original or the pasted regions. The proposed method trains on a
large number of source images and a few target domain images. The proposed
method achieves higher accuracy than conventional methods in a very different
domain problem setting, where RGB images are the source domain, and thermal
infrared images are the target domain. Similarly, the proposed method achieves
higher accuracy in the cases of simulation images to real images.

##  1 Introduction

Systems used in various environments throughout the day, such as autonomous
driving and surveillance robots, are being put to practical use. These systems
require high accuracy throughout the day. Infrared cameras can capture visible
images even in such situations, and they can robustly detect objects. To
achieve high accuracy, object detection models require a large amount of
labeled training data. It is easy to use a large amount of labeled training
data of RGB images [1, 2, 3, 4, 5, 6]. However, most of the images are
captured using an RGB camera, and a few images are captured using an infrared
camera (hereinafter infrared images). Therefore, to achieve high accuracy,
detection models need to train with a few labeled infrared images.

One of the methods for training high accuracy detection model is transfer
learning using pre-trained model trained with RGB images. However, it is
difficult to improve the accuracy if the domain gap between RGB and infrared
images is large [7, 8]. This phenomenon, called negative transfer, occurred
under the large domain gap between the training images for pre-trained model
and the ones for fine-tuning model. As one of the conventional methods for
overcoming the domain gap, there are style transformation methods such as
CycleGAN [9, 10]. GAN-based style transformations can easily convert between
images with similar spectra. However, the style transformation is difficult
when the spectral distributions are significantly different, such as in RGB
and infrared images. A method using GRL [11, 12, 13] is proposed as a training
method for domain adaptation in the feature space. However, if the spectral
distributions between the input images are significantly different, it is
difficult to align the distributions of different domains because the feature
distributions are extremely different. The methods for training features that
interpolate between two images are as follows: Mixup [14], BC-learning [15],
CutMix [16], and CutDepth [17]. These methods train features located between
two images by mixing the two images or by replacing a portion of the image
with the other image. These data augmentation methods synthesize features with
a mixture of different domains.

We propose a few shot adaptive object detection with cross-domain CutMix. We
take advantage of the fact that data augmentation can reduce the domain gap by
mixing features of domains with a large domain gap. Our method enables highly
accurate object detection even for a few annotated infrared images based on a
pre-trained model of RGB images. We paste a part of one domain’s image onto a
part of another domain’s image, such as CutMix, because we overcome large
domain gap. Particularly, in object detection task, the size of detection
targets is smaller than that of the background. Therefore, to perform domain
adaptation of small detection object, we cut out the detection object and
paste it onto the other domain instead of randomly cutting out the image. Even
if there is a significant difference in appearance between domains, the
features of the detected objects between domains are trained to be similar to
each other.

Additionally, we adapt the domain using feature-based adversarial learning. In
conventional methods, the discriminator of the domain identification label
does not change during training. However, the domain identification label also
needs to change the label according to the pasted area because our CutMix-
based method changes the pasting area during training. The conventional domain
identification label cannot be used by pasting an image of another domain.
Therefore, the domain identification label should be the same domain label as
the input image to which the image of another domain is pasted. Since the
correct domain label can be assigned according to the pasting position of the
object, feature-level domain alignment can be performed even when the input
image is changed, as in the proposed method.

Our contributions are as follows: we propose a few shot adaptive object
detection with cross-domain CutMix so that we can adapt the domain, which
looks significantly different. Furthermore, we propose an input image
synthesizing method based on CutMix for cross-domains and domain
identification label in discriminator for that. Through experiments, we show
the effectiveness of the proposed method using RGB images as a pre-trained
model and data from multiple domains such as RGB and thermal infrared images.

##  2 Related Work

###  2.1 Object Detection for Each Domain

Most object detection methods have been studied for RGB images [18, 19, 20].
These methods can be roughly divided into two-stage and one-stage detection
methods. R-CNN and its extended technologies [21, 22, 23] represent the two-
stage methods. YOLO [24], SSD (Single Shot Multi-box Detector) [25], and their
extended technologies represent the one-stage methods. Additionally, object
detection techniques that use transformer have been proposed [26, 27, 28, 29,
30], and they are expected that it will be applied to various environments.

There is research on applications in the real environment such as robots [31,
32], drones [33, 34], object detection for in-vehicle cameras [35, 36], and
license plate detection [37]. Many datasets [1, 2, 3, 4, 5, 6] that can be
used to train object detection are available to the public. The night scenes
on these datasets are fewer than daytime scenes. Furthermore, visibility is
poor because the pixel value of the subject in the RGB image captured at night
is small. Thus, it is difficult to achieve high detection performance using
RGB images both day and night.

Highly accurate object detection with in-vehicle cameras and outdoor drones is
required for both day and night. Some methods for detecting objects using
spectra information other than RGB images were proposed to detect objects with
high accuracy on both day and night. Lu et al. [38] proposed object detection
using RGB and infrared images in the framework of weakly supervised learning.
This method focuses on the use of multispectral information, and it detects
objects using a roughly aligned RGB image and an infrared image. Liu et al.
[39] and Konig et al. [40] proposed methods for inputting RGB and thermal
infrared images into a deep learning model and fusing their features. Highly
accurate object detection is possible under various lighting conditions using
RGB and infrared images simultaneously. These methods are algorithms that
assume that there are numerous RGB and infrared images. Thus, they can be used
if the RGB and the infrared images can be photographed in large quantities and
annotated in the same environment as the inference scene. However, the cost of
collecting data and annotating in each application environment is high in
reality.

###  2.2 Knowledge Transfer to Different Domain

To reduce the cost of preparing infrared images and training high accurate
object detection models, domain adaptation and transfer learning use a pre-
trained model with a small number of labeled infrared and RGB images.

Akkaya et al. [41] proposed unsupervised domain adaptation between a model
taken from numerous RGB and thermal infrared images for image classification.
Vibashan et al. [42] used paired RGB and thermal infrared images to perform
domain adaptation for object detection. When only the recognized object is
displayed as in the image classification, the domains of both the difference
in the sensor and shooting scene can be applied. However, when the background
area without objects occupies a large area in object detection, it is
difficult to adapt the domain of both the sensor difference and the shooting
scene difference without using a pair of datasets. Thus, it is still a problem
to adapt between different sensors and scenes for object detection.

There is a knowledge transfer method, which is by fine-tuning infrared images,
using a model trained with RGB images as pre-trained model. For fine-tuning to
be effective, the feature of training data between source domain and target
domain must be similar. RGB and infrared images have extremely different
spectrum to be imaged, and they look very different even if they have the same
object and color. Negative transfer [7, 8] occurs because of the difference in
the distribution of these data. Therefore, knowledge transfer using a small
number of labeled infrared images is difficult. In both domain adaptation and
fine-tuning, the key to improving performance is transferring knowledge while
making the differences between domains closer.

##  3 Proposed Method

###  3.1 Overview

In this paper, we propose high-accuracy object detection on infrared images
using a large number of labeled RGB images and a small number of labeled
infrared images. RGB and infrared images receive different spectra; thus,
there is a significant difference in appearance, which is a large gap between
domains. Therefore, we not only align the gaps between domains at the feature
level using methods such as adversarial learning but also explicitly reduce
the gaps between domains at the image level. This improves the accuracy of
domain adaptation by converting the input image to conditions that make it
adapt the domain easier.

We propose Object aware Cross-Domain CutMix (OCDC) and OCDC-based
Discriminator Label (OCDCDL) based on the domain for each location. Figure 1
shows our proposed framework. We explain the outline of the proposed method
using the model of the domain adaptation method based on adversarial learning
proposed by Han-Kai et al. [43] as an example. This method trais using the
loss of both object detection and adversarial learning to reduce the
difference between domains. This proposed method is simple and easy to
incorporate into the type of domain adaptation that uses adversarial learning
in object detection problems, which uses few labeled images.

OCDC (Fig. 1 (a)) is a method for cutting out an object area and pasting a
part of the image between domains to reduce the gap between the source and the
target domains. Zhou et al. showed that there is a domain generalization
effect by mixing images with different domains in a batch [44]. Inspired by
that study, we focused on mixing object units, which is important for object
detection. When the entire image is mixed, it is trained to reduce the
distance between the background domains that occupy most of the image. In the
proposed method, the distance between domains is emphasized rather than the
background features, so the object detection performance is expected to
improve.

OCDCDL (Fig. 1 (b)) is a method for adaptively changing the domain label of
the discriminator based on the pasting position of other domain images. By
converting the input image using OCDC, the domain identification label is no
longer one value. The output feature using the input image includes
information from multiple domains by pasting an image of another domain on the
input image using OCDC. Thus, the conventional single identification label
cannot correctly discriminate the domain. By adaptively changing the label
based on the OCDC, the discriminator makes it possible to discriminate the
domain even if there is information on different domains in the image.

Figure 1:  The framework of our proposed method; (a) OCDC mitigates the image-
level domain gap by cutting out the object area and pasting them in each other
domain. (b) We adaptively determine the domain identification label based on
the area pasted by the OCDC. The pasted object, which serve areas as new
ground truth, are input into the detection network, and the detection and the
adversarial losses are calculated.

###  3.2 Object aware Cross-Domain CutMix (OCDC)

For domain adaptation to data with a large gap between domains such as RGB and
infrared images, we propose a method of aligning domains at the image level in
addition to the conventional method of aligning domains using features. A
method based on data augmentation called DomainMix augmentation [45] was
proposed: as a method for reducing the domain gap at the image level. This
method reduces the difference in appearance between images by simply
connecting different domains without using a deep generative model. Mixup
[14], BC-learning [15], CutMix [16], and CutDepth [17] are data augmentation
methods to be mixed at the image level. These studies mention that mixing
images can generate features that interpolate two images. By mixing them at
the image level, the features between the domains in the input image can be
brought closer to each other.

However, the background area without objects occupies most of the image, and
the objects that should align the domains are only part of the image. Thus, in
Mixup or CutMix, which uses the entire image or mixes random regions,
respectively, the gap in the background domain is small, but the gap in the
domain of the object to be detected is not always small. Therefore, we propose
an OCDC to cut out the object area existing in one domain and paste it onto
the other domain. The domain gap is reduced at the image level by cutting out
the object area from the source and target and pasting them together.

The pasting process is performed for each training iteration. The images of
the source domain and that of the target domain included in the batch are
used. The object to be detected is cut out from those images based on the
ground truth coordinates. The object image cut out from the image of the
source domain is pasted onto the image of the target domain selected at
random. On the other hand, the image cut out from the object image of the
target domain is pasted onto the image of the source domain selected at
random. The ground truth labels for detection loss are updated by adding the
ground truth label of the pasted image of another domain to that of the image
where the image of another domain is pasted.

If the object images are pasted while the objects already exist, the
originally existing objects will be hidden. The loss of object detection
becomes large because it is difficult to detect such hidden objects.
Therefore, we decide the pasting position of the image based on the overlap
between the image to be pasted and the object of the image to which the object
image is pasted. If the area where the object is hidden increases more than a
certain percentage after pasting the image, the pasting position is reselected
when deciding the pasting position. This prevents the original object from
being hidden after pasting.

Additionally, to train the position and label of an object, the object
detection model pastes an object image to a real-world location. For example,
if the domain is adapted between the images of the in-vehicle camera, which is
installed at almost the same position, the coordinates before and after
pasting do not change significantly. Alternatively, the object image is pasted
at the same position as the position before pasting or at a slightly shifted
position.

The domain adaptation for object detection that has been used so far has
insufficient consideration for object detection. On the other hand, our method
using cutting out the position of objects is a new point of view that the
domain gap of the object to be recognized can be reduced. In addition, we
argue that it causes problems in adversarial learning and propose a solution.
In the following subsections, each proposed method is explained concretely.

###  3.3 OCDC-based Discriminator Label (OCDCDL)

In domain adaptation using adversarial learning such as a method [43],
features calculated from the source or target domains are input into the
discriminator. In our proposed OCDC, the information of another domain is
included in a part of the feature because the image of another domain is
pasted on the part of the input image. Therefore, when one value is used for
the domain identification label of the discriminator as in the conventional
method, the loss of the area where the image of another domain is pasted
cannot be calculated correctly.

The discriminator is trained so that neither of the two domains can be
discriminated. We append the discriminator $D$ after the backbone $F$. The
input image of the source domain and the target domain are $I_{S}$ and
$I_{T}$, respectively. $D$ outptus a domain prediction map of each pixel
$D(\cdot)_{h,w}$. The source domain identification label and target domain
identification label are $d=0$ and $d=1$, respectively. Eq. 1 is a adversarial
loss $\mathcal{L}_{adv}$ and Eq. 2 is an overall loss $\mathcal{L}$.

$\mathcal{L}_{adv}(F(I))=-\Sigma_{h,w}[d\log
D(F(I))_{h,w}+(1-d)\log(1-D(F(I))_{h,w})],$ (1)
$\min_{F}\max_{D}\mathcal{L}(I_{S},I_{T})=\mathcal{L}_{det}(I_{S})+\mathcal{L}_{det}(I_{T})+\lambda_{adv}[\mathcal{L}_{adv}(F(I_{S}))+\mathcal{L}_{adv}(F(I_{T}))],$
(2)

where $\mathcal{L}_{det}(\cdot)$ is the detection loss, and $\lambda_{adv}$ is
a weight that determines the loss balance. We set $\lambda_{disc}$ is 0.1 in
our experiments.

However, because the features near the boundary of the area where the object
of another domain is pasted using OCDC are a mixture of the features of the
two domains, it is difficult to distinguish which domain is near this
boundary. Thus, the loss near the boundary of the trained discriminator is
smaller than that in other regions. However, information from different
domains is rarely mixed at a position far from the boundary of the pasted
object. Therefore, the loss of discriminator in the feature is large in that
area.

Furthermore, we adaptively determine the domain identification label based on
the position and domain of the image pasted by OCDC. After the pasting
process, the RGB domain label is replaced with the domain identification label
corresponding to the infrared image region, and the infrared domain label is
replaced with the domain identification label corresponding to the RGB image
region.

##  4 Experiments

We evaluate the effectiveness of the proposed methods using RGB images [5, 46]
and thermal infrared images [47, 48] with a large domain gap. In these
experiments, there are differences in both the spectrum and the captured
scene. We compare the performance with a large amount of RGB images and a
small amount of thermal infrared images. All datasets are labeled.
Additionally, to verify the generalization performance of the proposed method,
we evaluate the performance using real images [6] and simulation images [49].

###  4.1 Comparison Methods

We explain the comparison method used for each experimental setting. In our
experiment, images of the target domain are used for evaluation. Source-only
and target-only labels in the tables of experimental results show the
evaluation results when the model is trained using only the image of the
source or target domains, respectively. The fine-tuning label shows the
results using a pre-trained model and fine-tuning using target domain data.
The target samples label shows the number of target domain data using fine-
tuning. The Domain-Adversarial Training of Neural Networks (DANN) [50], one of
the adversarial learning methods, is used as the baseline of the adversarial
learning method. DANN label shows the detection results using domain
adaptation with DANN. We use Faster R-CNN [23] as the detection network and
VGG16 [51] for the backbone. In domain adaptation, the model parameters are
pre-trained in the source domain. The height size is 600 of the image
resolution, but if the maximum width size is more than 1,000, we set it to
1,000 while maintaining the aspect ratio. Ours label shows the result using
the proposed method, which uses the OCDC and OCDCDL. The optimizer is SGD; the
learning rate, the weight decay, and the momentum are 0.001, 0.0005, and 0.9,
respectively. The batch size is one. The evaluation metrics are the average
precision at an intersection over union (IoU), which threshold is 0.5. The
front of the arrow indicates the source domain, and the tip of the arrow
indicates the target domain.

Table 1:  Results on BDD100k $\rightarrow$ FLIR

Method | Target Samples | Person | Bicycle | Car | mAP
---|---|---|---|---|---
Source-only | — | 39.9 | 24.9 | 68.2 | 44.4
| Full | 74.2 | 57.9 | 84.1 | 72.1
| 1/2 | 71.5 | 56.0 | 82.3 | 69.9
| 1/4 | 66.7 | 48.7 | 78.5 | 64.6
Target-only | 1/8 | 61.4 | 41.4 | 75.4 | 59.4
| 1/16 | 57.0 | 42.5 | 71.8 | 57.1
| 1/32 | 51.3 | 34.9 | 67.3 | 51.2
| 1/64 | 44.4 | 32.0 | 63.6 | 46.7
| Full | 75.0 | 60.5 | 86.3 | 73.9
| 1/2 | 74.8 | 58.3 | 86.1 | 73.1
| 1/4 | 72.4 | 53.1 | 85.4 | 70.1
fine-tuning | 1/8 | 69.1 | 47.8 | 82.9 | 66.6
| 1/16 | 64.6 | 45.4 | 79.5 | 63.2
| 1/32 | 65.9 | 46.7 | 83.3 | 65.3
| 1/64 | 64.5 | 41.2 | 82.0 | 62.6
| Full | 78.1 | 63.8 | 87.0 | 76.3
| 1/2 | 77.6 | 63.1 | 87.2 | 76.0
| 1/4 | 75.2 | 56.5 | 86.2 | 72.6
DANN | 1/8 | 72.4 | 58.8 | 84.5 | 71.9
| 1/16 | 70.6 | 55.8 | 83.8 | 70.1
| 1/32 | 69.4 | 53.8 | 82.3 | 68.5
| 1/64 | 67.7 | 51.8 | 81.9 | 67.1
| Full | 77.8 | 63.5 | 86.9 | 76.1
| 1/2 | 78.3 | 62.6 | 87.2 | 76.1
| 1/4 | 76.9 | 59.9 | 86.9 | 74.5
Ours | 1/8 | 75.4 | 60.9 | 85.7 | 74.0
| 1/16 | 72.2 | 57.9 | 84.5 | 71.5
| 1/32 | 71.1 | 53.8 | 82.0 | 69.3
| 1/64 | 68.5 | 51.6 | 82.3 | 67.5

BDD100k $\rightarrow$ FLIR: The BDD100k dataset [5] is collected based on six
types of weather conditions, six different scenes, and three categories of
time of data; the number of images is 100,000. This dataset is annotated in
ten categories. FLIR ADAS dataset [47] is an image captured by a FLIR Tau2
camera, and the number of images is 10,228. Only thermal infrared images from
this dataset are used. In our experiment, the training data includes 36,728
images labeled as daytime from the BDD100k dataset as the source domain data
and 8,862 thermal infrared images used as training splits from the FLIR ADAS
dataset. The categories person, bicycle, and car, which are common categories.

Table 1 shows the evaluation results. The detection accuracy of source-only is
the lowest because this does not use knowledge of the target domain. In
target-only, mAP is 72.1 % using all target data named Full. However,
performances considerably deteriorates when the amount of data decreased. In
fine-tuning, performances are higher than the performances of target-only
because fine-tuning models had a knowledge that performance improves somewhat
even in different domains. The performances of DANN are higher than fine-
tuning because of the effects of domain adaptation. There is no significant
performance degradation due to negative transfer, but the effect of domain
adaptation can be confirmed. The performances of the proposed method tend to
improve overall, but in particular, the performance of the person label
outperformed DANN.

When the target samples are Full, the number of each object is large, so even
if the object areas are small, there are enough numbers to improve the
performance by domain adaptation. However, in the case of Full, there is only
a difference of 0.3 points, so the conventional domain adaptation does not
have a difference drastically. In particular, in this experimental result, it
should be noted that the target samples, which are our targets, are smaller
than 1/2, rather than the detection result of Full. Under these conditions,
the our method has higher performance than the conventional method in almost
all cases. We confirmed about 4 point improvements over DANN even if there is
a few data. For example, if we consider person label, this is because the
percentage of people in the dataset was high, and the overall percentage of
pasting person’s images onto another domain based on the input image was high.

Table 2:  Results on Caltech $\rightarrow$ KAIST (Person) and SIM10K
$\rightarrow$ Cityscapes (Car)

Method | Target Samples |  | Person
---
(KAIST)
| Car
---
(Cityscapes)
Source-only | — | 2.8 | 43.1
| Full | 67.0 | 60.3
| 1/2 | 67.6 | 58.1
| 1/4 | 63.0 | 54.3
Target-only | 1/8 | 58.7 | 52.0
| 1/16 | 57.1 | 48.3
| 1/32 | 51.3 | 45.2
| 1/64 | 47.4 | 41.1
| Full | 63.4 | 58.1
| 1/2 | 63.4 | 58.1
| 1/4 | 63.5 | 57.8
fine-tuning | 1/8 | 63.2 | 56.8
| 1/16 | 61.7 | 55.7
| 1/32 | 59.5 | 52.5
| 1/64 | 57.1 | 49.8
| Full | 69.4 | 62.0
| 1/2 | 71.7 | 61.0
| 1/4 | 70.4 | 58.6
DANN | 1/8 | 69.1 | 59.6
| 1/16 | 66.9 | 55.2
| 1/32 | 57.0 | 54.1
| 1/64 | 54.5 | 52.3
| Full | 68.4 | 63.6
| 1/2 | 73.3 | 61.8
| 1/4 | 72.4 | 60.0
Ours | 1/8 | 69.7 | 59.6
| 1/16 | 67.5 | 57.4
| 1/32 | 59.1 | 56.5
| 1/64 | 57.1 | 54.2

Caltech $\rightarrow$ KAIST: The Caltech Pedestrian dataset [46] is a dataset
that contains labeled images of pedestrians captured using an in-vehicle
camera. 42,782 images from this dataset are used for the images. The KAIST
Multispectral Pedestrian dataset [48] is a dataset captured using the FLIR A35
microbolometer LWIR camera and contains 95,000 images labeled for pedestrians.
The thermal infrared image of this dataset is used. In KAIST, 7,688 thermal
infrared images are used for training, and 2,252 thermal infrared images are
used for testing. This is based on the procedure in the paper [52]. In this
experiment, only person is used as the category.

Table 2 shows the evaluation results. Source-only detection accuracy is
extremely low because the appearance of RGB and thermal infrared images
differs significantly. Target-only and fine-tuning detection accuracy
deteriorated as the target sample decreased, as demonstrated in the FLIR
experiment. Particularly, fine-tuning detection accuracy was lower than
target-only detection accuracy by pre-training in the source domain. However,
the performance degradation is suppressed when the target samples decrease.
DANN and the proposed method perform better than target-only when there are
many target samples. Performance is higher than fine-tuning when the number of
samples decreased. The proposed method showed more than two points higher
performance than DANN when target samples are 1/32 and 1/64. This experiment
shows that the proposed method is more effective in single-class object
detection than in multi-class object detection. In single-class object
detection, when an image of another domain is pasted on an image of another
domain, the image is rarely pasted to the detected object. Therefore, few
occlusion problems due to pasting occurred in the BDD100k $\rightarrow$ FLIR
experiment. Since the proposed method has a remarkable effect when the number
of targets is small, it is expected to be effective in applications that are
often used, such as pedestrian detection.

SIM10K $\rightarrow$ Cityscapes: The SIM10K dataset [49] is a composite of
10,000 images generated by the Grand Theft Auto (GTA) game engine and is
annotated with cars and other similar images. The Cityscapes dataset [6]
consists of real images captured in multiple urban areas and segmentation
labels. We used the circumscribed rectangle of the object segmentation label
as the bounding box for evaluation in the car categories. Furthermore, we used
10,000 composite images from SIM10K as a training set. Note that 2,975 images,
which are training splits from Cityscapes, are used as training data, and 500
images, which are validation splits, are used as evaluation data. The
evaluation is performed using the common category of car.

Table 2 shows the evaluation results. Similar to the previous results, the
source-only detection accuracy is the lowest. If the target domain detection
accuracy has a large amount of data, the target-only detection performance is
high, but when the amount of data is small, the performance is significantly
reduced. Fine-tuning detection accuracy reduces performance degradation when
the amount of data is small. However, when the amount of data is small, the
performance of DANN improves, so the effect of domain adaptation can be
confirmed. When the target samples were 1/8, the proposed method and DANN had
the same accuracy. Cityscapes are in-vehicle camera images, and the size of
the car in the image is much larger than that of a person. Therefore, even if
the number of data is reduced a little, the vehicle area can be used for
domain adaptation in the same area as the background, so the accuracy is not
so decrease. In this experiment, we confirmed that the proposed method is
effective not only for domain adaptation between RGB and infrared images but
also for conventional problem settings. This experiment showed that the
proposed method is a general-purpose technique that can be used in various
source and target data domains.

###  4.2 Ablation Study

This section shows the comparison results under different experimental
conditions. In either case, a comparison is made under the evaluation
conditions of BDD $\rightarrow$ FLIR.

Contribution of Components: We evaluate the effect of OCDC and OCDCDL. Table 3
shows the results. For Person, the method using both OCDC and OCDCDL had high
performance. On the other hand, Bicycle and Car have high performance even
with OCDC alone. Our experiment do not consider labels near the boundaries of
objects. Thus, even if a person with a small area makes a mistake in the
discriminator label near the object boundary, the effect on detection accuracy
is small. However, Bicycle and Car have a large area. Therefore, the
performance decrease if the discriminator label near the object boundary is
mistaken. However, in comparison with mAP, our proposed methods have the
highest accuracy and effectiveness.

Table 3:  Results on contribution of components

Target Samples | OCDC | OCDCDL | Person | Bicycle | Car | mAP
---|---|---|---|---|---|---
|  |  | 78.1 | 63.8 | 87.0 | 76.3
Full | ✓ |  | 77.8 | 63.2 | 86.9 | 76.0
| ✓ | ✓ | 77.8 | 63.5 | 86.9 | 76.1
|  |  | 77.6 | 63.1 | 87.2 | 76.0
1/2 | ✓ |  | 78.2 | 63.2 | 87.4 | 76.3
| ✓ | ✓ | 78.3 | 62.6 | 87.2 | 76.1
|  |  | 75.2 | 56.5 | 86.2 | 72.6
1/4 | ✓ |  | 76.7 | 61.9 | 86.8 | 75.1
| ✓ | ✓ | 76.9 | 59.9 | 86.9 | 74.5
|  |  | 72.4 | 58.8 | 84.5 | 71.9
1/8 | ✓ |  | 74.4 | 58.5 | 85.5 | 72.8
| ✓ | ✓ | 75.4 | 60.9 | 85.7 | 74.0
|  |  | 70.6 | 55.8 | 83.8 | 70.1
1/16 | ✓ |  | 72.1 | 54.9 | 84.9 | 70.6
| ✓ | ✓ | 72.2 | 57.9 | 84.5 | 71.5
|  |  | 69.4 | 53.8 | 82.3 | 68.5
1/32 | ✓ |  | 71.0 | 54.0 | 82.6 | 69.2
| ✓ | ✓ | 71.1 | 53.8 | 82.0 | 69.3
|  |  | 67.7 | 51.8 | 81.9 | 67.1
1/64 | ✓ |  | 68.1 | 52.5 | 81.8 | 67.5
| ✓ | ✓ | 68.5 | 51.6 | 82.3 | 67.5

Region Selection Strategies: We compare the accuracy of whether the pasting
position and scale are the same before and after pasting in OCDC. Table 4
shows the experimental results. A fixed label indicates position or scale are
the same before and after pasting, and a random label indicates that the
position and scale are set randomly. The detection accuracy in many cases is
higher if the same position is maintained before and after pasting. For
example, in the case of an in-vehicle camera, objects are concentrated on the
lower side of the image. The object detection model trains a set of the
position and the class. To train the relationship between the position and the
class, which is unlikely to occur, does not have a positive effect on the
inference result. In our experiment, by making the pasting position and size
the same, the detection model is able to train the positions and scales that
are likely to occur during inference.

Table 4:  Results on region selection strategies

Target Samples | Position | Scaling | Person | Bicycle | Car | mAP
---|---|---|---|---|---|---
Full | Fixed | Fixed | 77.8 | 63.5 | 86.9 | 76.1
Fixed | Random | 77.9 | 62.8 | 87.0 | 75.9
Random | Fixed | 76.0 | 62.5 | 86.5 | 75.0
Random | Random | 76.4 | 62.2 | 86.6 | 75.0
1/2 | Fixed | Fixed | 78.3 | 62.6 | 87.2 | 76.1
Fixed | Random | 78.3 | 63.4 | 87.4 | 76.4
Random | Fixed | 77.2 | 62.5 | 87.0 | 75.5
Random | Random | 77.0 | 61.2 | 87.0 | 75.1
1/4 | Fixed | Fixed | 76.9 | 59.9 | 86.9 | 74.5
Fixed | Random | 77.3 | 61.4 | 86.9 | 75.2
Random | Fixed | 75.9 | 59.8 | 86.3 | 74.0
Random | Random | 76.1 | 59.4 | 86.3 | 73.9
1/8 | Fixed | Fixed | 75.4 | 60.9 | 85.7 | 74.0
Fixed | Random | 74.6 | 58.1 | 85.5 | 72.7
Random | Fixed | 73.4 | 58.6 | 85.2 | 72.4
Random | Random | 73.8 | 58.0 | 85.1 | 72.3
1/16 | Fixed | Fixed | 72.2 | 57.9 | 84.5 | 71.5
Fixed | Random | 72.1 | 56.4 | 84.4 | 71.0
Random | Fixed | 71.3 | 53.9 | 84.1 | 69.8
Random | Random | 70.7 | 56.3 | 84.0 | 70.4
1/32 | Fixed | Fixed | 71.1 | 53.8 | 82.0 | 69.3
Fixed | Random | 70.2 | 52.0 | 82.9 | 68.4
Random | Fixed | 70.5 | 53.4 | 82.3 | 68.7
Random | Random | 69.2 | 51.9 | 82.4 | 67.8
1/64 | Fixed | Fixed | 68.5 | 51.6 | 82.3 | 67.5
Fixed | Random | 68.4 | 51.1 | 81.9 | 67.1
Random | Fixed | 67.7 | 46.9 | 81.8 | 65.5
Random | Random | 66.8 | 51.1 | 82.1 | 66.6

###  4.3 Qualitative Results

Figure 2 is object detection results of FLIR, KAIST, and Cityscapes after
domain adaptation performed in subsection 4.1. At the top of each dataset is
the result when target samples is 1/16, and at the bottom of each dataset is
the result when it is 1/64. The comparison methods are (a) fine-tuning, (b)
DANN, and (c) Ours, respectively, and (d) Ground Truth. The car detection
result is shown in magenta, and the person detection result is shown in cyan.

In the FLIR results, there is no difference in the car detection results, but
there is a difference in the person detection results. In fine-tuning and
DANN, even people with similar reflection intensities in thermal infrared
images are not detected. On the other hand, the proposed method detects
objects with similar reflection intensities, even if they are people far away.
The proposed method adapted the domain to information about the reflection
intensity of persons, which is a small area in the image. On the other hand,
the conventional methods did not fully adapt the domain, so some objects could
not be detected.

In the KAIST results, the conventional methods did not detect some small
persons. In domain adaptation, it is difficult to adapt information on small
objects because even if the information in a small object is ignored, the loss
of the detection model decreases. However, the proposed method makes it easier
to detect even small objects by explicitly giving information from other
domains to the input image.

In the Cityscapes results, in both 1/16 and 1/64, the farthest car on the left
side was not detected by conventional methods, but the proposed method could
detect the car. This is because it is difficult to adapt the domain of a small
object, which is the same reason as in the case of KAIST. In this experiment,
we clarified the importance of explicitly giving information of other domains
to the input image in domain adaptation of small objects as in the proposed
method.

Target
Samples  |  |  |  |
---|---|---|---|---
1/16 |  |  |  |
1/64 |  |  |  |
| FLIR
1/16 |  |  |  |
1/64 |  |  |  |
| KAIST
1/16 |  |  |  |
1/64 |  |  |  |
| Cityscapes
|  (a) fine-tuning |  (b) DANN |  (c) Ours |  (d) Ground Truth

Figure 2:  There are the detection results using (a) fine-tuning, (b) DANN,
(c) our method, and (d) ground truth, respectively. The first and second rows
are the detection results of FLIR images using BDD100k $\rightarrow$ FLIR
adaptation, The third and fourth rows are the detection results of KAIST
images using Caltech $\rightarrow$ KAIST adaptation, The fifth and sixth rows
are the detection results of Cityscapes using SIM10K $\rightarrow$ Cityscapes.
The odd rows are 1/16 target samples and the even rows are 1/64, respectively.
The bounding box colored cyan indicates person, and the bounding box colored
magenta indicates car, respectively.

##  5 Conclusion

We proposed few-shot supervised domain adaptation for object detection in
cases with large domain gaps, such as RGB and thermal infrared images.
Although the number of infrared images is smaller than that of RGB images, the
performance is improved compared with the conventional domain identification
via OCDC method for reducing the gap between domains for the input image we
proposed and the corresponding change in the domain identification label of
the discriminator (OCDCDL). Furthermore, it was confirmed that the proposed
method is effective by a comparative experiments in which the number of data
was changed, and the versatility of the method was shown by a comparative
experiments using various data.

## References

  * [1] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.:  Microsoft coco: Common objects in context.  In: European conference on computer vision, Springer (2014) 740–755
  * [2] Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.:  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision 111 (2015) 98–136
  * [3] Krasin, I., Duerig, T., Alldrin, N., Veit, A., Abu-El-Haija, S., Belongie, S., Cai, D., Feng, Z., Ferrari, V., Gomes, V., Gupta, A., Narayanan, D., Sun, C., Chechik, G., Murphy, K.:  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github.com/openimages (2016)
  * [4] Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.:  Objects365: A large-scale, high-quality dataset for object detection.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2019) 8430–8439
  * [5] Kim, J., Rohrbach, A., Darrell, T., Canny, J., Akata, Z.:  Textual explanations for self-driving vehicles.  Proceedings of the European Conference on Computer Vision (ECCV) (2018)
  * [6] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.:  The cityscapes dataset for semantic urban scene understanding.  In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  * [7] Pan, S.J., Yang, Q.:  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering 22 (2009) 1345–1359
  * [8] Weiss, K., Khoshgoftaar, T.M., Wang, D.:  A survey of transfer learning.  Journal of Big data 3 (2016) 1–40
  * [9] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.:  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In: Proceedings of the IEEE international conference on computer vision. (2017) 2223–2232
  * [10] Shang, Q., Hu, L., Li, Q., Long, W., Jiang, L.:  A survey of research on image style transfer based on deep learning.  In: 2021 3rd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM), IEEE (2021) 386–391
  * [11] Osumi, K., Yamashita, T., Fujiyoshi, H.:  Domain adaptation using a gradient reversal layer with instance weighting.  In: 2019 16th International Conference on Machine Vision Applications (MVA), IEEE (2019) 1–5
  * [12] Bolte, J.A., Kamp, M., Breuer, A., Homoceanu, S., Schlicht, P., Huger, F., Lipinski, D., Fingscheidt, T.:  Unsupervised domain adaptation to improve image segmentation quality both in the source and target domain.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. (2019) 0–0
  * [13] Ganin, Y., Lempitsky, V.:  Unsupervised domain adaptation by backpropagation.  In: International conference on machine learning, PMLR (2015) 1180–1189
  * [14] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.:  mixup: Beyond empirical risk minimization.  In: International Conference on Learning Representations. (2018)
  * [15] Tokozume, Y., Ushiku, Y., Harada, T.:  Between-class learning for image classification.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018) 5486–5494
  * [16] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.:  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In: Proceedings of the IEEE/CVF international conference on computer vision. (2019) 6023–6032
  * [17] Ishii, Y., Yamashita, T.:  Cutdepth: Edge-aware data augmentation in depth estimation.  arXiv preprint arXiv:2107.07684 (2021)
  * [18] Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M.:  Deep learning for generic object detection: A survey.  International journal of computer vision 128 (2020) 261–318
  * [19] Kang, J., Tariq, S., Oh, H., Woo, S.S.:  A survey of deep learning-based object detection methods and datasets for overhead imagery.  IEEE Access 10 (2022) 20118–20134
  * [20] Wu, X., Sahoo, D., Hoi, S.C.:  Recent advances in deep learning for object detection.  Neurocomputing 396 (2020) 39–64
  * [21] Girshick, R., Donahue, J., Darrell, T., Malik, J.:  Rich feature hierarchies for accurate object detection and semantic segmentation.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2014) 580–587
  * [22] Girshick, R.:  Fast r-cnn.  In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). (2015)
  * [23] Ren, S., He, K., Girshick, R., Sun, J.:  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems 28 (2015)
  * [24] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.:  You only look once: Unified, real-time object detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  * [25] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.:  Ssd: Single shot multibox detector.  In: European conference on computer vision, Springer (2016) 21–37
  * [26] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.:  An image is worth 16x16 words: Transformers for image recognition at scale.  In: International Conference on Learning Representations. (2020)
  * [27] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.:  End-to-end object detection with transformers.  In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I. (2020) 213–229
  * [28] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.:  Deformable detr: Deformable transformers for end-to-end object detection.  In: International Conference on Learning Representations. (2020)
  * [29] Yao, Z., Ai, J., Li, B., Zhang, C.:  Efficient detr: improving end-to-end object detector with dense prior.  arXiv preprint arXiv:2104.01318 (2021)
  * [30] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.:  Swin transformer: Hierarchical vision transformer using shifted windows.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision. (2021) 10012–10022
  * [31] Khokhlov, I., Davydenko, E., Osokin, I., Ryakin, I., Babaev, A., Litvinenko, V., Gorbachev, R.:  Tiny-yolo object detection supplemented with geometrical data.  In: 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring), IEEE (2020) 1–5
  * [32] Szemenyei, M., Estivill-Castro, V.:  Fully neural object detection solutions for robot soccer.  Neural Computing and Applications (2021) 1–14
  * [33] Hossain, S., Lee, D.j.:  Deep learning-based real-time multiple-object detection and tracking from aerial imagery via a flying robot with gpu-based embedded devices.  Sensors 19 (2019) 3371
  * [34] Wu, Z., Suresh, K., Narayanan, P., Xu, H., Kwon, H., Wang, Z.:  Delving into robust object detection from unmanned aerial vehicles: A deep nuisance disentanglement approach.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision. (2019) 1201–1210
  * [35] Lin, C., Lu, J., Wang, G., Zhou, J.:  Graininess-aware deep feature learning for pedestrian detection.  In: Proceedings of the European conference on computer vision (ECCV). (2018) 732–747
  * [36] Zhou, C., Yuan, J.:  Bi-box regression for pedestrian detection and occlusion estimation.  In: Proceedings of the European Conference on Computer Vision (ECCV). (2018) 135–151
  * [37] Xu, Z., Yang, W., Meng, A., Lu, N., Huang, H., Ying, C., Huang, L.:  Towards end-to-end license plate detection and recognition: A large dataset and baseline.  In: Proceedings of the European conference on computer vision (ECCV). (2018) 255–271
  * [38] Zhang, L., Zhu, X., Chen, X., Yang, X., Lei, Z., Liu, Z.:  Weakly aligned cross-modal learning for multispectral pedestrian detection.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision. (2019) 5127–5137
  * [39] Liu, J., Zhang, S., Wang, S., Metaxas, D.N.:  Multispectral deep neural networks for pedestrian detection.  arXiv preprint arXiv:1611.02644 (2016)
  * [40] Konig, D., Adam, M., Jarvers, C., Layher, G., Neumann, H., Teutsch, M.:  Fully convolutional region proposal networks for multispectral person detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. (2017) 49–56
  * [41] Akkaya, I.B., Altinel, F., Halici, U.:  Self-training guided adversarial domain adaptation for thermal imagery.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2021) 4322–4331
  * [42] Vs, V., Poster, D., You, S., Hu, S., Patel, V.M.:  Meta-uda: Unsupervised domain adaptive thermal object detection using meta-learning.  In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. (2022) 1412–1423
  * [43] Hsu, H.K., Yao, C.H., Tsai, Y.H., Hung, W.C., Tseng, H.Y., Singh, M., Yang, M.H.:  Progressive domain adaptation for object detection.  In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. (2020) 749–757
  * [44] Zhou, K., Yang, Y., Qiao, Y., Xiang, T.:  Domain generalization with mixstyle.  In: International Conference on Learning Representations. (2020)
  * [45] Wang, W., Liao, S., Zhao, F., Kang, C., Shao, L.:  Domainmix: Learning generalizable person re-identification without human annotations.  arXiv preprint arXiv:2011.11953 (2020)
  * [46] Dollár, P., Wojek, C., Schiele, B., Perona, P.:  Pedestrian detection: A benchmark.  In: 2009 IEEE conference on computer vision and pattern recognition, IEEE (2009) 304–311
  * [47] Group, F., et al.:  Flir thermal dataset for algorithm training.  URL: https://www. flir. co. uk/oem/adas/adas-dataset-form/(May 2019) (2018)
  * [48] Hwang, S., Park, J., Kim, N., Choi, Y., So Kweon, I.:  Multispectral pedestrian detection: Benchmark dataset and baseline.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 1037–1045
  * [49] Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.:  Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?  arXiv preprint arXiv:1610.01983 (2016)
  * [50] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.:  Domain-adversarial training of neural networks.  The journal of machine learning research 17 (2016) 2096–2030
  * [51] Simonyan, K., Zisserman, A.:  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * [52] Zhang, S., Benenson, R., Omran, M., Hosang, J., Schiele, B.:  How far are we from solving pedestrian detection?  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)