[a,b]Tania Robens

# Constraining extended scalar sectors at current and future colliders

###### Abstract

In this proceeding, I summarize results on various new physics extensions of
the Standard Model, for models with and without dark matter candidates. I
discuss current constraints as well as rates and discovery prospects at future
colliders.
RBI-ThPhys-2022-12, CERN-TH-2022-056

##  1 Introduction

In this proceeding, I discuss various new physics models that extend the
Standard Model (SM) by adding additional fields that transform as either
singlets or doublets under the standard model gauge group. Most of the results
presented here have already been discussed elsewhere, and I summarize our
previous findings here. In particular, I discuss

  * •

The real singlet extension of the SM, which comes with an additional scalar
that transforms as a singlet. The model features one additional CP even
neutral scalar. See [1, 2, 3, 4, 5] for original literature as well as [6] for
results presented here;

  * •

The Inert Doublet Model (IDM) [7, 8, 9], a two Higgs doublet model (THDM) with
an additional $\mathbb{Z}_{2}$ symmetry that provides a dark matter (DM)
candidate. Our original work can be found in [10, 4, 11, 12];

  * •

The two real singlet extension (TRSM), where the SM scalar sector is extended
by two additional gauge singlets, featuring in total three CP even neutral
scalars that also allow for interesting cascade decays. The results discussed
here have first been presented in [13, 14];

  * •

The THDMa, a two higgs doublet model that is enhanced by an additional
pseudoscalar which serves as a portal to the dark matter sector. In the
version of the model discussed here, the DM candidate is fermionic. See [15,
16, 17, 18, 19, 20, 21] for original work and [22] for work containing the
results discussed here.

All models are confronted with most recent theoretical and experimental
constraints. Theory constraints include the minimization of the vacuum as well
as the requirement of vacuum stability and positivity. We also apply
constraints from perturbative unitarity and perturbativity of the couplings at
the electroweak scale.

Experimental bounds include the agreement with current measurements of the
properties of the 125 GeV resonance discovered by the LHC experiments, as well
as agreement with the null-results from searches for additional particles at
current or past colliders. Furthermore, we impose constraints from electroweak
precision observables (via $S,\,T,\,U$ parameters [23, 24, 25]), B-physics
observables
$\left(B\,\rightarrow\,X_{s}\,\gamma,\,B_{s}\,\rightarrow\,\mu^{+}\,\mu^{-},\,\Delta
M_{s}\right)$, and astrophysical observables (relic density and direct
detection bounds). We use a combination of private and public tools in these
analyses. In particular, we use HiggsBounds [26], HiggsSignals [27], 2HDMC
[28], SPheno [29], Sarah [30], micrOMEGAs [31, 32], and MadDM [33].
Experimental numbers are taken from [34, 35] for electroweak precision
observables, [36] for $B_{s}\,\rightarrow\,\mu^{+}\,\mu^{-}$, [37] for $\Delta
M_{s}$ and [38] and [39] for relic density and direct detection, respectively.
Bounds from $B\,\rightarrow\,X_{s}\gamma$ are implemented via a fit function
from [40, 41]. Predictions for production cross sections shown here have been
obtained using Madgraph5 [42].

##  2 Real singlet extension

As a first simple example, we discuss a real singlet extension of the SM with
a $\mathbb{Z}_{2}$ symmetry previously reported on in [1, 2, 3, 43, 4, 5]. The
$\mathbb{Z}_{2}$ symmetry is softly broken by a vacuum expectation value (vev)
of the singlet field, inducing mixing between the gauge-eigenstates which
introduces a mixing angle $\alpha$. The model has in total 5 free parameters.
Two of these are fixed by the measurement of the $125\,{\rm GeV}$ resonance
mass and electroweak precision observables. We then have

$\sin\alpha,\,m_{2},\,\tan\beta\,\equiv\,\frac{v}{v_{s}}$ (1)

as free parameters of the model, where $v\,(v_{s})$ are the doublet and
singlet vevs, respectively. We concentrate on the case where
$m_{2}\,\geq\,125\,{\rm GeV}$, where SM decoupling corresponds to
$\sin\alpha\,\rightarrow\,0$.

Limits on this model are shown in figure 1, taken from [6]111Updates will be
presented in the proceedings of Moriond2022., including a comparison of the
currently maximal available rate of $H\,\rightarrow\,h_{125}h_{125}$ with the
combination limits from ATLAS [44]. The most constraining direct search bounds
are in general dominated by searches for diboson final states [45, 46, 47,
48]. In some regions, the Run 1 Higgs combination [49] is also important.
Especially [47, 48] currently correspond to the best probes of the models
parameter space222We include searches currently available via HiggsBounds..

Figure 1: Results for the singlet extension, taken from [6]. Left: comparison
of current constraints for a fixed value of $\tan\beta\,=\,0.1$. Right:
maximal $H\,\rightarrow\,h\,h$ allowed, with electroweak constraints at the
electroweak scale (blue) or including RGE running to a higher scale (red), in
comparison with results from the ATLAS combination.

##  3 Inert Doublet Model

The Inert Doublet Model is a two Higgs doublet model with an exact discrete
$\mathbb{Z}_{2}$ symmetry. It provides a dark matter candidate that stems from
the second doublet [7, 8, 9]. The particle content of the model consists of
four additional scalar states $H,\,A,\,H^{\pm}$, and has in total 7 free
parameters prior to electroweak symmetry breaking:

$v,\,m_{h},\,\underbrace{m_{H},\,m_{A},\,m_{H^{\pm}}}_{\text{second
doublet}},\,\lambda_{2},\,\lambda_{345}\,\equiv\,\lambda_{3}+\lambda_{4}+\lambda_{5},$
(2)

Here, the $\lambda_{i}$s denote standard couplings appearing in the THDM
potential. Two parameters ($m_{h}$ and $v$) are fixed by current measurements.
The model has been subjected to various experimental and theoretical
constraints [10, 4, 11, 12, 50]. One important observation is the existance of
a relatively strong degeneracy between the additional masses of the second
doublet, as well as a minimal mass scale for the dark matter candidate $H$
resulting from a combination of relic density and signal strength measurement
constraints (see [10, 12] for a detailed discussion). We display these
features in figure 2.

Figure 2:  Left: Masses are requested to be quite degenerate after all
constraints have been taken into account. In the
$\left(M_{H^{\pm}}-M_{H},\,M_{A}-M_{H}\right)$ plane (taken from [11]). Right:
Interplay of signal strength and relic density constraints in the
$\left(M_{H},\,\lambda_{345}\right)$ plane, using XENON1T results, with golden
points labelling those points that produce exact relic density (taken from
[4]).

###  3.1 Sensitivity study at current and future colliders

I here present results first discussed in [12]. In that work, a sensitivity
comparison for selected benchmark points [11, 51, 12] was used, which relies
on a simple counting criteria: a benchmark point is considered reachable if at
least 1000 signal events are produced using nominal luminosity of the
respective collider (c.f. also [52]). Table 1 shows the results using this
simple criterium. The accompagnying figures, displaying production cross
sections for pair-production of the novel scalars at various collider options
and center-of-mass energies are shown in figure 3, taken from [12]. We here
have used Madgraph5 [42] with a UFO input file from [53] for cross-section
predictions. Results for CLIC were taken from [51, 54].

Figure 3:  Predictions for production cross sections for various processes and collider options. Top left: Predictions for various pair-production cross sections for a $pp$ collider at 13 TeV, as a function of the mass sum of the produced particles. Top right: Same for various center-of-mass energies. Bottom left: VBF-type production of $AA$ and $H^{+}\,H^{-}$ at various center-of-mass energies for $pp$ colliders. Bottom right: Same for $\mu^{+}\mu^{-}$ colliders. Taken from [12]. The lines correspond to the cross-sections required to prodce at least 1000 events using the respective design luminosity. collider | all others | $AA$ |  $AA$ +VBF
---|---|---|---
HL-LHC | 1 TeV | 200-600 GeV | 500-600 GeV
HE-LHC | 2 TeV | 400-1400 GeV | 800-1400 GeV
FCC-hh | 2 TeV | 600-2000 GeV | 1600-2000 GeV
CLIC, 3 TeV | 2 TeV | - | 300-600 GeV
$\mu\mu$, 10 TeV | 2 TeV | - | 400-1400 GeV
$\mu\mu$, 30 TeV | 2 TeV | - | 1800-2000 GeV
Table 1: Sensitivity of different collider options, using the sensitivity
criterium of 1000 generated events in the specific channel. $x-y$ denotes
minimal/ maximal mass scales that are reachable.

##  4 TRSM

In the TRSM [13], the SM scalar sectors is augmented by two real scalars
obeying a discrete $\mathbb{Z}_{2}\,\otimes\,\mathbb{Z}_{2}^{\prime}$
symmetry. Both fields acquire a vev, which induces a mixing between all scalar
states. The model then has 9 a priori parameters after electroweak symmetry
breaking,

$m_{1},\,m_{2},\,m_{3},\,v,\,v_{X},\,v_{S},\,\theta_{hS},\,\theta_{hX},\,\theta_{SX},$

where $m_{i},\,v,\,\theta$ denote masses333We use the convention
$m_{1}\,\leq\,m_{2}\,\leq\,m_{3}$., vevs, and mixing angles. One mass
$m\,\sim\,125\,{\rm GeV}$ and $v\,\sim\,246\,{\rm GeV}$ are fixed by current
measurements.

Various benchmark planes (BPs) where proposed within this model [13], allowing
for novel production and decay processes, including decay chains which by that
time had not been investigated by the LHC experiments. Production and decay
modes can be characterized as

$p\,p\,\rightarrow\,h_{3}\,\rightarrow\,h_{1}\,h_{2},\;p\,p\,\rightarrow\,h_{a}\,\rightarrow\,h_{b}\,h_{b},$

where for the symmetric decays we assume none of the scalars corresponds to
the SM-like 125 GeV resonance.

In [14], we focussed on one particular benchmark plane (BP3), that features
the first production mode, in the scenario with $h_{1}\,\equiv\,h_{125}$. This
allows for a $h_{125}\,h_{125}\,h_{125}$ final state; production cross
sections depend on the masses of the two additional scalars and are displayed
in figure 4. We investigated the scenario where all $h_{125}$ further decay
into $b\,\bar{b}$ final states and conducted a complete phenomenological study
for a 14 TeV LHC. We made use of a customized loop_sm model implemented in
MadGraph5_aMC@NLO (v2.7.3) [55, 56], that was interfaced to HERWIG (v7.2.1)
[57, 58, 59, 60, 61, 62, 63]. We have applied a semi-automated cut
prescription to suppress SM background. Results are shown in table 2. We see
that several benchmark points are already accessible with a relatively low
integrated luminosity.

Figure 4:  Production cross sections for $h_{1}\,h_{1}\,h_{1}$ production in
BP3 at leading order, taken from [14]. See text for more details.

$(M_{2},M_{3})$ | $\sigma(pp\rightarrow h_{1}h_{1}h_{1})$ | $\sigma(pp\rightarrow 3b\bar{b})$ | $\text{sig}|_{300\rm{fb}^{-1}}$ | $\text{sig}|_{3000\rm{fb}^{-1}}$
---|---|---|---|---
${[{\rm GeV}]}$ | ${[{\rm fb}]}$ | ${[{\rm fb}]}$ |  |
$(255,504)$ | $32.40$ | $6.40$ | $2.92$ |  $9.23$
$(263,455)$ | $50.36$ | $9.95$ |  $4.78$ |  $15.11$
$(287,502)$ | $39.61$ | $7.82$ |  $4.01$ |  $12.68$
$(290,454)$ | $49.00$ | $9.68$ |  $5.02$ |  $15.86$
$(320,503)$ | $35.88$ | $7.09$ |  $3.76$ |  $11.88$
$(264,504)$ | $37.67$ | $7.44$ |  $3.56$ |  $11.27$
$(280,455)$ | $51.00$ | $10.07$ |  $5.18$ |  $16.39$
$(300,475)$ | $43.92$ | $8.68$ |  $4.64$ |  $14.68$
$(310,500)$ | $37.90$ | $7.49$ |  $4.09$ |  $12.94$
$(280,500)$ | $40.26$ | $7.95$ |  $4.00$ |  $12.65$
Table 2:  6 b final state leading-order production cross sections at 14 TeV,
as well as significances for different integrated luminosities. Taken from
[14].

We refer the reader to the above work for details of the analysis as well as
SM background simulation. Several of the benchmark points are in the 4-5
$\sigma$ range already for a relatively low luminosity, and all have
significances above the discovery reach after the full run of HL-LHC.

Finally, we can ask whether other channels can not equally constrain the
allowed parameter space at the HL-LHC. We therefore extrapolated various
analyses assessing the heavy Higgs boson prospects of the HL-LHC in final
states originating from $h_{i}\rightarrow h_{1}h_{1}$ [64, 44],
$h_{i}\rightarrow ZZ$ [47, 65] and $h_{i}\rightarrow W^{+}W^{-}$ [66, 67], for
$i=2,3$, and combined these with extrapolations of results from 13 TeV where
appropriate. We display the results in figure 5.

Figure 5: Constraints on the $\left(M_{2},\,M_{3}\right)$ plane from
extrapolation of other searches at the HL-LHC from extrapolation (see text for
details). Taken from [14].

In particular $ZZ$ final states can probe nearly all of the available
parameter space. These however depend on different model parameters than the
$h_{1}\,h_{1}\,h_{1}$ final state rates. These searches are testing a
different part of the parameter space and new physics potential.

##  5 THDMa

The THDMa has been widely promoted within the LHC Dark matter working group.
It is a type II two-Higgs-doublet model that is extended by an additional
pseudoscalar $a$ mixing with the "standard" pseudoscalar $A$ of the THDM. In
the gauge-eigenbasis, the additional scalar serves as a portal to the dark
sector, with a fermionic dark matter candidate, denoted by $\chi$. More
details can e.g. be found in [15, 16, 17, 18, 19, 20, 21].

The following mass eigenstates are incorporated within this model in the
scalar and dark matter sector: ${h,\,H,\,H^{\pm}}$, ${a,}\,{A,}\,{{\chi}}$. It
depends on 12 additional new physics parameters

$\displaystyle{v,\,m_{h},\,m_{H},}\,{m_{a},}\,{m_{A},\,m_{H^{\pm}},}\,{m_{\chi}};\;{\cos\left(\beta-\alpha\right),\,\tan\beta,}\,{\sin\theta;\;y_{\chi},}\,{\lambda_{3},}\,{\lambda_{P_{1}},\,\lambda_{P_{2}}},$

where $v$ and either $m_{h}$ or $m_{H}$ are fixed by current measurements in
the electroweak sector.

In [21], a scan was presented that allows all of the above novel parameters
float in specific predefined ranges. It is then not always straightforward to
display bounds from specific constraints in 2-dimensional planes. Two examples
where this is possible are shown in figure 6. The first plot displays bounds
in the $\left(m_{H^{\pm}},\,\tan\beta\right)$ plane from B-physics
observables, and shows that in general low masses
$m_{H^{\pm}}\lesssim\,800\,{\rm GeV}$ as well as values $\tan\beta\lesssim\,1$
are excluded. The second plot displays the relic density as a function of the
mass difference $m_{a}-2\,m_{\chi}$. In the region where this mass difference
remains small, relic density annihilates sufficiently to stay below the
observed relic density bound. On the other hand, too large differences lead to
values $\Omega\,h_{c}\,\gtrsim\,0.12$ and therefore are forbidden [38].

For these scans, it was investigated which cross-section values would still be
feasible for points that fulfill all constraints [21] at $e^{+}e^{-}$
colliders. A particular interest lies on signatures that include missing
energy and therefore distinguish this model from signatures that would be
realized in a THDM without a portal to the dark sector. Processes like
$e^{+}e^{-}\,\rightarrow\,hA,ha$ are suppressed due to alignment, which makes
$e^{+}e^{-}\,\rightarrow\,HA,Ha$ the most interesting channel that contains
novel signatures. However, such parameter points typically have mass scales
$\gtrsim\,1\,{\rm TeV}$. In such a case, production cross sections for an
$e^{+}e^{-}$ collider with a center-of-mass energy of 3 TeV are of interest.
The corresponding production cross sections are shown in figure 7, which
displays predictions for $t\,\bar{t}\,t\,\bar{t}$ and $t\,\bar{t}+\not{E}$
final states using a factorized approach. There is a non-negligible number of
points where the second channel is dominant. A "best" point with a large rate
for $t\,\bar{t}+\not{E}_{\perp}$ has been presented in [21].

Figure 6:  Left: Bounds on the $\left(m_{H^{\pm}},\,\tan\beta\right)$ plane
from B-physics observables, implemented via the SPheno [29]/ Sarah [30]
interface, and compared to experimental bounds [36, 37]. The contour for low
$\left(m_{H^{\pm},\,\tan\beta}\right)$ values stems from [40, 41]. Right: Dark
matter constraints in the THDMa model. Right: Dark matter relic density as a
function of $m_{a}-2\,m_{\chi}$, with $m_{\chi}$ defining the color coding.
The typical resonance-enhanced relic density annihilation is clearly visible.
Figures taken from [21].

Figure 7:  Production cross sections for $t\bar{t}t\bar{t}$ (x-axis) and
$t\bar{t}+\not{E}$ (y-axis) final state in a factorized approach, for an
$e^{+}e^{-}$ collider with a 3 TeV center-of-mass energy. Left: mediated via
$HA$, right: mediated via $HA$ and $Ha$ intermediate states. Color coding
refers to $m_{H}+m_{A}$ (left) and $M_{H}+0.5\times\,\left(m_{A}+m_{a}\right)$
(right). Figures taken from [21].

##  6 Summary and Conclusion

In this work, I have reported on some previously published results for models
that extend the scalar sector of the SM by additional gauge singlets or
doublets. Some of the models discussed here in addition feature a dark matter
candidate. I have presented results of applying current constraints on these
models, and rendered predictions for rates or significances at various future
collider options. Some of the models presented here, in particular the IDM and
TRSM, have not yet been fully explored by current collider experiments. I
therefore strongly encourage the experimental collaborations to consider these
at LHC Run III.

##  7 Acknowledgements

This research was supported in parts by the National Science Centre, Poland,
the HARMONIA project under contract UMO-2015/18/M/ST2/00518 and OPUS project
under contract UMO-2017/25/B/ST2/00496 (2018-2021), by the European Union
through the Programme Horizon 2020 via the COST actions CA15108 -
FUNDAMENTALCONNECTIONS and CA16201 - PARTICLEFACE, and by the UK’s Royal
Society.

## References

  * [1] G. M. Pruna and T. Robens,  Phys. Rev. D88, 115012 (2013), 1303.1150.
  * [2] T. Robens and T. Stefaniak,  Eur. Phys. J. C75, 104 (2015), 1501.02234.
  * [3] T. Robens and T. Stefaniak,  Eur. Phys. J. C76, 268 (2016), 1601.07880.
  * [4] A. Ilnicka, T. Robens, and T. Stefaniak,  Mod. Phys. Lett. A 33, 1830007 (2018), 1803.03594.
  * [5] J. Alison et al.,  (2019), 1910.00012,  [Rev. Phys.5,100045(2020)].
  * [6] T. Robens,  Extended scalar sectors at current and future colliders,  in 55th Rencontres de Moriond on QCD and High Energy Interactions, 2021, 2105.07719.
  * [7] N. G. Deshpande and E. Ma,  Phys. Rev. D18, 2574 (1978).
  * [8] Q.-H. Cao, E. Ma, and G. Rajasekaran,  Phys. Rev. D76, 095011 (2007), 0708.2939.
  * [9] R. Barbieri, L. J. Hall, and V. S. Rychkov,  Phys. Rev. D74, 015007 (2006), hep-ph/0603188.
  * [10] A. Ilnicka, M. Krawczyk, and T. Robens,  Phys. Rev. D 93, 055026 (2016), 1508.01671.
  * [11] J. Kalinowski, W. Kotlarski, T. Robens, D. Sokolowska, and A. F. Zarnecki,  JHEP 12, 081 (2018), 1809.07712.
  * [12] J. Kalinowski, T. Robens, D. Sokolowska, and A. F. Zarnecki,  Symmetry 13, 991 (2021), 2012.14818.
  * [13] T. Robens, T. Stefaniak, and J. Wittbrodt,  Eur. Phys. J. C 80, 151 (2020), 1908.08554.
  * [14] A. Papaefstathiou, T. Robens, and G. Tetlalmatzi-Xolocotzi,  JHEP 05, 193 (2021), 2101.00037.
  * [15] S. Ipek, D. McKeen, and A. E. Nelson,  Phys. Rev. D90, 055021 (2014), 1404.3716.
  * [16] J. M. No,  Phys. Rev. D93, 031701 (2016), 1509.01110.
  * [17] D. Goncalves, P. A. N. Machado, and J. M. No,  Phys. Rev. D95, 055027 (2017), 1611.04593.
  * [18] M. Bauer, U. Haisch, and F. Kahlhoefer,  JHEP 05, 138 (2017), 1701.07427.
  * [19] P. Tunney, J. M. No, and M. Fairbairn,  Phys. Rev. D96, 095020 (2017), 1705.09670.
  * [20] LHC Dark Matter Working Group, T. Abe et al.,  Phys. Dark Univ. 27, 100351 (2020), 1810.09420.
  * [21] T. Robens,  Symmetry 13, 2341 (2021), 2106.02962.
  * [22] J. Kalinowski, T. Robens, and A. F. Zarnecki,  New Physics with missing energy at future lepton colliders - Snowmass White Paper,  in 2022 Snowmass Summer Study, 2022, 2203.07913.
  * [23] G. Altarelli and R. Barbieri,  Phys. Lett. B253, 161 (1991).
  * [24] M. E. Peskin and T. Takeuchi,  Phys. Rev. Lett. 65, 964 (1990).
  * [25] M. E. Peskin and T. Takeuchi,  Phys. Rev. D46, 381 (1992).
  * [26] P. Bechtle et al.,  Eur. Phys. J. C80, 1211 (2020), 2006.06007.
  * [27] P. Bechtle et al.,  Eur. Phys. J. C 81, 145 (2021), 2012.09197.
  * [28] D. Eriksson, J. Rathsman, and O. Stål,  Comput. Phys. Commun. 181, 189 (2010), 0902.0851.
  * [29] W. Porod and F. Staub,  Comput. Phys. Commun. 183, 2458 (2012), 1104.1573.
  * [30] F. Staub,  Comput. Phys. Commun. 185, 1773 (2014), 1309.7223.
  * [31] G. Bélanger, F. Boudjema, A. Goudelis, A. Pukhov, and B. Zaldivar,  Comput. Phys. Commun. 231, 173 (2018), 1801.03509.
  * [32] G. Belanger, A. Mjallal, and A. Pukhov,  Eur. Phys. J. C 81, 239 (2021), 2003.08621.
  * [33] F. Ambrogi et al.,  Phys. Dark Univ. 24, 100249 (2019), 1804.00044.
  * [34] Gfitter Group, M. Baak et al.,  Eur. Phys. J. C74, 3046 (2014), 1407.3792.
  * [35] J. Haller et al.,  Eur. Phys. J. C78, 675 (2018), 1803.01853.
  * [36] CMS-PAS-BPH-20-003, LHCb-CONF-2020-002, ATLAS-CONF-2020-049.
  * [37] HFLAV, Y. S. Amhis et al.,  Eur. Phys. J. C81, 226 (2021), 1909.12524.
  * [38] Planck, N. Aghanim et al.,  Astron. Astrophys. 641, A6 (2020), 1807.06209,  [Erratum: Astron.Astrophys. 652, C4 (2021)].
  * [39] XENON, E. Aprile et al.,  Phys. Rev. Lett. 121, 111302 (2018), 1805.12562.
  * [40] M. Misiak, A. Rehman, and M. Steinhauser,  JHEP 06, 175 (2020), 2002.01548.
  * [41] M. Misiak,  Private communication.
  * [42] J. Alwall, M. Herquet, F. Maltoni, O. Mattelaer, and T. Stelzer,  JHEP 06, 128 (2011), 1106.0522.
  * [43] LHC Higgs Cross Section Working Group, D. de Florian et al.,  (2016), 1610.07922.
  * [44] ATLAS, G. Aad et al.,  Phys. Lett. B800, 135103 (2020), 1906.02025.
  * [45] CERN Report No.,  , 2013 (unpublished),  CMS-PAS-HIG-13-003.
  * [46] CMS, V. Khachatryan et al.,  JHEP 10, 144 (2015), 1504.00936.
  * [47] CMS, A. M. Sirunyan et al.,  JHEP 06, 127 (2018), 1804.01939,  [Erratum: JHEP03,128(2019)].
  * [48] ATLAS, M. Aaboud et al.,  Phys. Rev. D98, 052008 (2018), 1808.02380.
  * [49] CERN Report No.,  , 2012 (unpublished),  CMS-PAS-HIG-12-045.
  * [50] T. Robens,  The IDM and THDMa – current constraints and future prospects,  in European Physical Society Conference on High Energy Physics 2021, 2021, 2110.07294.
  * [51] J. Kalinowski, W. Kotlarski, T. Robens, D. Sokolowska, and A. F. Zarnecki,  JHEP 07, 053 (2019), 1811.06952.
  * [52] T. Robens, J. Kalinowski, A. F. Zarnecki, and A. Papaefstathiou,  Extended scalar sectors at future colliders,  in 27th Cracow Epiphany Conference on Future of particle physics , 2021, 2104.00046.
  * [53] A. Goudelis, B. Herrmann, and O. Stål,  JHEP 09, 106 (2013), 1303.3010.
  * [54] J. de Blas et al.,  3/2018 (2018), 1812.02093.
  * [55] J. Alwall et al.,  JHEP 07, 079 (2014), 1405.0301.
  * [56] V. Hirschi and O. Mattelaer,  JHEP 10, 146 (2015), 1507.00020.
  * [57] M. Bahr et al.,  Eur. Phys. J. C58, 639 (2008), 0803.0883.
  * [58] S. Gieseke et al.,  (2011), 1102.1672.
  * [59] K. Arnold et al.,  (2012), 1205.4902.
  * [60] J. Bellm et al.,  (2013), 1310.6877.
  * [61] J. Bellm et al.,  Eur. Phys. J. C76, 196 (2016), 1512.01178.
  * [62] J. Bellm et al.,  (2017), 1705.06919.
  * [63] J. Bellm et al.,  Eur. Phys. J. C80, 452 (2020), 1912.06509.
  * [64] CMS, A. M. Sirunyan et al.,  Phys. Rev. Lett. 122, 121803 (2019), 1811.09689.
  * [65] M. Cepeda et al.,  CERN Yellow Rep. Monogr. 7, 221 (2019), 1902.00134.
  * [66] ATLAS, M. Aaboud et al.,  Eur. Phys. J. C78, 24 (2018), 1710.01123.
  * [67] ATLAS,  CERN Report No. ATL-PHYS-PUB-2018-022, 2018 (unpublished).


[fieldsource=url, match={
_}—{_}—
_, replace=_] [fieldsource=url, match={$
sim$}—{}̃—$
sim$, replace=~] [fieldsource=url, match={
26}, replace=26] Date of publication xxxx 00, 0000, date of current version
xxxx 00, 0000.

This work is supported by Knowledge Hub Aichi Priority Research Project (third
term). The work of Shigemichi Matsuzaki is also supported in part by the
Leading Graduate School Program, “Innovative program for training brain-
science-information-architects by analysis of massive quantities of highly
technical information about the brain,” by the Ministry of Education, Culture,
Sports, Science and Technology, Japan.

Corresponding author: Shigemichi Matsuzaki (e-mail:
matsuzaki@aisl.cs.tut.ac.jp).

# Image-based scene recognition for robot navigation considering traversable
plants and its manual annotation-free training

SHIGEMICHI MATSUZAKI1  HIROAKI MASUZAWA1
and JUN MIURA1  Department of Computer Science and Engineering, Toyohashi
University of Technology, Toyohashi 441-8580, Japan

###### Abstract

This paper describes a method of estimating the traversability of plant parts
covering a path and navigating through them for mobile robots operating in
plant-rich environments. Conventional mobile robots rely on scene recognition
methods that consider only the geometric information of the environment. Those
methods, therefore, cannot recognize paths as traversable when they are
covered by flexible plants. In this paper, we present a novel framework of
image-based scene recognition to realize navigation in such plant-rich
environments. Our recognition model exploits a semantic segmentation branch
for general object classification and a traversability estimation branch for
estimating pixel-wise traversability. The semantic segmentation branch is
trained using an unsupervised domain adaptation method and the traversability
estimation branch is trained with label images generated from the robot’s
traversal experience during the data acquisition phase, coined traversability
masks. The training procedure of the entire model is, therefore, free from
manual annotation. In our experiment, we show that the proposed recognition
framework is capable of distinguishing traversable plants more accurately than
a conventional semantic segmentation with traversable plant and non-
traversable plant classes, and an existing image-based traversability
estimation method. We also conducted a real-world experiment and confirmed
that the robot with the proposed recognition method successfully navigated in
plant-rich environments.

###### Index Terms:

Agricultural mobile robots, Deep learning, Manual annotation-free training,
Mobile robots, Navigation, Traversability estimation, Unstructured
environments

=-15pt

((a)) Network architecture

((b)) Navigation pipeline

Figure 1: Overview of the proposed method. (a) Our image-based recognition
module consists of two modules: Semantic Segmentation Module (SSM) for pixel-
wise general object classification, and Traversability Estimation Module (TEM)
for pixel-wise traversability estimation. SSM is trained by a pseudo-label
learning method. TEM is trained with the traversability masks generated based
on the robot’s experience. (b) During the robot navigation, object classes and
the traversability are predicted, fused with depth data and projected in the
3D space to build a semantic 3D map. The voxels predicted as plants and with
high traversability are not considered as obstacles and thus the robot is able
to traverse paths covered by traversable plant parts.

##  I INTRODUCTION

The mobile robot technologies have come nearly to the level of practical
realization and commercialization in some situations. For example, self-
driving cars are already being tested in public areas [1, 2]. Another example
is the service robots operating in public places such as airports and stores
[3, 4]. Those applications mainly target structured environments such as urban
areas and the inside of buildings. In plant-rich environments such as
agricultural fields and forest paths, however, it is more difficult for robots
to autonomously navigate than in structured environments because the paths are
possibly covered by plant parts such as branches and leaves, which, though,
can be driven through by the robots. Most of conventional mobile robots only
consider the presence of objects and do not consider such traversable objects.

One of such environments is greenhouses and agricultural fields.
Traditionally, in such environments, plants are extracted from sensor readings
and then two plant rows that form a path are detected via simple line fitting
algorithms [5, 6, 7, 8, 9]. Those methods assume that paths are straight and
open, and not covered by any plant parts. A recent work [10] applies an end-
to-end policy learning to learn a function that directly maps an image to an
appropriate control. Although their method succeeded to navigate in outdoor
scenes with tall grass, it requires “trial and error” experiences achieved via
a robot’s random walk in the real environments, which are difficult to acquire
in environments like greenhouses and other environments with objects that the
robot must not collide with. Imitation learning (IL) is another possible
approach for end-to-end policy learning that takes a human’s demonstration of
control for supervision. However, IL requires abundant amount of high-quality
demonstrations given by experts, which are not easy to collect. Moreover, even
when using learned control policies, explicit scene recognition is still
important for safe deployment of navigation systems in such environments.
Therefore, we aim at an image-based scene recognition considering traversable
plants for robot navigation in plant-rich environments such as greenhouses.

In this paper, we present a novel framework of scene recognition and robot
navigation that takes into account traversable plants covering the paths. To
recognize objects as traversable, it is not sufficient to just recognize
plants because plants have both traversable parts such as branches and leaves,
and non-traversable parts such as stems. Datasets with such fine-grained
classes are not publicly available to the best of our knowledge. To provide
information of traversability of objects, which cannot be acquired from
existing datasets, we use label data indicating the regions with objects that
the robot has traversed during data acquisition phases, coined traversability
masks. Since the traversability masks are generated based on a limited number
of the robot’s traversal experiences, all traversable regions are not labeled
but some of them are left unlabeled (See Fig. III-A). Directly using such
masks as labels of traversable plants will confuse the training. To take full
advantage of the incomplete information of the traversability masks, we
present a deep neural network (DNN) architecture and its training method
inspired by PU (positive and unlabeled) learning [11], where a model is
trained with labeled positive examples and unlabeled examples including both
positive and negative ones.

The overview of the proposed method is shown in Fig. 1. The proposed DNN
architecture consists of Semantic Segmentation Module (SSM) for pixel-wise
general object classification and Traversability Estimation Module (TEM) for
estimating traversability which indicates how likely that each pixel can be
traversed by robots. SSM is trained without a manually labeled greenhouse
dataset by means of an unsupervised domain adaptation (UDA) method proposed by
the authors [12]. TEM is then trained using the features from SSM as inputs
and the traversability masks. For the training of TEM, we introduce a training
method inspired by PU learning [11] in order to take full advantage of the
information given by the traversability masks. The overall network is
therefore trained without manual annotation of the training images.

The prediction results (the object class and the traversability of each pixel)
are then projected to the 3D space to build a semantic 3D voxel map used in
navigation. For each data frame, the prediction results (observations) are
temporally fused with the prior probabilities of the object classes and the
traversability in each voxel using Bayes’ update. By treating as free spaces
voxels where the probability of plant is the highest and the traversability is
greater than a threshold, the robot is able to navigate through traversable
plant parts covering the paths.

The contributions of this paper are as follows:

  1. 1.

A novel framework of scene recognition considering both general object classes
and the traversability of objects for robot navigation in plant-rich
environments.

  2. 2.

A manual annotation-free training method for the scene recognition model.

  3. 3.

A PU learning-based training for traversability estimation which requires only
positive labels given to a part of traversable regions in images.

  4. 4.

Applying the scene recognition model to the navigation tasks in a real
greenhouse with plants partially covering the paths.

##  II RELATED WORK

###  II-A Traversability estimation

####  II-A1 Geometry-based methods

Traverasability estimation is a crucial task in robot navigation.
Traditionally, rigid obstacles and negative obstacles such as ditches and
holes are detected using geometric information from range sensors such as
laser range finders (LRFs), stereo cameras, and RGB-D sensors. In unstructured
scenes, many studies have focused on estimating the traversability of terrains
by analyzing the steepness, roughness and so on [13, 14, 15, 16]. Bogoslavsky
et al. [13] estimated the traversability of indoor environments using Kinect
sensor by analyzing the normals. Zhu et al. [14] exploited disparity
information from a stereo camera to estimate traversable regions consisting of
a flat plane. Martínez et al. [15] used static obstacle detection and terrain
analysis using 2D LRF in their reactive navigation scheme. The near-to-far
approach is also a popular approach, where the terrain traversability is
estimated within the measurement range of a range sensor and the estimated
traversability is then propagated to farther areas based on the similarity of
image features [17, 18, 19].

The researches mentioned above are for estimating the traversability of the
terrain, i.e., whether the robot can drive over the terrain. Unlike those
methods, we deal with the estimation of the traversability of objects such as
plant parts hanging over and covering the paths, i.e., whether the robot is
allowed to touch and push them while navigating.

####  II-A2 Semantics-based methods

Other methods exploit object semantics estimated from RGB images [20, 21, 22,
23, 24, 25]. The rich information of RGB images such as colors and texture
allows for estimating object semantics such as object class and traversability
from the appearance. Zhou et al. [20] trained a terrain classifier on visual
features with class labels automatically given using structural information
from a 3D LiDAR. Matsuzaki et al. [21] used examples of traversable path lines
given by human annotators to train a probabilistic model to classify the image
regions into either of the object classes: building, road, or grass. In the
last decade, DNNs played a crucial role to provide high recognition accuracy
and generalization ability. Maturana et al. [22] generated a 2.5D semantic map
using DNN-based semantic segmentation for path planning of autonomous off-road
vehicles. Gao et al. [23] proposed a fine-grained traversability estimation to
deal with different level of traversability of the terrain. Guan et al. [24]
proposed a hybrid method of DNN-based semantic segmentation and analysis of
geometric information for autonomous navigation of excavators.

None of the aforementioned methods, however, addressed traversable objects
such as plants covering the paths, which are the main target of our method.
Although the authors’ previous work [26] constructed a semantic 3D map inside
a greenhouse, traversable and non-traversable plants are not distinguished and
robot navigation was not realized. In addition, those DNN-based methods to
estimate object semantics require an abundant amount of manual annotation by
humans, while our method provides manual annotation-free training.

####  II-A3 Self-training based on experience

Using information of robot’s traversal is a preferred approach for identifying
traversable regions in recent studies. Barnes et al. [27] used trajectories of
an autonomous vehicle for automatically labeling images for semantic
segmentation. Similarly, Wellhausen et al. [28] proposed to use footprints of
a legged robot to label images and showed that sufficient performance can be
achieved even with such sparse labels. Lee et al., [29] trained a multi-layer
perceptron (MLP) on geometric features in a self-training manner with data
labeling using both a robot’s trajectory and structural information from an
elevation map. Those methods are, however, not applicable to traversable
objects hanging over three-dimensionally. Some studies consider the
traversability of objects as an affordance depending on both the environment
and the robots. Kim et al. [30] trained a classifier with simple hand-crafted
features of image patches by exploiting a robot’s experience of successful and
unsuccessful traversals acquired by its interaction with objects. In another
work [31], they proposed a superpixel-based image classification to improve
the accuracy of region segmentation. The studies by Kim et al. share the most
similar challenge with our work in terms of considering the traversability of
plants.

Recently, methods based on Reinforcement Learning (RL) and Imitation Learning
(IL) have also been presented. Kahn et al. [10] applied in outdoor robot
navigation such an end-to-end policy learning based on robot’s experience of
trial and error gathered through its random walk motion in real environments.
Despite their success, we claim that such a training method is not suitable
for navigation in greenhouses or other plant-rich environments because there
may be objects that the robot must not hit such as crop rows and equipments,
or steep slopes beside the path. Random motions in such environments may lead
to a fatal failure. IL methods instead exploit high-quality demonstrations by
human experts. Such demonstrations are, however, not always available. Even
when a kind of policy learning can be used, explicit scene recognition is
still necessary for ensuring the safety. In this sense, our proposed method
can be rather complementary to those methods.

###  II-B Agricultural mobile robots

Agricultural mobile robots are one of the practical and actively studied
applications of mobile robots in plant-rich environments. Many of currently
used agricultural robots move on rails or heating pipes equipped on the paths
[32, 33]. Robotic systems with such locomotion is able to move stably, while
movable areas for operation are limited. In addition, such systems require
installation of the rails when not installed. More flexible robot operation is
beneficial for extending the field of tasks that the robots can automate.

In outdoor, precise localization by GNSS is adopted in various studies [34,
35]. In indoor fields such as greenhouses, however, such a method is not
available due to noise imposed by surrounding obstacles. Therefore, a popular
approach is recognizing a traversable path using sensors equipped on the
robot. The sensors used in path recognition include ultrasonic sensors [36,
37], LIDARs [7, 38, 9] and monocular / stereo cameras [39, 5, 6, 8].

Most of those methods detect two plant rows that form a path with a
combination of plant detection based on sensor readings, and line fitting
algorithms such as Hough transformation [5, 8] and iterative methods like
RANSAC [9, 7]. Those methods assume that the lines of plants are visible.
However, this is not the case in environments where some plant parts cover the
path. In one of pioneering work by Mandow et al. [36], it has been reported
that while they succeeded to autonomously operate a mobile robot using an
ultrasonic sensor, some failure cases observed where the robot recognized
plant parts overhanging the path as obstacles and stopped the navigation. To
the best of our knowledge, the problem of recognizing traversable plant parts
covering the paths has not been explicitly addressed.

###  II-C Semantic segmentation

Semantic segmentation is a task to assign an object class to each pixel of an
image. Long et al. [40] first proposed to convert classification networks into
fully convolutional network to produce a probability map for each pixel. Based
on ResNet proposed by He et al. [41] as a backbone network, very deep networks
have been shown to achieve high accuracy [42, 43]. While it is proven to be
effective to stack layers in depth to get a high accuracy, those models impose
high computational cost. For real-time visual tasks, computationally efficient
architectures have also been studied, targeting the applications such as
robotics and autonomous driving systems [44, 45, 46]. Recent approaches
further improved the accuracy while maintaining the efficiency by introducing
more efficient convolution approaches such as factorized convolution [47],
depth-wise separable convolution [48], efficient spatial pyramid [49, 50] etc.

In our previous work, we have shown that a semantic segmentation network for
greenhouses can be trained without requiring hand-labeled images, by utilizing
multiple publicly available datasets of urban scenes and employing an
unsupervised domain adaptation (UDA) method [12].

###  II-D PU learning

PU (positive and unlabeled) learning is a problem where part of the data is
labeled as positive, and the rest is unlabeled and could be either positive or
negative. Unlike the setting where positive and negative data are assumed, PU
setting cannot be solved by just applying a simple classification of positive
and unlabeled data since the unlabeled data include positive ones. Elkan and
Noto [11] showed that it is possible to estimate the label probabilities by
modeling the probability that given samples are labeled. As a practical
application, Yang et al. [51] applied PU learning framework in object
detection task where some of objects are not labeled.

In our work, we exploit binary-labeled images where pixels that the robot has
traversed are labeled, and the rest is left unlabeled. We can treat the
labeled pixels as “traversable” based on the robot’s traversals, whereas it is
not always true that the unlabeled pixels are “non-traversable”, since there
is a substantial amount of unlabeled pixels of traversable objects as later
shown in III-A. This problem is exactly an example of PU learning. We
therefore introduce a training method inspired by Elkan and Noto [11] to the
training of TEM in III-B2.

##  III PROPOSED METHOD

Fig. 1 shows the overview of the proposed framework. We employ two network
modules for the scene recognition: Semantic Segmentation Module (SSM) for
semantic segmentation, and Traversability Estimation Module (TEM) for
traversability estimation. SSM is trained by an unsupervised domain adaptation
method. For the training of TEM, we generate traversability masks based on the
robot’s traversal experience. The network is then trained with the generated
traversability masks (Fig. 1(a)). Predictions of the trained network are
projected to the 3D space to build semantic local 3D map that indicates the
object classes and traversability of the regions around the robot and the
navigation is operated based on the map (Fig. 1(b)).

###  III-A Traversability mask

The traversability masks are label data that indicate the image regions
traversed by the robot or a human during the data acquisition phase. Using the
traversability masks, we automatically label the regions of images dominated
by traversable plant parts to train a network so that it can distinguish
traversable and non-traversable plants based on the robot’s experience.

We first acquire RGB-D data by manually operating a robot or walking inside
the target greenhouse with an RGB-D sensor moving through the plants. We then
build a 3D voxel map using RTAB-Map [52] from the RGB-D data. After that, by
utilizing the robot’s path estimated in the mapping process, the voxels
traversed by the robot or the human are labeled. Here, we approximate the
shape of the robot or the human with a rectangular, and from each observation
point of the 3D voxel map, voxels within the rectangular are labeled as
traversed. Finally, traversability masks are generated by projecting the
traversability labels of the voxels on an image plane from each sensor pose on
the map.

[t!](topskip=0pt, botskip=0pt,
midskip=0pt)[width=0.9999]figs/matsu2_trav_mask_example.pdf Examples of
traversability masks. Regions traversed by the robot are superimposed on the
corresponding camera images Examples of the traversability masks are shown in
Fig. III-A. One may notice that while much of traversable plant parts such as
leaves and branches are labeled, there is also quite a lot of unlabeled
regions of traversable plants. This is because the traversability masks are
based on the limited number of traverses during the data acquisition phase.
This nature of the traversability masks makes it inappropriate to simply treat
them as positive and negative examples. By our training method inspired by PU
learning [11] described in III-B2, the traversability masks are effectively
exploited to distinguish traversable and non-traversable plants.

###  III-B Network architecture

In this section, we describe a network architecture for pixel-wise semantic
segmentation and traversability estimation. The overview of our network is
shown in Fig. 1(a). The network consists of Semantic Segmentation Module (SSM)
for pixel-wise object label assignment and Traversability Estimation Module
(TEM) for estimating how likely that each pixel can be traversed by robots.
SSM is first trained independently. TEM is then trained with intermediate
features of SSM as inputs and traversability masks as training labels. The
reason for not training them simultaneously is to avoid overfitting of the
entire network to the traversability masks which include unlabeled traversable
regions.

Our network is based on ESPNetv2 [50] and we make two modifications. First, an
auxiliary segmentation branch is attached to the middle of the main network of
ESPNetv2. We refer to the segmentation network including the auxiliary branch
as SSM (Semantic Segmentation Module). The auxiliary branch is used to
estimate pixel-wise uncertainty for the training of SSM proposed in [12].
Second, a branch for estimating traversability, i.e., TEM is attached. The
intermediate feature from the SSM is fed into a $3\times 3$ convolution and
the sigmoid function to produce pixel-wise probability predictions. For the
complete network architecture, see Fig. 7 in the Appendix. The additional
layers do not affect the computational efficiency and still allows for
prediction at sufficient speed as explained in IV-B.

The advantages of our architecture are three-fold. Firstly, the architecture
enables the PU learning-based training to model the traversability as
probability values. Using the semantic segmentation task provides
discriminative features for the PU learning. Secondly, by separating the tasks
into semantic segmentation and traversability estimation, the branches for
each task can be trained with the maximum possible number of data. Because the
traversability masks are generated from a map, the number of available
training data for TEM is less than that for SSM, which only requires RGB
images. Thanks to the network structure, the SSM can fully exploit available
training images for a better performance of semantic segmentation individually
from the training of TEM. In fact, as shown later in IV-A, the SSM was trained
with 6689 images while the TEM was trained with 889 images with traversability
masks in our experiment. Thirdly, the two modules provide complementary
information. Misclassifications of TEM can be refined by a prediction of SSM
by utilizing prior knowledge that, e.g., artificial objects are not
traversable. We evaluate this effect in IV-A4.

####  III-B1 Semantic Segmentation Module

Semantic Segmentation Module (SSM) is responsible for pixel-wise general
object classification. Here we use three general object classes: plants,
artificial objects, and the ground. For training SSM, we introduce a training
method in our previous work [12]. In the method, we employ multiple publicly
available datasets with pixel-wise labels (the source datasets) and an
unlabeled greenhouse dataset (the target dataset). First, segmentation models
are trained with each source dataset. Images from the target dataset are then
fed into the models and corresponding outputs are yielded. If a pixel is
assigned with the same object label from all the source models, the label is
treated as a pseudo-label of the pixel. For training of SSM with the pseudo-
labels, we adopt a method of adaptively weighting a loss value on each pixel
based on the uncertainty of the prediction using the main and auxiliary
branches [53].

####  III-B2 Traversability Estimation Module

Traversability Estimation Module (TEM) estimates the probabilities that the
pixels are traversable. In the training of TEM with the traversability masks,
we introduce the PU (positive and unlabeled) learning framework. The purpose
of PU learning is to model a probability function $p(y=1|x)$, where $x$ is an
input data and $y\in\\{0,1\\}$ is a binary label of the data $x$, in a
situation where a part of the positive data ($y=1$) is labeled as positive,
and the rest is unlabeled and can be either positive or negative. Elkan and
Noto [11] state that under the “selected completely at random” assumption,
meaning that the labels are given completely at random, PU learning can be
solved by modeling a classifier $g(x)$ such that $g(x)=p(s=1|x)$, where $s$
denotes a random variable indicating that data $x$ is labeled if $s=1$. Here,
note that $y=1$ if $s=1$ but not vice versa. The equation can then be
transformed as follows, given the “selected completely at random” assumption,
i.e., $p(s=1|y=1,x)=p(s=1|y=1)$:

$\displaystyle g(x)$ $\displaystyle=$ $\displaystyle p(s=1|x)$ (1)
$\displaystyle=$ $\displaystyle p(y=1\land s=1|x)$ $\displaystyle=$
$\displaystyle p(y=1|x)p(s=1|y=1,x)$ $\displaystyle=$ $\displaystyle
p(y=1|x)p(s=1|y=1).$

Therefore, the traversable probability of a pixel $p(y=1|x)$ can be calculated
as follows:

$p(y=1|x)=\frac{g(x)}{c},$ (2)

where $c=p(s=1|y=1)$ which denotes the conditional probability that a positive
data is labeled. $c$ is approximated by feeding the training data to the
estimated $g(x)$ and average the probabilities of the labeled features, i.e.,

$c=p(s=1|y=1)\approx\frac{1}{n}\sum_{x\in P}g(x),$ (3)

where $P$ denotes a set of all labeled pixels in the training dataset and $n$
is the number of elements in $P$.

In TEM, we first model the label probability $g(x)=p(s=1|x)$ by a convolution
and a sigmoid activation. The output is then divided by $c$ calculated by eq.
(3) to produce $p(y=1|x)$. The feature map from the SSM is fed in TEM. The
size of the feature map is $(B,C,H,W)$ where $B$ denotes the batch size, $C$
denotes the channel size, $H$ and $W$ denote the height and the width,
respectively. Here we consider a vector of size $C$ as an input $x$ for the
corresponding pixel. $3\times 3$ convolution is applied to the features,
followed by a sigmoid function to scale the output in the range of $[0,1]$. We
use $3\times 3$ convolution instead of $1\times 1$ convolution to take
information of adjacent pixels into consideration in the estimation.

Note that the “selected completely at random” assumption does not strictly
hold in our task. In practice, however, we confirm that this formulation is
effective to estimate pixel-wise traversability.

###  III-C 3D semantic voxel map

Using the results of the recognition explained in section III-B, we build 3D
semantic voxel map around the robot. At first, an RGB image acquired from the
RGB-D sensor is passed to the segmentation network. The predictions from SSM
and TEM are then mapped into the 3D space using the corresponding depth image.
The 3D space around the robot is divided into voxels with a side of 0.1 [m],
and the predictions mapped into the 3D space are assigned to a corresponding
voxel that they fall into. For the object class information, a histogram of
the object classes is constructed in each voxel and the class label with
highest frequency within the voxel is used as an observation. For the
traversability information, the traversability values of the points within a
voxel are averaged and treated as an observation.

In each time step, those observations are temporally fused by Bayesian update.
The observation of the object class $z^{o}_{t}$ at time $t$ is fused by the
following equation:

$P(l_{t}|z^{o}_{t})=\eta P(z^{o}_{t}|l_{t-1})P(l_{t-1}),$ (4)

where $l_{t}$ denotes an object label of the voxel at time $t$, $\eta$ is a
normalization term and $P(l_{t-1})$ is the prior of label $l_{t-1}$ at time
$t-1$. $P(z^{o}_{t}|l)$ is the likelihood of label $l$ with observation
$z^{o}_{t}$. The likelihood is calculated using greenhouse images not used in
the training of SSM as follows. Firstly, pseudo-labels are generated for each
image. The same images are then fed in SSM and outputs of pixel-wise class
label are produced. For each object class $l$, a histogram of predicted object
classes over all pixels of the pseudo-labels with the label $l$. A conditional
probability $P(z_{o}|l)$ is calculated by normalizing the histogram and used
as a likelihood function. The object class with the highest probability is
assigned to the voxel as its label.

Similarly, the observation of traversability is fused by the following
equation:

$P(\tau_{t}|z^{\tau}_{t})=\eta P(z^{\tau}_{t}|\tau_{t-1})P(\tau_{t-1}),$ (5)

where $\tau_{t}\in\\{0,1\\}$ denotes an event that the voxel is traversable
(1) or not traversable (0) at time $t$. The likelihood $P(z^{\tau}_{t}|\tau)$
is calculated using the training images with traversability masks as follows.
The input images are fed in TEM and outputs of traversability values are
yielded. For each label $\tau\in\\{0,1\\}$ in the traversability masks, a
histogram of estimated traversability values is calculated over all pixels
with the traversability label $\tau$. The histograms are then normalized to
calculate a conditional probability $P(z^{\tau}_{t}|\tau)$ and it is used as a
likelihood.

After the integration of observed points to the semantic voxel map, obstacle
point cloud is generated by treating the voxels whose object class is “plant”
and the traversability is higher than the threshold as free spaces and the
others as obstacles. The location of a point for a voxel is the centroid of
the points that have been accumulated in the voxel over the frames. By feeding
the obstacle point cloud to a conventional navigation module, the navigation
in plant-rich environments is realized. For mitigating the degradation of the
processing speed as the number of voxels increases, the voxels where no point
has been observed for a certain number of consecutive frames are removed from
the map. In the experiment below, the number of frames is set to 10.

##  IV EXPERIMENTS

###  IV-A Evaluation of TEM

####  IV-A1 Used datasets

TABLE I: Greenhouse datasets used in the training. The type of each set is shown in the brackets | Train | Test | Date
---|---|---|---
A | 6684 (unlabeled) | 33 (true trav. labels) | May 25, 2018
B | 899 (w/ trav. masks) | - | July 12, 2019

For the training of SSM by the UDA method, we used three source datasets:
CamVid [54], Cityscapes [55], and Freiburg Forest [56], for generating pseudo-
labels. As target data, we used 6684 unlabeled greenhouse images from
Greenhouse A dataset, taken in a greenhouse growing tomatoes.

For the training of TEM, we used 899 pairs of an RGB image and the
corresponding traversability mask from Greenhouse B dataset, taken in the same
greenhouse as Greenhouse A on a different date. The data were collected
through a human operator’s control of the robot for about 20 minutes in total.
During the data collection, the robot traversed each of the paths with plant
rows on both sides only once. The labels are therefore imbalanced as can be
seen in Fig. III-A. Approximately 40 to 50 % of the traversable plant regions
were actually labeled. Greenhouse A and B have a different appearance due to a
different level of growth of the plants. For the evaluation, we manually gave
true labels of traversable regions on 33 images from Greenhouse A. Overview of
the greenhouse datasets is shown in Table I.

####  IV-A2 Training conditions and hyperparameters

The DNN model is implemented with PyTorch [57] and all of the training is
performed on one NVIDIA Quadro RTX 8000 with 48GB of memory. In the training
of SSM, the training epoch is set to 200 with a fixed learning rate of
$5\times 10^{-5}$, and the batch size of 64. TEM is then trained with weights
of SSM fixed. The number of epoch is set to 200 and the batch size is 64. We
use cyclical learning rate scheduling [58]. The initial learning rate is
$5\times 10^{-5}$ and linearly increases by a factor of 10 in 10 epochs and
then decreases to the original value in 20 epochs.

####  IV-A3 Baselines for traversability estimation

As baseline methods of image-based traversability estimation, we use the
following methods.

Semantic segmentation We train a semantic segmentation model with “traversable
plant” class as well as the classes used in the training of SSM. A model
identical to SSM is used as a semantic segmentation model. For training, we
used the same dataset as the one used in the training of TEM (899 RGB images
with traversability masks). Pseudo-labels are generated for each image with
the three classes. Labels of the pixels with the “plant” class are then
changed to “traversable plant” if the value of the corresponding
traversability mask is 1 (traversable). As a result, the object classes used
in the training are as follows: traversable plant, other plant, artificial
object, and ground. We initialize the network via pre-training with Greenhouse
A dataset. A constant learning rate of $1\times 10^{-4}$ is used and an epoch
size is 50.

Kim et al. Out of few studies that explicitly predict the traversability of
objects such as plants, we use Kim et al. [30, 31] as another baseline. They
addressed the problem of training a classifier based on a robot’s experience
of successes/failures of traversals, which shares the research target with
ours. To evaluate the baseline method, we conducted 10-fold cross validation
using the 33 test images with ground truth labels. We use those data instead
of the training images with traversability masks for two reasons: 1\. the
baseline method assumes complete labels of traversable/non-traversable while
the traversability masks are not complete and include unlabeled traversable
regions. 2\. the baseline method is for incremental online learning with
several image frames instead of hundreds of training data. In the original
work [31], ten consecutive frames of images are used for training.

####  IV-A4 Results

Figure 2: Precision-Recall curve and IoU-Recall curve of traversability
estimation TABLE II: Performance of TEM

| IoU (%) | Accuracy (%) | Precision (%) | Recall (%)
---|---|---|---|---
Raw | 44.69 | 82.73 | 50.39 | 81.41
Refined |  45.03 (+0.34) |  83.05 (+0.32) |  51.00 (+0.61) | 81.04 (-0.37)
Segmentation | 2.97 | 82.76 | 45.45 | 3.10
Kim et al. [31] | 30.95 | 80.42 | 35.87 | 72.12

Figure 3: Prediction results of TEM. From the top: Camera images, Ground
truth, predicted binary images with threshold 0.75, refined binary images,
binary images generated from the ordinary semantic segmentation method
(“traversable plant” pixels are shown in white), and binary images generated
from the probability maps predicted by the baseline method [31]. Note that the
traversable regions are not identical to the plant regions, but plant parts
such as stems are not traversable. TEM is capable of distinguishing the
traversable plants from the non-traversable ones (highlighted in yellow
ellipses). The refinement by predicted object classes mainly worked as noise
suppression (highlighted in red rectangles).

We evaluate the binary traversability images generated from the predicted
traversability with different thresholds. Here we compare two types of binary
images: raw and refined. The raw binary images are generated by just
binarizing the predicted traversability with a threshold. The refined binary
images are generated by additionally setting all the pixels predicted as an
artificial object or the ground to non-traversable. This refinement provides
the filtering effect of false positive traversability predictions even in the
case of high predicted traversability, which is equivalent to the calculation
in the voxel described in III-C where the final decision of traversability is
given considering both the object class and the predicted traversability. Fig.
2 shows a Precision-Recall curve as well as an IoU-Recall curve to analyze the
performance of TEM. The best IoU was achieved when the threshold was 0.75.
Therefore, we use this traversability threshold in the rest of the paper.
Table II shows the accuracy, precision, and recall of the model on the test
set with the threshold of 0.75.

From Fig. 2 and Table II, we can see high recall and relatively low precision.
The refined predictions resulted in better precision and IoU. This result
indicates that fusing the predictions of object classes and traversability
decreased the false positive rate and thus is suitable for safe robot
operation compared to only predicting the traversability.

Fig. 3 is the visualization of the prediction by TEM. Much of the traversable
regions are classified as positive. In addition, some non-traversable plants
are predicted correctly. This result shows that our model can learn the
differences between traversable and non-traversable plants. In terms of the
refined results, we cannot qualitatively see obvious improvements, but we
confirmed that small prediction noise was suppressed. Note that these
segmentation results are yielded from automatically generated traversability
masks, which are incomplete and noisy. Even from such data, our PU learning-
based training framework effectively learns the features of traversable
objects, and provides reasonable estimation.

Comparison to the semantic segmentation TEM provided better performance than
the segmentation-only method in all the metrics. Especially, the IoU and the
recall were significantly low in the ordinary semantic segmentation. In Fig.
3, we can see that the semantic segmentation method could not capture the
traversable plant regions well. We suppose it is because the labeling of
“traversable plant” regions does not cover all the traversable plant regions
as mentioned in III-A and results in inconsistent labels with traversable
plant regions labeled as non-traversable plant. Such labels confuse the
training. Therefore, the segmentation-only method does not get along with the
sparse traversable masks. On the other hand, our proposed method enables more
accurate and effective training of the model to estimate traversability of
image regions.

Comparison to Kim et al. We report the average of the metrics over the 10
different partitions for the cross-validation in Table II. For each data
partition, the threshold of the probability is determined so that it yields
the best mean IoU for the test images. As a result, our proposed method
outperformed the baseline in all the metrics even though the baseline model is
trained with ground truth labels while our method is trained with noisy and
incomplete traversability masks. As can be seen in Fig. 3, although the plant
regions are mostly classified correctly, other regions of artificial objects
and the ground are also classified as traversable. Moreover, the traversable
leaves and branches and non-traversable stem parts are not distinguished and
almost equally classified as traversable.

Our method is advantageous over the baseline by Kim et al. in three aspects.
First, our PU learning-based method enables the training without negative
labels that are difficult to acquire. In [30], negative labels are collected
using the information of the robot actually bumping into an obstacle, which is
not appropriate in environments such as greenhouses. Our learning framework
allows for safer and more practical way of training the scene recognition
model by utilizing only positive labels acquired via the control by a human
operator. Second, our model provides semantics about object classes to
complement the estimation of traversability. As can be seen in Fig. 3, there
are a lot of regions wrongly classified as traversable in the prediction of
the baseline method. There is no way to correct the misclassifications in the
baseline, while our method can amend them using the prediction by SSM as
mentioned above. Third, our method can utilize more discriminative features
thanks to the DNN model trained in the task of semantic segmentation, which
led to the better performance of the proposed method over the baseline even by
the raw predictions of TEM.

###  IV-B Navigation in a greenhouse

Finally, we conducted a real-world navigation experiment in a greenhouse. All
the software is processed on a laptop with an Intel Core i7-6700HQ, an NVIDIA
GeForce 960M, and 32 GB RAM. Revast Mercury is used as a robot platform (See
Fig. 4(a)). As sensors, the robot is equipped with a Kinect v2, an RGB-D
sensor, and Hokuyo UTM-30LX, a laser range finder.

An image of a path in the greenhouse is shown in Fig. 4(b). The length of the
paths is approximately 7.5 [m]. The software for the DNN prediction is
implemented on Robot Operating System (ROS) [59] with C++ and Libtorch, a C++
library for PyTorch. The frame rate of the prediction of the network is
approximately 8 [fps] and the 3D mapping runs at approximately 5 [fps]. The
frame rate of the original ESPNetv2 is also approximately 8 [fps] on the same
computer and thus our modification to the network does not affect the
computational efficiency.

As a baseline for navigation, we also test a system that considers all the
voxels as obstacles. It corresponds to using only geometric information, which
is a major approach in most of the current mobile robots.

((a)) Robot configuration ((b)) Inside the greenhouse

Figure 4: Experiment environment

((a))  ((b))  ((c))

Figure 5: Generated 3D maps during the experiment. An input image, a predicted
segmentation image, and a predicted pixel-wise traversability are shown on the
left side of each figure. In the segmentation images, blue is the plant, red
is the artificial object and yellow is the ground. In the figures of 3D map,
green indicates traversable voxels with the plant class and the traversability
higher than the threshold. The rest of the colors indicates the same object
classes as in the segmentation images.

####  IV-B1 Obstacle detection in a simple forward motion

As a primary experiment, we conducted an experiment of obstacle detection
using the proposed 3D semantic map during the robot’s act of moving forward.
We assume that the path is straight and constant control signals of linear
velocity are given to the robot while the robot recognizes the traversability
of objects in front. The linear velocity is set to 0.1 [m/s]. An obstacle
point cloud is generated from the 3D semantic voxel map by outputting only
voxels other than “traversable plant” as obstacle points. When an obstacle
point is detected within a range in front of the robot, the control signal is
stopped.

We carried out 5 trials on the same path. The baseline navigation method
consistently failed the task due to the part of plants grown out to the path.
In contrast, our system was able to recognize the traversable plants and
navigate through the path between the plant rows in all the trials. This
result shows the ability of our scene recognition method to recognize the
traversable plants covering the path. There were, however, some cases where
the robot stops for a while and then restarts even though there was no
obstacle. This was due to misclassification of voxels in front of the robots
as obstacles. The misclassification of voxels stems either from the error in
the image-based semantic segmentation or from the registration error between
the predicted segmentation images and the corresponding depth images, which
RGB-D sensors inevitably have. Errors temporarily occurring on a few pixels
are suppressed by the voxelization and the Bayes’ update. When the error
consistently occurs on a large area, however, the corresponding voxel can be
misclassified.

Fig. 5 is the visualization of the semantic 3D maps generated during the
experiment at different times in chronological order. There were a lot of
plant parts partially covering the path. Our proposed method was able to
recognize the traversable plants and to traverse them. Fig. 5(b) shows the
case where the system wrongly recognized the regions in a far front of the
robot as obstacles shown in the red voxels. The robot stopped moving in
response to the obstacles. After observing a few data frames while stopping,
the obstacle voxels were removed from the map and the robot resumed the
navigation. We suppose the cause of this phenomenon as follows. When the
misclassified regions are far from the robot, the segmentation results and the
depth readings can be more noisy and it results in the misclassification of
the voxels or creating voxels on wrong locations. In the case in Fig. 5(b), it
is more likely that the voxels were spawned on wrong locations due to the
depth noise. As the robot approaches the voxels, both the segmentation and the
depth values become more accurate and the voxels are updated with those
observations. As a result, in the case in Fig. 5(c), the misclassified voxels
were deleted from the voxel map because points in those voxels were not
observed for the pre-defined number of frames (10 frames) when the robot was
stopping near them.

####  IV-B2 Integration with move_base

We also conducted an experiment of applying our 3D semantic voxel map to
navigation using move_base [60], a de facto standard navigation software in
ROS [59].

The obstacle point cloud is fed in move_base as a sensor reading. In addition
to the obstacle point cloud, a 2D laser range finder (LRF) is used to
complement the blind spot of the RGB-D sensor and stabilize the navigation.
Because of this purpose, the obstacle range of the LRF is limited to 1.0 [m].
The configuration of the LRF is the same in the baseline method. Sub-goals are
given manually via Rviz, an interactive GUI tool for visualization.

Here, we define criteria for judging the success or failure of the trials. In
all the trials, we observed that the accuracy of the SSM consistently degrades
about 2.5 [m] before the end of the paths and the robot fails to detect the
traversable path (We discuss this problem in IV-C). We therefore consider a
trial in which the robot reached about 2.5 [m] before the end of the path as
“traversed“. When the robot stops due to misclassification, we allow a human’s
intervention by manually restarting move_base to clear the costmap at most
once. Trials where the robot gets stuck even after the human intervention are
marked as “stuck”.

The baseline navigation method consistently failed to find a path on three
paths before entering them. An example of a costmap generated with the
baseline method is shown in Fig. 6(b). We carried out 12 trials of navigation
on the three paths (4 each) with the proposed method. The result is shown in
Table III.

TABLE III: Result of the navigation experiment using move_base Trials | Traversed (intervened) | Stuck
---|---|---
12 | 8 (5) | 4

The robot with the proposed method succeeded to traverse through the path
partially covered by plants in 8 trials, 5 of which involved a human’s
intervention. The costmaps generated with the proposed method are compared in
Fig. 6(c). From these results, we can see that our proposed method enables the
robot to recognize the traversable plant parts, which would otherwise be
recognized as obstacles, and traverse through them. However,
misclassifications affected the robot’s motion as can be seen from the number
of required interventions and failures.

((a)) Input image

((b)) Baseline for navigation

((c)) Ours

Figure 6: Costmaps in the baseline and the proposed method. (a) An input
image. (b) Semantic 3D voxel map and the costmap of the baseline. All the
voxels are considered as obstacles and thus the path is recognized as blocked.
(c) Semantic 3D voxel map and the costmap generated by the proposed method.
The voxels of traversable plant are not considered as obstacles and the path
is recognized as traversable.

###  IV-C Discussion

####  IV-C1 Cause of the failures of navigation

We showed that our method enabled navigation through the paths covered by
traversable plant parts in the greenhouse, while the baseline method
constantly failed.

In all the trials, however, the robot with our proposed method consistently
failed to navigate about 2.5 [m] before the end of the paths. In addition, in
some of the successful trials, it required a human’s intervention due to
misclassifications. On the other hand, in the experiment in IV-B1, all the 5
trials were successful without any interventions.

We suppose that the difference of the degrees of the success rates stems from
the difference of the density of leaves. In such a case, majority of the
regions are dominated by a single class of “plant” and a few
misclassifications are filtered out in the probability calculation in each
voxel. In the experiment in IV-B2, however, leaves were sparse and artificial
objects behind the plant rows were also visible through the leaves. This was
especially obvious near the end of the path. Since the SSM is trained with
pseudo-labels, which are generated by merging the outputs from multiple pre-
trained models, the model lacks pixel-level accuracy. This is the current
limitation of our method.

For more reliable navigation, the accuracy of the recognition model needs to
be improved. In addition, to deal with the prediction noise in DNN models, a
better filtering and/or refinement methods should be implemented.

####  IV-C2 Adaptability of the proposed system

The success of the training of our scene recognition model depends on the UDA
of semantic segmentation, since TEM takes the intermediate features of the
semantic segmentation network (SSM) for estimation. In terms of the
segmentation tasks in greenhouses, we confirmed in the previous work [12] that
our UDA method is applicable to multiple greenhouses and multiple seasons. The
proposed method will thus also be applicable to various greenhouse
environments. When applying the proposed system in a new environment, in
general, the segmentation model will need to be newly trained to adapt to the
environment. Once the segmentation model is trained, TEM can be trained in the
same manner as described in this paper. Note that our proposed model and the
training method provide advantages of the training with a little images and
incomplete positive labels, which lead to a minimum burden in deployment.

####  IV-C3 Influence of the sensor setting

In this work, we employed a Kinect v2 for simplicity of implementation of the
sensor system to acquire calibrated RGB and depth images. Although it is
sensitive to disturbance from strong light, we did not observe any severe
effect of the sunlight on the depth readings. in our experiments. We suppose
it is because the sunlight was cut by the translucent roof and walls of the
greenhouse and the leaves, and thus the sensor was able to work in moderate
lighting conditions. When we apply the system to outdoor environments,
however, the choice of sensors will influence the accuracy and reliability of
the system. Note that our proposed method is compatible with any sensors that
can provide registered RGB images and depth readings and the sensors can be
selected based on the lighting condition of the target environment. Applicable
sensor settings include a stereo sensor, and well-calibrated 3D LiDAR and a
monocular camera.

In terms of sensor calibration, Kinect v2 was used out-of-the-box in our
experiments. In multi-sensor setting, it will indeed require a precise
calibration, although the Bayes’ update and the voxelization process can to
some extent deal with a little noise due to the misalignment of the sensors.

##  V CONCLUSIONS

In this paper, we described a method of estimating the traversability of
plants covering a path and navigating through them in plant-rich environments
for mobile robots. We proposed the following novel methods. 1\. A scene
recognition model that estimates general object classes and the traversability
of the objects. 2\. A manual annotation-free training of the model using an
unsupervised domain adaptation method for the semantic segmentation module
(SSM) and the information of the robot’s traversals, named Traverasability
masks for the traversability estimation module (TEM). 3\. A PU learning-based
training method to effectively train TEM with the traversability masks, which
include unlabeled traversable regions. Our proposed method enables easy and
safe deployment of mobile robots with the capability of recognizing
traversable plants with a minimum burden. In the comparative analysis, we
confirmed that the proposed method is able to estimate the traversability of
image regions more accurately than a conventional semantic segmentation and an
existing work of image-based traversability estimation. Moreover, we applied
the scene recognition model to the navigation tasks in a real greenhouse with
plants covering the paths and confirmed its ability to recognize traversable
plants and navigate through them.

As future work, we are going to work on the improvement of the navigation
system. In particular, the accuracy of the scene recognition and the filtering
of misclassification should be improved. We are also looking to applying this
work to a navigation task in unstructured environments such as forest paths.

##  Appendix A Details of the network structure

Fig. 7(a) shows a complete structure of our network architecture. Our network
is based on ESPNetv2 [50], a light-weight semantic segmentation network. It
consists of modules such as Extremely Efficient Spatial Pyramid (EESP),
Efficient Point-Wise Convolution (EffPWConv), Efficient Pyramid Pooling
(EffPyrPool), which consist of several convolutions with different types of
kernels, batch normalization, and activation layers. For detailed descriptions
of those modules, we refer the readers to [50]. An auxiliary segmentation
branch and a branch for estimating traversability, i.e., TEM are attached to
the middle of the main network of ESPNetv2. The features for TEM are generated
by concatenating the intermediate features from the EffPyrPool layer in each
segmentation branch (See Fig. 7(b)). The intermediate feature from the SSM is
fed into a $3\times 3$ convolution and the sigmoid function to produce pixel-
wise probability predictions.

((a)) Network architecture ((b)) Efficient Pyramid Pooling (EffPyrPool)

Figure 7: A detailed diagram of our architecture. The numbers beside each
layer shows the spatial resolution relative to the input image and the channel
size of the output of the layer, respectively. Based on ESPNetv2 [50], an
auxiliary segmentation branch is added to estimate pixel-wise uncertainty for
the training of SSM proposed in [12]. For estimating traversability, the
intermediate features of the two segmentation branches are concatenated and
fed into a $3\times 3$ convolution followed by the sigmoid function. For the
detail of each network layer, refer to [50].

## References

  * [1] Waymo  “Waymo Safety Report”, 2020  URL: https://storage.googleapis.com/sdc-prod/v1/safety-report/2020-09-waymo-safety-report.pdf
  * [2] Tier IV  “Tier IV Safety Report”, 2020  URL: https://assets.ctfassets.net/rfp71c5fx4wl/3YwptpgxDOmVddje77hxqX/f9468a13518bee9876ed7270c8ba76de/Safety_Report_Eng_9_29_master_compressed.pdf
  * [3] Amit Kumar Pandey and Rodolphe Gelin  “A Mass-Produced Sociable Humanoid Robot: Pepper: the First Machine of Its Kind”  In _IEEE Robotics and Automation Magazine_ 25.3 IEEE, 2018, pp. 40–48  DOI: 10.1109/MRA.2018.2833157
  * [4] Rudolph Triebel et al.  “SPENCER: A Socially Aware Service Robot for Passenger Guidance and Help in Busy Airports”  In _Field and Service Robotics_ New York: Springer International Publishing, 2016, pp. 607–622  DOI: 10.1007/978-3-319-27702-8˙40
  * [5] Tijmen Bakker et al.  “A vision based row detection system for sugar beet”  In _Computers and Electronics in Agriculture_ 60.1, 2008, pp. 87–95  DOI: 10.1016/j.compag.2007.07.006
  * [6] Jinlin Xue, Lei Zhang and Tony E Grift  “Variable field-of-view machine vision based row guidance of an agricultural robot”  In _Computers and Electronics in Agriculture_ 84, 2012, pp. 85–91  DOI: 10.1016/j.compag.2012.02.009
  * [7] Flavio B.P. Malavazi et al.  “LiDAR-only based navigation algorithm for an autonomous agricultural robot”  In _Computers and Electronics in Agriculture_ 154 Elsevier B.V., 2018, pp. 71–79  DOI: 10.1016/j.compag.2018.08.034
  * [8] Wera Winterhalter, Freya Fleckenstein, Christian Dornhege and Wolfram Burgard  “Localization for precision navigation in agricultural fields—Beyond crop row following”  In _Journal of Field Robotics_ 38.3, 2020, pp. 429–451  DOI: 10.1002/rob.21995
  * [9] Tuan D. Le, Vignesh R. Ponnambalam, Jon G.O. Gjevestad and Pål J. From  “A low-cost and efficient autonomous row-following robot for food production in polytunnels”  In _Journal of Field Robotics_ 37.2, 2020, pp. 309–321  DOI: 10.1002/rob.21878
  * [10] Gregory Kahn, Pieter Abbeel and Sergey Levine  “BADGR: An Autonomous Self-Supervised Learning-Based Navigation System”  In _IEEE Robotics and Automation Letters_ 6.2, 2021, pp. 1312–1319  DOI: 10.1109/LRA.2021.3057023
  * [11] Charles Elkan and Keith Noto  “Learning Classifiers from Only Positive and Unlabeled Data”  In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2008, pp. 213–220
  * [12] Shigemichi Matsuzaki, Jun Miura and Hiroaki Masuzawa  “Multi-source Pseudo-label Learning of Semantic Segmentation for the Scene Recognition of Agricultural Mobile Robots”, 2021  arXiv: http://arxiv.org/abs/2102.06386
  * [13] Igor Bogoslavskyi et al.  “Efficient traversability analysis for mobile robots using the Kinect sensor”  In _European Conference on Mobile Robots_ , 2013, pp. 158–163  DOI: 10.1109/ECMR.2013.6698836
  * [14] Xiaozhou Zhu et al.  “Stereo vision based traversable region detection for mobile robots using u-v-disparity”  In _Chinese Control Conference_ , 2013, pp. 5785–5790
  * [15] Jorge L. Martínez et al.  “Reactive navigation on natural environments by continuous classification of ground traversability”  In _Sensors_ 20.22, 2020, pp. 1–18  DOI: 10.3390/s20226423
  * [16] Yan Zhou, Ying Huang and Zhenhua Xiong  “3D Traversability Map Generation for Mobile Robots Based on Point Cloud”  In _IEEE/ASME International Conference on Advanced Intelligent Mechatronics_ IEEE, 2021, pp. 836–841  DOI: 10.1109/AIM46487.2021.9517463
  * [17] Michael Happold, Mark Ollis and Nik Johnson  “Enhancing Supervised Terrain Classification with Predictive Unsupervised Learning”  In _Robotics: Science and Systems_ , 2006  URL: http://roboticsproceedings.org/rss02/p06.pdf
  * [18] Raia Hadsell et al.  “Deep Belief Net Learning in a Long-Range Vision System for Autonomous Off-Road Driving”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2008, pp. 628–633  URL: http://yann.lecun.com/exdb/publis/pdf/hadsell-iros-08.pdf
  * [19] Mingjun Wang, Jun Zhou, Jun Tu and Chengliang Liu  “Learning Long-range Terrain Perception for Autonomous Mobile Robots”  In _International Journal of Advanced Robotic Systems_ 7.1, 2010, pp. 55–66
  * [20] Shengyan Zhou et al.  “Self-Supervised Learning to Visually Detect Terrain Surfaces for Autonomous Robots Operating in Forested Terrain”  In _Journal of Field Robotics_ 29.2, 2012, pp. 277–297  DOI: 10.1002/rob.21417
  * [21] Sango Matsuzaki, Kimitoshi Yamazaki, Yoshitaka Hara and Takashi Tsubouchi  “Traversable Region Estimation for Mobile Robots in an Outdoor Image”  In _Journal of Intelligent & Robotic Systems_ 92.3-4 Journal of Intelligent & Robotic Systems, 2018, pp. 453–463  DOI: 10.1007/s10846-017-0760-x
  * [22] Daniel Maturana, Po-Wei Chou, Masashi Uenoyama and Sebastian Scherer  “Real-time Semantic Mapping for Autonomous Off-Road Navigation”  In _Field and Service Robotics_ 5, 2018, pp. 335–350  URL: https://www.ri.cmu.edu/wp-content/uploads/2017/11/semantic-mapping-offroad-nav-compressed.pdf
  * [23] Biao Gao, Shaochi Hu, Xijun Zhao and Huijing Zhao  “Fine-Grained Off-Road Semantic Segmentation and Mapping via Contrastive Learning”, 2021  arXiv: http://arxiv.org/abs/2103.03651
  * [24] Tianrui Guan, Zhenpeng He, Dinesh Manocha and Liangjun Zhang  “TTM: Terrain Traversability Mapping for Autonomous Excavator Navigation in Unstructured Environments”, 2021  arXiv: http://arxiv.org/abs/2109.06250
  * [25] Sadegh Hosseinpoor et al.  “Traversability Analysis by Semantic Terrain Segmentation for Mobile Robots”  In _IEEE International Conference on Automation Science and Engineering_ IEEE, 2021, pp. 1407–1413  DOI: 10.1109/CASE49439.2021.9551629
  * [26] Shigemichi. Matsuzaki, Hiroaki. Masuzawa, Jun. Miura and Shuji. Oishi  “3D Semantic Mapping in Greenhouses for Agricultural Mobile Robots with Robust Object Recognition Using Robots’ Trajectory”  In _IEEE International Conference on Systems, Man, and Cybernetics_ , 2018, pp. 357–362  DOI: 10.1109/SMC.2018.00070
  * [27] Dan Barnes, Will Maddern and Ingmar Posner  “Find your own way: Weakly-supervised segmentation of path proposals for urban autonomy”  In _IEEE International Conference on Robotics and Automation_ , 2017, pp. 203–210  DOI: 10.1109/ICRA.2017.7989025
  * [28] Lorenz Wellhausen et al.  “Where should i walk(Predicting terrain properties from images via self-supervised learning”  In _IEEE Robotics and Automation Letters_ 4.2, 2019, pp. 1509–1516  DOI: 10.1109/LRA.2019.2895390
  * [29] Hyunsuk Lee and Woojin Chung  “A Self-Training Approach-Based Traversability Analysis for Mobile Robots in Urban Environments”  In _IEEE International Conference on Robotics and Automation_ IEEE, 2021, pp. 3389–3394  DOI: 10.1109/ICRA48506.2021.9561394
  * [30] Dongshin Kim et al.  “Traversability Classification using Unsupervised On-line Visual Learning for Outdoor Robot Navigation”  In _IEEE International Conference on Robotics and Automation_ , 2006, pp. 518–525  URL: http://sangminoh.org/Publications_files/Kim06icra.pdf
  * [31] Dongshin Kim, Sang Min Oh and James M Rehg  “Traversability Classification for UGV Navigation: A Comparison of Patch and Superpixel Representations”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2007, pp. 3166–3173
  * [32] Lars Grimstad, Rémy Zakaria, Tuan Dung Le and Pål Johan From  “A Novel Autonomous Robot for Greenhouse Applications”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2018, pp. 8270–8277  DOI: 10.1109/IROS.2018.8594233
  * [33] The SweeperProject  “Homepage Sweeper”, 2018  URL: http://www.sweeper-robot.eu/
  * [34] Yoshisada Nagasaka et al.  “Autonomous guidance for rice transplanting using global positioning and gyroscopes”  In _Computers and Electronics in Agriculture_ 43.3, 2004, pp. 223–234  DOI: 10.1016/j.compag.2004.01.005
  * [35] Luciano Cantelli et al.  “A Small Versatile Electrical Robot for Autonomous Spraying in Agriculture”  In _AgriEngineering_ 1.3, 2019, pp. 391–402  DOI: 10.3390/agriengineering1030029
  * [36] A Mandow et al.  “The Autonomous Mobile Robot Aurora for Greenhouse Operation”  In _IEEE Robotics and Automation Magazine_ 3.4, 1996, pp. 18–28  DOI: 10.1109/100.556479
  * [37] Fuhong Dong, Wolfgang Heinemann and Roland Kasper  “Development of a row guidance system for an autonomous robot for white asparagus harvesting”  In _Computers and Electronics in Agriculture_ 79.2, 2011, pp. 216–225  DOI: 10.1016/j.compag.2011.10.002
  * [38] El Houssein Harik, Audun Korsaeth, El Houssein Chouaib Harik and Audun Korsaeth  “Combining Hector SLAM and Artificial Potential Field for Autonomous Navigation Inside a Greenhouse”  In _Robotics_ 7.2 Multidisciplinary Digital Publishing Institute, 2018, pp. 22  DOI: 10.3390/robotics7020022
  * [39] P J Younse and T F Burks  “Greenhouse Robot Navigation Using KLT Feature Tracking for Visual Odometry”  In _Agricultural Engineering International: CIGR Journal_ 9, 2007
  * [40] Jonathan Long, Evan Shelhamer and Trevor Darrell  “Fully Convolutional Networks for Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3431–3440
  * [41] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778
  * [42] Hengshuang Zhao et al.  “Pyramid scene parsing network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6230–6239  DOI: 10.1109/CVPR.2017.660
  * [43] Liang-Chieh Chen et al.  “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 40.4, 2018, pp. 834–848  DOI: 10.1109/TPAMI.2017.2699184
  * [44] Vijay Badrinarayanan, Alex Kendall and Roberto Cipolla  “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39.12, 2017, pp. 2481–2495  DOI: 10.1109/TPAMI.2016.2644615
  * [45] Olaf Ronneberger, Philipp Fischer and Thomas Brox  “U-Net: Convolutional Networks for Biomedical Image Segmentation”  In _Medical Image Computing and Computer-Assisted Intervention_ 9351, 2015, pp. 234–241  DOI: 10.1007/978-3-319-42999-1˙15
  * [46] Adam Paszke, Abhishek Chaurasia, Sangpil Kim and Eugenio Culurciello  “ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation”  In _arxiv_ , 2016  arXiv: https://arxiv.org/pdf/1606.02147.pdf
  * [47] Eduardo Romera, Jose M. Alvarez, Luis M. Bergasa and Roberto Arroyo  “ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation”  In _IEEE Transactions on Intelligent Transportation Systems_ 19.1, 2018, pp. 263–272  DOI: 10.1109/TITS.2017.2750080
  * [48] Mark Sandler et al.  “MobileNetV2: Inverted Residuals and Linear Bottlenecks”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4510–4520  DOI: 10.1109/CVPR.2018.00474
  * [49] Sachin Mehta et al.  “ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation”  In _European Conference on Computer Vision_ , 2018, pp. 561–580  DOI: 10.1007/978-3-030-01249-6˙34
  * [50] Sachin Mehta, Mohammad Rastegari, Linda Shapiro and Hannaneh Hajishirzi  “ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9190–9200  URL: https://github.com/sacmehta/
  * [51] Yuewei Yang, Kevin J. Liang and Lawrence Carin  “Object detection as a positive-unlabeled problem”, 2020  arXiv: https://arxiv.org/pdf/2002.04672.pdf
  * [52] Mathieu Labbé and François Michaud  “RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation”  In _Journal of Field Robotics_ 36.2, 2018, pp. 416–446  DOI: 10.1002/rob.21831
  * [53] Zhedong Zheng and Yi Yang  “Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation”  In _International Journal of Computer Vision_ 129, 2020, pp. 1106–1120  DOI: 10.1007/s11263-020-01395-y
  * [54] Gabriel J. Brostow, Julien Fauqueur and Roberto Cipolla  “Semantic object classes in video: A high-definition ground truth database”  In _Pattern Recognition Letters_ 30.2, 2009, pp. 88–97  DOI: 10.1016/j.patrec.2008.04.005
  * [55] Marius Cordts et al.  “The Cityscapes Dataset for Semantic Urban Scene Understanding”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3213–3223  DOI: 10.1109/CVPR.2016.350
  * [56] Abhinav Valada, Gabriel L. Oliveira, Thomas Brox and Wolfram Burgard  “Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion”  In _International Symposium for Experimental Robotics_ , 2017, pp. 465–477  DOI: 10.1007/978-3-319-50115-4˙41
  * [57] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Conference on Neural Information Processing Systems_ , 2019, pp. 8024–8035
  * [58] Leslie N. Smith  “Cyclical learning rates for training neural networks”  In _IEEE Winter Conference on Applications of Computer Vision,_ , 2017, pp. 464–472  DOI: 10.1109/WACV.2017.58
  * [59] Morgan Quigley et al.  “ROS: an open-source Robot Operating System”  In _IEEE International Conference on Robotics and Automation Workshop on Open Source Robotics_ , 2009  URL: http://stair.stanford.edu
  * [60] ROS.org  “move_base”  URL: http://wiki.ros.org/move_base

|  Shigemichi Matsuzaki (Graduate Student Member, IEEE) received the associate
degree in engineering from National College of Technology, Kumamoto College in
2016, and the bachelor’s and the master’s degrees in engineering from
Toyohashi University of Technology, Aichi, Japan in 2018 and 2020,
respectively. He is currently pursuing the Ph.D. degree at Toyohashi
University of Technology. His research interests include deep learning in
robot perception, mobile robots in unstructured environments, and mobile
service robots.
---|---
|  Hiroaki Masuzawa received the associate degree in engineering from National
College of Technology, Hachinohe College in 2006, received the bachelor’s and
the master’s degree in engineering from Toyohashi University, Aichi, Japan in
2008 and 2010, respectively. He is currently a project research associate of
Department of Computer Science and Engineering, Toyohashi University of
Technology. His research interests include application of artificial
intelligence to agricultural robots.
---|---
|  Jun Miura (Member, IEEE) received the B.Eng. degree in mechanical
engineering, and the M.Eng. and Dr.Eng. degrees in information engineering
from The University of Tokyo, Tokyo, Japan, in 1984, 1986, and 1989,
respectively. In 1989, he joined the Department of Computer-Controlled
Mechanical Systems, Osaka University, Suita, Japan. Since April 2007, he has
been a Professor with the Department of Computer Science and Engineering,
Toyohashi University of Technology, Toyohashi, Japan. From March 1994 to
February 1995, he was a Visiting Scientist with the Computer Science
Department, Carnegie Mellon University, Pittsburgh, PA. He has published over
220 articles in international journal and conferences in the areas of
intelligent robotics, mobile service robots, robot vision, and artificial
intelligence. He received several awards, including the Best Paper Award from
the Robotics Society of Japan, in 1997, the Best Paper Award Finalist at
ICRA-1995, and the Best Service Robotics Paper Award Finalist at ICRA-2013
---|---