# Robotic Irrigation Water Management: Estimating Soil Moisture Content by
Feel and Appearance

Marsela Polic, Marko Car, Jelena Tabak and Matko Orsag Authors are with
Faculty of Electrical Engineering and Computing, University of Zagreb, Unska
3, 10000 Zagreb, Croatia marsela.polic, marko.car, jelena.tabak, matko.orsag
@fer.hr

###### Abstract

In this paper we propose a robotic system for Irrigation Water Management
(IWM) in a structured robotic greenhouse environment. A commercially available
robotic manipulator is equipped with an RGB-D camera and a soil moisture
sensor. The two are used to automate the procedure known as ”feel and
appearance method”, which is a way of monitoring soil moisture to determine
when to irrigate and how much water to apply. We develop a compliant force
control framework that enables the robot to insert the soil moisture sensor in
the sensitive plant root zone of the soil, without harming the plant. RGB-D
camera is used to roughly estimate the soil surface, in order to plan the soil
sampling approach. Used together with the developed adaptive force control
algorithm, the camera enables the robot to sample the soil without knowing the
exact soil stiffness a priori. Finally, we postulate a deep learning based
approach to utilize the camera to visually assess the soil health and moisture
content.

##  I Introduction

The recent rise of robots in agriculture includes various application, ranging
from picking [1], pruning [2], pollination [3] etc. However, taking care of
the plant includes maintaining the health of the soil in which it grows. The
_feel and appearance method_ is actually a well defined and proscribed
procedure farmers use to schedule irrigation of their crops [4]. It is a way
of monitoring soil moisture to determine when to irrigate and how much water
to apply. It is a common knowledge that plants need water to live and grow.
However, applying too much water wastes this precious resource, and causes the
loss of nutrients available for the plant. The feel and appearance of soil
vary with texture and moisture content. Experienced farmers can estimate soil
moisture conditions, to an accuracy of about 5 percent. Even though it is best
to vary the number of sample sites and depths according to crop, field size,
soil texture, and soil stratification, unfortunately to save time and effort
the soil is typically sampled at three or more sites per field. For each
sample the ”feel and appearance method” involves various steps of tactile and
visual inspection, comparing observations with photographs and/or charts to
estimate percentage of the available water.

One of the goals within the SpECULARIA project [5] is to automate procedures
like this, reducing human labor input in small indoor farms by replacing it
with a heterogeneous team of robots. This team of robots is used in structured
greenhouse cultivation, where plants are grown in container units so that they
can be transported around the greenhouse by an unmanned ground vehicle (UGV).
The UGV transports the plants to the workstation, where a robotic manipulator
treats the plants under controlled conditions. The structure of the controlled
workspace around the robot gives it an upper hand when compared to mobile
robots that manipulate plants in various conditions all over the farm.

Figure 1: Collaborative robot Franka Panda performing soil moisture
measurement procedure on a sweet pepper (Capsicum anuum) plant with a smart
IoT moisture sensor.

This paper focuses on the adaptive compliant control algorithm that enables
the robot to sample the soil around the sensitive parts of the plant, and
measure the water content close to the root of the plant. The method allows
manipulation of objects of variable and unknown stiffness, ranging from
manipulation of soft, wet ground, to handling collisions with rigid object
such as roots or stones. The collaborative manipulator, which is the focus of
this paper, is equipped with an RGB-D camera and a soil moisture sensor. The
camera is used to estimate the position of the soil in the pot. Even though a
rough estimate of the soil surface can be known a priori, the exact position
varies during the entire vegetation process, as well as across the plant
containers. Furthermore, the compliant control method additionally provides an
estimate of the equivalent stiffness of the soil, thus providing another
characteristic of the considered soil that can be used for describing the
conditions. Combining different modalities will ultimately enable us to derive
an AI based Expert system capable of soil moisture condition estimation
utilized to plan optimal irrigation strategies for the greenhouse.

##  II Related work

The work presented here relies on a widespread IoT solution for soil moisture
measurement. The Soil Moisture Sensor utilizes a simple breakout for a
straightforward method to measure water content in the soil [6]. The two
exposed pads function as probes acting as a variable resistor. The amount of
water in the soil is reflected in the electrical conductivity between the
pads, and is observed as a lower effective electrical resistance. The standard
farming approach is to place the sensors and keep them in the ground for
continuous measurements. Unfortunately, the exposed pads are quick to corrode,
causing inconsistent measurements and harming the plants. Therefore, this
sensor solution is recently being replaced with a more expensive versions of
the soil moisture sensors. However, in the envisioned scenario we aim to
utilize the existing resistance-based sensors, for two main reasons. First,
such an end effector adapter is cost-effective, and the deployment with the
robotic arm as opposed to fixed long-term measurements reduces the corrosion
issue. The second motivation to use this type of sensor is in its physical
resemblance to the U-fork, patented in France 1963, also known as the
grelinette [7]. This tool is intended for soil aeration and drainage performed
by digging the dirt around the plant to gently loosen it.

Inspired by the pioneering work of Hogan [8], impedance control of robotic
manipulators has been extensively researched over the past decades, resulting
in a development of a whole range of different impedance control strategies
[9]. Impedance controllers have been applied for a wide variety of tasks,
including, but not limited to, robotic rehabilitation, industrial
manipulation, micro-manipulation and agricultural grasping and picking [10].
It has been shown that both accurate force tracking and soft grasping can be
achieved using impedance controller [11, 12], which has been successfully
tested in agricultural applications as well [13, 14]. In [13], authors
presented an impedance control strategy for compliant fruit and vegetable
grasping, while an impedance-based compliant plant exploration framework has
been presented in [14]. In this work, the classical impedance control is
extended with an adaptive control law that enables the robot to sample the
soil without a priori knowing its exact stiffness.

To enable the adaptation law to estimate the stiffness of the ground, however,
prior to placing the sensor in the ground, it is necessary to estimate the
exact position of the soil surface inside the pot. Common approaches to the
plane detection problem include computing surface normals and using standard
Random Sample Consensus (RANSAC) algorithm [15], or its improved, noise-
resistant version [16], on the point cloud obtained with an RGB-D camera.
Alternative approaches are based on region growing methods [17] and Hugh
transform [18], or a combination of both [19]. The approach proposed in this
work uses the high performing RANSAC algorithm and addresses the problem of
spurious planes by fitting the plane model only on the subset of the point
cloud extracted with the custom ground plane detection method. This custom
ground plane detection uses the existing information about the structured
conditions in which the plants are grown.

##  III Ground plane detection

As the plant is watered while it grows over a period of time, the surface of
the ground surrounding the plant tends to gradually drop. This adds to the
uncertainty of the motion control parameters. Even though a rough estimate of
the surface of the soil can be assumed from the structured nature of the
greenhouse and the plant containers, knowing the precise position of the
surface allows for commanding the compliant control motion with a
predetermined force, and using the adaptive control strategy to estimate the
stiffness of the ground. The ground plane was detected using information from
a consumer RGB-D camera, Intel RealSense D435, which was mounted on the
manipulator in an eye-in-hand configuration. The camera calibration procedure
was conducted in an autonomous manner, as described in [20], yielding the
transformation between the camera ($L_{c}$) and the flange ($L_{f}$) where the
camera is mounted $\textbf{T}_{f}^{c}\in\mathbb{R}^{4\times 4}$.

The obtained camera-flange transformation, along with the known robot
kinematics $\textbf{T}_{0}^{f}(\textbf{q})$ , is used to transform the point
cloud measurments $\boldsymbol{\pi}_{i}$ from the local camera frame to the
global reference frame
$\textbf{p}_{i}=\textbf{T}_{0}^{f}(\textbf{q})\cdot\textbf{T}_{f}^{c}\cdot\boldsymbol{\pi}_{i}$.
Here we used $L_{0}$ to denote the base frame, and
$\textbf{q}\in\boldsymbol{R}^{n}$ to denote the n joints of the robot. For
clarity we assume mapping between homogenoeus and Cartesian space is done
implicitly when using vectors. Once the point cloud is transformed, NaN values
and points which are out of the robot reach are filtered. As the robot is
operating within the structured environment, the z coordinate of the table top
position $z_{TT}$, on which the pot is placed, as well as the height of the
pot $z_{pot}$, is known a priori, enabling us to filter points
$\textbf{p}_{i}\cdot\hat{\textbf{z}}_{0}<z_{TT}$ and
$\textbf{p}_{i}\cdot\hat{\textbf{z}}_{0}>z_{TT}+z_{pot}+\epsilon$.
$\hat{\textbf{z}}_{0}$ denotes the z axis of the base frame $L_{0}$. Finally,
the remaining set of points $\mathbb{E}_{0}$ in the reachable environment of
the robot is defined as:

$\begin{gathered}\mathbb{E}_{0}=\\{\textbf{p}\in\\{\mathbb{PC}\\}|x_{min}<x_{p}<x_{max},\\\
y_{p}<y_{max},\\\ z_{TT}<z_{p}<z_{TT}+z_{pot}+\epsilon\\},\end{gathered}$ (1)

where the point p from the point cloud $\mathbb{PC}$ belongs to the set of
points $\mathbb{E}_{0}$ if it is within the robot reach and if its $z$
coordinate is bigger than the $z_{TT}$ and smaller than the sum of $z_{TT}$
and $z_{pot}$, increased by the small positive value $\epsilon$. The set of
points $\mathbb{E}_{0}$ is sorted based on the ascending z value.

The points from the set $\mathbb{E}_{0}$ are split into bins based on their z
coordinates with the initial $\Delta z$ being set to $7cm$. Score for each bin
is calculated according to the set of equations 2.

$\begin{gathered}s1=\dfrac{||B_{c-1}|-|B_{c}||}{|B_{c-1}|+|B_{c}|+1}\cdot|B_{c}|\\\
s2=\dfrac{||B_{c}|-|B_{c+1}||}{|B_{c}|+|B_{c+1}|+1}\cdot|B_{c}|\\\
s=\dfrac{s1+s2}{2},\end{gathered}$ (2)

where $|B_{c}|$ denotes the number of points in the current bin for which the
score s is being calculated. The calculated score represents the number of
points in the current bin, scaled with the relative difference between the
number of points in the current bin and the number of points in the
neighbouring bins. It is expected that the bin with the highest score contains
the points which belong to the ground plane. After selecting the highest
scoring bin, the set of points $\mathbb{E}_{k}$ is updated as in Eq. 3:

$\mathbb{E}_{k}=\\{\textbf{p}\in\\{\mathbb{E}_{k-1}\\}|(z_{b_{min}}-\dfrac{\Delta
z}{2})<z_{p}<(z_{b_{max}}+\dfrac{\Delta z}{2})\\},$ (3)

where $z_{b_{min}}$ and $z_{b_{max}}$ stand for the minimum and the maximum
value of the $z$ coordinates of the points in the highest scoring bin,
respectively. After each iteration, the value of $\Delta z$ is reduced by 25%.
The algorithm is terminated once the value of $\Delta z$ decreases to $1cm$.
The visualization of the bins throughout the single experiment is shown in
Fig. 2.

Figure 2: The ground plane is segmented by iteratively dividing the subset of
points into bins, shown in different colors, and selecting the highest scoring
bin (denoted with red rectangle).

Plane model is fitted on the remaining $\mathbb{E}_{k}$ set of points using
open sourced Point Cloud Library [21] implementation of the RANSAC algorithm
[22]. The center of the ground plane $\textbf{g}_{c}$ is defined as the median
value across all three coordinates of the plane inliers and the point closest
to the robot $\textbf{g}_{min}$ is obtained by replacing the $y$ value of the
$\textbf{g}_{c}$ with the minimum value of the $y$ coordinates of the plane
inliers.

Prior to conducting an experimental validation, the proposed method is
validated in the simulation environment. Custom pepper plant models were
generated for simulation validation, along with the plastic growth container
and the soil surface inside it. The soil surface was modelled with variation
along the $z$ axis for a more realistic morphology. In the simulation
environment, RGB and depth images of pepper plants are generated using 3D
modeling software, Blender [23]. The generated images served as an input to
the custom ROS package, blender_rgbd_ros [24], which converts them to the
point cloud data and publishes both images and point clouds on separate ROS
topics. Camera intrinsic parameters used for image generation correspond to
the parameters of the camera used in the real experiments, Intel RealSense
D435.

Figure 3: Blender setup for validation of the soil surface detection method.
The simulated RGB-D camera records a realistic pepper plant model grown in a
container. Robot approach vector $y$ is shown along with the ground truth
point closest to the robot ($\textbf{g}_{min}$) and ground truth center
($\textbf{g}_{c}$).

Blender setup is shown in Fig. 3. As in the real environment, $y$ axis of the
reference coordinate system points towards the plant container. In the
simulation environment, the exact position of $\textbf{g}_{c}$ and
$\textbf{g}_{min}$ can easily be extracted, enabling the accurate comparison
of the estimates and the ground truth. Both the mean value and the standard
deviation of the soil surface estimates for a single plant recorded from 10
random positions, along with the ground truth value, are shown in Table I. The
error and the dissipation of the $x_{median}$, $y_{median}$ and $y_{min}$ are
at a millimeter level, meaning that, even in the worst case scenario,
estimated ground center lies within the close environment of the ground truth
center, as can be seen in Fig. 4. The dissipation of the z estimate,
$z_{median}$, is even smaller, which was expected, considering that the plane
model was fitted on the modeled surface plane points, which differed in the
value of their z coordinate for up to $1cm$. The error and the dissipation of
the z estimate are sufficiently small for the successful implementation of the
adaptive compliant control strategy. Knowing the central position of the soil
with respect to the plant enables the robot to sample at the safe distance
from plant’s sensitive roots.

TABLE I: Ground truth and soil surface estimates in the simulation environment for the single plant recorded from 10 random positions. Diameter of the top of the pot is $20cm$. | ground truth | mean | std
---|---|---|---
$x_{median}$ [mm] | 1.8 | 3.4 | 6.8
$y_{median}$ [mm] | 0.8 | 4.0 | 5.0
$y_{min}$ [mm] | -100 | -96.2 | 3.8
$z_{median}$ [mm] | 110.2 | 111.2 | 0.1
Figure 4: Visualization of the ground center estimates of the soil surface
modeled in Blender. Red sphere denotes the ground truth center and the blue
spheres denote the estimates for each of the 10 experiment repetitions. The
radius of the modeled pot is $10cm$ and the maximum distance between the
ground truth and the estimate is $2cm$. Texture of the soil surface is semi-
transparent for the better visualization.

##  IV Adaptive compliant control

In this work, interaction of the end-effector with the environment relies on
impedance control in the Cartesian space. The impedance filter models the
robot-environment interaction system with an equivalent spring. The force
tracking error between the desired contact force $\textbf{F}_{r}$ and the
measured force F as in eq. 4, drives robot motion according to the desired
mass-spring-damper system properties.

$\textbf{E}=\textbf{F}_{r}-\textbf{F}.$ (4)

Here we consider force control along three spatial axes, so that all the
vectors used are from $R^{3\times 1}$. The user-defined target impedance
behavior of the system determines the dynamic relationship between the robot
position and the force tracking error so that it mimics a mass-spring-damper
system as in eq. 5,

$\textbf{E}=\textbf{M}(\ddot{\textbf{X}}_{c}-\ddot{\textbf{X}}_{r})+\textbf{B}(\dot{\textbf{X}}_{c}-\dot{\textbf{X}}_{r})+\textbf{K}(\textbf{X}_{c}-\textbf{X}_{r}),$
(5)

where $\textbf{X}_{c}$ and $\textbf{X}_{r}$ are $R^{3\times 1}$ commanded and
reference position vectors of the end-effector, and M, B and K are the
$R^{3\times n}$ mass, damping and stiffness matrices of the target impedance,
respectively. The reference position is the one provided by the user or by a
higher level control. The commanded position is the actual input reference for
the robot Cartesian position control. When in contact with the environment,
the commanded robot position $\textbf{X}_{c}$ changes with the measured
contact force through eq. 5, and the position tracking is in general not
accurate. In other words, the impedance filter balances the position and force
tracking errors. Switching to analysis along a single spatial axis without
loss of generality, it can be shown [25] that the desired contact forces can
only be realized in case both precise environment position and the environment
equivalent stiffness are known, by generating an adequate position reference
$x_{r}$ using eq. 6

$x_{r}=\frac{F_{r}}{k_{e}}+x_{e},$ (6)

where $F_{r}$ is the desired contact force, and $k_{e}$ and $x_{e}$ are the
environment stiffness and position, respectively. However, in the case
presented in this work, the soil stiffness is unknown a-priori, and varies
with the soil moisture. The controller used in this work is an extension of
the classic position based impedance controller developed in
[contact2021markovic], with online adaptation of the impedance filter inputs
based on the estimated environment stiffness. The adaptation law for the
position reference is based on the adaptive parameter $\kappa(t)$ that
accounts for unknown elastic properties of environment under external force,

$x_{r}(t)=\kappa(t)F_{r}+x_{e},$ (7)

where the position reference is a function of the estimate of the initial
position of the environment $x_{e}$ and the force reference $F_{r}$. The exact
adaptation law is given with eq. 8 for one spatial dimension,

$\displaystyle k\dot{\kappa}(t)+b\ddot{\kappa}(t)+m\dddot{\kappa}(t)$
$\displaystyle=-\gamma_{1}q(t)+\gamma_{1}^{*}\dot{q}(t),$ $\displaystyle q(t)$
$\displaystyle=p_{1}e(t)+p_{2}\dot{e}(t)$ (8)

where $k$, $b$, and $m$ are the impedance filter parameters, $e(t)$ is the
force tracking error, and $p_{1}$ and $p_{2}$ are the free parameters tuned
based on the particular application. The derivation and convergence proof for
this adaptation law can be found in [26]. The adaptation law 7 used for
ensuring force reference tracking implicitly yields a stiffness estimate of
the manipulated object. In this case, the soil stiffness measure is used both
for compliant manipulation in moisture measurement, and as a feature in the
proposed soil moisture monitoring framework.

##  V Experimental Validation

We conducted a series of experiments to verify how well the system estimates
the position and the stiffness of the soil. In practice, the information
gathered in such a way can help the system estimate the health of the soil and
manage its water content.

###  V-A Ground plane detection precision

In the same manner as in the simulation environment, a pepper plant was
recorded from 10 random positions in the laboratory conditions. The center of
the ground plane was estimated for each of the 10 frames as the median value
of the fitted plane inliers across all three dimensions. The maximum
difference and the dissipation of the estimates are shown in Table II. While
the estimated value of z coordinate of the ground center remained sufficiently
small and robust to the change in the camera recording angle, the estimated x
and y values significantly deteriorated compared to the simulation
environment. This can be explained by the imperfections of the depth module of
the RealSense camera. An example of an inaccurate point cloud of pot is shown
in Fig 5. This problem was addressed by extracting the closest point along the
y coordinate on the detected plane with respect to the robot base, instead of
the median value. The final y coordinate of the reference position is defined
as $3cm$ further along the $y$ coordinate, in order to reach the soil instead
of the pot edge. As visible from the Table II, the dissipation is smaller when
working with the $y_{min}$ instead of $y_{median}$. This was not the case in
the simulation environment, as the camera model simulated in Blender did not
incorporate the fluctuations in the depth data. However, considering that the
diameter of the top of the pot is equal to $20cm$, dissipation in both x and y
values, though bigger than in the simulation environment, is still negligible.

TABLE II: Dissipation of the estimates in the real environment for the single plant recorded from 10 random positions. Diameter of the top of the pot is $20cm$. | $x_{median}$ | $y_{median}$ | $y_{min}$ | $z_{median}$
---|---|---|---|---
$\Delta_{max}$ [mm] | 26.3 | 46.6 | 26.1 | 5.0
std [mm] | 8.7 | 13.6 | 7.3 | 1.5
Figure 5: Erroneous point cloud of pot recorded with Intel RealSense D435.

###  V-B Control

(a)

(b)

(c)

Figure 6: Force tracking of the robot end effector in experiments on the
softest object, moist soil 6a, on the stiffer object, namely dry soil 6b, and
in case of collision with a rigid object 6c. The robot motion is safe for the
manipulated objects regardless of their stiffness.

Soil sampling experiments were conducted with a Franka Panda collaborative
robot arm, based on the described soil surface detection using impedance
control methods. The experiments were conducted in three scenarios analyzing
the framework behavior with respect to different soil conditions. The moist
soil conditions represent the softest scenario, i.e. the least stiff
environment if considering the estimated stiffness values. The dry soil
conditions represent a stiff scenario, while the extreme was tested through
collision with a very rigid object, representing e.g. a stone in the soil that
could potentially break the sensory equipment, or a part of the root system
that should not be harmed. Five experimental repetitions were conducted for
the first two scenarios, and three repetitions for the collision scenario. The
results are represented in the graphs in figures 6b-7c.

The force responses in the figures 6b-6c are obtained with the same impedance
filter, and the same parameters of the adaptation controller. The results show
that the adaption in the framework is capable of reaching the desired contact
force setpoint regardless of the stiffness of the manipulated object. When
considering the responses, it should be noted that instead of a precise
external force/torque sensor, the measurement is provided by the Franka Panda
dynamics estimation model. The imprecision in the model can be observed
particularly in the variable baseline offset in the measurements at the
beginning and at the end of each experiment. Here, the robot is not in contact
with the environment, and there are no external forces acting on the end-
effector. However, the estimated forces are non-zero due to model imprecision,
and vary depending on the robot pose and velocity.

(a)

(b)

(c)

Figure 7: Adaptation dynamics during experiments with moist soil 7a, dry soil
7b, and in case of collision with a rigid object 7c. The estimation also
compensates manipulator dynamics and surface detection pipeline. Adaptation
tuned for fastest convergence on stiffest objects.

The adaptation framework implicitly models the stiffness of the manipulated
object, as shown in 6b-6c. Instead of the estimated stiffness, the responses
show the dynamics of the inverse variable $1/K_{e}$, that could be considered
compliance of the manipulated object (soil). For safety reasons, the initial
assumption is that the manipulated object is infinitely stiff (zero
compliance), and the adaptation of the estimated stiffness gradually reaches
the actual value along with the desired contact forces. This is one of the
reasons behind faster adaptation for a stiffer object, the other being the
chosen adaptation parameters. These parameters would, in case of very
compliant objects, probably have to be tuned for more aggressive (faster)
adaptation. The compliance (stiffness) measure not only models the soil, but
inherently takes into account the elasticity of the robot manipulator, as well
as the imprecision of the soil surface detection. Regardless, the results show
that throughout the repetitions, the estimation converges to the same region
of values, proving the adaptation method is stable with respect to robot
dynamics and detection imprecision.

##  VI Discussion

This paper presents a soil moisture measurement method and its experimental
validation applicable for robotic plant cultivation. The detection method in
the initial step relies on an RGB-D visual setup for detection of the soil
surface within the container. Instead of relying on color based segmentation,
which is sensitive with respect to the lighting and other external conditions,
or on other complex segmentation techniques such as CNNs, the method relies on
an iterative 3D pointcloud segmentation and RANSAC based plane fitting. The
method validation in simulation was confirmed in experiments with real plants,
showing that the method is suitable for detection of soil plane inside the
growth container.

The actual measurement method relies on compliant control framework for a
collaborative robot, that can easily be applied to industrial manipulator
equipped with an external force torque sensor as well. The method enables
compliant manipulator control during interaction with a deformable fragile
objects, without prior information on the mechanical properties. Moreover, the
method implicitly models them during motion control in attempt to reach the
desired contact force setpoint. However, the obtained compliance measure not
only models the soil but other system components as well, such as manipulator
dynamics, and the error in soil surface estimation, which explains the
variation in the estimations obtained over experiment repetitions. Even though
not converging to precisely the same value over several experiment
repetitions, the estimates of the compliance measures for three tested
scenarios still distinguish between various manipulation conditions, and imply
that such a measure could be used as a part of the feel and appearance method.
Most importantly, the method is shown suitable and safe both for the robot,
and for the underground plant parts, on a variety of soil conditions ranging
from the soft moist soil to stiff soils with rigid debris in an autonomous
manner without additional pre-tuning.

In the final setup, the visual part of the ”feel and appearance method” will
be achieved using the equipped RGB-D camera. Combining all three modalities
(i.e. vision, measured resistance and stiffness) will enable us to derive an
AI based Expert system capable of estimating soil moisture conditions to plan
optimal irrigation strategies for the greenhouse.

## ACKNOWLEDGMENT

This work has been supported by Croatian Science Foundation under the project
Specularia UIP-2017-05-4042 [27].

## References

  * [1] Y. Xiong, Y. Ge, L. Grimstad, and P. J. From, “An autonomous strawberry-harvesting robot: Design, development, integration, and field evaluation,” Journal of Field Robotics, vol. 37, no. 2, pp. 202–224, 2020\.
  * [2] Q. Zhang, M. Karkee, and A. Tabb, “The use of agricultural robots in orchard management,” in Robotics and automation for improving agriculture (J. Billingsley, ed.), Burleigh Dodds Science Publishing, 2019.
  * [3] J. Strader, C. Yang, Y. Gu, J. Nguyen, C. Tatsch, Y. Du, K. Lassak, B. Buzzo, R. Watson, H. Cerbone, and N. Ohi, “Flower interaction subsystem for a precision pollination robot,” pp. 5534–5541, 11 2019.
  * [4] N. L. Klocke and P. E. Fischbach, “G84-690 estimating soil moisture by appearance and feel,” 1984.
  * [5] M. Polic, A. Ivanovic, B. Maric, B. Arbanas, J. Tabak, and M. Orsag, “Structured ecological cultivation with autonomous robots in indoor agriculture,” in 2021 16th International Conference on Telecommunications (ConTEL), pp. 189–195, 2021.
  * [6] M. Saleh, I. H. Elhajj, D. Asmar, I. Bashour, and S. Kidess, “Experimental evaluation of low-cost resistive soil moisture sensors,” in 2016 IEEE International Multidisciplinary Conference on Engineering Technology (IMCET), pp. 179–184, IEEE, 2016.
  * [7] K. Farhat, “Digging fork device for at least one digging fork,” WO/2014/154983.
  * [8] N. Hogan, “Impedance control: An approach to manipulation,” in 1984 American Control Conference, pp. 304–313, 1984.
  * [9] H. Al-Shuka, S. Leonhardt, W.-H. Zhu, R. Song, C. Ding, and Y. Li, “Active impedance control of bioinspired motion robotic manipulators: An overview,” Applied Bionics and Biomechanics, 06 2018.
  * [10] P. Song, Y. Yu, and X. Zhang, “A tutorial survey and comparison of impedance control on robotic manipulation,” Robotica, vol. 37, pp. 1–36, 01 2019\.
  * [11] Z. Ting, L. Jiang, S. Fan, X. Wu, and W. Feng, “Development and experimental evaluation of multi-fingered robot hand with adaptive impedance control for unknown environment grasping,” Robotica, vol. -1, pp. 1–18, 08 2014.
  * [12] Y. Sano, R. Hori, and T. Yabuta, “Comparison between admittance and impedance control method of a finger-arm robot during grasping object with internal and external impedance control,” Transactions of the Japan Society of Machanical Engineers Series C, vol. 79, pp. 4330–4334, 11 2013.
  * [13] X. Wang, Y. Xiao, S. Bi, X. Fan, and H. Rao, “Design of test platform for robot flexible grasping and grasping force tracking impedance control,” Nongye Gongcheng Xuebao/Transactions of the Chinese Society of Agricultural Engineering, vol. 31, pp. 58–63, 01 2015.
  * [14] M. Polic, M. Car, F. Petric, and M. Orsag, “Compliant plant exploration for agricultural procedures with a collaborative robot,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2768–2774, 2021.
  * [15] D. Holz, S. Holzer, R. B. Rusu, and S. Behnke, “Real-time plane segmentation using rgb-d cameras,” in RoboCup 2011: Robot Soccer World Cup XV (T. Röfer, N. M. Mayer, J. Savage, and U. Saranlı, eds.), (Berlin, Heidelberg), pp. 306–317, Springer Berlin Heidelberg, 2012.
  * [16] L. Li, F. Yang, H. Zhu, D. Li, Y. Li, and L. Tang, “An improved ransac for 3d point cloud plane segmentation based on normal distribution transformation cells,” Remote Sensing, vol. 9, no. 5, 2017.
  * [17] A.-V. Vo, L. Truong-Hong, D. F. Laefer, and M. Bertolotto, “Octree-based region growing for point cloud segmentation,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 104, pp. 88–100, 2015.
  * [18] R. Hulik, M. Spanel, P. Smrz, and Z. Materna, “Continuous plane detection in point-cloud data based on 3d hough transform,” Journal of Visual Communication and Image Representation, vol. 25, no. 1, pp. 86–97, 2014.  Visual Understanding and Applications with RGB-D Cameras.
  * [19] X. Leng, J. Xiao, and Y. Wang, “A multi-scale plane-detection method based on the hough transform and region growing,” The Photogrammetric Record, vol. 31, pp. 166–192, 06 2016.
  * [20] B. Maric, M. Polic, T. Tabak, and M. Orsag, “Unsupervised optimization approach to in situ calibration of collaborative human-robot interaction tools,” in 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pp. 255–262, IEEE, 2020.
  * [21] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in IEEE International Conference on Robotics and Automation (ICRA), (Shanghai, China), IEEE, May 9-13 2011.
  * [22] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, vol. 24, pp. 381–395, 1981.
  * [23] R. Hess, Blender Foundations: The Essential Guide to Learning Blender 2.6.  Focal Press, 2010.
  * [24] A. Ivanovic, “blender_rgbd_ros.” https://github.com/larics/blender_rgbd_ros, 2021.  Accessed: 2021-10-10.
  * [25] H. Seraji and R. Colbaugh, “Force tracking in impedance control,” in [1993] Proceedings IEEE International Conference on Robotics and Automation, pp. 499–506, IEEE Comput. Soc. Press.
  * [26] L. Marković, M. Car, M. Orsag, and S. Bogdan, “Adaptive stiffness estimation impedance control for achieving sustained contact in aerial manipulation,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 117–123, IEEE, 2021.
  * [27] M. Orsag et al., “Specularia.” https://sites.google.com/view/specularia-pepper-picking, 2021.  Accessed: 2021-03-31.


11institutetext: Nanjing University of Aeronautics and Astronautics, Jiangsu
Nanjing, China 22institutetext: School of Computing and Mathematical Sciences,
University of Leicester, U.K 22email<EMAIL_ADDRESS>

# A Task-aware Dual Similarity Network for Fine-grained Few-shot Learning

Yan Qi 11 Han Sun(✉) 11 Ningzhong Liu 11 Huiyu Zhou 22

###### Abstract

The goal of fine-grained few-shot learning is to recognize sub-categories
under the same super-category by learning few labeled samples. Most of the
recent approaches adopt a single similarity measure, that is, global or local
measure alone. However, for fine-grained images with high intra-class variance
and low inter-class variance, exploring global invariant features and
discriminative local details is quite essential. In this paper, we propose a
Task-aware Dual Similarity Network(TDSNet), which applies global features and
local patches to achieve better performance. Specifically, a local feature
enhancement module is adopted to activate the features with strong
discriminability. Besides, task-aware attention exploits the important patches
among the entire task. Finally, both the class prototypes obtained by global
features and discriminative local patches are employed for prediction.
Extensive experiments on three fine-grained datasets demonstrate that the
proposed TDSNet achieves competitive performance by comparing with other
state-of-the-art algorithms.

###### Keywords:

Fine-grained image classification Few-shot learning Feature enhancement

##  1 Introduction

As one of the most important problems in the field of artificial intelligence,
fine-grained image classification [8, 6] aims to identify objects of sub-
categories under the same super-category. Different from the traditional image
classification task [21, 22], the images of sub-categories are similar to each
other, which makes fine-grained recognition still a popular and challenging
topic in computer vision.

Benefiting from the development of Convolution Neural Networks (CNNs), fine-
grained image classification has made significant progress. Most approaches
typically rely on supervision from a large number of labeled samples. In
contrast, humans can identify new classes with only few labeled examples.
Recently, some studies [31, 25] focus on a more challenging setting, which
aims to recognize fine-grained images from few samples, and is called fine-
grained few-shot learning(FG-FSL). Learning from fine-grained images with few
samples brings two challenges. On the one hand, images in the same category
are quite different due to poses, illumination conditions, backgrounds, etc.
So how to capture invariant features in limited samples is a particularly
critical problem. On the other hand, it is complicated to distinguish subtle
visual appearance clues on account of the small differences between
categories. Therefore, we consider that the invariant global structure and the
discriminative local details of objects are both crucial for fine-grained few-
shot classification.

To effectively learn latent patterns from few labeled images, many approaches
[33, 7] have been proposed in recent years. These methods can be roughly
divided into two branches: the meta-learning methods and the metric learning
ones. Metric learning has attracted more and more attention due to its
simplicity and effectiveness, and our work will focus on such methods.
Traditional approaches such as matching network [29] and relation network [27]
usually utilize global features for recognition. However, the distribution of
these image-level global features cannot be accurately estimated because of
the sparseness of the samples. In addition, discriminative clues may not be
detected only by relying on global features. CovaMNet [15] and DN4 [16]
introduce the deep local descriptors which are exploited to describe the
distribution with each class feature. Furthermore, although these methods
learn abundant features, they deal with each support class independently and
cannot employ the contextual information of the whole task to generate task-
specific features. In conclusion, the importance of different parts changes
with different tasks.

In this paper, we propose a Task-aware Dual Similarity Network(TDSNet) for
fine-grained few-shot learning, which makes full use of both global invariant
features and discriminative local details of images. More specifically, first,
a local feature enhancement module is employed to activate discriminative
semantic parts by matching the predicted distribution between objects and
parts. Second, in the dual similarity module, the proposed TDSNet calculates
the class prototypes as global invariant features. Especially, in the local
similarity branch, task-aware attention is adopted to select important image
patches for the current task. By considering the context of the entire support
set as a whole, the key patches in the task are selected and weighted without
paying too much attention to the unimportant parts. Finally, both global and
local similarities are employed for the final classification. We conduct
comprehensive experiments on three popular fine-grained datasets to
demonstrate the effectiveness of our proposed method. Especially, our method
can also have good performance when there is only one training image.

##  2 Related Work

####  2.0.1 Few shot learning.

Few-shot learning aims at recognizing unseen classes with only few samples.
The recently popular literature on few-shot learning can be roughly divided
into the following two categories: meta-learning based methods and metric-
learning based methods.

Meta-learning based methods attempt to learn a good optimizer to update model
parameters. MAML [9] is dedicated to learning a good parameter initialization
so that the model can adapt to the new task after training on few samples.
Ravi et al. [24] propose a meta-learner optimizer based on LSTM to optimize a
classifier while also studying an initialization for the learner that contains
task-aware knowledge.

Metric-learning based methods aim to measure the similarity by learning an
appropriate metric that quantifies the relationship between the query images
and support sets. Koch et al. [13] adopt a siamese convolutional neural
network to learn generic image representations, which is performed as a binary
classification network. Lifchitz et al. [18] directly predict classification
for each local representation and calculates the loss. DN4 [16] employs
k-nearest neighbors to construct an image-to-class search space that utilizes
deep local representations. Unlike DN4, which is most relevant to our work, we
argue that considering each support class independently may capture features
shared among classes that are unimportant for classification. In this paper,
task-aware local representations will be detected to explore richer
information.

Fine-grained image classification. Because some early approaches [1, 3]
require a lot of bounding boxes or part annotations as supervision that needs
a high cost of expert knowledge, more and more researchers are turning their
attention to weakly supervised methods [20, 23] that rely only on image-level
annotations. Inspired by different convolutional feature channels
corresponding to different types of visual modes, MC-Loss [5] proposes a
mutual-channel loss that consists of a discriminality component and a
diversity component to get the channels with locally discriminative regions
for a specific class. TDSA-Loss [4] obtains multi-regional and multi-
granularity features by constraining mid-level features with the attention
generated by high-level features. Different from these methods, we consider
that the discriminability of local features obtained only by the attention
maps may not be guaranteed. In order to overcome this limitation, the proposed
TDSNet activates the local representations with strong discriminability by
matching the distribution between the global features and their sub-features,
so that the discriminability of global features at fine-grained scales is
improved.

##  3 Method

###  3.1 Problem Definition

In this paper, the proposed TDSNet also follows the common setup of other few-
shot learning methods. Specifically, few-shot classification is usually
formalized as N-way K-shot classification problems. Let $S$ denote a support
set that contains $N$ distinct image classes, and each class contains $K$
labeled samples. Given a query set $Q$, the purpose of few-shot learning is
classifying each unlabeled sample in $Q$ according to the support set $S$.
However, limited samples in $S$ make it difficult to efficiently train a
network. Therefore, auxiliary set $A$ is always introduced to learn
transferable knowledge to improve classification performance. Note that $S$
and $A$ have their own distinct label spaces without intersections.

In order to learn transferable knowledge better, the episode training
mechanism [29] is adopted in the training phase. Specifically, at each
iteration, support set $AS$ and query set $AQ$ are randomly selected from
auxiliary set $A$ to simulate a new few-shot classification task. In the
training process, multiple episodes are constructed to train the model.

###  3.2 Overview

The overall framework of our method is shown in Fig. 1. First of all, images
are fed into the feature extractor which is usually implemented by CNN or
ResNet [11] to get image embeddings. In this stage, a LFE module is designed
to explore local details with strong discriminability by the supervision of
global features. Next, the features are used as inputs to the metric module.
Our metric module adopts dual similarity that is composed of global and local
metric branches, which can not only exploit the intra-class invariance of
global features but also explore rich clues hidden in local details.
Specially, in the local similarity branch, the discriminative patches are
reweighted to eliminate noise, i.e., patches shared in the task, and enhance
the significant regions. The proposed TDSNet focuses on the relationships
among local patches rather than isolated individuals. Finally, the mean value
of global and local classification scores is reported as the final result. LFE
module is only used during training.

Figure 1: Framework of the proposed TDSNet.

###  3.3 Local Feature Enhancement(LFE) Module

Weakly supervised attention generation. Parts of the objects are predicted
first. In this paper, we explore discriminative regions in a weakly supervised
manner. Besides, instead of using a pre-trained convolutional neural network,
we employ an attention generation strategy.

For the input image $X$, the feature $F\in R^{H\times W\times C}$ is explored
through the feature embedding module $f_{\varphi}$, where $H$, $W$, and $C$
denote the height, width, and the number of channels of the feature
respectively. Then, the attention maps $A^{a}\in R^{H\times W\times m}$ for
each image can be determined by

$A^{a}=f(F)=\bigcup^{m}_{k=1}A^{a}_{k}$ (1)

$f(\cdot)$ represents a convolution operation and $A^{a}_{k}$ is the k-th
local attention map. Similar to the bilinear pooling [10, 28], element-wise
multiplication between $A^{a}$ and $F$ is performed to produce the part
feature map $f$, which can be expressed as

$f_{k}=g(A^{a}_{k}\odot F),k=1,2,\cdot\cdot\cdot,m$ (2)

$f_{k}$ is the k-th local feature, $\odot$ denotes element-wise
multiplication, and $g(\cdot)$ is the global average pooling operation.
Finally, we stack these part maps to obtain the final part feature matrix. It
can be represented as

$M^{P}=\left(\begin{array}[]{c}f_{1}\\\ f_{2}\\\ \cdot\cdot\cdot\\\
f_{m}\end{array}\right)$ (3)

Local feature enhancement. Weakly supervised attention generation is capable
of activating some local parts of the objects, however, the discriminability
of these local parts may not be guaranteed. Therefore, we propose a feature
regularization method to constrain object representation, which extracts
knowledge from global features to local features and guides the parts with
strong discriminability to be encouraged. An effective way to achieve this
effect is to match the prediction distributions between objects and their
parts. Let $P_{g}$ and $P_{a}$ describe the predicted distributions of the
global features and part features $M^{P}$ respectively. We optimize a KL
divergence loss[19] that is applied for measuring the difference between two
probability distributions as follows,

$L_{KL(P_{g}\parallel P_{a})}=-H(P_{g})+H(P_{g},P_{a})$ (4)

where $H(P_{g})=-\sum P_{g}logP_{g}$, $H(P_{g},P_{a})=-\sum P_{g}logP_{a}$

This regularization loss forces the feature representation learning to focus
on the discriminative details from a particular local region, by which we can
further filter out unnecessary and misleading information to improve the
discriminability of global features at fine-grained scales. Only the one with
global features is used for final target prediction.

###  3.4 Dual Similarity Module

Global similarity. This branch adopts the global feature maps for
classification, which employs the cosine distance as a metric function. The
feature maps are fed into two convolution blocks($h_{conv}$) to learn generic
knowledge with global representations in the images, followed by a cosine
similarity $cos(\cdot)$ to measure the similarity. For the query image
$X_{q}$, the global similarity score corresponding to the support class
$X_{s}$ is determined as follows,

$S_{g}=cos(h_{conv}(\frac{1}{K}\sum^{K}_{s=1}f_{\varphi}(X_{s})),h_{conv}(f_{\varphi}(X_{q})))$
(5)

We take the mean value of feature mappings from each support class as the
class prototype, which is used to calculate the global structure so that the
invariant features within the class can be learned.

Figure 2: The framework of task-aware attention.

Task-aware local similarity. Some recent works, such as DN4 and ConvMNet, have
shown that features based on local descriptors are richer than global.
Specifically, local descriptors are able to capture local subtle information
which is of greater benefit for fine-grained image recognition. For an image
feature $F\in R^{H\times W\times C}$, it is regarded as a set of r(r=HW)
C-dimensions local feature descriptors, which can be expressed as

$f_{\varphi}(X)=[x_{1},...,x_{r}]\in R^{C\times r}$ (6)

where $x_{i}$ denotes the i-th depth local descriptor. These local descriptors
correspond to spatial local patches in the raw image. Basically, for each
query image $X_{q}$, we get HW local descriptors to estimate its distribution,
denoted as $L^{q}=f_{\varphi}(X_{q})\in R^{C\times HW}$. Similarly, for
support set, all local descriptors of class prototypes will be employed
together as $L^{s}=f_{\varphi}(X_{s})\in R^{C\times NHW}$. Next, we calculate
the similarity matrix $M$ between query image and support set by

$M_{i,j}=cos(L^{q}_{i},L^{s}_{j})$ (7)

where $i\in{1,...,HW}$, $j\in{1,...,NHW}$, $cos(\cdot)$ represents cosine
similarity. Each row in matrix $M$ represents the similarity of a specific
query patch to all patches of the support set.

We then construct the task-aware attention map to reweight these parts. As
shown in Fig. 2, we built another relation matrix $M^{\phi}$ for the next
operation, which is obtained by a convolution layer and a cosine similarity
layer. We consider that local descriptors shared by multiple classes in the
task do not contribute to the classification. For instance, if the query image
patch has a high level of similarity to multiple patches in the support set,
it has a minuscule contribution to classification. Therefore, we distract
attention to make local descriptors that are shared in the task get relatively
small attention values. The attention matrix $M^{A}$ is defined as

$M_{i,j}^{A}=\frac{I(M^{\phi}_{i,j})}{\sum_{j}I(M^{\phi}_{i,j})}$ (8)
$I(x)=\left\\{\begin{array}[]{ll}x,&if\ x>\beta\\\
0,&otherwise.\end{array}\right.$ (9)

$\beta$ is the threshold, which is set as the minimum of the top-k elements
obtained by k-NN [2] from the relationship matrix $M^{\phi}$ to eliminate the
noises. Since $I(\cdot)$ is indifferentiable, we approximate it by a variant
function of sigmoid with a hyperparameter t as

$I^{*}(x)=x/(1+exp^{-t(x-\beta)})$ (10)

Theoretically, when t is large enough, it can be approximated as $I(\cdot)$.
We perform element-wise multiplication between the weight matrix $M^{A}$ and
the relation matrix $M$. Finally, the local similarity score for n-th class
between the query image $X_{q}$ and the support class $X_{s}$ can be
calculated by applying the attention map to the similarity matrix $M$ as
follows:

$S_{l}=\frac{1}{HW}\sum^{HW}_{i=1}\sum^{HW}_{j=1}(M^{A}\odot M)_{i,j}$ (11)

The total classification score is formulated as follows which is used to make
a final prediction:

$S_{total}=\frac{1}{2}(S_{g}+S_{l})$ (12)

In particular, our local similarity branch only introduces a small number of
parameters and the overfitting problem in the few-shot learning can also be
alleviated to some extent.

###  3.5 Loss Function

In the training phase, the purpose is to learn a task agnostic network for
classification. We can obtain $y^{g}_{q}$ and $y^{l}_{q}$ as two predicted
results by global and local branches respectively. Then, predicted values are
compared with the ground-truth label $y_{q}$ to calculate two classification
losses.

$L^{g}_{q}=\sum^{N}_{j}(y^{g}_{q,j}-y_{q,j})^{2},q=1,...,|Q|$ (13)
$L^{l}_{q}=\sum^{N}_{j}(y^{l}_{q,j}-y_{q,j})^{2},q=1,...,|Q|$ (14)

In the end, the whole loss function can be written as:

$L_{total}=L^{g}_{q}+L^{l}_{q}+\lambda L_{KL}$ (15)

Where $\lambda$ is a trade-off parameter used to control the relative
importance of the loss $L_{KL}$. Empirically, we set $\lambda=0.4$.

##  4 Experiment

###  4.1 Datasets and Experimental Setting

We evaluate our method on three widely used fine-grained datasets, namely
CUB-200-2011 [30], Stanford Dogs [12], and Stanford Cars [14]. We conduct
experiments under the 5-way 1-shot and 5-way 5-shot settings. All images are
resized to 84 × 84 before being fed into the feature extraction module. In the
training process, episode training mechanism is used to train our model. For
the three datasets, models are trained for 600 and 400 epochs corresponding to
the 5-way 1-shot and 5-way 5-shot tasks, respectively. We use Adam optimizer
to train the network with the initial learning rate of 0.001, decaying by half
every 100,000 episodes. In the testing phase, the top-1 accuracy with 95%
confidence interval will be reported by random sampling of 600 episodes from
the test set.

Table 1: Comparison with typical FSL and FG-FSL methods on three fine-grained datasets. The best and the second best results are highlighted in red and green respectively. Dataset | CUB Birds | Stanford Dogs | Stanford Cars
---|---|---|---
Setting | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
Matching Net [29] | 45.30±1.03 | 59.50±1.01 | 35.80±0.99 | 47.50±1.03 | 34.80±0.98 | 47.50±1.03
Prototype Net [26] | 37.36±1.00 | 45.28±1.03 | 37.59±1.00 | 48.19±1.03 | 40.90±1.01 | 52.93±1.03
Relation Net [27] | 59.58±0.94 | 77.62±0.67 | 43.05±0.86 | 63.42±0.76 | 45.48±0.88 | 60.26±0.85
MAML [9] | 54.92±0.95 | 73.18±0.67 | 44.64±0.89 | 60.20±0.80 | 46.71±0.89 | 60.73±0.85
PCM [32] | 42.10±1.96 | 62.48±1.21 | 28.78±2.33 | 46.92±2.00 | 29.63±2.38 | 52.28±1.46
CovaMNet [15] | 52.42±0.76 | 63.76±0.64 | 49.10±0.76 | 63.04±0.65 | 56.65±0.86 | 71.33±0.63
DN4 [16] | 46.84±0.81 | 74.92±0.64 | 45.41±0.76 | 63.51±0.62 | 59.84±0.80 | 88.65±0.44
BSNet [17] | 65.89±1.00 | 78.48±0.65 | 51.68±0.95 | 67.93±0.75 | 54.39±0.92 | 73.37±0.77
FOT [31] | 67.46±0.68 | 83.19±0.43 | 49.32±0.74 | 68.18±0.69 | 54.55±0.73 | 73.69±0.65
ours | 69.34±0.89 | 80.34±0.59 | 54.48±0.87 | 69.45±0.69 | 62.14±0.91 | 75.64±0.72

###  4.2 Comparison with State-of-the-art Methods

To evaluate the validity of the proposed TDSNet, we conduct extensive
experiments on three classic fine-grained datasets with 5-way 1-shot and 5-way
5-shot task settings and compare them with some SOTA methods.

As is demonstrated in Table. 1, the results compared with four classical few-
shot methods and five fine-grained few-shot methods illustrate that our method
achieves good performance on all three datasets. Specifically, the proposed
TDSNet performs better on the more challenging 1-shot task and shows high
stability. The reason for this progress is that our approach focuses on the
discriminative parts and gives them higher weights. Additionally, the changes
in visual appearance may not affect our TDSNet because we also pay attention
to invariant global structure.

###  4.3 Ablation Study

Table 2: Ablation study on the proposed components on CUB. LFE: local feature enhancement module, LS: local similarity module, att: task-aware attention. Method | 1-shot | 5-shot
---|---|---
(a)Baseline | 63.33±1.01 | 77.64±0.67
(b)Baseline+LFE | 65.20±0.99 | 78.77±0.67
(c)Baseline+LFE+LS w/o att | 67.02±0.96 | 79.59±0.64
(d)Baseline+LFE+LS w/ att | 69.34±0.89 | 80.34±0.59

Effectiveness of local feature enhancement module. We use the feature
embedding module and the global similarity metric module of this paper as the
baseline. Table. 2 shows that the addition of the local enhancement module
makes the accuracy significantly improved by 1.87% and 1.13% respectively,
which is mainly due to the activation of the features with strong
discriminability. In this way, we can further filter out misleading
information so as to improve the discrimination of features.

Effectiveness of dual-similarity. Compared with the results that only employ
the global similarity measurement module, dual similarity demonstrates its
superiority. It proves that only global features are hard to detect some
detailed information that is suitable for fine-grained features, while only
local features are sensitive to some intra-class changes.

Effectiveness of task-aware attention. We verify the effectiveness of task-
aware attention on CUB, with 2.32% and 0.75% improvements respectively. Task-
aware attention makes the network pay more attention to the features that are
most relevant to the current task and reweight the key parts so that the
features shared between classes will obtain less attention.

###  4.4 Visualization

As can be seen from Fig. 3, our TDSNet has less activation in the background
and is more concentrated on the discriminative regions of objects, which
demonstrates that the features can be semantically enhanced by our approach.
It highlights the local details on raw images that represent different
semantic patches with strong discriminability, such as the head and wings of a
bird.

Figure 3:  Visualization of the features under Relation Network, DN4, BSNet
and the proposed TDSNet on CUB. The redder the region, the more discriminative
it is.

###  4.5 Number of Trainable Parameters

As Table. 3 shows, compared to the BSNet, which is also a bi-similarity method
on the global feature, we have only half the number of parameters of this
approach. This illustrates that although we adopt additional architectures to
improve the performance, the proposed TDSNet only introduces a small number of
the trainable parameters.

Table 3: Comparsion of the number of trainable parameters along on CUB. Model | Params | 5-way 5-shot
---|---|---
Prototype network | 0.113M | 45.28
Relation network | 0.229M | 77.62
DN4 | 0.113M | 74.92
BSNet(R&C) | 0.226M | 78.84
TDSNet(ours) | 0.114M | 80.34

##  5 Conclusion

In this paper, we propose a Task-aware Dual Similarity Network (TDSNet) for
FG-FSL, which consists of two designed components, a local feature enhancement
module and a measurement module that combines global and task-aware local
similarity. Specifically, the former is designed to fully explore the
discriminative details suitable for fine-grained classification, while the
latter explores similarity by taking multiple perspectives, both global and
local. Extensive experiments demonstrate that our proposed TDSNet achieves
competitive results. In the future, we intend to reinforce the features of the
foreground object and eliminate the negative effect of complicated
backgrounds.

## References

  * [1] Berg, T., Belhumeur, P.N.: Poof: Part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In: Computer Vision & Pattern Recognition (2013)
  * [2] Boiman, O., Shechtman, E., Irani, M.: In defense of nearest-neighbor based image classification. In: IEEE Conference on Computer Vision & Pattern Recognition (2008)
  * [3] Branson, S., Horn, G.V., Belongie, S., Perona, P.: Bird species categorization using pose normalized deep convolutional nets. In: British Machine Vision Conference 2014 (2014)
  * [4] Chang, D., Zheng, Y., Ma, Z., Du, R., Liang, K.: Fine-grained visual classification via simultaneously learning of multi-regional multi-grained features. arXiv abs/2102.00367 (2021)
  * [5] Chang, D., Ding, Y., Xie, J., Bhunia, A.K., Li, X., Ma, Z., et.al: The devil is in the channels: Mutual-channel loss for fine-grained image classification. IEEE Transactions on Image Processing 29, 4683–4695 (2020)
  * [6] Chang, D., Pang, K., Zheng, Y., Ma, Z., Song, Y.Z., Guo, J.: Your “flamingo” is my “bird”: Fine-grained, or not. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11471–11480 (2021)
  * [7] Chen, H., Li, H., Li, Y., Chen, C.: Multi-scale adaptive task attention network for few-shot learning. CoRR abs/2011.14479 (2020)
  * [8] Ding, Y., Ma, Z., Wen, S., Xie, J., Chang, D., Si, Z., et.al: Ap-cnn: Weakly supervised attention pyramid convolutional neural network for fine-grained visual classification. IEEE Transactions on Image Processing 30, 2826–2836 (2021)
  * [9] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning (2017)
  * [10] Fu, J., Zheng, H., Tao, M.: Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In: IEEE Conference on Computer Vision & Pattern Recognition (2017)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [12] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.: L.: Novel dataset for fine-grained image categorization. CVPR workshop on fine-grained visual categorization (2013)
  * [13] Koch, G., Zemel, R., Salakhutdinov, R.: Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning 37 (2015)
  * [14] Krause, J., Stark, M., Deng, J., Li, F.F.: 3d object representations for fine-grained categorization. In: IEEE International Conference on Computer Vision Workshops (2014)
  * [15] Li, W., Xu, J., Huo, J., Wang, L., Luo, J.: Distribution consistency based covariance metric networks for few-shot learning. In: Thirty-Third AAAI Conference on Artificial Intelligence (2019)
  * [16] Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., Luo, J.: Revisiting local descriptor based image-to-class measure for few-shot learning. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 7260–7268 (2019)
  * [17] Li, X., Wu, J., Sun, Z., Ma, Z., Cao, J., Xue, J.H.: Bsnet: Bi-similarity network for few-shot fine-grained image classification. IEEE Transactions on Image Processing 30, 1318–1331 (2021)
  * [18] Lifchitz, Y., Avrithis, Y., Picard, S., Bursuc, A.: Dense classification and implanting for few-shot learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [19] Luo, W., Yang, X., Mo, X., Lu, Y., Davis, L., Li, J., Yang, J., Lim, S.N.: Cross-x learning for fine-grained visual categorization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8241–8250 (2019)
  * [20] Lz, A., Sh, B., Wei, L.A.: Learning sequentially diversified representations for fine-grained categorization. Pattern Recognition (2021)
  * [21] Ma, Z., Lai, Y., Kleijn, W.B., Song, Y.Z., Wang, L., Guo, J.: Variational bayesian learning for dirichlet process mixture of inverted dirichlet distributions in non-gaussian image feature modeling. IEEE Transactions on Neural Networks and Learning Systems 30(2), 449–463 (2019)
  * [22] Ma, Z., Xie, J., Lai, Y., Taghia, J., Xue, J.H., Guo, J.: Insights into multiple/single lower bound approximation for extended variational inference in non-gaussian structured data modeling. IEEE Transactions on Neural Networks and Learning Systems 31(7), 2240–2254 (2020)
  * [23] Rao, Y., Chen, G., Lu, J., Zhou, J.: Counterfactual attention learning for fine-grained visual categorization and re-identification. In: IEEE/CVF International Conference on Computer Vision. pp. 1025–1034 (2021)
  * [24] Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning. In: International Conference on Learning Representations (2017)
  * [25] Shermin, T., Teng, S.W., Sohel, F., Murshed, M., Lu, G.: Integrated generalized zero-shot learning for fine-grained classification. Pattern Recognition 122, 108246 (2022)
  * [26] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems. vol. 30 (2017)
  * [27] Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P., Hospedales, T.: Learning to compare: Relation network for few-shot learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1199–1208 (2018)
  * [28] Tao, H., Qi, H.: See better before looking closer: Weakly supervised data augmentation network for fine-grained visual classification. arXiv 1901.09891 (2019)
  * [29] Vinyals, O., Blundell, C., Lillicrap, T., kavukcuoglu, k., Wierstra, D.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems. vol. 29 (2016)
  * [30] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset. california institute of technology (2011)
  * [31] Wang, C., Song, S., Yang, Q., Li, X., Huang, G.: Fine-grained few shot learning with foreground object transformation. Neurocomputing 466, 16–26 (2021)
  * [32] Wei, X., Wang, P., Liu, L., Shen, C., Wu, J.: Piecewise classifier mappings: Learning fine-grained learners for novel categories with few examples. IEEE Transactions on Image Processing 28(12), 6116–6125 (2019)
  * [33] Wu, Y., Zhang, B., Yu, G., Zhang, W., Wang, B., Chen, T., Fan, J.: Object-Aware Long-Short-Range Spatial Alignment for Few-Shot Fine-Grained Image Classification, pp. 107–115 (2021)