# Meta-Tasks: An alternative view on Meta-Learning Regularization

Mohammad Rostami, Atik Faysal, Huaxia Wang, Avimanyu Sahoo, and Ryan Antle
This research is partially funded in part by the Oklahoma Center for
Advancement of Science and Technology (OCAST) #R21-070 and Department of
Energy #DE-FE0032196. Mohammad Rostami, Atik Faysal, and Huaxia Wang are with
the Department of Electrical and Computer Engineering, Rowan University,
Glassboro, NJ; Avimanyu Sahoo is with the Department of Electrical and
Computer Engineering, University of Alabama in Huntsville; and Ryan Antle is
with Baker Hughes, Oklahoma City, OK. e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and ryan.antle@BakerHughes.com).

###### Abstract

Few-shot learning (FSL) is a challenging machine learning problem due to a
scarcity of labeled data. The ability to generalize effectively on both novel
and training tasks is a significant barrier to FSL. This paper proposes a
novel solution that can generalize to both training and novel tasks while also
utilizing unlabeled samples. The method refines the embedding model before
updating the outer loop using unsupervised techniques as “meta-tasks”. The
experimental results show that our proposed method performs well on novel and
training tasks, with faster and better convergence, lower generalization, and
standard deviation error, indicating its potential for practical applications
in FSL. The experimental results show that the proposed method outperforms
prototypical networks by 3.9%.

###### Index Terms:

Prototypical Network, Few-Shot Learning, Meta-learning, Regularization, Deep
Leanring, Artificial Intelligence.

##  I Introduction

Few-Shot Learning (FSL) emerges as an appealing solution to the problem of
limited data availability in computer vision tasks. FSL approaches use meta-
learning techniques to learn how to learn[1] from a small number of instances,
allowing effective generalization to novel tasks with limited training data.
The primary goal of FSL is to efficiently extract meaningful representations
from a small sample size. Recent developments in deep learning, notably in the
field of meta-learning, have led to the development of powerful FSL techniques
such as Matching Networks [2], Relation Networks [3], Model Agnostic Meta-
Learning (MAML) [4], Prototypical Networks [5], Reptile [6]. FSL enables
models to learn about anomalies and recognize uncommon occurrences, which can
be particularly useful in medical applications, such as locating uncommon
diseases like COVID-19. By minimizing the quantity of data needed to train a
model, FSL can drastically reduce the expenses involved with data gathering
and annotation [7, 8, 9].

Meta-learning [1] has been the subject of much research in recent years,
However, FSL still requires a thorough regularization to prevent overfitting
and ensure good generalization performance on samples from training datasets
and novel classes. Due to the nature of these algorithms, which involve
backpropagation from a single sample multiple times on a limited number of
samples, regularization is particularly difficult in meta-learning and a
variety of regularization methods have been proposed to improve its
generalization capability [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
22]. Most conventional regularization methods from other areas of machine
learning, such as weight decay [23], dropout [24], and incorporating noise
[10, 11, 12], are still applicable to meta-learning. However, the specialized
regularization techniques for meta-learning have proven more efficient.

Regularization in the meta-learning process is divided into two major
components: meta-generalization, in which the meta-model must generalize to
unknown tasks, and adaptation-generalization, in which the adapted model must
generalize to the domain of a specific task. The ability of the meta-model to
adapt to new tasks determines how effective meta-learning is. The majority of
the cited works concentrate on meta-generalization. However, they do not
address the issue of adaptation-generalization. Moreover, The effectiveness of
meta-learning algorithms varies with the task at hand. Choosing the right
regularization approach and hyperparameters can significantly improve the
meta-learning model’s performance.

Motivated by the involved challenges in regularization, in this paper, we
proposed the addition of new tasks that can regularize individual tasks or
episodes called “meta-tasks” (Figure 1). The rationale behind considering the
regularization problem as a task is that it can be solved similarly to other
tasks by introducing an additional loss function. This will address the
complexity of selecting a proper regularization term that satisfies all the
tasks. This will allow unsupervised or semi-supervised tasks to be used as
regularization terms or tasks. For example, an autoencoder [25] can be used as
a regularizer for FSL by restricting the embedding vector to be
reconstructable to the original image. Therefore, we also proposed a new
regularization task for meta-learners called ‘meta-autoencoder’. We show that
using a meta-autoencoder leads to faster convergence with higher accuracy and
lower generalization error and standard deviation, highlighting its potential
for practical applications in FSL. This novel framework is intended to improve
the adaptability and generalization capabilities of meta-learning models,
opening up a promising avenue for overcoming current limitations in the field.

The remainder of the paper is organized as follows. Section II presents a
brief literature review to motivate the problem followed by the problem
statement in Section III. Our proposed approach and results are presented in
Sections IV and V, respectively, followed by conclusions in Section VI.

Figure 1: A High-level view of Meta-Task: Each episode is made up of labeled
examples (the support set) and unlabeled instances (the query set). The
episode samples are then utilized to generate meta-task samples that are
unique to that episode. The episode is subsequently processed by an embedding
model, which maps the samples to a feature space and produces a compact
representation of the episode. The meta-learning approach then uses this
feature representation to learn and extract relevant information from the
support set. Using the learned patterns and relationships from the support
set, the obtained knowledge is applied to predict or make inferences on the
query set.

##  II Background/Related Work

In this section, we briefly review the relevant literature on meta-learning
and regularization issues. The specialized regularization techniques for meta-
learning are 1) Explicit Regularization (ER) [13, 14, 17, 18], 2) Episode
Augmentation Regularization (EAR) [15, 16], 3) Task Augmentation
Regularization (TAR) [21, 19, 22], 4) Adversarial Task Up-sampling (ATU) [20].

The ER methods impose explicit regularization terms on the meta-learning
update, such as iMAML [13], MR-MAML [14], and their combinations or extensions
[17, 18]. These methods directly constrain the model optimization process,
aiming to improve the generalization performance of meta-learning models. For
instance, minimax-meta regularization [18] combines two types of
regularizations at the inner and outer levels to enhance the generalization
performance of bi-level meta-learning.

The EAR approach controls the meta-training by making modifications to
individual tasks through noise or mixup [15, 16]. By expanding the amount of
training data available for sampling, data augmentation supports various
aspects of the meta-learning pipeline. It can be applied to support data in
the inner loop of fine-tuning and enlarge the pool of evaluation data to be
sampled during training.

TAR methods generate new tasks by interpolating between or augmenting existing
tasks [21, 22]. The goal is to expand the task distribution for more robust
generalization. For instance, TAR methods, such as MetaMix [26] and Channel
Shuffle [19] have been proposed to increase the dependence of target
predictions on the support set and provide additional knowledge to optimize
the model initialization. Additionally, the ATU approach [20] seeks to
generate tasks that match the true task distribution and can contribute to the
latest meta-learner by maximizing an adversarial loss, thereby improving the
meta-testing performance and the quality of up-sampled tasks. These task
augmentation techniques are designed to enhance the generalization capability
of meta-learning models by effectively expanding the task distribution and
mitigating overfitting during meta-training.

##  III Problem Statement

In this section, we formally introduce the problem in meta-learning and
propose a solution. The central concept is to learn how to adapt a base
learner to a new task for which only a few labeled samples are available by
using a large number of comparable few-shot tasks. Training numerous tasks are
referred to as ‘meta,’ and learning to scale and modify the functions of Deep
Neural Networks (DNN) weights for each task is based on transfer learning,
meaning using the trained meta-learning model as a pre-trained model for each
task [4]. Meta-learning often uses shallow neural networks (SNNs), improving
its effectiveness by reducing intra-class variation [27]. The first part of a
few-shot learning method involves embedding the data into a low-dimensional
feature space in a way that captures the important aspects of the data. The
second part involves using the labeled examples to learn a classifier that can
accurately predict the labels of new, unlabeled examples. In many few-shot
learning methods, the embedding and classification stages are often combined
into a single neural network architecture. Although the two parts are closely
related, they can still be viewed as separate components of a typical FSL
method.

Let’s go through FSL basics briefly, An $N$-way, $K$-shot classification
consists of $N$ classes and $K$ examples of each class. They serve as the
task’s ‘support set’ and aid the model in learning like a training set. The
performance of this task is also assessed using additional examples of the
same classes, referred to as ‘query set.’ Samples from one task might not
appear in others because each task can be entirely independent of the others.
The notion is that throughout training, the system continually encounters
tasks with a structure similar to the final few-shot task but contains
different classes. The meta-learning framework for few-shot learning involves
selecting a random training task at each meta-learning step and updating model
parameters based on the classification accuracy of the query set. The aim is
to differentiate class labels in general rather than a specific subset of
classes because the task at each time step is different. Few-shot performance
is evaluated on a set of unseen classes, referred to as novel classes, that
were not included in the training.

We hypothesize that we can further enhance the generalization capability in
meta-learning by developing auxiliary tasks called “meta-tasks” that act as
regularization terms for both meta-generalization and adaptation-
generalization as regularization methods. Tasks have the potential to encode a
diverse range of information about the world and the relationships between
different concepts, enabling the model to learn generalizable patterns and
ultimately improve its performance on new tasks. The reasoning behind this
approach is based on the concept of FSL, which is inspired by the human
ability to grasp new tasks with only a few examples and then generalize this
knowledge to unfamiliar scenarios. Meta-tasks can also be viewed as a
generalization of TAR, transforming them into simple, explicit regularization
terms that seamlessly integrate into existing meta-learning algorithms.
Consider Generative Adversarial Networks (GANs) as a more complex example.
GANs are primarily tasked with creating new examples that closely resemble an
existing sample distribution, such as creating images that are similar to but
distinct from a dataset of pre-existing images. GANs are composed of two
neural network models: the generator and the discriminator. The generator
attempts to deceive the discriminator, and the discriminator is capable of
distinguishing between real and generated samples. In this context, the
discriminator serves as the generator’s regulator and vice versa. Applying
this concept to meta-learning, we can assign a new task to our meta-learning,
such as acting as a discriminator, by conceptualizing the embedding model as
the generator. This idea has been exhaustively explored in [28, 29]

Our goal in extending this concept to meta-learning is to generate additional
tasks and samples, similar to task and episode augmentation. However, we
consider the regularization of each task to be a task in and of itself. As a
result, each regularization approach for a specific task can be introduced as
a new task to improve the model’s understanding of that task. Studies related
to TAR are examples of meta-tasks that work. For the first step, we introduced
an autoencoder’s task to learn a lower-dimensional representation of a higher-
dimensional input, a simple but important task that is linked to all vision
tasks in FSL.

##  IV Approach

In This section, we formulate meta-tasks for prototypical networks and explain
why we can consider meta-tasks as regularization terms. The rationale behind
incorporating meta-tasks regularization is similar to that of meta-learning.
As meta-learning aims to help a meta-learner better understand the
commonalities between different tasks, meta-task learns to generalize more
effectively and develop a more robust set of features that can be applied to
new tasks. By training on related meta-tasks, a meta-learner can leverage the
knowledge gained from these tasks to learn more efficiently and adapt to new
situations. The meta-dataset is split into the train, validation, and test
sets, and denoted by $D^{\textrm{tr}}$, $D^{\textrm{val}}$, $D^{\textrm{te}}$
where $D^{\textrm{tr}}$ consist of tasks $T^{\textrm{tr}}_{1}$,
$D^{\textrm{val}}$ consist of tasks $T^{\textrm{val}}_{1}$,
$T^{\textrm{val}}_{2}$, $\dots$, $T^{\textrm{val}}_{J}$ and $D^{\textrm{te}}$
consists of tasks $T^{\textrm{te}}_{1}$, $T^{\textrm{te}}_{2}$, $\dots$,
$T^{\textrm{te}}_{I}$.

We introduce a new task, denoted as $T^{\textrm{tr}}_{L1}$, which only
requires images from $D^{\textrm{tr}}$ and does not need labels, or labels can
be created using available labels in $D^{\textrm{tr}}$. In each episode, we
take $N$ tasks from $\\{T^{\textrm{tr}}_{i}\\}_{i=1}^{L}$ and use
$T^{\textrm{tr}}_{L1}$ (and any other applicable $\\{T^{\textrm{tr}}_{Li}\\}$)
as the regularizer for $\\{T^{\textrm{tr}}_{i}\\}_{i=1}^{L}$ or fine-tuned
separately for each task.

To clarify this method, we can define a new task $T^{\textrm{tr}}_{L1}$ as
generating an image in an autoencoder. In this task, the loss can be defined
as the mean squared error (MSE) loss.

Prototypical networks aim to minimize the negative log probability (log-
softmax loss). For a given query sample $x$, if $n$ is the correct label, then
the loss function is defined as

$J_{x}(\theta)=-\log(p_{\theta}(y=n|x)),$ (1)

where $p_{\theta}(y=n|x)$ is the probability of assigning the correct label
$n$ to query sample $x$. Alternatively, $p_{\theta}$ can be represented as

$p_{\theta}(y=n|x)=\frac{\exp(-d(f_{\theta}(x),c_{n})}{\sum_{k=1}^{N}\exp(-d(f_{\theta}(x),c_{k}))},$
(2)

where $f_{\theta}$ is the embedding network parameterized by $\theta$, $c_{n}$
is the prototype of the $n$-th class, and $d(a,b)$ is the Euclidean distance
between vectors $a$ and $b$. The logarithm is used to increase the loss when
the model fails to predict the correct class. Thus, we can rewrite the loss
function and, subsequently, the update role as

$\min\limits_{\theta}J_{x}(\theta)=d\left(f_{\theta}(x),c_{n}\right)+\log\sum_{k=1}^{N}\exp\left(-d(f_{\theta}(x),c_{k})\right).$
(3)

We add an additional term to the loss function to incorporate the
regularization term into our method. This term is the loss function of the new
task we introduced, which we denote as $T^{\textrm{tr}}_{L1}$. In our case,
$T^{\textrm{tr}}_{L1}$ is the generation of an image using an autoencoder, and
we use mean squared error as the loss function for this task. To be more
specific, we add the autoencoder update role to the original update role of
Prototypical Networks, resulting in the loss function for a query sample $x$
with correct label $n$ given by

$J_{x}(\theta)=\underbrace{d(f_{\theta}(x),c_{n})+\log\sum_{k=1}^{N}\exp(-d(f_{\theta}(x),c_{k}))}_{\text{Meta-
Learning Update}}\\\
+\overbrace{\lambda||g_{\theta^{{}^{\prime}}}(f_{\theta}(x))-x||}^{\text{AutoEncoder
Update }},$ (4)

where $g$ is the decoder network with parameters $\theta^{{}^{\prime}}$, and
$\lambda$ is the learning rate for the autoencoder. This regularization term
can be added to any other arbitrary loss function given by

$J_{x}(\theta)=-\log(p_{\theta}(y=n|x))+\lambda||g_{\theta^{{}^{\prime}}}(f_{\theta}(x))-x||$
(5)

We calculate the sum of losses overall query samples in a task
$T^{\textrm{tr}}_{i}$ to obtain the overall loss for that task as

$J_{T^{\textrm{tr}}_{i}}(\theta)=\sum_{x\in
T^{\textrm{tr}}_{i}}J_{x}(\theta).$ (6)

During each episode, we compute the loss function for each task and sum them
to get the overall loss function[4]

$J(\theta)=\sum_{i=1}^{L}J_{T^{\textrm{tr}}_{i}}(\theta).$ (7)

Now, if we assume that we have a custom meta-task regularizer for each task by
additive separability, we can rewrite (7) as

$J(\theta)=\sum_{i=1}^{L}\left(J_{T^{\textrm{tr}}_{i}}(\theta)+\lambda
J_{T^{\textrm{tr}}_{Li}}(\theta)\right).$ (8)

If the regularization term is separable (like the autoencoder update), we can
simplify (8) as

$J(\theta)=\sum_{i=1}^{L}J_{T^{\textrm{tr}}_{i}}(\theta)+\lambda\sum_{i=1}^{L}J_{T^{\textrm{tr}}_{Lr_{i}}}(\theta).$
(9)

Now, if we want to generalize this equation to form where each task may have
multiple or no meta-task attached to it. We will have:

$J(\theta)=\sum_{i=1}^{L}J_{T^{\textrm{tr}}_{i}}(\theta)+\lambda\sum_{r=1}^{R}J_{T^{\textrm{tr}}_{Lr}}(\theta),$
(10)

where $R$ is the total number of regularizers applied to the model.

###### Remark 1

It is important to mention the proposed approach for defining regularization
is model agnostic. Alternatively, the proposed regularization approach is
generic and can be applied to any FSL method.

###### Remark 2

Equations (10), (9) work only if meta-tasks and tasks are separable or
independent.

Figure 2: Detailed architecture of Meta-Autoencoder: The training examples in
the autoencoder task are identical copies of the support set samples but
without their labels. For these examples, the embedding model generates
feature vectors employed in two ways. They are first processed through a
decoder to reconstitute the original input images, which aids the model in
learning meaningful representations. Second, the feature vectors are used to
generate prototypes for each support set class, which are then used to
categorize query photos based on their similarity. The training procedure
includes two backpropagation steps: one for updating model parameters based on
the classification loss, which measures the match between prototypes and query
images, and another for updating model parameters based on the decoder’s
reconstruction loss, which ensures accurate reconstruction of the original
images.

Although the use of encoder-decoders is not new in meta-learning [30], we use
a novel autoencoder (figure 2) method to illustrate the effectiveness of this
regularization. Our developed method is called Meta-Autoencoder. In this
method, we use the MSE loss of the decoder as the regularization term for each
episode. In other words, for each episode, we pass the images from the episode
through the decoder to generate the regularization term for the update of the
prototypical network. This regularization helps to improve the generalization
performance of the model by encouraging the latent representations learned by
the encoder to be more robust and informative. It is worth noting that the
main distinction from [30] lies in our focus on regulating the training
process. In contrast, their approach revolves around finding a better starting
point using the episode.

x

##  V Results and Discussions

We applied the proposed approach using ResNet50 [31] and CovNet architecture.
Our findings in this section show that the proposed method improves the
training process by including a decoder update as a regularization term.
Specifically, we found that introducing artificial tasks during training can
improve performance and standard deviation. These findings suggest that our
approach has the potential to improve the training of meta-learning methods in
a variety of applications. It is also important to note that meta-task updates
can be used in testing but to achieve more fair results, we avoided using
meta-tasks during testing so that the meta-tasks only influence the model’s
training weights. To compare meta-generalization and adaptation-
generalization, we measured training, validation, and testing accuracy. We
used the default splits for each dataset, but we argue that the validation
split currently has no value and is usually combined with the training split;
however, if the validation split included both training and novel tasks, it
would be a good representation of both meta-generalization and adaptation-
generalization rather than just meta-generalization, which would add useful
information when training models.

For Prototypical Networks training, we used a pre-trained ResNet50 autoencoder
from the bird 525 species dataset. The trained encoder was then used as an
embedding model in our experiments. Model training was carried out using a
single-threaded NVIDIA Tesla P100 computing resource. The experiments covered
$50,000$ episodes, with a learning rate of $10^{-4}$ and a query size of $15$.
The Prototypical Network training time averaged 10.5 hours, while the Meta-
Autoencoder averaged 12 hours. The autoencoder was trained using a separate
learning rate of $10^{-6}$. The standard deviation for Prototypical accuracy
and loss were 0.7% and 0.0003, while Meta-Autoencoder accuracy and loss were
0.08% and 0.00005. For MAML training, we used Mini-ImageNet to train a CovNet
with random initialization, a shallow CovNet decoder (75K parameters), and a
moderately deep CovNet decoder (250K parameters). We trained the models on two
NVIDIA RTX 4090s, using MAML, Meta-AutoEncoder with shallow decoder, and Meta-
AutoEncoder with moderately deep decoder, which took 7.2, 7.9, and 8.4 hours,
respectively. The training lasted 300 epochs, with a learning rate of 0.01 for
the encoder and classifier and 1e-6 for the AutoEncoder and The query size was
15. The complete source code and results for this paper can be found at the
following GitHub repository: HIDDEN

To compare the effectiveness of various approaches, we conducted a comparative
study of FSL’s core algorithms (Prototypical Network and MAML) with their
meta-autoencoder counterparts. Initially, we tuned the hyperparameters of both
models for the 5-Way 5-Shot task on the Mini-ImageNet dataset[32]. The
adjusted hyperparameters were then used in all subsequent training runs. It is
important to note that assessing few-shot learning methods is difficult due to
a lack of suitable evaluation metrics. Standard accuracy and precision
measurements, which are intended for non-repetitive samples, may not be
suitable for few-shot learning scenarios. For our evaluation, we used
available metrics, such as training/val/test accuracy and loss, as well as
their standard deviation, to compare meta-learning methods, as these are
currently the primary tools for comparison. In addition, we evaluated the
performance of the two models using traditional evaluation metrics, such as
the f1-score, assuming that each sample is unique, as shown in the
supplementary materials. The results of our study are summarized in the
following tables I.

Tiered ImageNet
---
5-Way 5-Shot |  | 5-Way 1-Shot
Model | Train | Val | Test |  | Model | Train | Val | Test
Prototypical Network | 66.00% | 61.30% | 65.30% |  | Prototypical Network | 46.10% | 41.90% | 46.70%
Meta-AutoEncoder | 74.60% | 68.00% | 71.30% |  | Meta-AutoEncoder | 53.80% | 49.70% | 50.00%
Mini-ImageNet
5-Way 5-Shot |  | 5-Way 1-Shot
Model | Train | Val | Test |  | Model | Train | Val | Test
Prototypical Network | 78.00% | 65.30% | 65.20% |  | Prototypical Network | 55.90% | 44.50% | 44.60%
Meta-AutoEncoder | 83.60% | 64.90% | 65.80% |  | Meta-AutoEncoder | 62.30% | 51.60% | 47.90%
FC100
5-Way 5-Shot |  | 5-Way 1-Shot
Model | Train | Val | Test |  | Model | Train | Val | Test
Prototypical Network | 87.20% | 49.30% | 48.70% |  | Prototypical Network | 70.50% | 29.50% | 29.30%
Meta-AutoEncoder | 90.30% | 48.50% | 49.20% |  | Meta-AutoEncoder | 75.30% | 31.30% | 39.00%

TABLE I: The table displays the training accuracy of the prototypical network
and meta-autoencoder. The results of our experiments clearly show that our
method maintains consistent accuracy and loss compared to the prototypical
network. This is especially evident when observing the accuracy of the
prototypical network over the next 10,000 episodes, where the meta-autoencoder
consistently outperforms it.

Mini-ImageNet
---
5-Shot 5-Way
Model | Train | Val | Test
MAML(CNN) | 76.64% | 61.11% | 62.23%
Meta-AutoEncoder(CNN) | 76.82% | 61.04% | 62.52%
Meta-AutoEncoder(moderately deep CNN Decoder) | 76.84% | 61.57% | 63.01%
|  |  |
5-Shot 1-Way
Model | Train | Val | Test
MAML(CNN) | 57.58% | 46.36% | 47.54%
Meta-AutoEncoder(CNN) | 58.10% | 45.24% | 47.81%
Meta-AutoEncoder(moderately deep CNN Decoder) | 57.94% | 46.13% | 47.67%

TABLE II: The table displays the training accuracy of MAML and meta-
autoencoder. Figure 3: The figure displays the loss of the prototypical
network and meta-autoencoder. The results of our experiments clearly show that
our method maintains consistent accuracy and loss compared to the prototypical
network. The results indicate that our method achieves faster convergence
compared to the prototypical network. This is especially evident when
observing the accuracy of the prototypical network over the next 10,000
episodes, where the meta-autoencoder consistently outperforms it. When the
losses are compared, the meta-autoencoder regularization outperforms the
Prototypical Network alone in terms of training regularization. This addition
reduces overall loss, indicating improved generalization performance. The
autoencoder regularization term assists the model in learning more robust and
informative latent representations through the encoder, resulting in better
training regularization.

The experiments listed in Table I show that our proposed method (Meta-
AutoEncoder) outperforms the prototypical network. The training and test sets’
standard deviations were consistently lower than the prototypical network.
Tiered-ImageNet training and validation accuracy improved by approximately 7%,
indicating improved adaptation-generalization. The 6% improvement on the test
set reflects an improvement in meta-generalization. The loss also decreased
more than in the prototypical network and was consistently lower during
testing. Table I shows that the results remained consistent across different
hyperparameters, including the number of episodes, ways, shots, and learning
rate, as well as datasets. The results of the 5-Way 1-shot show a larger gap
between the two models, indicating better adaptation-generalization and meta-
generalization with fewer shots.

Table II showcases the performance of Meta-Autoencoder with MAML.
Interestingly, both the shallow and moderately deep decoder configurations
outperform the baseline MAML approach. This suggests that Meta-Autoencoder and
its extension meta-tasks can be also effective with the addition of a shallow
network if need be. Furthermore, the shallower decoder performed better on
5-Shot 1-Way, demonstrating that using a deeper model does not always produce
better results. The moderately deep decoder demonstrates how a deeper network
performs better with more shots. While Meta-AutoEncoder with MAML showed
improvement, it did not reach the same level as the prototypical network. This
could be partly attributed to the difficulty of finding a suitable learning
rate for MAML. Due to its computational complexity, tuning the learning rate
effectively was a challenge. Our decision to use the same learning rate as the
prototypical network, while simplifying comparison, might have limited MAML’s
potential performance. Exploring different learning rate schedules or
optimization techniques for MAML could be promising avenues for future
research.

##  VI Conclusion

This paper addresses the regularization problem in FS methods. To achieve
this, we introduced a novel approach called Meta-Task, which serves as an
additional regularization task that can be easily incorporated into any FSL
method. This Meta-Task is constructed using semi-supervised or unsupervised
techniques, such as autoencoders. We focused on the N-way K-shot image
classification problem and extensively investigated Meta-Tasks’ impact on
improving model performance. By incorporating Meta-Tasks, we observed a
significant reduction in the noise present in the input data, leading to
faster convergence and lower generalization error. Our findings highlight the
effectiveness of using autoencoders to enhance accuracy while simultaneously
reducing loss. By thoroughly exploring the concept of Meta-Tasks and their
relationship with the meta-learning problem, we have provided a clear
understanding of how this approach can effectively address the regularization
problem. Meta-tasks offer a convenient and efficient way to improve the
performance of FSL methods by leveraging additional auxiliary tasks. Our
results demonstrate the benefits of incorporating Meta-Tasks, emphasizing
their potential for enhancing model accuracy and reducing overfitting.

We also introduced a novel Meta-Task called meta-autoencoder. The meta-
autoencoder leverages the power of autoencoders within the meta-learning
framework to enhance the learning process. By incorporating the meta-
autoencoder task, we aim to improve the model’s ability to extract meaningful
features and representations from the input data. This novel approach offers a
promising direction for advancing meta-learning algorithms and opens up new
opportunities for improving performance in various domains. We recognized the
importance of fine-tuning the learning rate for optimal performance and plan
to explore this further in future work.

While meta-tasks are effective in addressing the issue of overfitting, it is
important to note that they are not immune to overfitting themselves. We
observed that increasing the number of episodes and learning rate could lead
to overfitting, particularly when working with a larger learning rate.
Furthermore, it is worth noting that simply adding more meta-tasks does not
necessarily result in better regularization, as meta-tasks are not inherently
prone to task overfitting [17]. Finding the right balance when incorporating
meta-tasks into the meta-learning process is crucial. Adding too many meta-
tasks can lead to task overfitting, where the model becomes overly specialized
to the specific tasks in the meta-training set and fails to generalize well to
new tasks. Therefore, careful consideration should be given to the selection
and number of meta-tasks to ensure the meta-learning algorithm’s optimal
performance and generalization ability.

ResNets, which have the potential for greater representational power, were
excluded from this study due to MAML’s inherent training time limitations.
Training MAML with deep architectures such as ResNets is often computationally
expensive and impractical, limiting its usefulness in this context.

When constructing a validation set for few-shot learning models, we advocate
for incorporating both training tasks and novel tasks. This approach offers a
more comprehensive evaluation by assessing the model’s performance on two key
aspects: its ability to learn on the initially provided training tasks
(adaptation error) and its ability to adapt to new, unseen tasks (meta-error).
Examining both types of tasks provides a clearer picture of the model’s
strengths and weaknesses compared to solely focusing on either training
performance or adaptation on completely novel tasks.

## References

  * [1] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Learning to learn how to learn: Self-adaptive visual navigation using meta-learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6750–6759, 2019.
  * [2] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29, 2016.
  * [3] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [4] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR, 2017.
  * [5] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [6] Alex Nichol and John Schulman.  Reptile: a scalable metalearning algorithm.  arXiv preprint arXiv:1803.02999, 2(3):4, 2018.
  * [7] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li.  Meta-sgd: Learning to learn quickly for few-shot learning.  arXiv preprint arXiv:1707.09835, 2017.
  * [8] Amrith Setlur, Oscar Li, and Virginia Smith.  Is support set diversity necessary for meta-learning?  arXiv preprint arXiv:2011.14048, 2020.
  * [9] Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song.  Metagan: An adversarial approach to few-shot learning.  Advances in neural information processing systems, 31, 2018.
  * [10] Alessandro Achille and Stefano Soatto.  Information dropout: Learning optimal representations through noisy computation.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2897–2905, 2018.
  * [11] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy.  Deep variational information bottleneck.  arXiv preprint arXiv:1612.00410, 2016.
  * [12] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In 2015 ieee information theory workshop (itw), pages 1–5. IEEE, 2015.
  * [13] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  Advances in neural information processing systems, 32, 2019.
  * [14] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn.  Meta-learning without memorization.  arXiv preprint arXiv:1912.03820, 2019.
  * [15] Shin’ya Yamaguchi, Daiki Chijiwa, Sekitoshi Kanai, Atsutoshi Kumagai, and Hisashi Kashima.  Regularizing neural networks with meta-learning generative models.  arXiv preprint arXiv:2307.13899, 2023.
  * [16] Jun Shu, Xiang Yuan, Deyu Meng, and Zongben Xu.  Dac-mr: Data augmentation consistency based meta-regularization for meta-learning.  arXiv preprint arXiv:2305.07892, 2023.
  * [17] Edwin Pan, Pankaj Rajak, and Shubham Shrivastava.  Meta-regularization by enforcing mutual-exclusiveness.  arXiv preprint arXiv:2101.09819, 2021.
  * [18] Lianzhe Wang, Shiji Zhou, Shanghang Zhang, Xu Chu, Heng Chang, and Wenwu Zhu.  Improving generalization of meta-learning with inverted regularization at inner-level.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7826–7835, 2023.
  * [19] Huaxiu Yao, Long-Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al.  Improving generalization in meta-learning via task augmentation.  In International conference on machine learning, pages 11887–11897. PMLR, 2021.
  * [20] Yichen Wu, Long-Kai Huang, and Ying Wei.  Adversarial task up-sampling for meta-learning.  Advances in Neural Information Processing Systems, 35:31102–31115, 2022.
  * [21] Huaxiu Yao, Linjun Zhang, and Chelsea Finn.  Meta-learning with fewer tasks through task interpolation.  arXiv preprint arXiv:2106.02695, 2021.
  * [22] Janarthanan Rajendran, Alexander Irpan, and Eric Jang.  Meta-learning requires meta-augmentation.  Advances in Neural Information Processing Systems, 33:5705–5715, 2020.
  * [23] Anders Krogh and John Hertz.  A simple weight decay can improve generalization.  Advances in neural information processing systems, 4, 1991.
  * [24] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv preprint arXiv:1207.0580, 2012.
  * [25] Mark A Kramer.  Nonlinear principal component analysis using autoassociative neural networks.  AIChE journal, 37(2):233–243, 1991.
  * [26] Yangbin Chen, Yun Ma, Tom Ko, Jianping Wang, and Qing Li.  Metamix: Improved meta-learning with interpolation-based consistency regularization.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 407–414. IEEE, 2021.
  * [27] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  arXiv preprint arXiv:1904.04232, 2019.
  * [28] Arvind Sridhar.  Meta-gan for few-shot image generation.  In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
  * [29] Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song.  Metagan: An adversarial approach to few-shot learning.  Advances in neural information processing systems, 31, 2018.
  * [30] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  arXiv preprint arXiv:1807.05960, 2018.
  * [31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [32] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In International conference on learning representations, 2017.

##  Appendix A Appendix

More information on the experiments can be found here.

Prototypical Network
---
| precision | recall | f1-score | support
0 | 29.12% | 28.49% | 28.80% | 22635
1 | 54.58% | 50.13% | 52.26% | 64455
2 | 26.88% | 29.89% | 28.31% | 32565
3 | 49.94% | 42.34% | 45.83% | 65115
4 | 36.33% | 33.04% | 34.60% | 23910
5 | 56.25% | 62.92% | 59.40% | 31275
6 | 53.73% | 41.41% | 46.77% | 39420
7 | 44.38% | 46.71% | 45.51% | 38310
8 | 26.97% | 30.00% | 28.40% | 31665
9 | 70.61% | 86.22% | 77.64% | 23130
10 | 53.81% | 49.29% | 51.45% | 54825
11 | 27.54% | 43.78% | 33.81% | 39690
12 | 34.26% | 34.13% | 34.19% | 46980
13 | 37.02% | 33.19% | 35.00% | 62745
14 | 30.11% | 19.93% | 23.99% | 32160
15 | 65.74% | 76.98% | 70.91% | 16020
16 | 56.42% | 51.69% | 53.96% | 24045
17 | 55.85% | 80.23% | 65.86% | 23205
18 | 46.94% | 38.72% | 42.44% | 39870
19 | 47.49% | 52.11% | 49.69% | 37980
accuracy | 44.48%
macro avg | 45.20% | 46.56% | 45.44% | 750000
weighted avg | 44.75% | 44.48% | 44.23% | 750000
meta-learning accuracy | 42.90%
|  |  |  |
TABLE III: 5-way 1-shot classification report Meta-autoencoder
---
| precision | recall | f1-score | support
0 | 41.58% | 37.27% | 39.31% | 64200
1 | 49.30% | 58.49% | 53.51% | 47295
2 | 50.43% | 50.11% | 50.27% | 39165
3 | 49.96% | 29.91% | 37.42% | 16200
4 | 56.85% | 38.64% | 46.01% | 70305
5 | 62.96% | 66.98% | 64.91% | 24285
6 | 39.47% | 44.93% | 42.02% | 23100
7 | 44.45% | 46.86% | 45.62% | 39465
8 | 48.39% | 39.82% | 43.69% | 22965
9 | 50.20% | 55.14% | 52.55% | 54960
10 | 43.57% | 37.51% | 40.31% | 39480
11 | 27.81% | 22.48% | 24.87% | 24030
12 | 36.03% | 25.30% | 29.73% | 47070
13 | 35.88% | 36.83% | 36.35% | 47415
14 | 35.86% | 50.05% | 41.79% | 15885
15 | 59.57% | 85.49% | 70.21% | 38685
16 | 51.83% | 84.35% | 64.21% | 39405
17 | 57.98% | 61.81% | 59.84% | 47970
18 | 20.93% | 14.95% | 17.44% | 32655
19 | 54.04% | 54.30% | 54.17% | 15465
accuracy | 47.28%
macro avg | 45.86% | 47.06% | 45.71% | 750000
weighted avg | 46.47% | 47.28% | 46.14% | 750000
meta-learning accuracy | 47.30%
|  |  |  |
TABLE IV: 5-way 1-shot classification report Hyper parameters
---
epoch | 5 | 5
train_num_episode | 10000 | 10000
test_num_episode | 10000 | 10000
train_way | 5 | 5
train_shot | 5 | 1
train_query | 15 | 15
test_way | 5 | 5
test_shot | 5 | 1
test_query | 15 | 15
optimizer | Adam | Adam
autoencoder_lr | 1.00E-06 | 1.00E-06
lr | 1.00E-04 | 1.00E-04


# System Configuration and Navigation of a Guide Dog Robot:
Toward Animal Guide Dog-Level Guiding Work

Hochul Hwang†, Tim Xia†, Ibrahima Keita, Ken Suzuki, Joydeep Biswas1, Sunghoon
I. Lee∗, and Donghyun Kim2 † Authors contributed equally, ∗ Co-corresponding
author1 University of Texas at Austin<EMAIL_ADDRESS>of
Massachusetts Amherst, 140 Governors Dr, U.S<EMAIL_ADDRESS>
###### Abstract

A robot guide dog has compelling advantages over animal guide dogs for its
cost-effectiveness, potential for mass production, and low maintenance burden.
However, despite the long history of guide dog robot research, previous
studies were conducted with little or no consideration of how the guide dog
handler and the guide dog work as a team for navigation. To develop a robotic
guiding system that is genuinely beneficial to blind or visually impaired
individuals, we performed qualitative research, including interviews with
guide dog handlers and trainers and first-hand blindfold walking experiences
with various guide dogs. Grounded on the facts learned from vivid experience
and interviews, we build a collaborative indoor navigation scheme for a guide
dog robot that includes preferred features such as speed and directional
control. For collaborative navigation, we propose a semantic-aware local path
planner that enables safe and efficient guiding work by utilizing semantic
information about the environment and considering the handler’s position and
directional cues to determine the collision-free path. We evaluate our
integrated robotic system by testing guide blindfold walking in indoor
settings and demonstrate guide dog-like navigation behavior by avoiding
obstacles at typical gait speed ($0.7\text{\,}\mathrm{m}\text{/}\mathrm{s}$).

##  I Introduction

According to the World Health Organisation (WHO), more than 2.2 billion people
in the world have near or distance vision impairments [1]. Guide dogs (GD)
have been considered as an effective way to improve the mobility and
independence of visually impaired individuals [2, 3, 4]. However, only about
2% of blind people in the U.S. work with guide dogs [5], and the percentage is
expected to decrease worldwide mainly due to the lack of supply. Fundamental
challenges of expanding the supply are the significant cost and time that are
required to train, deploy, and maintain animal guide dogs. The cost for
training one guide dog is over $50,000 USD and the time for breeding, raising,
training, and deploying takes up to two years. Guide dogs’ work span is
typically less than ten years and they require lifelong care from their
handler, including medical care, feeding, and daily walks. As a result, they
are not suitable for people who cannot raise a dog due to various reasons,
such as financial affordability, allergies, and physical limitations (e.g.,
due to motor impairments or natural aging) [2, 4, 6]. To remedy the issues and
embrace the potential of mass production, a cost-effective and sustainable
guide dog robot could be a significant contribution to the visually impaired
population.

Figure 1: Development of a guide dog robot based on qualitative research.
Summer collaboratively works with the handler by reflecting the handler’s
directional suggestions to reach the destination in a safe and efficient
manner by seamlessly avoiding obstacles similar to an animal guide dog.

This strong need inspired researchers, and various navigation-assistant
devices have been developed for a long time. The early study by [7] evidenced
how a robotic system can support the navigation works of a person with visual
impairment. Multiple wheeled mobile robots have been developed from 1976 to
1983 and demonstrated autonomous navigation and obstacle avoidance while
guiding a person [8]. On the other hand, researchers also developed various
forms of assistive devices such as smart canes [9, 10], vibrating belts [11],
hand-held camera traversability estimators [12], and head mount devices [13].
Throughout the research, the perception assistant has been actively studied,
but the mobility of the assistant device was less highlighted. The recent
progress in locomotion control of quadruped robots opened the chance to
utilize the legged robots for guide dog robots, and several studies using a
quadruped robot have been reported [14, 15, 16]. However, we discover that the
existing guide dog robots do not adequately address how the handlers interact
with their animal guide dogs in the real world, which has been established
over several decades to support comfortable mobility and navigation
efficiency.

For example, having a rigid connection is crucial for the handlers to perceive
the motion of the guide dog and navigate in a safe manner [17]. In addition,
positioning the handler right next to the robot is important to navigate
through sharp turns and narrow paths. However, [14, 15] used a soft leash and
[16] located a person behind a robot. Moreover, previous studies have mainly
focused on robot control under the handler’s pulling force or autonomous
navigation rather than interactive communication during a guiding work of a
handler-guide dog unit. Besides the handle design and the handler’s position,
various aspects should be considered in the guiding system design. Rich
literature about guide dog work and training methods exists [18, 19, 20], but
less guidelines have been disclosed for even basic robot specifications such
as size, weight, battery life, sensors, walking speed, required robustness of
walking controllers, and so on [6]. For the navigation algorithm development,
we have to precisely understand the roles of a handler and a dog because
guiding work is different from autonomous driving, which brings a person to
the target location without intermittent direction commands from the driver.

In this study, we aim to identify important design components of a guide dog
robot to support effective navigation and obstacle avoidance based on the
established interaction mechanisms between the handler and animal guide dogs.
To understand the interaction mechanisms, we conducted interviews with guide
dog handlers and trainers, followed by several observation sessions to
investigate how the handler navigates with the guide dog in real-world
settings. Our team also performed blindfold walking with four different guide
dogs under a guide dog trainer’s supervision to understand the navigation
process better. Based on qualitative data analysis, we configure the
development directions of a guide dog robot and build an interactive
navigation scheme and local path planning algorithm.

In terms of hardware design, we adopted the handle developed for animal guide
dog users and installed a custom-designed button interface to allow the
handlers to adjust speed and give walking direction cues, which are the most
frequently used cues in guiding work. The handle can be completely detachable
from the robot’s body to help the handlers to easily keep the robot underneath
their seat when they use public transportation. For the local path planner, we
primarily focus on rapid response to the environment change to safely navigate
a dynamic environment. Special effort is made to enable seamless pedestrian
avoidance while maintaining the handler’s normal walking speed to provide a
comfortable guiding work experience.

The proposed semantics-aware local path planner selects a path from a cost map
made with a pretrained semantic segmentation network [21] without extra
geometrical route computation to save computation power. In addition, our
method relies only on built-in RGB cameras without depth estimation and a
standard NVIDIA GPU computing unit. Similar navigation approaches that use
pretrained image processing backbones [22, 23] have been proposed, but less
have been explored to directly utilize semantic segmentation as a local path
planner. [24] proposed to utilize off-the-shelf semantic segmentation for
indoor mobile robots but did not deploy the methods on actual robotic systems.
[25] explored the idea of indoor navigation based on semantic segmentation, in
which they trained various segmentation networks and analyzed their
computational efficiency, and in [26, 27], the researchers deployed semantic
segmentation on an actual robot for indoor navigation, and both relied on the
egocentric view of the segmented scene for target point creation and
subsequent target point following. However, such methods are sensitive to the
noise of the segmentation network, which is inappropriate for safety-critical
applications. Our planner provides robustness against the segmentation error
by employing simple cost map-based path selection and rule-based stop
criteria.

Primary contributions of our work are summarized as follows: 1) human-centered
system integration of a guide dog robot based on qualitative research with
guide dog handlers and trainers, 2) development of a guiding work framework
with a safety-oriented local path planner that imitates animal guide dogs’
navigation works, and 3) experimental evaluation of the robot in real-world
indoor environments and demonstration of animal guide dog-like behavior at a
preferred walking speed. The video

TABLE I: Participant demographics Subject | Age | Gender | Experience (years) | Session type
---|---|---|---|---
H01 | 60 | F | 36 | Interview, observation
H02 | 69 | M | 30 | Interview, observation
H03 | 63 | M | 11 | Interview, observation
H04 | 69 | F | 30 | Interview, observation
H05 | 59 | M | 21 | Interview, observation
T01 | 65 | M | 45 | Interview
T02 | 52 | M | 19 | Interview, observation
T03 | 58 | M | 35 | Interview, observation

##  II Guide dog robot configuration

Guide dogs work with the handler as a team, called a unit, to safely navigate
to the desired destination. Also, unlike a white cane, guide dogs provide
efficient navigation by seamlessly avoiding obstacles at a walking speed.
Based on qualitative research with five guide dog handlers (GDHs) and three
guide dog trainers (GDTs), we discovered 1) core components that must be
considered when developing a guide dog robot to ensure safety and 2) advanced
components to enhance efficiency in navigation and increase comfort for the
handler. We elaborate on our guide dog robot’s configuration by specifying the
hardware requirements for real-world deployment. The participants’
demographics are summarized in Table I.

Figure 2: Pulling force measurement of animal guide dogs and the guide dog
robot, Summer. (a) We performed blindfold walk with different guide dogs. A
digital force gauge was used to measure the pulling force while walking. (b)
Go1’s built-in controller is evaluated on a testbed that pulls the robot with
the horizontal force up to $34.32\text{\,}\mathrm{N}\text{/}$
($3.5\text{\,}\mathrm{kg}\text{/}$). (c) Pulling forces measured during
blindfold walking with two different guide dogs.

###  II-A Hardware requirements to serve as a guide dog robot

To address the various physical attributes of various blind or visually
impaired individuals, guide dog trainers walk with the potential guide dog
handler beforehand and match with the appropriate guide dog considering
features including the size and pulling force. For a guide dog robot, compact
size and small weight are desirable as they allow handlers to be able to
independently carry and put the robot under seats, especially when they use
public transportation such as buses, trains, or taxi cabs. For an animal dog,
this can be easier as they have a flexible body as one of the handlers
mentioned:

> “Whereas a guide dog, being squishy, you train him to, even a dog his size,
> get on the floor area and lay down.” \- H01

Although minimizing the weight is beneficial for portability, it should still
be heavier than a certain mass to handle the nominal pulling force of a
handler and to address emergent stops. To identify the range of pulling force
in guide dog work, we measured the pulling force with a force gauge during our
blindfold walking with different dogs as in Fig. 2’ (a). We found that the
pulling force of a guide dog remained mostly under $40$\mathrm{N}\text{/}$$
even in the case of a dog with strong pulling strength (guide dog 1 in Fig. 2
(c)). We experimentally validated that the robot’s locomotion controller is
robust enough to manage a similar external disturbance by letting the robot
walk while being pulled by $3\text{\,}\mathrm{kg}\text{/}$ weight as shown in
Fig. 2 (b).

###  II-B Core components in GD work to ensure safety

####  II-B1 Rigid connection and correct positioning

> “I think it can’t be done without some sort of device that you have some
> sort of handle. … Because, when you’re working with whatever to get
> someplace, you are feeling that thing move. … if I’m holding a dog or
> holding a two-point handle to something if it stops, I would stop.” \- H01

As one of the GDH subjects mentioned, in guide dog work, the rigid connection
between the handle and the guide dog is critical. Most guide dog handlers
primarily use a harness while walking since it can provide instant feedback of
the guide dog’s motion while it is challenging to interpret the motion of the
guide dog using only a soft leash.

Walking at a proper position relative to the guide dog (next to the dog’s
hindquarters) while walking is also critical for various safety purposes. For
example, when the animal guide dog stops at a curb, the handler can use the
foot within a single step to reach out to recognize and locate the curb. Also,
it is ideal to be in the recommended position to reduce the range of motion
and turning speed for safety. Additionally, note that most guide dogs are
trained to walk on the handler’s left-hand side because most handlers are
right-handed. This enforces the handler to use their left hand to hold the
harness handle and to use their right hand to perform additional tasks or
search around when a dog stops. This setup can be uncomfortable for left-
handed handlers, but it is hard to accommodate them because switching the
position requires additional training.

Considering the aforementioned safety concerns, we use a rigid harness handle
structure provided by Ruffwear. The length-adjustable design is expected to
support various potential handlers, having different physical features, to
maintain the proper position while walking. On top of the safety benefits, the
rotating handle structure enables easy transition between left-handed and
right-handed handlers. Also, the detachable harness structure is desirable
considering the handler’s portability.

Figure 3: Observation of the guide dog handler’s walking with their guide
dogs. (a) The handler-guide dog unit is turning based on the handler’s
directional cue. (b) The handler-guide dog units are seamlessly avoiding
obstacles (e.g., a chair, a tree, and a person) in their local community.

####  II-B2 Collaborative navigation

Unlike autonomous navigation, in most guiding work, the handler needs to
provide cues to the guide dog to some extent, especially when initiating turns
and deciding when to cross a street, which can also be observed in Fig 3 (a).
In the basic setup between a handler and a guide dog, the handler is
responsible for knowing the directions to the target location based on
cognitive maps and making decisions about whether it is safe to make a move by
interpreting nonvisual information [19, 28].

> “We have to have some idea of where you want to go. So it becomes the
> person’s job to tell them straight, forward, left, right, find the way, look
> for whatever they’re aiming for.” \- T03

We implement such collaborative navigation by 1) designing a button interface
that takes the combination of touch sensor signals provided by the handler as
input and 2) allowing our proposed semantics-aware local path planner to take
this input into the path selection process. With such implementation, Summer
interprets the handler’s directional suggestions to safely take turns when
available.

Based on the feedback obtained from the GDHs, we designed a wireless button
interface that allows the handler to suggest directional cues to Summer. We
minimized the number of buttons to reduce the complexity of usage. These two
buttons are digitally connected to an Arduino Nano which is attached to a
Bluetooth HC-05 module. Note that the wireless communication paradigm allows
the harness handle to maintain its detachability. With a single press of the
button, a handler can give a directional command to the local path planner
that takes the command into consideration when deciding the most desirable
path which the details are provided in Section III.

###  II-C Advanced components in GD work to enhance efficiency and comfort

####  II-C1 Seamless obstacle avoidance at high walking speed

Guide dogs not only ensure the safety of the handler by avoiding collision but
they allow higher travel speed by seamlessly maneuvering around obstacles that
we found from interviews and the observation session (Fig 3(b)). This smooth
mobility is difficult to get with a white cane because a user needs to find a
way to avoid obstacles in a zig-zag manner based on tactile information [29].
One of the guide dog trainers gives a clear idea of how seamlessly an animal
guide dog can avoid obstacles.

> “Little bumps in the road and you’ll realize that sometimes if there’s a
> bunch of obstacles, whether it’s traffic cones, trees, fire hydrants,
> whatever it is, you might never know them there. You can have a guide dog
> for 10 years and you’ll go, ‘Oh I didn’t know that was there,’ because the
> dog just fluidly goes around. It’s a little different than using your cane
> where your cane you’re aware of all the obstacles.” \- T03

Here, we focus on developing a local path planner to provide smooth obstacle
avoidance in guiding work without slowing down or stopping the handler unless
they are necessary as explained in Section III-B. We also allow the handler
can control the walking speed using the button interface because the walking
speed varies by handlers; the dog’s walking speed is one of the most important
parameters to matching a proper handler-guide dog unit.

> ”Fidelco (training school), before they give you a dog, will match up your
> walking speed with a harness that they pull around to see how fast, slow,
> medium, or fast.”\- H03

The walking speed can be controlled incrementally by pressing the button
consecutively. As an example, the walking speed of Summer will increase by
$0.05\text{\,}\mathrm{m}\text{/}\mathrm{s}$ if the upper button is pressed
twice as shown in Fig. 4.

Figure 4: Handle interface design that features speed/direction control
button, adjustable handle, and detachable hook for portability. Figure 5:
Navigation pipeline. Images from front and side camera are passed into a
segmentation network. The segmented images are then projected down to bird-
eye-view and subsequently converted into a binary traversable and non-
traversable local cost map. Trajectories are rolled out on this cost map. The
button interface is used to bias trajectory selection. When the best path is
selected, the steering command is fed into the built-in controller. The button
interface can also be used to adjust the forward velocity of the controller.
Velocity is also adjusted based on cost boxes in the front and the left local
cost maps in case of emergencies for obstacle clearing.

##  III Guiding work framework

As explained in Section. II-B2, a handler provides orientation cues such as
forward, left, or right and a guide dog take care of local path planning by
safely and seamlessly avoid collision or other risk for the handler. Based on
the standard guiding work setup, we focused on developing a local path planner
that can provide rapid response to the environment change, avoiding various
obstacles including walking pedestrians without incurring uncomfortable jerky
movements or sudden stops unless they are necessary. One challenge was
avoiding walking pedestrians coming from opposite direction while maintaining
the handler’s walking speed. Considering the average speed of humans ($\approx
1~{}$\mathrm{m}\text{/}\mathrm{s}$$) and perception camera sight
($2-3~{}$\mathrm{m}\text{/}$$), the robot has around only
$1~{}$\mathrm{s}\text{/}$$ before it hit the person. Therefore, we targeted to
maximize the update frequency by removing computationally complex process and
by minimizing sensor inputs. Our image segmentation-based local planner plans
and evaluates paths at a rate of 20 Hz and is capable of robust static and
dynamic obstacle avoidance. Here we present the main components of our
navigation pipeline.

###  III-A Local path planner

We assume the robot is constantly moving forward, and employ a local planner
based on a trajectory roll-out algorithm that samples 40 trajectories
parameterized by minimum and maximum curvature. The 40 curvatures are rolled
out in front of Summer and each is then individually evaluated according to
length and visual cost. At each time step, the best path is selected and
Summer adjusts its steering angle according to the curvature of the selected
best path.

In order to evaluate the visual cost associated with the rolled-out
trajectories, a visual cost map is built based on the semantic categories of
the objects in the Summer’s field of view. We use the front- and side-facing
camera equipped on Summer as our main input. The camera operates at the rate
of $30~{}$\mathrm{fps}\text{/}$$, and two $464\times 400$ sized RGB images are
streamed from Summer to a laptop in the form of ROS messages [30]. Once the
image is received by the laptop, it is processed by a pretrained network with
a MobileV2Dialted encoder [31] and a convolutional decoder and converted into
two $464\times 400$ sized matrix, where pixel values are converted into one of
150 semantic classes. As we are only concerned about the traversable class,
floor, all 150 classes are converted into binary classes, i.e.,
$\text{{floor}}=0$ vs. $\text{{non-floor}}=1$. The resulting binary $464\times
400$ matrices are then scaled by 200 and converted into a grayscale image. The
grayscale image is then mapped down to BEV using a homography matrix and is
used as a cost map to evaluate trajectories.

Figure 6: Indoor guiding work experiments (a) The bird-eye view and the cost
map show how the robot recognize the environment and convert the view to the
cost map. Among the 40 path candidates (green), the path with minimal cost is
selected (red). (b) The right image shows how the pedestrian is represented in
the cost map. (c) Summer uses a side-view camera to secure the enough
traversable space for the handler. When upcoming obstacles are detected, which
is shown as the white area under the red cost box in the cost map, Summer
makes side stepping until the obstacle is cleared. (d) (e) Our method can find
a path as long as the side walk is properly detected. Therefore, guiding work
is robust to the situations where unfamiliar objects exist (e.g., construction
site) (f) Summer can navigate through narrow path with
$1.5~{}$\mathrm{m}\text{/}$$ width.

###  III-B Trajectory adjustment, emergency stop, and path clearance

We augmented the trajectory roll-out algorithm by simultaneously rolling out
parallel trajectories, thus evaluating an extended trajectory space that takes
into account Summer’s body width as well as the handler’s position.
Furthermore, we implemented an emergency stop mechanism in the form of a
visual cost box located directly in front of Summer in it’s BEV local cost
map. If an obstacle comes too close into proximity or if navigation itself
chose the wrong path and was heading straight into an obstacle, then the
segmented object or non-traversable path would enter the visual cost box, and
based on a cost threshold, the forward velocity command will halt, preventing
Summer and the handler from crashing into the obstacle. An additional cost box
is instantiated in Summer’s left BEV local cost map. The logic of the
mechanism is identical, however, this box’s function is to instruct Summer to
step to the right whenever there is not enough space on it’s left side for the
handler to traverse.

###  III-C Handler’s direction and speed control

When the up button attached to the handle is pressed, a right turn command is
issued, and this influences the trajectory evaluation algorithm by reducing
the costs assigned to the paths that lead to the right side, thereby biasing
Summer to turn right. The same mechanism applies to the left when the down
button is pressed. Moreover, we implemented a speed control mechanism, where
if the up button is pressed twice, Summer’s forward velocity increases,
enabling the guided unit to traverse at a faster pace. The same mechanism
applies to slowing down if the down button is pressed twice.

##  IV Experimental Results

To demonstrate the robustness of the proposed local path planner, we performed
blindfold guiding work in two different indoor environments, which differ in
basic layout, flooring, and wall texture. In the first environment, the total
length of the navigation task is $105~{}$\mathrm{m}\text{/}$$, and in the
second environment, the total length of the navigation task is
$90~{}$\mathrm{m}\text{/}$$. In both environments, the task is to traverse
through a hallway while avoiding collision with static and dynamic obstacles.
Obstacles included in the experiments are card boxes, blue mats, trash cans, a
wet sign, and a moving person. Both environments contain turns where the
handler needs to issue directional commands and a long hallway where the path
is cluttered with obstacles as close as $2~{}$\mathrm{m}\text{/}$$ apart.
During the experiment, a sighted person is blindfolded and acts as a handler.
The handler is situated on the left side of Summer to demonstrate that the
handle can be easily adjusted for handedness preferences. An operator will be
following the handler while carrying a GPU-containing laptop. In real guiding
work, a handler gives direction cues, but in our experiment, the operator
provides signals on when to give directional cues to the handler because the
blindfolded handler is not well accustomed to the layout of the test
environments.

###  IV-A Obstacle Avoidance and Command Following

In both environments, Summer and the handler successfully complete the task
without a third person’s intervention except for the operator’s directional
cues. During the tests, Summer smoothly walks around obstacles and keeps
itself and the handler from collision. In cases where the turn is narrow,
Summer stops and relies on its left visual cost map to adjust its position
such that the handler has enough traversable space. In walking pedestrian
avoidance, Summer leverages its high-frequency planner and responsibly swerves
around the incoming person. Because the direction command does not override
but only biases the path selection algorithm, the timing of the command does
not need to be precise. In our experiments, as long as the blindfolded handler
issued a direction cue before the turn, Summer was able to follow the
direction issued by the handler.

###  IV-B Robust Behavior in Unplanned Scenarios

Several unplanned scenarios occurred during the experimentation process.
Construction took place in one experimental condition that drastically changed
the appearance of the hallway and reduced the width of the traversable area.
Despite the appearance change, the reduction in traversable area, vaguely
identified objects, Summer finds a traversable path as long as the side walk
is properly detected, which is true most cases. Another unplanned scenario
occurred when a large trash carrier was placed on the planned route, blocking
a significant portion of the traversable area. Despite this, Summer was able
to successfully navigate around the obstacle.

##  V Conclusions

In this letter, we present a safe and resource-light guide dog robot that is
designed based on insights gathered from qualitative research. We utilize two
built-in RGB cameras on the Go1 robot and an off-the-shelf semantic
segmentation network to establish a local path planner. We implement
navigational mechanisms that allows the robot to stop during emergencies and
avoid obstacles while accounting for the robot’s body and position of the
human handler. Custom harness handle interface is designed and fabricated for
the interactive guiding work. Finally, we demonstrate the robustness of our
method in two different indoor environments.

## ACKNOWLEDGMENT

The study materials used in this study has been reviewed and approved by the
University of Massachusetts Amherst IRB, Federal Wide Assurance # 00003909
(protocol ID: 3124). We acknowledge Unitree Robotics for providing a Go1 robot
featured with low-level controller accessible software and for technical
support to enable this study. We also acknowledge Ruffwear for providing the
initial harness design suitable for our application.

## References

  * [1] Blindness and vision impairment. World Health Organization. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment
  * [2] R. J.-T. Miner, “The experience of living with and using a dog guide,” _RE: view_ , vol. 32, no. 4, p. 183, 2001.
  * [3] L. Whitmarsh, “The benefits of guide dog ownership,” _Visual impairment research_ , vol. 7, no. 1, pp. 27–42, 2005.
  * [4] C. Wiggett-Barnard and H. Steel, “The experience of owning a guide dog,” _Disability and Rehabilitation_ , vol. 30, no. 14, pp. 1014–1026, 2008.
  * [5] Faqs. Guiding Eyes for the Blind. [Online]. Available: https://www.guidingeyes.org/about/faqs/
  * [6] M. A. Hersh and M. A. Johnson, “A robotic guide for blind people. part 1. a multi-national survey of the attitudes, requirements and preferences of potential end-users,” _Applied Bionics and Biomechanics_ , vol. 7, no. 4, pp. 277–288, 2010.
  * [7] S. Tachi and K. Komoriya, “Guide dog robot,” _The Robotics Research 2 (The Second International Symposium 1984)_ , vol. 2, pp. 333–349, 1985.
  * [8] Guide dog robot (meldog). Tachi Laboratory, The University of Tokyo. [Online]. Available: https://tachilab.org/en/projects/meldog.html
  * [9] M. H. A. Wahab, A. A. Talib, H. A. Kadir, A. Johari, A. Noraziah, R. M. Sidek, and A. A. Mutalib, “Smart cane: Assistive cane for visually-impaired people,” _CoRR_ , vol. abs/1110.5156, 2011. [Online]. Available: http://arxiv.org/abs/1110.5156
  * [10] P. Slade, A. Tambe, and M. J. Kochenderfer, “Multimodal sensing and intuitive steering assistance improve navigation and mobility for people with impaired vision,” _Science Robotics_ , vol. 6, no. 59, p. eabg6594, 2021.
  * [11] E. J. A. Prada and L. M. S. Forero, “A belt-like assistive device for visually impaired people: Toward a more collaborative approach,” _Cogent Engineering_ , vol. 9, no. 1, p. 2048440, 2022.
  * [12] N. Hirose, A. Sadeghian, M. Vázquez, P. Goebel, and S. Savarese, “Gonet: A semi-supervised deep learning approach for traversability estimation,” _CoRR_ , vol. abs/1803.03254, 2018. [Online]. Available: http://arxiv.org/abs/1803.03254
  * [13] S. S. Bhatlawande, J. Mukhopadhyay, and M. Mahadevappa, “Ultrasonic spectacles and waist-belt for visually impaired and blind person,” in _2012 National Conference on Communications (NCC)_ , 2012, pp. 1–4.
  * [14] A. Xiao, W. Tong, L. Yang, J. Zeng, Z. Li, and K. Sreenath, “Robotic guide dog: Leading a human with leash-guided hybrid physical interaction,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 11 470–11 476.
  * [15] Y. Chen, Z. Xu, Z. Jian, G. Tang, Y. Yangli, A. Xiao, X. Wang, and B. Liang, “Quadruped guidance robot for the visually impaired: A comfort-based approach,” _arXiv preprint arXiv:2203.03927_ , 2022.
  * [16] K. A. Hamed, V. R. Kamidi, W.-L. Ma, A. Leonessa, and A. D. Ames, “Hierarchical and safe motion control for cooperative locomotion of robotic guide dogs and humans: A hybrid systems approach,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 1, pp. 56–63, 2019.
  * [17] J. Guerreiro, D. Sato, S. Asakawa, H. Dong, K. M. Kitani, and C. Asakawa, “Cabot: Designing and evaluating an autonomous navigation robot for blind people,” in _The 21st International ACM SIGACCESS conference on computers and accessibility_ , 2019, pp. 68–82.
  * [18] M. Tucker, _The Eyes that lead_. Howell, 1984.
  * [19] W. R. Wiener, R. L. Welsh, and B. B. Blasch, _Foundations of orientation and mobility_. American Foundation for the Blind, 2010, vol. 1.
  * [20] C. Bane, _Forward Together_. Independently published, 2020.
  * [21] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ADE20K dataset,” _CoRR_ , vol. abs/1608.05442, 2016. [Online]. Available: http://arxiv.org/abs/1608.05442
  * [22] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive mapping and planning for visual navigation,” _CoRR_ , vol. abs/1702.03920, 2017. [Online]. Available: http://arxiv.org/abs/1702.03920
  * [23] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” 2016. [Online]. Available: https://arxiv.org/abs/1609.05143
  * [24] W. Kim and J. Seok, “Indoor semantic segmentation for robot navigating on mobile,” in _2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN)_ , 2018, pp. 22–25.
  * [25] Y. Yeboah, C. Yanguang, W. Wu, and Z. Farisi, “Semantic scene segmentation for indoor robot navigation via deep learning,” in _Proceedings of the 3rd International Conference on Robotics, Control and Automation_ , ser. ICRCA ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 112–118. [Online]. Available: https://doi.org/10.1145/3265639.3265671
  * [26] M. Adachi, S. Shatari, and R. Miyamoto, “Visual navigation using a webcam based on semantic segmentation for indoor robots,” in _2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)_, 2019, pp. 15–21.
  * [27] D. Teso-Fz-Betoño, E. Zulueta, A. Sánchez-Chica, U. Fernandez-Gamiz, and A. Saenz-Aguirre, “Semantic segmentation to develop an indoor navigation system for an autonomous mobile robot,” _Mathematics_ , vol. 8, p. 855, 05 2020.
  * [28] N. A. Giudice and G. E. Legge, “Blind navigation and the role of technology,” _The engineering handbook of smart technology for aging, disability, and independence_ , vol. 8, pp. 479–500, 2008.
  * [29] A. D. P. dos Santos, F. O. Medola, M. J. Cinelli, A. R. Garcia Ramirez, and F. E. Sandnes, “Are electronic white canes better than traditional canes? a comparative study with blind and blindfolded participants,” _Universal Access in the Information Society_ , vol. 20, no. 1, pp. 93–103, 2021.
  * [30] Stanford Artificial Intelligence Laboratory et al., “Robotic operating system.” [Online]. Available: https://www.ros.org
  * [31] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” _CoRR_ , vol. abs/1801.04381, 2018. [Online]. Available: http://arxiv.org/abs/1801.04381