# Heuristic-free Optimization of Force-Controlled Robot Search Strategies in
Stochastic Environments

Benjamin Alt1,2, Darko Katic1, Rainer Jäkel1 and Michael Beetz2 This work was
supported by the German Federal Ministry of Education and Research under the
grant 01DR19001B.1ArtiMinds Robotics, Karlsruhe, Germany
<EMAIL_ADDRESS>for Artificial Intelligence, University
of Bremen, Germany

###### Abstract

In both industrial and service domains, a central benefit of the use of robots
is their ability to quickly and reliably execute repetitive tasks. However,
even relatively simple peg-in-hole tasks are typically subject to stochastic
variations, requiring search motions to find relevant features such as holes.
While search improves robustness, it comes at the cost of increased runtime:
More exhaustive search will maximize the probability of successfully executing
a given task, but will significantly delay any downstream tasks. This trade-
off is typically resolved by human experts according to simple heuristics,
which are rarely optimal. This paper introduces an automatic, data-driven and
heuristic-free approach to optimize robot search strategies. By training a
neural model of the search strategy on a large set of simulated stochastic
environments, conditioning it on few real-world examples and inverting the
model, we can infer search strategies which adapt to the time-variant
characteristics of the underlying probability distributions, while requiring
very few real-world measurements. We evaluate our approach on two different
industrial robots in the context of spiral and probe search for THT
electronics assembly.111See github.com/benjaminalt/dpse for code and data.

©2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Despite years of research, manipulating objects whose pose can only be
determined with uncertainty is still very challenging for robots. In service
robotics, examples include the manipulation of occluded objects or imprecise
pose estimation due to noisy perception. Industrial applications often require
robots to manipulate objects whose pose in the workspace is known only within
given tolerances and whose precise pose varies stochastically between cycles.
In the context of electronics assembly, for example, the poor positioning
accuracy of conveyor belts, manufacturing tolerances of parts from different
suppliers or wear and tear of materials make it impossible to precisely know
the pose of connectors or holes during offline programming [1]. Force-
controlled search strategies, which probe the environment with a defined
series of motions until a change in forces is detected, are a commonly used
method for resolving such uncertainties [2]. While greatly increasing the
robustness of robot tasks, however, force-controlled search comes at the cost
of increased time required to complete the task. Determining an optimal set of
search motion parameters, which maximizes robustness while keeping execution
times minimal, currently requires lengthy parameter tuning by human robot
programmers. We posit that a method for the automatic parameterization of
search strategies must meet four crucial requirements to be applicable for
most real-world use cases:

  1. 1.

Heuristic-freeness: It must make no prior assumptions about the underlying
stochastic processes.

  2. 2.

Data efficiency: It must require only unsupervised or self-supervised data.
The amount of real-world training data must be small and data collection must
not interfere with the completion of regular tasks, such as the ongoing
operation of a production line.

  3. 3.

Handling nonstationary processes: It must be robust to nonstationary noise
processes whose characteristics change over time.

  4. 4.

Black-box optimization: It must be agnostic with respect to the concrete robot
program representation to ensure cross-domain applicability and compatibility
with various robot frameworks.

Figure 1:  Differentiable Programming in Stochastic Environments
($\partial$PSE) leverages transfer learning and differentiable robot program
representations to optimize search motions in the presence of stochastic
noise.

In this paper, we introduce Differentiable Programming in Stochastic
Environments ($\partial$PSE), a heuristic-free, data-efficient black-box
approach to optimizing robot search motions even in the face of nonstationary
stochastic processes.

Our contribution is threefold: (1) A method for learning predictive models of
robot skills which capture the stochastic characteristics of the environments
they are executed in. To that end, we present a transfer-learning based
approach to efficiently condition such models on the stochastic environment at
hand. By pre-training in simulated environments and fine-tuning on a small
dataset of real-world action executions, the learned models predict an
expected robot trajectory given the robot skill’s parameters (e.g. the search
pattern of a probe search) and the (learned) probability distributions
governing the environment. We show that continuous fine-tuning can keep the
learned models synchronized with time-varying, nonstationary stochastic
environments.
(2) A method to optimize motion parameters to ensure minimal search time and
maximal roboustness in the face of nonstationary stochastic environments. We
embed these environment-conditioned models in a state-of-the-art framework for
differentiable robot programming [3], in which robot programs can be expressed
as differentiable computation graphs. A gradient-based optimizer jointly
optimizes the program’s input parameters with respect to relevant metrics such
as cycle time or failure rate.
(3) Experimental validation of our method in two real-world mechanical and
electronics assembly tasks for both stationary and nonstationary environments,
and empirical comparison against several meta-learning approaches.

Figure 2: Overview of our proposed approach. $\partial$PSE enables the data-
efficient, heuristic-free optimization of arbitrary parametric search
strategies by pretraining a predictive model (“shadow program”) of the search
strategy on a large simulated dataset (1), finetuning it on a few samples of
the concrete real-world environment (2), and inferring optimal search
parameters via SPI (3). By repeatedly finetuning on data collected at test
time (4), nonstationary noise processes can be addressed.

##  II Related Work

##### Robot program parameter optimization

Several black-box optimizers for robot program parameters have been proposed,
most often relying on zero-order methods such as evolutionary algorithms [4,
5], whose requirement of repeatedly executing the robot program for every
optimization renders them impractical for real-world application. Bayesian
optimization [6, 7, 8] avoids this, but requires the goal function to be known
at data collection time, is difficult to extend to nonstationary processes [9,
10] and cannot exploit gradient information. First-order neural network
(NN)-based optimizers have been proposed [11, 12], but remain impractical in
productive environments due to their data requirements and reliance on
supervised or reinforcement learning.

#####  Differentiable programming ($\partial$P)

In $\partial$P, programs are composed of differentiable operations, which
permits the first-order optimization of a program’s parameters with respect to
its outputs [13]. PDP [14] is a $\partial$P framework for differentiable
motion control. However, PDP is not black-box, cannot represent hierarchical
programs and the learned parameters become outdated in nonstationary
environments. SPI [3] is a black-box $\partial$P framework for representing
robot programs, which, like Bayesian optimization, relies on surrogate models
learned via unsupervised learning. It exploits gradients for efficient
optimization, but assumes stationarity and requires substantial amounts of
real-world data. We leverage SPI for optimization, but avoid its stationarity
assumption and drastically reduce the amount of required real-world data.

#####  Transfer learning (TL)

TL is concerned with improving the performance of systems trained on some
source task on different, but related target tasks [15]. In sequential TL
schemes, a NN first learns a shared set of features on a large source dataset
before fine-tuning on a much smaller target dataset [16]. In natural language
processing [17, 18, 19, 20, 21] and computer vision [22, 23, 24], unsupervised
pretraining outperforms learning from scratch [25, 26], particularly for small
target datasets [27]. In robotics, sequential TL has been applied for inter-
robot learning [28], Sim2Real adaption [29] or few-shot learning [30, 31]. We
show that in the context of parameter optimization, sequential TL not only
greatly reduces the amount of required real-world data, but also that
continuous fine-tuning as a form of “lifelong learning” enables parameter
optimization in nonstationary stochastic environments.

##### Meta-learning

Meta-learning is concerned with efficiently learning to solve unseen tasks and
has been applied with particular success in few-shot problems [32, 33]. While
some approaches such as Meta Networks [34] or Prototypical Networks [35]
propose specific architectures, model-agnostic methods such as MAML [36] or
Reptile [37] optimize arbitrary networks by learning initial network
parameters particularly conducive to test-time fine-tuning. We provide an
evaluation of several model-agnostic meta-learners as alternatives to our
proposed TL-based approach in the context of search strategy parameter
optimization.

##  III Parameter Optimization in Stochastic Environments

###  III-A Definitions and Notation

As an example for an industrial assembly task subject to stochastic
variations, we consider a peg-in-hole task, where the actual pose of the hole
can be described by a random variable $H_{t}$ drawn from the (possibly
nonstationary) stochastic process $\\{H_{t}\\}$. Because the actual hole pose
at time $t$ is uncertain, an industrial robot will require the use of search
strategies to perform this task reliably. We define a search strategy as a
robot program which accepts a vector of parameters $x\in\mathcal{X}$ and
executes a sequence of end effector motions until a hole has been found or
some other termination criterion was reached. Intuitively, a robot program can
be defined as a function $\pi$ mapping some program parameters $x$ to the tool
center point (TCP) trajectory $y$ resulting from executing it. However,
because this trajectory depends on the uncertain pose of the hole, we model it
instead as a random variable $Y_{t}$, which takes values $y\in\mathcal{Y}$
according to $P(Y_{t}|\pi,x,H_{t})$, the probability distribution over
trajectories resulting from the execution of program $\pi$ at time $t$ with
inputs $x$ and hole distribution $H_{t}$.

###  III-B Parameter Optimization via Differentiable Programming

In the context of robotics, $\partial$P approaches center around learning or
constructing a differentiable representation of $\pi$, which permits to
backpropagate losses over the program’s outputs to its inputs $x$. SPI [3]
approximates $P(Y_{t}|\pi,x)$ by training an auxiliary, differentiable graph
of NNs (“shadow program”, $\hat{\pi}$) via unsupervised learning from past
executions of $\pi$. It then optimizes the program’s input parameters by
inverting the shadow program via gradient descent in the shadow program’s
input space. This optimization can be performed with respect to some user-
defined function $\mathcal{L}$ of the shadow program’s output (the expected
trajectory $\hat{y}$), such as cycle time. SPI’s compatibility with most
existing robot program or skill frameworks and exclusive reliance on
unsupervised learning are highly advantageous in industrial robot
applications. SPI is a heuristic-free and black-box optimizer according to the
criteria outlined in section I. In its model training phase, however, SPI
requires training tuples $(x_{i},y_{i})$ which cover the region of the input
space $\mathcal{X}$ relevant for optimization. This effectively requires
sampling the input space during data collection, which is impractical in a
running production line. Moreover, if features of the environment such as the
poses of holes follow nonstationary stochastic processes, the learned shadow
program immediately becomes outdated as the probability distributions change
over time.

###  III-C Environment-Conditioned Skill Models

Transfer-learning methods are capable of efficiently adapting learned models
to changing data distributions [38]. By treating the model learning phase of
SPI as a TL problem and applying a pretraining and finetuning technique, we
obtain a heuristic-free black-box differentiable program representation which
supports the optimization of search motions in the face of nonstationary noise
processes with high data efficiency.
We first define a task
$\mathcal{T}=\big{\\{}\\{H_{t}\\},P(Y_{t}|\pi,x,H_{t})\big{\\}}$: For a
stochastic process $\\{H_{t}\\}$ generating hole distributions, SPI must learn
the probability distribution over the resulting trajectories when executing
program $\pi$ with inputs $x$ in this environment. The corresponding task
dataset $D_{\mathcal{T}_{0}}=\\{(x_{0},y_{0}),...,(x_{N},y_{N})\\}$ for a task
$\mathcal{T}_{0}$ consists of input-trajectory pairs collected by executing
$\pi$ $N$ times, each time sampling a new hole distribution $H_{n}$ from
$\\{H_{t}\\}_{0}$. Following a pretraining/finetuning regime, we propose to
pretrain SPI’s shadow program on a large source dataset
$D_{S}=\bigcup_{m=0}^{M}D_{\mathcal{T}_{m}}$, the collection of $M$ task
datasets, each for a different hole-distribution-generating process
$\\{H_{t}\\}_{m}$ (see fig. 2 (1)). The shadow model can then be finetuned on
the much smaller target dataset $D_{T}=\\{D_{\mathcal{T}_{curr}}\\}$ (see fig.
2 (2)), containing only data from the current task $\mathcal{T}_{curr}$ \- in
the case of a running production line, the input-trajectory pairs from the
last $N$ executions of the program. The core intuition is to learn a prior
over many possible environments (hole-distribution-generating processes,
$\\{H_{t}\\}_{m}$) offline and finetune on the concrete process at hand.

####  III-C1 Data efficiency

We found that for most peg-in-hole tasks under uncertainty, only $N$=128
samples per task are required to effectively finetune on the concrete task.
Moreover, we show that pretraining on a large ($M$=1000, $N$=128) source
dataset collected in a simulated environment and finetuning on one single
real-world target task bridges the sim-to-real-gap sufficiently for SPI to
perform meaningful parameter optimization in the real-world environment. Both
spiral and probe search strategies considered in experiments IV-A and IV-B
involve force-sensitive interactions with the environment; in both cases,
pretraining exclusively on simulated data sufficed to find near-optimal
parameterizations in the real world. This efficiency can be attributed to the
fact that after pretraining, the shadow program can rely on useful priors at
two levels: The differentiable motion planner at the heart of the SPI shadow
program architecture provides a strong prior for the expected trajectory in an
ideal environment, and the new pretraining step trains a prior over a wide
range of possible environments, providing a very good initialization of the
shadow program for finetuning.

####  III-C2 Passive data collection

As a corollary of these strong priors, the proposed pretraining/finetuning
regime avoids the need to sample the parameter space in the real production
line. Because the relationship between program inputs and outputs is learned
across a wide variety of environments and for inputs $x$ sampled from the
entire parameter space, the finetuning dataset $D_{T}$ does not require such
diversity anymore. In fact, we find that SPI still performs meaningful
parameter optimization even if all finetuning examples in $D_{T}$ contain the
same inputs, i.e.
$D_{T}=\\{D_{\mathcal{T}_{curr}}\\}=\\{(x_{0},y_{0}),...,(x_{0},y_{N})\\}$.
Consequently, the required 128 real-world finetuning examples can be collected
in a completely passive manner by simply collecting robot trajectories from
the running assembly line, without needing to sample a diverse set of program
parameters and potentially disrupting production.

####  III-C3 Continuous learning and optimization for nonstationary processes

Nonstationary processes require continuous re-optimization of program
parameters, ideally at every timestep $t$. With the proposed
pretraining/finetuning regime, this becomes straightforward: After finetuning
once, running SPI’s optimization step and updating the program with the
optimized parameters, the next trajectory at timestep $t+1$ can be observed
and added to the finetuning dataset. SPI’s shadow program can then be
finetuned again and the optimal parameters for timestep $t+2$ can be computed.
Repeating this cycle of finetuning the shadow program and optimizing program
parameters ensures that the shadow program’s estimation of
$P(Y_{t}|\pi,x,H_{t})$ does not deteriorate as $t$ increases. We implement
$D_{T}=\\{D_{\mathcal{T}_{curr}}\\}$ as a ring buffer of fixed size $N$=128 to
ensure that outdated data is eventually removed from the finetuning dataset.
To avoid catastrophic forgetting, we finetune the original pretrained model at
each iteration instead of repeatedly finetuning the finetuned model.

(a) a (b) b (c) c

Figure 3: Results for experiment IV-A (spiral search, stationary): Experiment
setup (a) and examples for an optimized search pattern generated by
$\partial$PSE (b). Heuristic-free $\partial$PSE outperforms the human and
NSGA-II baselines and is competitive with an application-specific heuristic
(c).

##  IV Experiments

We validate our approach by comparing its performance against state-of-the-art
baselines on two real-world assembly tasks involving spiral and probe search
strategies, respectively. We also evaluate its performance on several
simulated nonstationary processes and provide an empirical analysis of our TL
scheme with respect to possible meta-learning based alternatives.

###  IV-A Spiral Search (stationary)

We consider the mechanical assembly task of inserting a tightly-fitting
cylinder into a hydraulic valve (see fig. 3). To simulate realistic process
variances, the position of the valve body in the XY-plane is sampled from a
bivariate Gaussian mixture with six components, forming the stationary process
$\\{H_{t}\\}=H_{t}=\sum_{i=1}^{6}w_{i}\mathcal{N}(\bm{\mu}_{i},\bm{\Sigma}_{i})$.
To successfully perform the insertion, the robot follows a spiral search
strategy, maintaining a constant pushing force against the surface until
dropping into the hole. We seek to optimize the parameters of the search
strategy (spiral position, orientation, extents, number of windings, velocity
and acceleration) to minimize a linear combination of cycle time
$\mathcal{L}_{cycle}(y)$ and failure probability $\mathcal{L}_{fail}(y)$, both
of which can be expressed in terms of the expected trajectory $y$ output by
$\partial$PSE’s shadow program. $\partial$PSE is pretrained on a source
dataset collected by simulating spiral searches on $M_{train}=1000$ different
Gaussian mixtures with $N_{train}=128$ sampled hole poses each, and finetuned
on $N_{test}=128$ training examples for the $M_{test}=1$ concrete hole
distributions at hand.***Simulations were conducted in a lightweight scripted
environment modelling Newtonian physics, damping and friction. We compare
against the following baselines:

  1. 1.

A fixed parameterization by a human expert

  2. 2.

A near-optimal Principal Component Analysis (PCA)-based heuristic fitting the
spiral orientation and extents to the ground-truth valve body poses

  3. 3.

$\mu+\lambda$ nondominated sorting genetic algorithm (NSGA-II) [39] with
$\mu=\lambda=25$

We repeat the experiment for 10 different distributions $\\{H_{t}\\}$ in a
purely simulated environment, and for 6 different $\\{H_{t}\\}$ on a real UR5
robot.

The results are shown in figure 3. $\partial$PSE provides near-perfect success
rates and significantly outperforms NSGA-II and the human expert, effectively
replicating the results in [3] with significantly fewer real-world training
examples. With respect to cycle time, $\partial$PSE outperforms both the human
and NSGA-II on the real task. The poor real-world performance of NSGA-II
illustrates the fundamental disadvantage of most heuristic-free, zero-order
black-box optimizers, which they require a large amount of program executions
to converge on a good solution. Here, was allowed 250 executions per test
distribution, nearly twice $\partial$PSE’s real-world data requirement of 128,
suggesting that $\partial$PSE owes its data efficiency at least partly to its
exploitation of gradient information.

###  IV-B Probe Search (stationary)

To evaluate $\partial$PSE for a more complex search strategy, we consider the
assembly of through-hole technology (THT) electronics components, where the
pose of mounting holes on a printed circuit board (PCB) is subject to
stationary noise $\\{H_{t}\\}$ with an a priori unknown distribution, e.g.
from imprecise positioning on a conveyor belt. To avoid scratching the surface
of the PCB, probe search repeatedly touches the surface according to a
predefined search pattern until the THT component’s pins drop into the holes
(see fig. 2 (4)). The search pattern is typically defined as a regular grid,
though the search strategy can be significantly optimized by tailoring the
pattern to the underlying noise distribution. Finding the optimal pattern
corresponds to a high-dimensional optimization problem over the 32-dimensional
input vector of the probe search strategy, describing the 16 touch points in
the XY-plane. We use $\partial$PSE to automatically optimize the touch points
without assuming knowledge about the hole distribution. $\partial$PSE is
trained on $M_{train}=1000$ simulated training tasks (hole distributions) with
$N_{train}=128$ probe searches each and finetuned on $N_{test}=128$ probe
searches over $M_{test}=1$ concrete hole distribution. We compare against the
following baselines:

  1. 1.

A fixed 4x4 grid covering the complete search region

  2. 2.

A heuristic tailored specifically to probe search, which fits a 16-mode
Gaussian Mixture Model (GMM) to the last points of those trajectories in the
finetuning dataset where the search was successful

  3. 3.

$\mu+\lambda$ NSGA-II with $\mu=\lambda=100$ (sim) or $\mu=\lambda=30$ (real)

The experiment is repeated for 10 different multimodal Gaussians $\\{H_{t}\\}$
in a simulated environment, and for 6 additional distributions on a real Fanuc
industrial robot. Both $\partial$PSE and NSGA-II minimize
$\mathcal{L}_{fail}(y)$, the probability of search failure. We also assess the
effects of adding additional regularization to $\partial$PSE, penalizing
either the L1 distance from the initial parameterization
($\mathcal{L}_{init}$, a heuristic-free regularizer) or the smallest distance
between any two optimized touch points ($\mathcal{L}_{cdist}$, a heuristic
regularizer specific to probe search).

(a) a (b) b (c) c (d) d

Figure 4: Results for experiment IV-B (probe search, stationary): Experiment
setup (a) and examples for optimized search patterns generated by
$\partial$PSE with the $\mathcal{L}_{init}$ (b) and $\mathcal{L}_{cdist}$ (c)
regularizers. Heuristic-free $\partial$PSE outperforms the human and NSGA-II
baselines and is competitive with an application-specific heuristic (d).

The results are summarized in figure 4. With both regularizers, the search
patterns produced by $\partial$PSE reflect the underlying hole distribution
without overfitting to its modes (see fig. 4b and 4c). Quantitatively, both
variants of $\partial$PSE outperform the human and NSGA-II baselines in both
simulated and real environments. While the application-specific heuristic is
nearly optimal in the simulated scenario, $\partial$PSE with
$\mathcal{L}_{cdist}$ regularization is competitive with it in the real-world
environment, while avoiding any assumptions on the underlying hole
distributions. The data efficiency of $\partial$PSE is particularly apparent
when compared to NSGA-II: In the real-world scenario, NSGA-II yielded subpar
results despite being allowed 300 executions, more than twice $\partial$PSE’s
128. $\partial$PSE achieves this efficiency by transferring knowledge learned
from simulated data during pretraining, and by exploiting gradient information
during optimization.

###  IV-C Probe Search (nonstationary)

In a further series of experiments, we analyze the capacity of $\partial$PSE
to optimize search strategies with respect to nonstationary processes. We
consider the same task as in experiment IV-B, but omit the stationarity
assumption on $\\{H_{t}\\}$. We evaluate $\partial$PSE in a simulated
environment on three different nonstationary processes:

  1. 1.

Linear drift, where the modes of $\\{H_{t}\\}$ are translated by a constant
offset at each timestep $t$

  2. 2.

Brownian motion, where the modes of $\\{H_{t}\\}$ are translated by an offset
sampled from a bivariate unimodal Gaussian at each timestep $t$

  3. 3.

Shift, where the modes of $\\{H_{t}\\}$ are translated by a uniformly random
offset with probability $p_{shift}=0.05$ at each timestep $t$

The results are summarized in table I. Both unregularized and regularized
$\partial$PSE increase the probability of success by over 20% over short time
horizons, confirming the results from experiment IV-B even as the underlying
distributions change over time. For long-horizon processes, the benefits of
$\partial$PSE are more pronounced, illustrating the capacity of $\partial$PSE
to continuously produce suitable parameterizations over longer timescales.

Figure 5: Optimization of a probe search strategy ($\partial$PSE, $\mathcal{L}_{init}$ regularization) in the presence of nonstationary noise processes (top: drift, bottom: brownian motion). For both noise processes, the search pattern (blue) follows the hole distribution, while respecting the constraints imposed by the regularizer. TABLE I: Results of experiment IV-C: Failure rates for different parameter optimizers and stochastic processes. $\partial$PSE with the $\mathcal{L}_{cdist}$ regularizer reduces failures by up to 53% over 100 timesteps. | Drift | Brownian | Shift
---|---|---|---
None | 0.679 | 0.679 | 0.600
$\partial$PSE ($\mathcal{L}_{fail}$) | 0.458 | 0.605 | 0.472
$\partial$PSE ($\mathcal{L}_{fail}+\mathcal{L}_{init}$) | 0.556 | 0.616 | 0.406
$\partial$PSE ($\mathcal{L}_{fail}+\mathcal{L}_{cdist}$) | 0.318 | 0.494 | 0.339

###  IV-D Comparison with Meta-Learning Approaches

TABLE II: Results of experiment IV-D: Comparison of pretraining/fine-tuning against meta-learning alternatives for training $\partial$PSE’s shadow program. Learning algorithm | Traj. loss | Success acc.
---|---|---
Pretrain | 0.80 | 0.72
Pretrain + Dropout | 0.83 | 0.71
MAML (5) + Dropout | 0.94 | 0.64
MAML (5) | 0.95 | 0.64
FOMAML (5) | 0.96 | 0.62
FOMAML (128) | 0.99 | 0.58
Reptile (128) | 1.35 | 0.56

Meta-learning, learning to efficiently learn from few training examples, is a
powerful paradigm fundamentally related to $\partial$PSE’s transfer-learning
scheme and has become state of the art for solving few-shot learning problems.
We find that in the context of gradient-based parameter optimization, simple
pretraining and fine-tuning outperforms various state-of-the-art meta-learning
approaches. We substantiate our finding by comparing $\partial$PSE as proposed
in III-C with variants of $\partial$PSE, where the pretraining and fine-tuning
of the shadow program is replaced by the meta-learning and adaptation schemes
of first- and second-order model-agnostic meta-learning (MAML) [36] and
Reptile [37]. Comparing against model-agnostic approaches ensures
comparability, as the network architectures as well as the training data
remain unchanged. Due to the high memory requirements of second-order MAML, it
could only be evaluated for meta-test sets of size $N=5$ (vs. $\partial$PSE’s
$N=128$). We evaluate first-order MAML (FOMAML) for both $N=5$ and $N=128$ and
Reptile for $N=128$. As shown in table II, the proposed pretraining and fine-
tuning scheme results in a better predictor than all tested meta-learning
alternatives. The poor performance of second-order MAML is likely due to its
limitation to only 5 adaptation examples, which do not suffice to learn
meaningful characteristics of the underlying distributions. However, the poor
performance of FOMAML on the larger meta-test set indicates that the benefits
of meta-learning diminish as the number of meta-test examples increases. This
suggests that simpler, much less computationally intensive transfer-learning
based approaches can compete with and even outperform meta-learning in the
small-data regime (between tens and hundreds of training examples). Our
findings also confirm the intuition of Sun et al. [40] that meta-learning is
most effective for shallow network architectures and requires a large amount
($M\sim 100k$) of meta-training tasks. The chained deep residual GRUs at the
core of SPI for precise trajectory prediction and the limited amount of
pretraining tasks ($M=1000$) likely limit the utility of meta-learning for
gradient-based parameter optimization, at least in conjunction with
$\partial$PSE.

##  V Conclusion and Outlook

In this paper, we propose Differentiable Programming in Stochastic
Environments ($\partial$PSE), a novel approach to optimizing robot search
strategies. By conditioning learned predictive models of robot skills via
transfer learning and embedding them in a differentiable program
representation, we obtain a heuristic-free optimizer for arbitrary search
strategies. By applying our method to two real-world assembly tasks with
stationary process noise and one simulated nonstationary process, we
demonstrated that $\partial$PSE outperforms other heuristic-free optimizers
and is competitive with task-specific, hand-crafted heuristics. $\partial$PSE
permits the effective automation of the labor-intensive optimization phase of
robot workcells, while its extreme data-efficiency as well as its black-box
nature renders it equally suitable for service robotics tasks. Our future work
is focused on extending $\partial$PSE to applications beyond search
strategies, such as force-controlled insertion or grasping. In addition, we
are exploring the inference of task-specific goal functions from human
demonstrations and symbolic knowledge.

## References

  * [1] M. Metzner, S. Leurer, A. Handwerker, E. Karlidag, A. Blank, F. Hefner, and J. Franke, “High-precision assembly of electronic devices with lightweight robots through sensor-guided insertion,” _Procedia CIRP_ , vol. 97, pp. 337–341, Jan. 2021.
  * [2] I. F. Jasim, P. W. Plapper, and H. Voos, “Position Identification in Force-Guided Robotic Peg-in-Hole Assembly Tasks,” _Procedia CIRP_ , vol. 23, pp. 217–222, Jan. 2014.
  * [3] B. Alt, D. Katic, A. K. Bozcuoglu, R. Jäkel, and M. Beetz, “Robot Program Parameter Inference via Differentiable Shadow Program Inversion,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , Xi’an, China, 2021.
  * [4] J. A. Marvel, W. S. Newman, D. P. Gravel, G. Zhang, Jianjun Wang, and T. Fuhlbrigge, “Automated learning for parameter optimization of robotic assembly tasks utilizing genetic algorithms,” in _2008 IEEE International Conference on Robotics and Biomimetics_ , Feb. 2009, pp. 179–184.
  * [5] S. Chernova and M. Veloso, “An evolutionary approach to gait learning for four-legged robots,” in _2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566)_ , vol. 3, Sep. 2004, pp. 2562–2567 vol.3.
  * [6] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth, “Bayesian optimization for learning gaits under uncertainty,” _Ann Math Artif Intell_ , vol. 76, no. 1, pp. 5–23, Feb. 2016.
  * [7] R. Akrour, D. Sorokin, J. Peters, and G. Neumann, “Local Bayesian Optimization of Motor Skills,” in _International Conference on Machine Learning_. PMLR, Jul. 2017, pp. 41–50.
  * [8] F. Berkenkamp, A. Krause, and A. P. Schoellig, “Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Robotics,” _ArXiv160204450 Cs_ , Apr. 2020.
  * [9] J. Snoek, K. Swersky, R. Zemel, and R. Adams, “Input Warping for Bayesian Optimization of Non-Stationary Functions,” in _International Conference on Machine Learning_. PMLR, Jun. 2014, pp. 1674–1682.
  * [10] A. Hebbal, L. Brevault, M. Balesdent, E.-G. Talbi, and N. Melab, “Bayesian Optimization using Deep Gaussian Processes,” _ArXiv190503350 Cs Stat_ , May 2019.
  * [11] Y. Zhou, J. Gao, and T. Asfour, “Movement Primitive Learning and Generalization: Using Mixture Density Networks,” _IEEE Robot. Automat. Mag._ , vol. 27, no. 2, pp. 22–32, Jun. 2020.
  * [12] J. Kulk and J. S. Welsh, “Evaluation of walk optimisation techniques for the NAO robot,” in _2011 11th IEEE-RAS International Conference on Humanoid Robots_ , Oct. 2011, pp. 306–311.
  * [13] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, “Automatic Differentiation in Machine Learning: A Survey,” _J. Mach. Learn. Res._ , vol. 18, no. 153, pp. 1–43, 2018.
  * [14] W. Jin, Z. Wang, Z. Yang, and S. Mou, “Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework,” _ArXiv191212970 Cs Eess Math_ , Jun. 2020.
  * [15] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A Comprehensive Survey on Transfer Learning,” _Proceedings of the IEEE_ , vol. PP, pp. 1–34, Jul. 2020.
  * [16] S. Ruder, “Neural Transfer Learning for Natural Language Processing,” Ph.D. dissertation, National University of Ireland, Feb. 2019.
  * [17] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-Tuning Language Models from Human Preferences,” _ArXiv190908593 Cs Stat_ , Jan. 2020.
  * [18] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to Fine-Tune BERT for Text Classification?” in _Chinese Computational Linguistics_ , ser. Lecture Notes in Computer Science, M. Sun, X. Huang, H. Ji, Z. Liu, and Y. Liu, Eds. Cham: Springer International Publishing, 2019, pp. 194–206.
  * [19] M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines,” in _International Conference on Learning Representations_ , Sep. 2020.
  * [20] Y. Liu, “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” _CoRR_ , vol. abs/1907.11692, 2019.
  * [21] Y. Peng, S. Yan, and Z. Lu, “Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets,” in _Proceedings of the 18th BioNLP Workshop and Shared Task_. Florence, Italy: Association for Computational Linguistics, Aug. 2019, pp. 58–65.
  * [22] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-Trained Image Processing Transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 299–12 310.
  * [23] X. Li, Y. Grandvalet, F. Davoine, J. Cheng, Y. Cui, H. Zhang, S. Belongie, Y.-H. Tsai, and M.-H. Yang, “Transfer learning in computer vision tasks: Remember where you come from,” _Image and Vision Computing_ , vol. 93, p. 103853, Jan. 2020.
  * [24] V. Miglani and M. Bhatia, “Skin Lesion Classification: A Transfer Learning Approach Using EfficientNets,” in _Advanced Machine Learning Technologies and Applications_ , ser. Advances in Intelligent Systems and Computing, A. E. Hassanien, R. Bhatnagar, and A. Darwish, Eds. Singapore: Springer, 2021, pp. 315–324.
  * [25] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why Does Unsupervised Pre-training Help Deep Learning?” _J. Mach. Learn. Res._ , vol. 11, no. 19, pp. 625–660, 2010.
  * [26] K. Zhang and S. Bowman, “Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis,” in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 359–361.
  * [27] S. Wang, M. Khabsa, and H. Ma, “To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks,” _ArXiv200608671 Cs Stat_ , Jun. 2020.
  * [28] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, “RoboNet: Large-Scale Multi-Robot Learning,” _ArXiv191011215 Cs_ , Oct. 2019.
  * [29] R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman, “Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning,” _ArXiv200410190 Cs Stat_ , Jul. 2020\.
  * [30] E. D. Coninck, T. Verbelen, P. V. Molle, P. Simoens, and B. D. IDLab, “Learning to Grasp Arbitrary Household Objects from a Single Demonstration,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Nov. 2019, pp. 2372–2377.
  * [31] L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y. Lin, “Learning to See before Learning to Act: Visual Pre-training for Manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2020, pp. 7286–7293.
  * [32] T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey, “Meta-Learning in Neural Networks: A Survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , pp. 1–1, 2021.
  * [33] A. Goyal and Y. Bengio, “Inductive Biases for Deep Learning of Higher-Level Cognition,” _ArXiv201115091 Cs Stat_ , Feb. 2021.
  * [34] T. Munkhdalai and H. Yu, “Meta Networks,” _ArXiv170300837 Cs Stat_ , Jun. 2017.
  * [35] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical Networks for Few-shot Learning,” _ArXiv170305175 Cs Stat_ , Jun. 2017.
  * [36] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_ , Aug. 2017, pp. 1126–1135.
  * [37] A. Nichol, J. Achiam, and J. Schulman, “On First-Order Meta-Learning Algorithms,” _ArXiv180302999 Cs_ , Oct. 2018.
  * [38] I. Prapas, B. Derakhshan, A. R. Mahdiraji, and V. Markl, “Continuous Training and Deployment of Deep Learning Models,” _Datenbank Spektrum_ , vol. 21, no. 3, pp. 203–212, Nov. 2021.
  * [39] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: NSGA-II,” _IEEE Trans. Evol. Comput._ , vol. 6, no. 2, pp. 182–197, Apr. 2002.
  * [40] Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele, “Meta-Transfer Learning for Few-Shot Learning,” _2019 IEEECVF Conf. Comput. Vis. Pattern Recognit. CVPR_ , 2019.

  *[$\partial$PSE]: Differentiable Programming in Stochastic Environments
  *[SPI]: Shadow Program Inversion
  *[NN]: neural network
  *[$\partial$P]: Differentiable programming
  *[TL]: Transfer learning
  *[TCP]: tool center point
  *[NNs]: neural network
  *[NSGA-II]: nondominated sorting genetic algorithm
  *[PCA]: Principal Component Analysis
  *[THT]: through-hole technology
  *[PCB]: printed circuit board
  *[GMM]: Gaussian Mixture Model
  *[MAML]: model-agnostic meta-learning
  *[FOMAML]: first-order MAML


# ReviewViz: Assisting Developers Perform Empirical Study on Energy
Consumption Related Reviews for Mobile Applications

Mohammad Abdul Hadi  University of British Columbia3187, University
WayKelownaCanada<EMAIL_ADDRESS>and  Fatemeh Hendijani Fard  University of
British Columbia3333, University WayKelownaCanada<EMAIL_ADDRESS>
(2020)

###### Abstract.

Improving energy efficiency of mobile applications is a topic that has gained
a lot of attention recently. It has been addressed in a number of ways such as
identifying energy bugs and developing a catalog of energy patterns. Previous
work shows that users discuss the battery related issues (energy inefficiency
or energy consumption) of the apps in their reviews. However, there is no work
that addresses the automatic extraction of the battery related issues from
users’ feedback.

In this paper, we report on a visualization tool that is developed to
empirically study machine learning algorithms and text features to
automatically identify the energy consumption specific reviews with the
highest accuracy. Other than the common machine learning algorithms, we
utilize deep learning models with different word embeddings to compare the
results. Furthermore, to help the developers extract the main topics that are
discussed in the reviews, two state of the art topic modeling algorithms are
applied. The visualizations of the topics represent the keywords that are
extracted for each topic along with a comparison with the results of string
matching.

The developed web-browser based interactive visualization tool is a novel
framework developed with the intention of giving the app developers insights
about running time and accuracy of machine learning and deep learning models
as well as extracted topics. The tool makes it easier for the developers to
traverse through the extensive result set generated by the text classification
and topic modeling algorithms. The dynamic-data structure used for the tool
stores the baseline-results of the discussed approaches and are updated when
applied on new datasets. The tool is open sourced to replicate the research
results. 111https://github.com/Mohammad-Abdul-
Hadi/Empirical_Study_on_Text_Classification

App review analysis, Energy consumption, Data-visualization, Machine learning,
Neural networks, Topic modeling

††journalyear: 2020††conference: IEEE/ACM 7th International Conference on
Mobile Software Engineering and Systems; October 5–6, 2020; Seoul, Republic of
Korea††booktitle: IEEE/ACM 7th International Conference on Mobile Software
Engineering and Systems (MOBILESoft ’20), October 5–6, 2020, Seoul, Republic
of Korea

Figure 1. Visualization tool for the easy traversal of result-set produced by
the elaborate empirical study

Teaser Figure

##  1\. Introduction

Figure 2. Different Illustrations of the Empirical Data

Improving energy efficiency of mobile applications is a topic that has gained
a lot of attention recently (Cruz and Abreu, 2019). This challenge is
addressed in a number of ways including energy consumption profiling
(Chowdhury et al., 2019), developing a catalog of energy patterns (Cruz and
Abreu, 2019), and investigating users’ reviews discussed on app stores (Wilke
et al., 2013). Previous work shows that users discuss the battery related
issues (energy inefficiency or energy consumption) of the apps in their
reviews (Wilke et al., 2013), (Noei et al., 2019). However, there is no work
that addresses the automatic extraction of the battery related issues from
users’ feedback. Although many works classify the app users’ reviews
automatically (Chen et al., 2014), there is still a need to retrieve reviews
on specific features (Dabrowski et al., 2019).

Considering the importance of energy efficiency in mobile apps (Cruz and
Abreu, 2019) and the fact that it is a main topic discussed by app users (Noei
et al., 2019), we empirically studied 90 different models to extract energy-
related issues from app reviews. These models include different machine
learning (with various feature combinations) and deep learning models. The
usage of deep learning models leveraged with three pre-trained word embeddings
is a first step toward transferring the knowledge for automatic extraction of
specific features from app users’ reviews.

The results of computational analysis can be effectively combined with human
reasoning with the help of powerful visualization and can lead to increased
efficiency and more valuable results (Fayyad, 2001). For the empirical study,
we developed an interactive web based visualization tool that enables the
users compare the accuracy of the results and the computational time of
different algorithms, as well as investigating the main topics in the filtered
reviews. We developed the tool for the usage of app developers in mind, giving
them the flexibility of choosing among different techniques based on their
requirements for running time or accuracy.

In this paper, we explain the architecture and technical details of the
developed tool. The tool is evaluated by 8 participants and is effective in
examining the results set, when exploring a large number of models, feature
combinations, and techniques. The main contributions of our work are empirical
studies of machine learning and deep learning models for automatic extraction
of app-specific features (energy consumption) from user reviews, and
developing a visualization tool for app developers to explore the results.

##  2\. Tool Architecture

For developing the tool, we have assessed different methods (Dirksen, 2017) to
find a convenient and concise way of presenting the generated data so that
developers can compare and utilize the extracted information with maximum
efficiency. The tool contains two main components that are structured based on
two essential purposes. Each component is necessary to accomplish insightful
data interpretation for helping developers to process all the information in a
consistently comparable way. This section discusses both components of the
visualization tool.

###  2.1. Component to Visualize Results of Different Classification
Techniques

Figure 3. Analysis of visualization technique presented in Figure 1 Figure 4.
Different Illustrations (Overall Representation) of the Empirical Data

This component of the tool provides various visualizations: (1) a tree
visualization that enables the users to explore the results by selecting a
model, features of the model, and the evaluation metrics (e.g. accuracy,
running time); (2) various plots such as bar chart and line chart that
provides comparisons of the evaluation metrics for different feature
combinations of the same algorithm (see Fig. 2) and comparison of one or more
evaluation metric among all of the trained models (see Fig. 4).

For the classification of energy-related data, four developers exhaustively
performed 90 combinations of different models and features, recording the
accuracy, F1-score, Precision, Recall, and Runtime information of each model.
In Fig. 1, we presented a comprehensive visualization technique for users’
effortless traversal of the result set. This horizontal interactive
visualization tree helps developers choose their itinerary without being
exposed to the complete information. We have used a JavaScript library named
Data-Driven Document (D3.js), which helps to produce dynamic, interactive data
visualizations in web browsers (Wikipedia contributors, 2020). The reason
behind choosing D3.js is that this library uses a wide variety of pre-built
JavaScript functions to create and manipulate Scalable Vector Graphics (SVG)
objects and apply suitable transitions and dynamic effects to them.
Furthermore, large datasets of various formats, such as JSON and CSV, can
easily be bound to these objects to generate rich graphic charts and diagrams.
Our visualization tool depends on the dataset compiled in JSON file format.
Using the data binding technique, we load the dataset from the generated file
and use the data to drive the creation of elements such as an SVG object.

The visualization tree consists of four levels. The dissection of the
structure is shown in Fig. 3. The levels are annotated as the Root, Model
Selection, Feature Selection, and Terminal. The Root works as the starting
point for the user; when completely collapsed, only Root node is visible. Upon
clicking this Root node, all the nodes (referencing to 10 different models) of
the second horizontal level, Model Selection, get expanded for user selection.
If we add any new untrained model, the associated node for that model will
appear in a different color. Each node has a dynamic tool-tip. The size of
each node in the Model Selection level is determined by the number of
different features performed on the respective model. A higher number of
features for a model would result in a bigger node.

A further selection of a node (model) from the second level would enforce all
its’ associated nodes form the third level (Feature Selection level) to
expand. These nodes represent features that are used to train the model (for
performance analysis). Selecting any node from the third level would expand
the three next-level nodes for Accuracy, F1-score, and Run-time. Each node has
tool tips to state the actual values of these metrics. These nodes also vary
in sizes depending on the value of the metrics.

All the nodes can be dragged, dropped, zoomed, and collapsed anywhere on the
screen to provide more flexibility and convenience for the users. Zooming in
can be performed by either double click or by scrolling the mouse-wheel;
zooming out can be performed by holding shift when double-clicking. Clicking
on the desired node achieves the expansion and collapse of the node. The tree
auto-calculates its sizes both horizontally and vertically so that the maximum
number of nodes can be presented on the screen while keeping the view
comfortable and aesthetically pleasing.

Users can choose from a range of techniques to visualize the data; the options
include horizontal, vertical bar-charts, line chart, and scatter plot. For
brevity, we chose to include only the outputs of horizontal column charts in
Fig. 2. Here, all traditional machine learning models have individual charts,
whereas the neural network models are presented together for better comparison
purposes (right side of Fig. 2). Each column of a chart represents a feature
utilized with the respective model; the first and seconds bars of each column
represent the accuracy and F1-score for the model-feature combination,
respectively. Users can sort each chart in the ascending or descending order.
We modified the tool to give users the option to combine all these
representations for any given metrics: accuracy, precision, recall, f1-score,
and run-time. Top and bottom parts of Fig. 4 demonstrate accuracy, f1-score
combination and run-time for all model feature combinations respectively.

The .json file used for data-binding in our visualization tool contains the
average values for all metrics. Whenever a user performs any model-feature
combination using the replication package of our empirical study, all the
resulting information gets updated in the file, which in turn is loaded by the
tool for data binding. Details of different parameterization of provided
models are saved and the tool-tips display the changing outcome for user-
convenience.

###  2.2. Component to Visualize Results of Different Topic Modeling
Techniques

Figure 5. Circle Packing Visualization technique for keywords presentation in
topic modeling

In our study, we compared two state of the art topic modeling algorithms,
namely OBTM (Cheng, 2014) and AOLDA (Gao et al., 2018) with the results of
string matching to automatically investigate the emerging issues responsible
for energy inefficiency of the apps. OBTM is used as it can extract topics
from short texts (Cheng, 2014) and AOLDA is chosen because it can capture
version-sensitive emerging topics from mobile app reviews (Gao et al., 2018).
We used D3.js to develop the visualization tool shown in Fig. 5 to exhibit the
result of a qualitative study on the performance of the techniques. The result
of a topic modeling algorithm is a number of topics with a set of keywords for
each extracted topic, which are shown in the visualization.

The topic visualization is divided into four levels: light gray (level-1),
dark gray (level-2), black (level-3), and white (level-4) circle. The level-1
circle is the main container holding the results of all algorithms, where each
one is represented in dark gray circles. The extracted topics for each
algorithm are shown as black level-3 circles (one for each topic) with
keywords inside them (level-4). The size of the white circles depends on the
probability of the term (keyword) to appear in a review providing that the
review falls into the term’s associated topic. For string matching, the size
of the circle depends on term’s frequency in the review-set.

For example, in Fig. 5, OBTM has two main topics that are shown by black
circles, with all of their keywords inside them respectively. In our study,
the developers explored the dataset with string matching which yielded to only
2 main topics that are related to energy efficiency of the apps. Therefore,
the number of topics (that should be set manually) for OBTM and AOLDA is set
to two.

###  2.3. Replication package

To encourage replicability of the research results, we make all scripts,
codes, and graphs available to the community 222https://github.com/Mohammad-
Abdul-Hadi/Result-Set-Visualization-d3js and will provide the annotated
dataset upon request.

##  3\. Experiments

We experimented the models on a well curated dataset of Google Play reviews.
The dataset contains more than 400 apps from 24 different app categories. The
training data includes ¡date¿, ¡rating¿, ¡review¿, ¡app_name¿, ¡app_category¿
records that are labeled as energy related reviews. The results of our
empirical study reveals that six deep learning models achieved the highest
accuracy among 90 different model-feature combinations tested. However, as
expected, the machine learning algorithms prevail when we take run-time into
consideration. Due to the space limits, we report on the experiments and
results of the study briefly. More details are provided in the tool page.

Four developers participated in our empirical study who had the independence
of not choosing the visualization tool for careful inspection of the dataset
and models. However, all of them found that the visualization tool was
effective for the exploration of the result set. Moreover, we invited four
undergraduate students (who had the basic knowledge of machine learning and
were paid) to evaluate the tool. The purpose of the tool development and the
results were explained to the participants. They worked with the tool to
explore the results and choose the models. Specifically, we asked (i) whether
the tool was necessary for exploring the results over a manual analysis, (ii)
comprehensiveness of the visualizations, (iii) which (sub)-components that are
not necessary for our purpose, and (iv) the sections of the tool that requires
improvement.

All of the participants reported that the visualization tool is required for
exploring and deciding the model to use for filtering the reviews. One of the
participants pointed out the lack of options presented in the topic modeling
part, which we found to be true. The comprehensiveness, smooth
interactability, aesthetic features, and user-friendliness of the tool was
also highly commended.

##  4\. Conclusion

We have successfully delivered an interactive visualization tool to support
the developers who frequently analyze and inspect users’ feedback to identify
energy efficiency issues. The tool is provided as part of the research on
empirical studies of automatic extraction of app-specific features from app
users’ reviews with transfer learning. For this study, we focused on energy
efficiency issues. In the future, we plan to work on providing different
choices for result-set visualization of different topic modeling algorithms
and perform an extensive qualitative study on the usage of the tool.

## References

  * (1)
  * Chen et al. (2014) Ning Chen, Jialiu Lin, Steven CH Hoi, Xiaokui Xiao, and Boshen Zhang. 2014.  AR-miner: mining informative reviews for developers from mobile app marketplace. In _Proceedings of the 36th ICSE_. 767–778.
  * Cheng (2014) X. Cheng. 2014.  BTM: Topic Modeling over Short Texts.  _IEEE Transaction on Knowledge and Data Engineering_ (December 2014).  https://doi.org/10.1109/TKDE.2014.2313872
  * Chowdhury et al. (2019) Shaiful Chowdhury, Stephanie Borle, Stephen Romansky, and Abram Hindle. 2019.  Greenscaler: training software energy models with automatic test generation.  _Empirical Software Engineering_ 24, 4 (2019), 1649–1692.
  * Cruz and Abreu (2019) Luis Cruz and Rui Abreu. 2019.  Catalog of energy patterns for mobile applications.  _Empirical Software Engineering_ 24, 4 (2019), 2209–2235.
  * Dabrowski et al. (2019) Jacek Dabrowski, Emmanuel Letier, Anna Perini, and Angelo Susi. 2019.  Finding and analyzing app reviews related to specific features: A research preview. In _International Working Conference on Requirements Engineering: Foundation for Software Quality_. Springer, 183–189.
  * Dirksen (2017) Jos Dirksen. 2017\.  _Expert Data Visualization: Advanced information visualization with D3.js_ (1st. ed.).  Packt Publishing, Birmingham, United Kingdom.
  * Fayyad (2001) U. Fayyad. 2001\.  _Information Visualization in Data Mining and Knowledge Discovery_ (1st. ed.).  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  * Gao et al. (2018) Cuiyun Gao, Jichuan Zeng, Michael R Lyu, and Irwin King. 2018\.  Online app review analysis for identifying emerging issues. In _Proceedings of the 40th International Conference on Software Engineering_. 48–58.
  * Noei et al. (2019) Ehsan Noei, Feng Zhang, and Ying Zou. 2019.  Too Many User-Reviews, What Should App Developers Look at First?  _IEEE TSE_ (2019).
  * Wikipedia contributors (2020) Wikipedia contributors. 2020\.  D3.js.  https://en.wikipedia.org/wiki/D3.js [Online; accessed 01-January-2020].
  * Wilke et al. (2013) C. Wilke, S. Richly, S. Götz, C. Piechnick, and U. Aßmann. 2013.  Energy consumption and efficiency in mobile applications: A user feedback study. In _2013 IEEE International Conference on Green Computing and Communications and IEEE Internet of Things and IEEE Cyber, Physical and Social Computing_. IEEE, 134–141.