11institutetext: Department of Computing, São Paulo State University
Av. Eng. Luiz Edmundo Carrijo Coube, 14-01, Bauru, 17033-360, Brazil
11email: {mateus.roder, leandro.passos<EMAIL_ADDRESS>Department of Production Engineering, Paulo State University
Rua Geraldo Alckmin, 519 - Vila Nossa Sra. de Fatima, Itapeva, 18409-010,
Brazil 22email<EMAIL_ADDRESS>
# Feature Selection and Hyperparameter Fine-tuning in Artificial Neural
Networks for Wood Quality Classification††thanks: The authors are grateful to
FAPESP grants #2016/06538-0, #2018/02822-1 and #2019/07825-1.

Mateus Roder 11 0000-0002-3112-5290 Leandro Aparecido Passos 11
0000-0003-3529-3109 João Paulo Papa 11 0000-0002-6494-7514
André Luis Debiaso Rossi 22 0000-0001-6388-7479

###### Abstract

Quality classification of wood boards is an essential task in the sawmill
industry, which is still usually performed by human operators in small to
median companies in developing countries. Machine learning algorithms have
been successfully employed to investigate the problem, offering a more
affordable alternative compared to other solutions. However, such approaches
usually present some drawbacks regarding the proper selection of their
hyperparameters. Moreover, the models are susceptible to the features
extracted from wood board images, which influence the induction of the model
and, consequently, its generalization power. Therefore, in this paper, we
investigate the problem of simultaneously tuning the hyperparameters of an
artificial neural network (ANN) as well as selecting a subset of
characteristics that better describes the wood board quality. Experiments were
conducted over a private dataset composed of images obtained from a sawmill
industry and described using different feature descriptors. The predictive
performance of the model was compared against five baseline methods as well as
a random search, performing either ANN hyperparameter tuning and feature
selection. Experimental results suggest that hyperparameters should be
adjusted according to the feature set, or the features should be selected
considering the hyperparameter values. In summary, the best predictive
performance, i.e., a balanced accuracy of $0.80$, was achieved in two distinct
scenarios: (i) performing only feature selection, and (ii) performing both
tasks concomitantly. Thus, we suggest that at least one of the two approaches
should be considered in the context of industrial applications.

###### Keywords:

Artificial Neural Network Optimization Wood Quality Classification Sawmill
Problem Hyperparameter Tuning Feature Selection.

##  1 Introduction

Industries have experienced many technological advances in recent years,
resulting in more complex processes, systems, and products. As a consequence,
the management of integrated manufacturing processes and operation analyses
are crucial to delivering high-quality products to clients. In this scenario,
the quality of raw materials is also paramount for high quality manufactured
products. However, imperfection detection, as well as quality classification
of raw materials, such as woods in sawmill companies, usually is still
performed by trained human operators [1]. Notwithstanding, the process is
inherently subjective, since it is a visual analysis, and these experts may
suffer from fatigue after a long working period performing repetitive
activities. Consequently, it is expected the increased number of incorrect
classifications [25].

These disadvantages stimulated the scientific community towards the
implementation of visual inspection systems, aiming to perform defect and
quality classification autonomously. Therefore, this work focuses exclusively
on Machine Learning (ML) techniques that have been successfully employed for
these tasks on wood boards [6]. Recently, some studies investigated the
performance of different ML techniques to classify the quality of wood surface
[21, 17]. These studies have used data generated from images captured in a
real sawmill company and classified by a specialist in three levels of
quality, according to the company rules, i.e., zero defect is found in the
wood piece (A), only small defects, such as knots, are found (B), and defects
that compromise the quality of the product, such as groups of knots and
exposed pith (C). Fig. 1 depicts some examples of wood images classified at
each level.

Wood quality A. Wood quality B. Wood quality C.

Figure 1: Three different qualities of wood boards (A, B, and C) according to
company’s rule.

Despite the success obtained in these studies, each ML algorithm has its
inherent tendency towards data specificity, which influences the model’s
induction and, thus, its predictive performance. Therefore, one can adjust
such tendencies through a proper hyperparameter (HP) selection. The task of
finding the best HP values is known as hyperparameter tuning and usually aims
at improving the model’s predictive performance while keeping the model as
simple as possible. Although some HP values may fit sufficiently well
different kinds of problems, it is a common practice to search for the
hyperparameter that provides the best solutions concerning each problem at
hand [21, 17].

Besides, another challenge in the context of this work is extracting the
images’ more representative features, i.e., the features that best describes
the problem, since the more descriptive they are, the higher the effectiveness
of the technique. Regarding classification tasks, features are usually
extracted through image descriptors, such as statistical measures from the
Gray Level Co-occurrence Matrix (GLCM) [8] and Local Binary Patterns (LBP)
[13]. However, many of these features may be correlated to each other or do
not add any relevant information for the ML technique. Therefore, the process
of selecting a subset of these features, referred to as feature selection
(FS), can be applied to select the most descriptive ones. Moreover, since
features are specific for each problem, FS has to be carried out for each data
set separately.

Considering that both the HP tuning and the task of feature selection relies
upon each other, they should be performed simultaneously to generate more
robust models, concerning generalization purposes. In general, ML techniques
require tuning more than one HP, since the tackled problems generally are
described by many features, thus implying on large search spaces. In this
context, metaheuristics approaches are commonly employed to solve such
problems by randomly initializing a collection of candidate solutions, which
interact among themselves and perform a directed exploration of the search
space toward the results that best fit a desirable target function with an
acceptable computational cost. Such approaches are commonly employed to solve
problems related to ML techniques hyperparameter tuning [19, 5, 14] and
feature selection [15], among others [4, 22].

In this paper, we investigate the problem of FS and Artificial Neural Network
(ANN) hyperparameter tuning applied in the context of wood boards quality
classification. Experiments were carried out using the population-based
metaheuristic Particle Swarm Optimization (PSO) [9, 20] to simultaneously
perform both tasks over a Multilayer Perceptron (MLP) ANN. Moreover, the
results compared against five distinct baselines, as well as a random search,
confirms the relevance of the proposed approach. We hypothesize that the
predictive performance of ANN models can be improved since they depend on the
HP values and the set of features used to describe the problem.

Therefore, the main contributions of this paper are twofold: (i) to propose a
method capable of simultaneously selecting the hyperparameters that best
performs over an MLP network as well as selecting the subset of features that
best describe each image sample, and (ii) to foster the scientific community
regarding material and wood quality classification. The remainder of this
paper is presented as follows. Section 2 defines the problem of hyperparameter
tuning and feature selection, and provides a brief description of some related
works. Section 3 presents the main concepts of ANN and PSO. The experimental
methodology employed to evaluate the effects of FS and MLP hyperparameter
tuning over the models’ performance is described in Section 4. Results are
presented and discussed in Section 5, and finally conclusions are presented in
Section 6.

##  2 Problem Definition and Related Work

Hyperparameter tuning and feature subset selection are two widely employed
tasks carried out in the data mining context, aiming to improve models’
predictive performance as well as simplifying them. Therefore, this section
formalizes the problem of simultaneously performing these two tasks. Besides,
it presents an overview of studies related to wood quality classification and
the importance of HP tuning and FS.

###  2.1 Problem Definition

The problem investigated in this paper consists of tuning the HP values of an
MLP Artificial Neural Network algorithm, as well as selecting a subset of
features that are relevant for the problem of wood quality classification
towards the improvement of the models’ predictive performance.

Let $A$ be an MLP algorithm that comprises the hyperparameter space $\Lambda$.
For each hyperparameter setting $\lambda\in\Lambda$, let $A_{\lambda}$
represent the learning algorithm $A$ that employs the hyperparameter setting
$\lambda$. Also, consider
$D=\\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n})\\}$ a dataset composed
of $n$ instances, such that $\bm{x}\in\mathbb{R}^{m}$ is a feature vector and
$y$ is the target value. Moreover, one can define $\kappa$ as the subset of
feature from $\bm{x}$. Finally, let $A^{\kappa}$ be an algorithm $A$ trained
with a subset of features $\kappa$ [12].

Therefore, the main goal of HP tuning is to finding $\lambda^{*}=\text{arg
min}_{\lambda\in\Lambda}M(A_{\lambda},D)\in\Lambda$ that minimizes some loss
function, such as the misclassification rate using the algorithm $A$ over
instances not used for training purposes. Moreover, one can estimate the
misclassification rate $M(A,D)$ achieved by $A$ when trained and tested on $D$
through a stratified multi-fold cross-validation resampling method.

Similarly to HP tuning, the goal of feature subset selection is to find
$\kappa^{*}=\text{arg
min}_{\kappa\subseteq\bm{x}}M(A^{\kappa},D)\subseteq\bm{x}$ which minimizes
the loss function achieved by $A$ when trained on $D$.

Therefore, the aim of combining hyperparameter tuning and feature subset
selection is to find the hyperparameter setting $\lambda^{*}\in\Lambda$ and
the feature subset $\kappa^{*}\subseteq\bm{x}$ which has the lowest
misclassification rate among all HP settings and features subsets, i.e.,
$(\kappa^{*},\lambda^{*})=\text{arg
min}_{\lambda\in\Lambda,\kappa\subseteq\bm{x}}M(A^{\kappa}_{\lambda},D)$.

###  2.2 Related Work

One of the first studies to investigate the problem of wood quality control
using an automated visual inspection system based on machine learning
techniques was accomplished by [16]. Since then, many others have investigated
this problem aiming to improve predictive performance by analyzing and
selecting different features, as well as optimizing HP values of ML techniques
[24, 21].

Tiryaki et al. [24] employed an ANN for modeling the wood surface roughness in
the machining process. The study highlights some variables that have impact
and influence in the surface roughness, such as wood species, the feed rate,
the number of the cutter, and the cutting depth. The model’s predictive
performance was good enough to allow its application in the wood industry in
order to optimize effort, time, and energy.

Others addressed the problem of combining FS and HP tuning for ML techniques
over different applications. As previously mentioned, some techniques are more
sensitive to HP tuning and FS than others. Besides, some optimization methods,
such as metaheuristic approaches based on evolutionary algorithms and swarm
intelligence [10], for instance, have successfully accomplished the task.

In this context, Roder et al. [21] used the PSO algorithm to tune the HP of an
ANN applied for wood quality classification aiming to enhance the model’s
predictive performance. The authors employed the GLCM [8] to extract features
from the same dataset used by [2], which is composed of five statistical
measures for two angles, i.e., $0$ and $90$ degrees: entropy, energy, maximum
intensity, inverse difference moment, and correlation. Experimental results
obtained up to $6\%$ of accuracy gain concerning the ANN classification, and
corroborate the necessity of tuning the ANN hyperparameters; stating the
efficiency of PSO for such a task.

##  3 Theoretical Background

This section briefly introduces the main concepts of the techniques employed
in this work, i.e., the Multilayer Perceptron neural network and the Particle
Swarm Optimization algorithm, as well as the process of wood image feature
extraction.

###  3.1 MultiLayer Perceptron ANN

MLPs are composed of an input and an output layer, as well as one or more
hidden layers of neurons, which can be fully or partially connected. A neural
network is called fully connected when each neuron from a given layer is
connected to all neurons of the next one. Similarly, it is said to be
partially connected when some neurons of adjacent layers are not connected.
These connections are represented by a weight matrix, which is usually
adjusted through gradient-based learning algorithms. With a single hidden
layer, an MLP is capable of representing a large number of functions, thus
sufficient for the purpose of this study. Figure 2 depicts the model
architecture.

Figure 2: MultiLayer Perceptron representation.

The conventional algorithm to train an MLP network is the backpropagation,
which is composed of the forward pass, which exposes the input data to a
series of linear operations followed by non-linear activations, and the
backward phases, which is responsible for propagating the output error and
update the network weights.

###  3.2 Particle Swarm Optimization

Particle Swarm Optimization [11] is a global optimization technique based on
the social behavior of birds, fishes, and insects, among others. The method
comprises a swarm composed of a set of individuals capable of sharing
information among themselves concerning their positions in the search space,
as well as the relative quality of this position, denoted by a fitness
function.

In short, each PSO particle is represented by its current position, velocity,
and the best position found during the training process. The position of a
particle $i$ is represented by a point in a $D$-dimensional space, given by
$\Psi_{i}=\\{\psi_{i1},\psi_{i2},\ldots,\psi_{iD}\\}$. Further, the particle
velocity is defined by $\bm{v}_{i}=\\{v_{i1},v_{i2},\ldots,v_{iD}\\}$, and
finally the best position found by this particle is represented by
$\bm{p}_{i}=\\{p_{i1},p_{i2},\ldots,p_{iD}\\}$. Besides, the best position
found among all particles is represented by $\bm{p}_{g}$.

A particle will move in a particular direction depending on its current
position, velocity, and best position. Additionally, it also depends on the
best position found by the other particles in the swarm. Therefore, the
position of a particle $\psi_{ij}(t+1)=\psi_{ij}(t)+v_{ij}(t)$ is computed for
each dimension $j\in\\{1,2,\ldots,D\\}$ at time step $t$. Further, the
velocity $v_{ij}(t)$ is updated using the following equation:

$v_{ij}(t+1)=\Gamma\cdot v_{ij}(t)+\varphi_{1}\cdot
r_{1}\cdot(p_{ij}-x_{ij}(t))+\varphi_{2}\cdot r_{2}\cdot(p_{gd}-x_{ij}(t)),$
(1)

where $\Gamma$ denotes the inertial weight, introduced by [23] to balance the
global and local search, $r_{1}$ and $r_{2}$ are two independent values
uniformly distributed in the range $[0,1]$, and $\varphi_{1}$ and
$\varphi_{2}$ are acceleration constants.

Such a representation of a particle is suitable for hyperparameter fine-
tuning, which considers real- and integer-valued numbers. However, it is not
adequate for feature selection tasks since it requires a categorical or binary
representation. Therefore, this work employs a variation of the method,
adapted for feature selection, as discussed in Section 4.

###  3.3 Feature Extractors

Haralick et al. [8] proposed a set of mathematical tools to extract
statistical features from images using a Gray Level Co-occurrence Matrix. The
GLCM is capable of describing the frequency of occurrences in grayscale
transitions for an image in a pixel-by-pixel fashion. Further, the features
are extracted considering the relationship of each pixel with its neighbors
over four different angles, i.e., $0$, $45$, $90$, and $135$ degrees, where
$0$ and $90$ degrees are the most employed ones. Moreover, the model is
capable of extracting a total of $14$ statistical features, named Angular
Second Moment, Contrast, Correlation, Variance, Inverse Difference Moment, Sum
Average, Sum Variance, Sum Entropy, Entropy, Difference Entropy, Information
Measures of Correlation ($1$ and $2$) and the Maximal Correlation Coefficient.

Another well-known texture descriptor is the Local Binary Pattern [13], which
converts the image to a gray-scale level and performs a pixel-by-pixel
comparison over the entire image considering a selected number of neighbors.
In this comparison, the central pixel of the square formed by its neighbor
assumes the value $1$ if it is greater or equal to its neighbors, or $0$
otherwise. Such value is stored in an array, which is further employed for
converting the binary intensity code to a decimal value for the pixel at hand.

Afterward, the same computation is performed for all pixels compounding the
image. Further, it is generated a histogram of the distribution of the values,
which will compose the final vector describing the image. Notice the method
was initially proposed with the number of neighborhood fixed in $3\times 3$.
Later, some changes allowed LBP to deal with a larger number of neighborhoods,
resulting in a non-square structure, usually employing a circular pattern
since it only requires the definition of the radius, instead of the $NxN$
arrangement.

##  4 Methodology

This section presents the methodology concerning the material and methods
employed during the experiments. It briefly describes the datasets, the
process of feature extraction, the modeling of the hyperparameter fine-tuning
and feature selection processes using PSO, the methods used for evaluation
purpose, and the baselines considered for comparison.

###  4.1 Dataset

This work employs a dataset $D$ composed of features extracted from $374$
instances of wood board images obtained in a Brazilian sawmill [25]. As stated
in the problem definition, each instance $D=(\bm{x}_{i},y_{i})$ is composed of
an $m$-dimensional feature vector $\bm{x}_{i}\in\mathbb{R}^{m}$, whose
features were extracted using both GLCM and LBP, as described in Section 4.2,
and a target value $y_{i}$, denoting the wood quality. Each sample’s target
value is established according to rules defined by the sawmill company, where
“A” stands for a high-quality standard and comprises $144$ instances, “B”
denotes an intermediate quality and comprises $177$ instances, and “C”
represents lower quality, comprising $53$ samples.

###  4.2 Image Texture Descriptors

The feature set was obtained by joining the features extracted from two
texture descriptors, namely, statistical measures extracted from GLCM and the
LBP. While the statistical measures have the advantage of enabling the
interpretation and comprehension of the image characteristics through
different measures, LBP is robust in the treatment of gray-scale images, with
a good performance for scale changes caused by illumination [13].

Concerning the GLCM, this paper employed $0^{o}$ and $90^{o}$ to extract six
measures: angular second momentum, energy, contrast, correlation,
dissimilarity, and homogeneity, resulting in $12$ characteristics for each
image. Besides, LPB uses $24$ neighbors as well as a radius of size $3$,
resulting in $26$ characteristics for each image. Thus, joining GLCM and LBP
features resulted in $38$ predictive attributes.

###  4.3 MLP Hyperparameter tuning and Feature Selection using PSO

This work employed a fully connected network composed of a single hidden layer
for the task of classification. Further, it also employed the PSO algorithm to
fine-tuning the three principal hyperparameters of the model, namely the
number of neurons in the hidden layer $\gamma$, the learning rate $\eta$, and
the momentum term $\mu$.

As mentioned previously, the PSO algorithm was employed to perform a combined
task of hyperparameter tuning and feature selection, hereafter referred to as
HP-FS-PSO. Considering the former task, PSO decision variables are modeled
admitting one integer values to represent the number of units in the hidden
layer, as well as two real numbers to represent the learning rate and
momentum.

Further, since the task of feature selection does not assume continuous
representation, PSO requires some modifications to work properly in this
context. The main change is made on the position representation, which must be
treated as a result of probability analysis from particles’ velocity to decide
what features are relevant to the context [9]. Therefore, considering the task
of feature selection, each particle’s decision variable is represented by a
binary value, where $1$ means to consider a feature whereas $0$ means to
discard a feature.

Since the particle’s decision variables employed for feature selection assume
binary values, it is necessary to binarize each position, $\psi_{ij}$ such
that $\psi_{ij}=1$ if $s(v_{ij})>r_{3}$ and $0$ otherwise. Notice $r_{3}$ is
the threshold, a real number generated randomly in the range $[0,1]$, and
$s(\cdot)$ is the logistic function.

Concerning the MLP hyperparameters’ search configuration, the hidden layer
size is optimized in the range $[2,60]$, the learning rate and momemtum assume
values in the range $[0,1]$, and the feature subset selection is defined by a
binary variable, i.e., assuming either $0$ or $1$. Additionally, it also
presents the MLP hyperparameter default values used by Weka [7]. The number of
neurons in the hidden layer is defined as
$\gamma=\frac{(\text{NA}+\text{NC})}{2}$, where $NA$ is the number of
attributes and $NC$ is the number of classes. Thus, the default value for our
problem, which has $38$ attributes and $3$ classes, is $20$ neurons.

Further, the PSO algorithm also has its own hyperparameters, defined as
follows: Number of Particles $N=30$, Acceleration Constant 1
$\varphi_{1}=1.494$, Acceleration Constant 2 $\varphi_{2}=1.494$, Inertia
Weight $\omega=0.729$, and Maximum Velocity $\upsilon$. Notice tuning such
hyperparameters would lead to a “never-ending” problem. Therefore, these
values were empirically selected based on similar works [9, 21]. Besides, the
maximum velocity $\upsilon$ varies according to the upper limit of its
respective hyperparameter.

The optimization process is performed until meeting the stop criterion, i.e.,
the maximum number of iterations, which was set in $300$.

###  4.4 Evaluation

Metaheuristic algorithms guide their search for the best solutions according
to the outcome of a fitness function. In this work, this measure is obtained
from the predictive performance of the ANNs over a validation subset.

Moreover, since the dataset used in this work is composed of an imbalanced
class distribution, i.e., there is a considerably smaller number of examples
labeled as “C” class. Therefore, measures that acknowledge this imbalance are
more suitable for the task. Thus, in this study, we considered the Balanced
Accuracy (BAC) [3]

The ANN training and evaluation were performed using a nested stratified
k-fold cross-validation (CV) re-sampling method. Such an approach splits the
dataset into $k=10$ partitions, where one of them is used to test, and the
remaining folds are employed for training purposes. In this context, the
training folds are used to train the model during the optimization process,
i.e., finding the best MLP hyperparameters and the best subset of features.
Therefore, PSO assesses the average BAC considering the fitness value over the
validation set. Further, the best set of hyperparameters and features found in
this process is them applied to train the model and induct the prediction of
the testing set samples’ labels. Such a process guarantees that the data used
to evaluate the model is never used in the model training steps and,
consequently, in the MLP hyperparameter tuning and feature selection
processes.

Finally, due to the stochastic process of PSO, the optimization process was
repeated during $10$ runs, aiming to perform a statistical analysis through
the Wilcoxon signed-rank test [26] with $5\%$ of significance.

###  4.5 Baselines

In order to evaluate and compare the results obtained by PSO, six baselines
methods were compared in the context of the combined MLP hyperparameter tuning
and feature selection:

  * •

Method 1 (M1) : Default hyperparameter values defined by Weka and the whole
set of features;

  * •

Method 2 (M2) : MLP hyperparameter tuned using PSO and the whole set of
features;

  * •

Method 3 (M3) : Default MLP hyperparameter values defined by Weka and feature
subset selected by PSO;

  * •

Method 4 (M4) : Default MLP hyperparameter values defined by Weka and
dimensionality reduction performed by Principal Components Analysis (PCA);

  * •

Method 5 (M5) : MLP hyperparameter values tuned by PSO and dimensionality
reduction performed by PCA;

  * •

Random Search (RS) : Random selection of MLP hyperparameter values and feature
subset. This approach considered the same number of solutions evaluated by
PSO.

The methodology adopted in this work aims at analyzing the PSO performance
from different $x$ perspectives. First, M1 is the baseline for both tasks,
i.e., the MLP hyperparameter using default parameters provided by Weka
considering the whole set of features. Further, M2 allows analyzing PSO
influence for the task of MLP hyperparameter tuning, with no feature
selection, while M3 investigates the opposite, i.e., PSO influence to the task
of feature selection with using MLP default hyperparameters. Moreover, M4
employs the default hyperparameter with a dimensionally reduced feature subset
performed by PCA, while M5 combines MLP hyperparameters tuning using PSO with
PCA. Finally, RS represents the analysis of random combinations of MLP
hyperparameter values and selected features.

The experiments carried out in this work were coded using
Python111https://www.python.org/ and R [18]. Further, the feature extraction
task was implemented in Python using the Scikit-image222http://scikit-
image.org/ package, while the MLP network was developed using the RWeka
package in R, which is an interface to Weka. Finally, PSO was also implemented
in R.

##  5 Experimental Results

This section presents the predictive performance of the ANNs assessed for the
HP-FS-PSO method, as well as the baselines techniques. Notice the results are
also provided with the $p$-values considering the Wilcoxon signed-rank test
compared to the HP-FS-PSO method as the reference for statistical analysis
purposes. Further, values presented in bold stand for the most accurate result
overall.

###  5.1 Optimization Evaluation

Regarding the optimization performance over the validation set, one can
observe in Table 1 that HP-FS-PSO and M3 have obtained the best results,
achieving a BAC average of $0.850$. Such techniques provided an improvement of
around $10\%$ compared to M1, which represents an ANN using default HP values
and the whole set of features. Therefore, the most important finding from
these results is that the optimization of MLP hyperparameters, and even more a
proper selection of the more suitable features, have a strong influence in the
induction of MLP models applied to wood boards quality classification. On the
other hand, results also show that performing only one task may also be enough
to increase performance since M3 performed only the task of feature selection,
while M2, which obtained an average BAC of $0.834$, performing only the task
of MLP hyperparameter tuning. Although by a narrow margin, in this case, the
set of features was more expressive for the model predictive performance than
tuning the network hyperparameters.

This behavior suggests that a user should, at least, select a set of features
for the hyperparameter values defined a priori. Besides the BAC values, Table
1 also provides the $p$-values considering the Wilcoxon signed-rank test
compared to the HP-FS-PSO method as the reference. These $p$-values support
our previous observations, considering only M3 and the random search obtained
a significance level higher than $\alpha=0.05$.

Table 1: Average BAC and the standard deviation concerning the task of MLP hyperparameter tuning and feature selection, considering the validation set over $10$ executions. Notice the “$p$-values” are compared against the HP-FS-PSO reference. | Validation | $p$-value
---|---|---
| (BAC$\pm$ SD)
M1 | 0.752 $\pm$ 0.008 | 0.002
M2 | 0.834 $\pm$ 0.015 | 0.008
M3 | 0.850 $\pm$ 0.015 | 0.557
M4 | 0.739 $\pm$ 0.011 | 0.002
M5 | 0.825 $\pm$ 0.022 | 0.006
RS |  0.845 $\pm$ 0.012 | 0.131
HP-FS-PSO |  0.850 $\pm$ 0.011 | Ref.

Notice the positive behavior of the random search approach, which is somehow
expected since the model presents itself as more sensitive to a proper
selection of the features instead of the network hyperparameter tuning, which
is expected to be a more straightforward task due to the binary nature of the
search space. Further, the data dimensionality reduction performed by PCA
showed to be inadequate for this problem, as denoted by methods M4, which
obtained BAC average results lower than using a default configuration, and M5.
The main reason lies in the fact that PCA may not be able to describe
sufficiently well the problem due to its linear nature.

For a better understanding of each method’s behavior during PSO convergence,
Figure 3 depicts the optimization performance (a) and the evolution of the BAC
values (b) during $300$ iterations. Figure 3 (a) considers the average values
over $10$ runs, where each iteration in the RS curve reflects the average
evaluation among $30$ executions, i.e., the same number of assessments
performed by PSO considering $30$ particles. Finally, M1 and M4 are
represented by fixed lines since no optimization was performed over such
approaches. Figure 3(b) corroborates our claim that HP-FS-PSO performed a
guided search through the MLP hyperparameters and features spaces, improving
its performance over the iterations. Notice PSO can reduce the number of
iterations required for finding reasonable BAC values since it obtained
relatively high accuracies (around $0.840$) after $80$ iterations only.

(a) (b)

Figure 3: PSO convergence considering the evaluation dataset (a) and evolution
of the BAC values of HP-FS-PSO and RS. In this case, the best BAC value found
by RS up to a iteration is kept in the next iterations (b).

Besides, Figure 3(b) depicts the HP-FS-PSO performance compared against a
random search considering the best results over each iteration, instead of an
average. One can observe the random search performed slightly better during
the first $30$ iterations. Afterward, the HP-FS-PSO surpassed RS and kept this
advantage until reaching the $300$ iterations. As previously mentioned, it is
possible to note that there is a considerable improvement of BAC for both
methods in the first $100$ iterations, and then there is a slowdown in the BAC
growth. Such a piece of information is of extreme relevance for industrial
applications, since it may save time and effort during the task of tuning the
model’s hyperparameter selecting the best subset of features.

###  5.2 Classification

This section investigates the ANN generalization power by evaluating the
predictive performance of the model over the testing set, considering the best
set of hyperparameters and a subset of features found during the optimization
process. Table 2 presents the BAC values obtained in this context. Notice the
values presented in bold stand for the most accurate approach overall.

Table 2: Average BAC and the standard deviation concerning the task of MLP wood quality classification considered the testing samples and the best set of hyperparameters and subfeatures found during the optimization process for each approach. Notice the “$p$-values” are compared against the HP-FS-PSO reference. | Test | p-value
---|---|---
| (BAC$\pm$ SD)
M1 | 0.755 $\pm$ 0.077 | 0.275
M2 | 0.779 $\pm$ 0.042 | 0.547
M3 |  0.798 $\pm$ 0.087 | 0.722
M4 | 0.705 $\pm$ 0.057 | 0.010
M5 | 0.773 $\pm$ 0.071 | 0.232
RS | 0.745 $\pm$ 0.080 | 0.131
HP-FS-PSO | 0.794 $\pm$ 0.062 | Ref.

In general, the results of test data are in agreement with those of the
validation set. The baseline method M3 led to the best BAC value, followed
closely by HP-FS-PSO (differing only in the third decimal place). These
performances are again superior to M1, supporting the idea of MLP
hyperparameter tuning and feature selection influence.

However, differently from Table 1, HP-FS-PSO was not statistically better than
M3, as observed in the $p$-value $>0.05$. Therefore, the improvement obtained
during the validation steps was not enough to provide a statistical
difference, considering the test data. The high standard deviation over the
testing set explains such behavior, which was considerably smaller regarding
the optimization steps.

##  6 Conclusion

This paper analyzed the compound problem of ANN hyperparameters tuning and
feature selection for the quality classification of wood boards in the sawmill
industry. Experiments showed that a solution based on PSO led to satisfactory
results compared to baseline methods. According to a statistical test, results
show a significant difference during the optimization task but not for the
generalization phase.

These experimental results suggest that MLP hyperparameter tuning and feature
selection are essential to obtain models with higher predictive performance.
Also, one can notice that these tasks are interdependent since the
hyperparameter values should be adjusted according to a subset of features and
vice versa. Consequently, for the problem investigated in this work,
performing only one of them was enough to reach substantial gain. Finally, the
accuracy obtained in this study supports employing machine learning models for
industrial implementation, contributing to overall cost reduction and
improvement in competitiveness. Regarding future works, we intend to perform a
transfer learning from a CNN trained using a dataset composed of a more
substantial number of wood image samples. Besides, we are willing to
investigate and compare different image descriptors, non-linear data reduction
techniques, and deep learning models.

## References

  * [1] Abdullah, A., Ismail, N.K.N., Kadir, T.A.A., Zain, J.M., Jusoh, N.A., Ali, N.M.: Agar wood grade determination system using image processing technique. Proceedings of the International Conference on Electrical Engineering and Informatics Institut Teknologi Bandung (2007)
  * [2] Affonso, C., Rossi, A.L.D., Vieira, F.H.A., Carvalho, A.C.P.L.F.: Deep learning for biological image classification. Expert Systems With Applications 85, 114–122 (2017). https://doi.org/http://dx.doi.org/10.1016/j.eswa.2017.05.039
  * [3] Brodersen, K.H., Ong, C.S., Stephan, K.E., Buhmann, J.M.: The balanced accuracy and its posterior distribution. In: 2010 20th International Conference on Pattern Recognition. pp. 3121–3124 (Aug 2010). https://doi.org/10.1109/ICPR.2010.764
  * [4] Cao, Y., Zandi, Y., Rahimi, A., Wu, Y., Fu, L., Wang, Q., Denić, N., Khadimallah, M.A., Milič, M., Paunović, M.: A new intelligence fuzzy-based hybrid metaheuristic algorithm for analyzing the application of tea waste in concrete as natural fiber. Computers and Electronics in Agriculture 190, 106420 (2021)
  * [5] De Souza, L.A., Passos, L.A., Mendel, R., Ebigbo, A., Probst, A., Messmann, H., Palm, C., Papa, J.P.: Fine-tuning generative adversarial networks using metaheuristics-a case study on barrett’s esophagus identification. In: Bildverarbeitung für die Medizin. pp. 205–210 (2021)
  * [6] Gu, I.Y.H., Andersson, H., Vicen, R.: Automatic classification of wood defects using support vector machines. In: Computer Vision and Graphics. pp. 356–367. Springer Berlin Heidelberg (2009)
  * [7] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009). https://doi.org/10.1145/1656274.1656278
  * [8] Haralick, R., Shanmugam, K., Distein, I.: Textual features for image classification. IEEE Transaction on Systems, Man and Cybernetics SMC-3(6), 610–621 (1973)
  * [9] Kennedy, J., Eberhart, R.C.: A discrete binary version of the particle swarm algorithm. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation. vol. 5, pp. 4104–4108 (Oct 1997). https://doi.org/10.1109/ICSMC.1997.637339
  * [10] Kennedy, J., Eberhart, R.: Swarm Intelligence. Morgan Kaufmann Publishers (2001)
  * [11] Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks. vol. 4, pp. 1942–1948. Perth, Australia (1995)
  * [12] Luo, G.: A review of automatic selection methods for machine learning algorithms and hyper-parameter values. Network Modeling Analysis in Health Informatics and Bioinformatics 5(1), 18 (May 2016). https://doi.org/0.1007/s13721-016-0125-6
  * [13] Ojala, T., Pietikainen, M., Harwood, D.: Comparative study of texture measures with classification based on feature distributions. Pattern Recognit pp. 51–59 (1996)
  * [14] Passos, L.A., ao Paulo Papa, J.: Fine-tuning infinity restricted boltzmann machines. In: 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). pp. 63–70. IEEE (2017)
  * [15] Pereira, C.R., Passos, L.A., Rodrigues, D., de Souza, A.N., Papa, J.P.: Jade-based feature selection for non-technical losses detection. In: ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. pp. 141–156. Springer (2019)
  * [16] Pham, D.T., Alcock, R.J.: Automatic detection of defects on birch wood boards. Proceedings of the Institution of Mechanical Engineers, Part E: Journal of Process Mechanical Engineering 210(1), 45–52 (1996). https://doi.org/10.1243/0954408991529852
  * [17] Qi, C., Fourie, A., Chen, Q.: Neural network and particle swarm optimization for predicting the unconfined compressive strength of cemented paste backfill. Construction and Building Materials 159, 473 – 478 (2018). https://doi.org/https://doi.org/10.1016/j.conbuildmat.2017.11.006
  * [18] R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2014)
  * [19] Roder, M., Passos, L.A., de Rosa, G.H., de Albuquerque, V.H.C., Papa, J.P.: Reinforcing learning in deep belief networks through nature-inspired optimization. Applied Soft Computing 108, 107466 (2021)
  * [20] Roder, M., de Rosa, G.H., Passos, L.A., Papa, J.P., Rossi, A.L.D.: Harnessing particle swarm optimization through relativistic velocity. In: 2020 IEEE Congress on Evolutionary Computation (CEC). pp. 1–8. IEEE (2020)
  * [21] Roder, M., Rossi, A.L.D., de Oliveira Affonso, C.: Boosting machine learning techniques for wood quality classification by particle swarm optimization. In: Encontro Nacional de Inteligência Artificial e Computacional. Sociedade Brasileira de Computação (2017)
  * [22] Rodrigues, D., de Rosa, G.H., Passos, L.A., Papa, J.P.: Adaptive improved flower pollination algorithm for global optimization. In: Nature-Inspired Computation in Data Mining and Machine Learning, pp. 1–21. Springer (2020)
  * [23] Shi, Y., Eberhart, R.: A modified particle swarm optimizer. In: 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360). pp. 69–73 (May 1998). https://doi.org/10.1109/ICEC.1998.699146
  * [24] Tiryaki, S., Malkoçoğlu, A., Şükrü Özşahin: Using artificial neural networks for modeling surface roughness of wood in machining process. Construction and Building Materials 66, 329 – 335 (2014). https://doi.org/https://doi.org/10.1016/j.conbuildmat.2014.05.098
  * [25] Vieira, F.H.A.: Image processing through machine learning for wood quality classification. Ph.D. thesis, Faculdade de Engenharia de Guaratinguetá (FEG), UNESP (2016)
  * [26] Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80–83 (1945)


#  PaCMO: Partner Dependent Human Motion Generation in Dyadic Human Activity
using Neural Operators

Md Ashiqur Rahman1, Jasorsi Ghosh1, Hrishikesh Viswanath1, Kamyar
Azizzadenesheli2, Aniket Bera1
1Department of Computer Science, Purdue University, West Lafayette, Indiana.
2Nvidia Corporation, Santa Clara, CA.
{rahman79, ghosh117<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We address the problem of generating 3D human motions in dyadic activities. In
contrast to the concurrent works, which mainly focus on generating the motion
of a single actor from the textual description, we generate the motion of one
of the actors from the motion of the other participating actor in the action.
This is a particularly challenging, under-explored problem, that requires
learning intricate relationships between the motion of two actors
participating in an action and also identifying the action from the motion of
one actor. To address these, we propose partner conditioned motion operator
(PaCMO), a neural operator-based generative model which learns the
distribution of human motion conditioned by the partner’s motion in function
spaces through adversarial training. Our model can handle long unlabeled
action sequences at arbitrary time resolution. We also introduce the
"Functional Fréchet Inception Distance" $\text{F}^{2}\text{ID}$ metric for
capturing similarity between real and generated data for function spaces. We
test PaCMO on NTU RGB+D and DuetDance datasets and our model produces
realistic results evidenced by $\text{F}^{2}\text{ID}$ score and the conducted
user study.

##  1 Introduction

Generating 3D human motion is widely studied and is a core component of
animation, games, human-robot interaction, and AR/VR applications. Despite
many efforts from the community, the problem of modeling 3D human motion is
still poorly understood. Most of the works are focused on generating the
motion of a single person [29, 45, 50, 2]. These works often generate human
motion without any interaction with the surrounding objects and rarely model
multi-agent interactions. Modeling the motion of multiple people in an
interactive setting is particularly difficult.

Recent works [29, 45, 43, 30] primarily deal with generating motion of a
single human with or without conditioned by action labels and sentences
describing the action. The advancement of joint language-image models (such as
CLIP) enables the production of motion sequences for unseen actions [45, 14].
But these models can only generate motion sequences of a fixed length at a
specific time resolution. To synthesize lengthy (long duration) human motions
many prior works assume an autoregressive or Markovian assumption [1, 9, 54,
19, 2], which lead to a static state after a certain time. Many recent works
[29, 43, 30, 45, 3] have utilized attention mechanisms [47] to capture non-
local dependencies in long motion sequences. Though these models can generate
long sequences but often in practice the produced sequences are of fixed
length at a particular resolution. They can only generate human motion at a
fixed time resolution which depends on the training data. But time continuous
3D human motion generation is crucial for animation and video games. At the
same time, the problem of multi-person motion generation in interactive
situations is under-explored. Though there have been attempts to address this
problem in recent works [43, 24], these methods require explicit action labels
or sentences describing the action which may be impractical for AR/VR or HRI
applications where we do not have explicit action labels. Rather, in many
cases, we need to model the motion of digital humans interacting with real
humans in arbitrary dyadic activities. In these situations, we need to
generate the motion of a virtual human only from the motion of participating
humans such that virtual humans can correctly collaborate with the intended
activity of the human partner.

To address these problems, inspired by the recent development of generative
modeling in functional spaces [33], we propose PaCMO, a conditional generative
adversarial neural operator model to generate 3D human motion for dyadic
activity from the movements of the other interacting human, i.e. the partner.
To the best of our knowledge, PaCMO is the first to propose a time-continuous
generative architecture to solve this problem. We model human motion sequence
as a stochastic function of time. Given the motion of one of the actors in a
dyadic activity, our model learns a push-forward measure from a Gaussian
random field [6] to the motion sequence of the other actor. Gaussian random
field or Gaussian processes (GP) are continuous functions such that any finite
collection of points on the function constitutes a multi-variate Gaussian
distribution. Unlike traditional deep neural networks, neural operators are
resolution invariant. As a result, our model can generate continuous time
human motion and is not restricted to the resolution of the training data. And
in evaluation time, PaCMO can generate motion at a resolution demanded by the
application (e.g., matching require frame-rate for video game application).
PaCMO can efficiently model global dependencies in long sequences and is not
limited by the receptive field of discrete convolution [50]. Also, PaCMO
outputs the entire motion sequence at once, making it suitable for real-time
applications and also avoiding the issue with autoregressive and Markovian
models.

Evaluating generative models is nontrivial and subject to human perception.
Fréchet Inception Distance (FID) [13] is a well used metric to quantitatively
evaluate generative models. To obtain a closed-form solution of Fréchet
distance, features from both generated and real data are modeled using
multivariate Gaussian distributions. But finite dimensional multivariate
Gaussian distributions cannot model features from infinite dimensional spaces.
Also, such FID is not resolution invariant, as a fixed finite dimensional
Gaussian distribution cannot model feature vectors of different sizes. As a
result, FID is ill-suited to evaluate generative models in infinite
dimensional spaces. To address this, we remove the finite-dimensional
assumption and propose Functional FID or $\text{F}^{2}\text{ID}$, which is
designed for general vector spaces and is resolution invariant.

In this work, we consider human motion as a sequence of skeletal poses but our
model does not depend on any particular representation of human motion and can
easily be extended to other representations, such as SMPL [23]. We have
evaluated our model on two widely used datasets: NTU-RGB+D-120 [22] and
DuetDance[18] by both quantitative metrics and perceptual human evaluations.
Our contributions can be summarized as follows:

  * •

We propose PaCMO, a novel deep neural operator for generating partner-
dependent 3D human motion in interactive dyadic human activities without any
action label or sentence describing the action.

  * •

We introduce a novel conditional generative neural operator which is a
resolution-invariant conditional generator for infinite dimensional spaces.

  * •

We propose ($\text{F}^{2}\text{ID}$), an evaluation metric for generative
models in function spaces.

  * •

Our model PaCMO can generate promising results outperforming baselines, taking
the first step in partner-dependent human motion generation in an interactive
setting.

In the rest of the paper, in section 2 we discuss related works for human
motion generation and introduce neural operators to the readers. In section 3
we introduce the novel PaCMO architecture and ($\text{F}^{2}\text{ID}$) score.
We discuss the details of our experiment with results in section 4.

##  2 Related Works

###  2.1 Human Motion Generation

Human motion prediction and generation are well-studied problems in the fields
of kinesiology, robotics, computer vision, and computer graphics. Especially
the field of human motion generation has experienced rapid development
following success in generative modeling such as GAN [7], VAE [16],
normalizing flow [15, 27], and diffusion [5] based models. The task of human
motion generation can be broadly divided into two categories : (a) conditional
human motion generation, where the generation is guided by external variables
such as action labels, action descriptions, or speech. (b) unconditional human
motion generation, where we aim to model the entire space of human motion.

Earlier works in unconstrained motion generation primarily depend on
autoregressive models [10, 51, 54]. CSGN [50] learns a mapping from latent
vector sampled from the Gaussian process to human motion sequence by a series
of convolution operations in an adversarial way. But due to the finite
receptive field, it has the inherent limitation to capture very long-term
dependencies and can only generate human motion at a fixed predetermined time
resolution. NeMF [11] directly extends the recent developments in neural
implicit represent action [31, 42] to generative modeling but such models are
extremely limited as the whole generated process which is infinite-dimensional
(continuous human motion) depends only on a finite-dimensional latent vector.

Generating human motion from action label or description largely dominates the
field on conditional motion generation [9, 2, 1]. Both ACTOR [30] and TEMOS
[29] use a transformer architecture to generate human motion, whereas the
former uses a pre-trained language model DistilBERT [36] to encode the
sentence description. GOAL [44] generates human motion with the specific goal
of grasping objects. TEACH [3] is also a text-conditioned human motion
generator but it provides temporal control over the generated motion.
MotionCLIP [45, 14] leverages CLIP [32] to generate unseen actions and body
shapes.

The problem of interactive multi-person motion generation is particularly
challenging and requires joint modeling of multiple human motions. ActFormer
[43] provides a framework to generate action-conditioned single and multi-
person motion sequences using a transformer architecture. MUGL [24] uses a
duration-aware autoregressive model to generate a multi-person motion sequence
from an action label.

Figure 1: The PaCMO generator architecture. It takes the motion of one actor
in a dyadic activity as the condition and generates the motion of the other
partner. The blue circles represent the concatenation operation in function
spaces.  Figure 2: The PaCMO discriminator architecture. It takes the motion
of two people involved in an interactive dyadic activity as input and maps it
to a real number $r\in\mathbb{R}$.

###  2.2 Neural Operators

Neural Operators [20, 21, 17] are a new paradigm in machine learning that
learns a mapping between function spaces. Traditional neural networks can only
map between finite-dimensional euclidean space i.e., can only process finite-
dimensional vectors or sets. Neural operator overcomes this limitation and is
discretization invariant. Among the different parameterizations of the neural
operator, Fourier Neural Operators (FNO) [20] have proven to be extremely
effective in achieving state-of-the-art results in learning to solve different
partial differential equations (PDE). An FNO $\mathcal{G}$ can be expressed as

$\mathcal{G}:\\{a:\mathcal{D}_{A}\rightarrow\mathbb{R}^{d_{a}}\\}\rightarrow\\{u:\mathcal{D}_{u}\rightarrow\mathbb{R}^{d_{u}}\\}$

where $a$ is the input function and $u$ is the output function. A neural
Operator architecture consists of three principal components.

  1. 1.

Point-wise operator $P$ which transforms the input function $a$ to higher
dimension spaces by mapping it to a function with higher co-domain dimension.

  2. 2.

A series of $n$ stacked non-linear neural integral operator layer
$\\{L_{i}\\}_{i=1}^{i=n}$, where $n$ is a hyper-parameter of the model. Each
layer $L_{i}$ is composed of a global kernel operator and a point-wise
operator. The layer $L_{i}$ maps input function $v_{i}$ to the function
$v_{i+1}$, which can be described as

$v_{i+1}=L_{i}(v_{i})=\sigma(\int\kappa_{i}(x,y)v_{i}(x)d\mu(y)+W_{i}v_{i})$

where $\mu$ is the measure over the domain of function $v_{i}$. The kernel
integration is usually computed in the Fourier domain as
$\mathcal{F}^{-1}(R_{i}\cdot\mathcal{F}(v_{i}))$, where $\mathcal{F}$ is the
Fourier transform and $R_{i}$ is a complex-valued matrix representing kernel
$\kappa_{i}$. And the $\sigma$ is a non-linearity of choice. In practice GELU
[12] is widely used [20, 34].

  3. 3.

The series of integral operators are followed by a projection operator $Q$. It
projects the output function of $L_{n}$ to match the desired output function
$u$.

Due to the ability to capture complex nonlinear dynamics, FNOs are being
employed to solve complex real-world problems such as CO2 injection [49],
weather modeling [28], and seismology [52]. Recently proposed GANO [33]
introduces a generative adversarial approach to learn probability on function
spaces generalizing GAN which is the first principled approach for generative
models in function spaces.

The problem of generating human motion for interactive activities conditioned
by the motion of other interacting humans is not well-explored. Often in
human-robot interaction, AR/VR, and VFX design, we need to generate the motion
of digital humans depending on the motion of the real actors. To the best of
our knowledge, PaCMO is the first work to address such an important problem
and we do that by introducing a conditional generative neural operator
architecture.

##  3 Methodology

In this section, we provide the mathematical formulation of our problem. Then
we formalize a general construction for the conditional generative adversarial
neural operator. Next, we propose the ($\text{F}^{2}\text{ID}$) score, an
adoption of Fréchet inception distance (FID) score for infinite dimensional
vector (function) spaces.

###  3.1 Problem Formulation

We consider human motion as a sequence of skeletal poses. In skeletal
representation with $J$ joints, the human pose or gesture can be represented
as a vector

$g=[x^{1},y^{1},z^{1},x^{2},y^{2},z^{2},\dots
x^{J},y^{J},z^{J}]\in\mathbb{R}^{(3\times J)}$

where $(x^{i},y^{i},z^{i})$ represents the location of $i$th joint in 3D
euclidean space. Let $\mathcal{M}$ be the motion space of humans such that for
$m\in\mathcal{M},m:t\rightarrow\\{g\in\mathbb{R}^{(3\times J)}\\}$, where
$t\in\mathbb{R}$ is the time. Then the probability space
$(\mathcal{M}\times\mathcal{M},\sigma(\mathcal{M})\times\sigma(\mathcal{M}),\mathcal{P}_{\mathcal{M}\times\mathcal{M}})$
represents the probability space of motions for all dyadic human activities.
In other words, for any
$(m_{1},m_{2})\sim\mathbb{P}_{\mathcal{M}\times\mathcal{M}}$, motion sequences
$m_{1}$ and $m_{2}$ form a valid dyadic human activity when
$\mathbb{P}_{\mathcal{M}\times\mathcal{M}}\in\mathcal{P}_{\mathcal{M}\times\mathcal{M}}$.
Also we assume that $(m_{1},m_{2})$ and $(m_{2},m_{1})$ represents the same
dyadic activity i.e., $m_{1}$ and $m_{2}$ are exchangeable.

Here we want to learn the conditional probability distribution
$\mathbb{P}_{\mathcal{M}|\mathcal{M}}$ such that for any $m_{2}\in\mathcal{M}$
and $m_{1}\sim\mathbb{P}_{\mathcal{M}|m_{2}}$, $m_{1},m_{2}$ constitute a
valid dyadic human activity. Note that we do not assume any particular time
discretization or length of the motion sequence.

###  3.2 Conditional Generative Adversarial Neural Operator (cGANO)

Let $\mathcal{U},\mathcal{A},$ and $\mathcal{C}$ be Polish function spaces
such that for any

$\forall
u\in\mathcal{U},u:\mathcal{D}_{\mathcal{U}}\rightarrow\mathbb{R}^{d_{\mathcal{U}}},~{}~{}\forall
a\in\mathcal{A},a:\mathcal{D}_{\mathcal{A}}\rightarrow\mathbb{R}^{d_{\mathcal{A}}},$
$\text{and~{}}\forall
c\in\mathcal{C},c:\mathcal{D}_{\mathcal{C}}\rightarrow\mathbb{R}^{d_{\mathcal{C}}}$

We define the space of operators $\bm{G}$ (Generators) such that for any
$\mathcal{G}\in\bm{G},\mathcal{G}:\mathcal{A}\times\mathcal{C}\rightarrow\mathcal{U}$
and we also define the space of functional $\bm{L}$ (Discriminators) that for
any $d\in\bm{L},d:\mathcal{U}\times\mathcal{C}\rightarrow\mathbb{R}$.

Let the probability space $(\mathcal{A},\sigma(A),\mathcal{P}_{\mathcal{A}})$
is induced by Gaussian Random Field (GRF) and
$(\mathcal{U}\times\mathcal{C},\sigma(\mathcal{U})\times\sigma(\mathcal{C}),\mathcal{P}_{\mathcal{U}\times\mathcal{C}})$
denotes the probability space on the product space
$\mathcal{U}\times\mathcal{C}$. We want to model the conditional probability
$\mathbb{P}_{\mathcal{U}|\mathcal{C}}$ with the push-forward measure of
$\mathbb{P}_{\mathcal{A}}$ under the map $\mathcal{G}$ given $\mathcal{C}$
i.e., $\bm{G}_{\mathcal{C}}\sharp\mathbb{P}_{\mathcal{A}}$. Hence we define
the Wasserstein distance between the two distributions as follows

$W(\mathbb{P}_{\mathcal{U}\times\mathcal{C}},\mathbb{P}_{\mathcal{C}}\cdot\bm{G}_{\mathcal{C}}\sharp\mathbb{P}_{\mathcal{A}})=\sup_{\begin{subarray}{c}d:d\in\bm{L}\\\
Lip(d)\leq
1\end{subarray}}\mathbb{E}_{\mathbb{P}_{\mathcal{U}\times\mathcal{C}}}[d]-\mathbb{E}_{\mathcal{G}\sharp\mathbb{P}_{\mathcal{A}},\mathbb{P}_{\mathcal{C}}}[d]$
(1)

where $\mathbb{P}_{\mathcal{C}}$ comes from the joint probability measure
$\mathbb{P}_{\mathcal{U}\times\mathcal{C}}$. We aim to find
$\mathcal{G}\in\bm{G}$ that minimizes the Wasserstein distance between the
true and generated probability measure. Assuming
$(\mathcal{U}\times\mathcal{C})^{*}$ denotes the dual space of
$\mathcal{U}\times\mathcal{C}$, we express the Lipshitz constraint as [8, 33]

$Lip(d)\leq 1\iff\|\partial d\|_{(\mathcal{U}\times\mathcal{C})^{*}}\leq 1$

Following the construction of GANO, we can rewrite the objective without the
constraint as

$\inf_{\mathcal{G}\in\bm{G}}\sup_{d\in\bm{L}}\mathbb{E}_{\mathbb{P}_{\mathcal{U}\times\mathcal{C}}}[d]-\mathbb{E}_{\mathbb{P}_{\mathcal{C}},\mathcal{G}\sharp\mathbb{P}_{\mathcal{A}}}[d]+\lambda\mathbb{E}_{\mathbb{P}^{\prime}_{\mathcal{U}\times\mathcal{C}}}(\|\partial
d\|_{(\mathcal{U}\times\mathcal{C})^{*}}-1)^{2}$ (2)

where
$\mathbb{P}^{\prime}_{\mathcal{U}\times\mathcal{C}}:=\lambda\mathbb{P}_{\mathcal{C}}\cdot\mathcal{G}\sharp\mathbb{P}_{\mathcal{A}}+(1-\lambda)\mathbb{P}_{\mathcal{U}\times\mathcal{C}}$
given $\lambda\sim U(0,1)$.

Now we construct the operator on the product space of
$\mathcal{A}\times\mathcal{C}$. Let’s decompose $\bm{G}$ as

$\bm{G}:=\bm{G}^{\prime}\circ\bm{G}_{con}$

where $\mathcal{G}_{con}$ defines concatenation operator such that for any
$\mathcal{G}_{con}\in\bm{G}_{con},\mathcal{G}_{con}:\mathcal{A}\times\mathcal{C}\rightarrow\mathcal{V}$.
Here the function space $\mathcal{V}$ is defined such that for any

$v\in\mathcal{V},v:(D_{\mathcal{A}}\times\mathcal{D}_{\mathcal{C}})\rightarrow\mathbb{R}^{d_{\mathcal{A}}+d_{\mathcal{C}}}$

and

$v(x\oplus y)=a(x)\oplus
c(y)~{}~{}\forall(x,y)\in(D_{\mathcal{A}}\times\mathcal{D}_{\mathcal{C}}).$

Where $\oplus$ denotes the concatenation of functions and $\bm{G}^{\prime}$ is
the space of operators mapping $\mathcal{V}$ to the target function space
$\mathcal{U}$.

In many practical scenarios, we want to learn the conditional probability
measure of the function space where the conditional variables are finite-
dimensional (e.g., action labels). In other words, we want to model
$\mathbb{P}_{\mathcal{U}|s}$ where $s\in\mathcal{D}_{s}$, where
$\mathcal{D}_{s}$ is a finite-dimensional vector space.

In such cases, we learn a mapping from $\mathcal{D}_{s}$ to $\mathcal{C}$. We
define the mapping space $\mathcal{E}$ such that for any
$E\in\mathcal{E},E:\mathcal{D}_{\mathcal{C}}\times\mathcal{D}_{s}\rightarrow\mathbb{R}^{d_{\mathcal{C}}}$.
For any $s\in D_{s}$ the map $E$ maps it to $c_{s}\in\mathcal{C}$ such that

$c_{s}(y)=E(y\oplus s)~{}~{}~{}~{}\forall y\in\mathcal{D}_{\mathcal{C}}$

In this work, we use U-NO [34], a U-shaped Fourier neural operator
architecture for the generator (see Fig. 1). For the discriminator, we also
use the U-NO architecture followed by a functional layer (see Fig. 2). A
functional layer maps functions to real numbers which can be described as
following

$r=\int_{D_{v}}\kappa_{f}(x)v_{n}(x)dx$

where $v:\mathcal{D}_{v}\rightarrow\mathbb{R}$ is the input function, $r$ is a
real number, and $\kappa_{f}:\mathcal{D}_{v}\rightarrow\mathbb{R}$ is a
learnable function in the functional layer implemented by a fully connected
neural network.

For conditional human motion generation, the function space $\mathcal{C}$ and
$\mathcal{U}$ is the 3D human motion space i.e.,
$\mathcal{C}=\mathcal{U}=\mathcal{M}$, i.e., for any operator
$\mathcal{G}\in\bm{G}$

$\mathcal{G}:\mathcal{A}\times\mathcal{M}\rightarrow\mathcal{M}$

where for any Gaussian process $a\in\mathcal{A}$, $m_{2}\in\mathcal{M}$, and
$m_{1}=\mathcal{G}(a,m_{2})$, $(m_{1},m_{2})$ constitutes a valid dyadic human
motion for two person.

###  3.3 Construction of ($\text{F}^{2}\text{ID}$)

Quantitative evaluation of high-dimensional generative models is an open
problem. Among different proposed metrics, Fréchet Inception Distance or FID
[13] is well used. Though it was first introduced to evaluate generative
models for images, it has been adapted for other generative tasks (e.g.,
motion, voice). FID measures the Wasserstein-2 distance between the
distribution of inception features of the generated and real images. For other
modalities, features from the real and generated data are extracted using
suitable models. To get a closed-form solution of Wasserstein-2 distance, the
extracted features from real and generated data are modeled by two
multivariate Gaussian distributions. Let $(\mu,\Sigma)$,
$(\mu^{\prime},\Sigma^{\prime})$ be the mean and covariance of real and
generated data respectively. Then the FID score is calculated as

$\displaystyle FID$
$\displaystyle=W_{2}^{2}({\mathcal{N}}(\mu,\Sigma),{\mathcal{N}}(\mu^{\prime},\Sigma^{\prime}))^{2}$
$\displaystyle=\lVert\mu-\mu^{\prime}\rVert_{2}^{2}+\operatorname{tr}\left(\Sigma+\Sigma^{\prime}-2\left(\Sigma^{\frac{1}{2}}\cdot\Sigma^{\prime}\cdot\Sigma^{\frac{1}{2}}\right)^{\frac{1}{2}}\right).$

But multi-variate Gaussian distributions are not suitable to model data (or
features) from infinite-dimensional function spaces or processes (both
stochastic and deterministic). Therefore, the FID score is not a suitable
metric for evaluating generative models in function spaces.

To address this problem, we introduce Functional FID ($\text{F}^{2}\text{ID}$)
scores. Here, we model the extracted feature by the Gaussian process. A
Gaussian process, $GP(m,k)$, is defined by a mean function $m$ and a co-
variance function $k$. Wasserstein-2 distance between Gaussian processes is
well defined and has a closed-form solution under the assumption that they are
indexed over a compact domain $X\subset\mathbb{R}^{n}$ with a metric $d(.,.)$,
$m\in L^{2}(x)$, and $k\in L^{2}(X\times X)$ [25]. If the features extracted
from real and generated data are modeled by two Gaussian processes
$GP(m_{1},k_{1})$ and $GP(m_{2},k_{2})$ with associated covariance operator
$K_{1}$ and $K_{2}$ respectively. Then the $\text{F}^{2}\text{ID}$ score is
defined as

$\text{F}^{2}\text{ID}=d_{2}(m_{1},m_{2})+Tr(K_{1}+K_{2}-2(K_{1}^{\frac{1}{2}}\cdot
K_{2}\cdot K_{1}^{\frac{1}{2}}))^{\frac{1}{2}}$ (3)

where for any operator $T$, $Tr(T)$ is the trace of the operator $T$ and we
take $d(.,.)$ as the metric induced by $L^{2}$ norm. In this work, we define
$GP$ over time $t$. Given the features from the generated and real motions, we
model them by the Gaussian processes $GP_{1}(m_{1},k_{1})$ and
$GP_{2}(m_{2},k_{2})$ respectively using moment matching. We evaluate mean and
covariance functions over a finite discretization of the domain. Finally, we
apply the Eqn.3 to calculate $\text{F}^{2}\text{ID}$ (details in
supplementary). Finite resolution computation of Wasserstein-2 distance is
proven to converge to true Wasserstein-2 distance with the increase of
discretization resolution [25, Theorem 8].

Figure 3: Samples of conditional generation of human motion by PaCMO. The top
row shows the input condition motion (Red and Green) on the left and the
complete motion on the right by combining the generated motion (in Magenta and
Blue) and input motion (Red and Green). The following rows only show the
complete motion combining the condition motions and generated motions from
PaCMO. Different task labels are indicated on the left. Though PaCMO does not
take action labels or descriptions as input, we can observe that PaCMO
generated motions interact with the input motions depending on the action.

##  4 Experiments

NTU-RGB+D
---
Methods |  $\text{F}^{2}\text{ID}$ $\downarrow$ | Diversity Score $\rightarrow$ | MMD-A $\downarrow$ | MMD-S$\downarrow$ | APE (root joint)$\downarrow$ | AVE(root joint)$\downarrow$
Real Motion | 0.001 | 1.90 | 0 | 0 | 0 | 0
PaCMO | 0.002 | 1.91 | 5.8e-3 | 3.8e-3 | 0.03 | 0.23
GRU baseline | 0.351 | 1.92 | 7.4e-3 | 4.4e-3 | 0.05 | 0.35
Table 1: Performance NTU-RGB+D 120 dataset. $(\downarrow)$ denotes the lower
the score the better and $(\rightarrow)$ denotes the closer to the real
dataset the better. Each metric is calculated five times and the mean is
reported. DuetDance
---
Methods |  $\text{F}^{2}\text{ID}$ $\downarrow$ | Diversity Score$\rightarrow$ | MMD-A $\downarrow$ | MMD-S$\downarrow$ | APE (root joint)$\downarrow$ | AVE(root joint)$\downarrow$
Real Motion | 1.6e-4 | 0.261 | 0 | 0 | 0 | 0
PaCMO | 4.0e-4 | 0.264 | 6.7e-3 | 6.67e-3 | 0.01 | 0.01
GRU baseline | 494.9e-4 | 0.190 | 23.36e-3 | 23.35e-3 | 0.04 | 0.08
Table 2: Performance DuetDance dataset. $(\downarrow)$ denotes the lower the
score the better and $(\rightarrow)$ denotes the closer to the real dataset
the better. Each metric is calculated five times and the mean is reported.

###  4.1 Datasets

For the purpose of our work, we need an interactive dyadic human activity
dataset with skeletal pose annotation. Unfortunately, such publicly available
datasets are scarce. We use two existing datasets: NTU-RGB+D 120[22] and the
DuetDance [18] datasets.

  * •

NTU-RGB+D 120 is the extended version of original NTU-RGB+D [39] dataset. It
contains 120 action types (such as handshake, pushing) of 106 subjects. The
provided skeleton data contains 3D locations of 25 major body joints. For our
purpose, we only considered actions with two persons resulting in 26K
sequences (motion pairs). Though we do not clip any motion to restrict the
length, we are not modeling the movement of the finger, as it requires finer
precision noise-free data [44] and we exclude two joints from each hand
representing the tip of the finger and thumb.

  * •

DuetDance The dataset is created from complex dance poses such as Cha-cha,
Jive, Rumba, Salsa, and Samba from the curated video tutorial on Youtube. Due
to the nature of the actions, the motion encompasses long-term correlations
between physical motions. From the curated videos the skeletal positions are
calculated using LCRNet++ [35]. The skeletal data contain 3D location of 15
major body joints. The dataset consists of one very long sequence ($\sim$ 48K
time steps) for each of the dance types. So to create more training instances,
we divide one long sequence into smaller sequences non-overlapping sequences.

###  4.2 Hyperparameters

We use U-shaped neural operator architecture [34] in both the generator and
discriminator network similar to GANO [33]. We use seven stacked integral
operator layers with GELU as non-linearity (see Fig. 1 and 2). We divide each
dataset into training and evaluation sets randomly at a ratio of $8:2$. For
each dataset, we train our model for 200 epochs with a single NVIDIA A100 GPU
on the training set. We used an ADAM optimizer with an initial learning rate
of $10^{-4}$ which is halved after every 50 epochs. We train our model on the
skeletal data represented as joint locations in 3D space. While training we do
not limit the length of the motion sequence by clipping. To maintain the
symmetry of the human body, We also add additional constraints on the
generator by penalizing asymmetric limb generations i.e., restricting the
generator to produce left and right legs of the same length with soft
constraints. Empirically we have found that these soft constraints increase
the convergence rate and the realism of the generated motions.

###  4.3 Experiment Results

####  4.3.1  Evaluation Metrics

  * •

$\text{F}^{2}\text{ID}$: We used our proposed $\text{F}^{2}\text{ID}$ score as
one of the the evaluation metric. To extract features from the motion
sequences, we have trained an auto-encoder neural operator to extract features
from human motion. The auto-encoder operator can be defined as
$\mathcal{G}:(\mathcal{M}\times\mathcal{M})\rightarrow\mathcal{M}\times\mathcal{M}$.
The auto-encoder also is composed of seven non-linear integral operator layers
and the output of the bottleneck layer is taken as a feature. We extract
features from randomly sampled (with replacement) 1000 generated and real
dyadic human motions and calculate the $\text{F}^{2}\text{ID}$. The process is
repeated five times and the mean $\text{F}^{2}\text{ID}$ score is reported.

  * •

Diversity Score: Diversity score represents the variance in the generated
data. We randomly sample (with replacement) 1000 motion from the generator and
extract the features using the auto-encoder. We divide the features into two
sets $S$ and $S^{\prime}$ each containing 500 motions. Then the diversity
score is calculated as

$\text{Diversity Score
}=\frac{1}{5e2}\sum_{i=1}^{5e2}L_{2}(S_{i},S^{\prime}_{i})^{2}$

where $L_{2}(.,.)$ is the $L_{2}$ norm. The average diversity score of five
repeated calculations is reported.

  * •

Maximum Mean Discrepancy (MMD) Maximum Mean Discrepancy measures the distance
between two distributions and is used to measure the discrepancy between
generated and real data [4, 53, 46]. Following the construction in [24], we
also used the kernel trick to calculate the MMD. We also present two MMD-based
metric MMD-A which measures the dissimilarity per time step basis and MMD-S
which measure the dissimilarity based on the whole sequence. We use the
official implementation of the work MUGL [24].

  * •

Average Position Error (APR) Average position error for joint $j$ between
generated motion $\\{m_{i}^{\prime}\\}_{i=0}^{n}$ and the corresponding ground
truth real motion $\\{m_{i}\\}_{i=0}^{n}$is defined as

$APE(j)=\frac{1}{N}\sum_{i=0}^{n}\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}||m^{\prime
j}_{i}(t)-m^{j}_{i}(t)||_{2}$

where $m^{j}_{i}(t)$ is the position of $jth$ joint of motion $m_{i}$ at time
$t$ and $T_{i}$ is length of motion $m_{i}$

  * •

Average Variance Error (AVE) Following the above notation, AVE can be defined
as

$AVE(j)=\frac{1}{N}\sum_{i=0}^{n}||\sigma(m^{\prime
j}_{i})-\sigma(m^{j}_{i})||_{2}$

where $\sigma(m^{j}_{i})$ is the variance of joint $j$ over time. We calculate
both APR and AVE without any normalization of the data.

It is pointed out in recent studies [24] that feature-based metrics (e.g.,
accuracy) are not reliable to evaluate action sequences. As a result, we kept
both feature-based (FID, Diversity score) metrics and metrics calculated
directly from the generated and real data (MMD, AVE, APE) as our evaluation
metrics. We report the mean value for every metric over five repeated
calculations on the randomly sampled data from the evaluation set.

####  4.3.2 Baseline

The problem of partner-conditioned human motion generation in function spaces
is new and we do not have any method to compare with. Previously, LSTM-based
models are used to generate body motion from audio [41]. Here we also design a
GRU base model to predict our desired human motion from the motion of their
partner in the action. The model consists of five stacked GRU layers with a
$200$ dimensional hidden state. This is followed by three layered fully
connected neural networks with GELU activation. This type of sequence-to-
sequence (seq-to-seq) architecture has been used successfully in many complex
real-world problems such as language translation, weather prediction [37, 38,
26, 40]. So, this model serves as an appropriate deterministic baseline for
this problem.

| Real data |  | PaCMO Generated
---
data
| Perceived as
---
Realistic
0.81 | 0.75
| Perceived as
---
Unrealistic
0.19 | 0.25
Table 3: Result of the survey of real and PaCMO generated motion.

####  4.3.3 Results

We present the performance of PaCMO on the DuetDance dataset shown in Table 2
and on the NTU-RGD+D 120 dataset shown in Table 1. $\text{F}^{2}\text{ID}$ is
most important in evaluating the quality of the generated motions. In both the
datasets, PaCMO achieves lower $\text{F}^{2}\text{ID}$ vastly outperforming
the baseline and the diversity score of PaCMO is closer to the real dataset.
Unlike motion generation from the action label where the input condition is
just a fixed label, here the input condition is the motion of the partner for
a dyadic activity and contains much variability even within a single action
category. As a result, both GRU base seq-to-seq models and PaCMO achieved a
diversity score close to the real dataset. For feature-free evaluation metrics
(MMD, APE, AVE), PaCMO achieves excellent results outperforming the baseline.
This shows that the motion generated by PaCMO is very realistic and close to
the real dataset.

####  4.3.4 Perceptual Evaluation

To bolster the finding in the previous section, we also conduct a visual
perceptual evaluation using of 50 web-based participants. In this study, we
provide the participants with both real and PaCMO-generated motions and ask
them to label them as real or fake (computer generated). Each participant is
asked to label 14 different randomly selected motions from both real and
generated data. The result of the study is shown in Table 3. We can observe
that our generated motion is labeled as real at a very high rate ($76\%$),
which makes it suitable for real applications such as AR/VR and animation. For
qualitative evaluation, we present a few motions generated by PaCMO along the
condition motion (see Fig. 3) and we observe that the generated motions are
realistic and in sync with the given motion of the partner as a condition.

##  5 Conclusion, Limitations, and Future

This work addresses an important yet under-explored problem of partner-
dependent human motion generation in dyadic human activities. And as a
solution, we propose PaCMO, a novel generative neural operator architecture.
Unlike existing conditional generative approaches, PaCMO is resolution
invariant, able to model global long-term dependencies, and also does not
depend on any particular representation of human poses. We also propose
$\text{F}^{2}\text{ID}$ as a suitable evaluation metric for generative models
for infinite dimensional spaces. Our work is the first step in generating
continuous conditional human motion generation. In the future, we plan to
extend the work from dyadic activities to multi-person group activities. Also,
we aim to model complected human motion such as grasping and emotions.

## References

  * [1] Hyemin Ahn, Timothy Ha, Yunho Choi, Hwiyeon Yoo, and Songhwai Oh.  Text2action: Generative adversarial synthesis from language to action.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5915–5920. IEEE, 2018.
  * [2] Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In 2019 International Conference on 3D Vision (3DV), pages 719–728. IEEE, 2019.
  * [3] Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol.  Teach: Temporal action composition for 3d humans.  arXiv preprint arXiv:2209.04066, 2022.
  * [4] Neeraj Battan, Yudhik Agrawal, Sai Soorya Rao, Aman Goel, and Avinash Sharma.  Glocalnet: Class-aware long-term human motion synthesis.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 879–888, 2021.
  * [5] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [6] Richard M Dudley.  Sample functions of the gaussian process.  In Selected works of RM Dudley, pages 187–224. Springer, 2010.
  * [7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [9] Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng.  Action2motion: Conditioned generation of 3d human motions.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020.
  * [10] Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura.  A recurrent variational autoencoder for human motion synthesis.  In 28th British Machine Vision Conference, 2017.
  * [11] Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou.  Nemf: Neural motion fields for kinematic animation.  arXiv preprint arXiv:2206.03287, 2022.
  * [12] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [14] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  arXiv preprint arXiv:2205.08535, 2022.
  * [15] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  Advances in neural information processing systems, 31, 2018.
  * [16] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [17] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Learning maps between function spaces.  arXiv preprint arXiv:2108.08481, 2021.
  * [18] Jogendra Nath Kundu, Himanshu Buckchash, Priyanka Mandikal, Anirudh Jamkhandi, Venkatesh Babu RADHAKRISHNAN, et al.  Cross-conditioned recurrent networks for long-term synthesis of inter-person human motion interactions.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2724–2733, 2020.
  * [19] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  Advances in neural information processing systems, 32, 2019.
  * [20] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  arXiv preprint arXiv:2010.08895, 2020.
  * [21] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Graph kernel network for partial differential equations.  arXiv preprint arXiv:2003.03485, 2020.
  * [22] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  IEEE transactions on pattern analysis and machine intelligence, 42(10):2684–2701, 2019.
  * [23] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [24] Shubh Maheshwari, Debtanu Gupta, and Ravi Kiran Sarvadevabhatla.  Mugl: Large scale multi person conditional action generation with locomotion.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 257–265, 2022.
  * [25] Anton Mallasto and Aasa Feragen.  Learning from uncertain curves: The 2-wasserstein metric for gaussian processes.  Advances in Neural Information Processing Systems, 30, 2017.
  * [26] Sanidhya Mangal, Poorva Joshi, and Rahul Modak.  Lstm vs. gru vs. bidirectional rnn for script generation.  arXiv preprint arXiv:1908.04332, 2019.
  * [27] George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  J. Mach. Learn. Res., 22(57):1–64, 2021.
  * [28] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al.  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  arXiv preprint arXiv:2202.11214, 2022.
  * [29] Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions supplementary material.
  * [30] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10985–10995, 2021.
  * [31] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [33] Md Ashiqur Rahman, Manuel A Florez, Anima Anandkumar, Zachary E Ross, and Kamyar Azizzadenesheli.  Generative adversarial neural operators.  arXiv preprint arXiv:2205.03017, 2022.
  * [34] Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli.  U-no: U-shaped neural operators.  arXiv preprint arXiv:2204.11127, 2022.
  * [35] Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid.  Lcr-net++: Multi-person 2d and 3d pose detection in natural images.  IEEE transactions on pattern analysis and machine intelligence, 42(5):1146–1161, 2019.
  * [36] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [37] Ljubisa Sehovac and Katarina Grolinger.  Deep learning for load forecasting: Sequence to sequence recurrent neural networks with attention.  IEEE Access, 8:36411–36426, 2020.
  * [38] Ljubisa Sehovac, Cornelius Nesen, and Katarina Grolinger.  Forecasting building energy consumption with deep learning: A sequence to sequence approach.  In 2019 IEEE International Congress on Internet of Things (ICIOT), pages 108–116. IEEE, 2019.
  * [39] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  * [40] Apeksha Shewalkar.  Performance evaluation of deep neural networks applied to speech recognition: Rnn, lstm and gru.  Journal of Artificial Intelligence and Soft Computing Research, 9(4):235–245, 2019.
  * [41] Eli Shlizerman, Lucio Dery, Hayden Schoen, and Ira Kemelmacher-Shlizerman.  Audio to body dynamics.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7574–7583, 2018.
  * [42] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  * [43] Ziyang Song, Dongliang Wang, Nan Jiang, Zhicheng Fang, Chenjing Ding, Weihao Gan, and Wei Wu.  Actformer: A gan transformer framework towards general action-conditioned 3d human motion generation.  arXiv preprint arXiv:2203.07706, 2022.
  * [44] Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas.  Goal: Generating 4d whole-body motion for hand-object grasping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13263–13273, 2022.
  * [45] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  arXiv preprint arXiv:2203.08063, 2022.
  * [46] Ilya O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf.  Minimax estimation of maximum mean discrepancy with radial kernels.  Advances in Neural Information Processing Systems, 29, 2016.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ${\mathcal{L}}$ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [48] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al.  Scipy 1.0: fundamental algorithms for scientific computing in python.  Nature methods, 17(3):261–272, 2020.
  * [49] Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson.  U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow.  Advances in Water Resources, 163:104180, 2022.
  * [50] Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin.  Convolutional sequence generation for skeleton-based action synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4394–4402, 2019.
  * [51] Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee.  Mt-vae: Learning motion transformations to generate multimodal human dynamics.  In Proceedings of the European conference on computer vision (ECCV), pages 265–281, 2018.
  * [52] Yan Yang, Angela F Gao, Jorge C Castellanos, Zachary E Ross, Kamyar Azizzadenesheli, and Robert W Clayton.  Seismic wave propagation and inversion with neural operators.  The Seismic Record, 1(3):126–134, 2021.
  * [53] Ping Yu, Yang Zhao, Chunyuan Li, Junsong Yuan, and Changyou Chen.  Structure-aware human-action generation.  In European conference on computer vision, pages 18–34. Springer, 2020.
  * [54] Yan Zhang, Michael J Black, and Siyu Tang.  Perpetual motion: Generating unbounded human motion.  arXiv preprint arXiv:2007.13886, 2020.
  * [55] Olek C Zienkiewicz, Robert Leroy Taylor, and Jian Z Zhu.  The finite element method: its basis and fundamentals.  Elsevier, 2005.

##  Appendix A Supplementary Material

###  A.1 Calculation of $\text{F}^{2}\text{ID}$

In this section we will discuss the approximation of Wasserstein-2 distance
for Gaussian processes and $\text{F}^{2}\text{ID}$.

Data: Features of real data $F_{t}$, features of generated data $F_{g}$, the
grid size $r$

Result: $\text{F}^{2}\text{ID}$ score

1 model $F_{t}$ by $GP(m_{1},k_{1})$;

2 model $F_{g}$ by $GP(m_{2},k_{2})$;

3 g[] $\leftarrow$ $r\text{ evenly spaced number on the interval }[0,1]$;

4 Initialize matrix $M_{t}$ and $M_{g}$ of size $r\times r$;

5 for  _i in $\\{1,2,\dots r\\}$_ do

6 for  _j in $\\{1,2,\dots r\\}$_ do

7 $M_{t}[i][j]$ $\leftarrow$ $\frac{1}{r}k_{1}(g[i],g[j])$ ;

8 $M_{g}[i][j]$ $\leftarrow$ $\frac{1}{r}k_{2}(g[i],g[j])$ ;

9

10

11L $\leftarrow$
$M_{t}+M_{g}-2(M_{t}^{\frac{1}{2}}M_{g}M_{t}^{\frac{1}{2}})^{\frac{1}{2}}$;

12 $\text{F}^{2}\text{ID}$=
$\frac{1}{r}\sum_{i=1}^{r}(m_{1}(g[i])-m_{2}(g[i]))^{2}+\sum_{i=1}^{r}L[g[i]][g[i]]$;

Algorithm 1 Calculation of $\text{F}^{2}\text{ID}$

Let $GP(m,k)$ be a Gaussian process where the mean function $m\in L^{2}(X)$,
and the covariance function $k\in L^{2}(X\times X)$. For the simplicity of
calculation, without any loss of generality we will assume that $X$ is a
compact subset of $\mathbb{R}^{d}$. We also assume that $\lambda(X)=1$ with
Lebesgue measure $\lambda$ on $R^{d}$. We assume that the function in
$L^{2}(X\times X)$ are discretized uniformly at the finite set of point
$x_{i}\in X_{i}$.

For numerical feasibility, we choose only a finite set of abnormal basis
$\\{e_{1},e_{2},..e_{r}\\}$ for the function spaces $L^{2}(X)$. The domain $X$
is discretized uniformly with a grid size of $r$ such that
$\lambda(X_{i})=\frac{1}{r}~{}~{}\forall i\in\\{1,2,\dots r\\}$ and
$\bigcup_{i=0}^{r}X_{i}=X$.

The basis function $e_{i}$ is defined to be equal to $\frac{1}{c}\sqrt{r}$ at
the point $x_{i}$ and $0$ outsize the domain $X_{i}$. Such choice constants
maintains the basis functions at unit norm. The $c$ is the universal constant
depends only the particular choice of $e_{i}$ [55], we assume it to be
$\approx 1$ for our calculation.

The covariance operator $K$ associated with covariance function $k$ is defined
as

$[K\phi](x)=\int_{X}k(x,s)\phi(s)ds$ (4)

Applying the operator $K$ on the the basis function $e_{i}$, we get

$\displaystyle[Ke_{i}](x)$ $\displaystyle=\int_{X}k(x,s)e_{i}(s)ds$
$\displaystyle=\int_{X_{i}}k(x,s)ds$
$\displaystyle\approx\frac{1}{\sqrt{r}}k(x,x_{i})~{}~{}~{}[\text{where
}x_{i}\in X_{i}]$

Here we assume that for a fixed $x$ the covariance function $k(x,s)$ is
constant $\forall s\in X_{i}$.

The trace of the operator $K$ can be defined as

$Tr(K)=\sum_{i=1}^{r}\langle Ke_{i},e_{i}\rangle$

Now,

$\displaystyle\begin{split}\langle
Ke_{i},e_{i}\rangle&=\int_{X}[Ke_{i}](x)e_{i}(x)dx\\\
&=\int_{X}\frac{1}{\sqrt{r}}k(x,x_{i})e_{i}(x)dx\\\
&\approx\int_{X_{i}}k(x,x_{i})dx\\\ &=\frac{1}{r}k(x_{i},x_{i})\end{split}$
(5)

where we assumes that for a fixed $x_{i}$, $k(x,x_{i})$ is constant $\forall
x\in X_{i}$. Now the trace can be written as

$Tr(K)\approx\frac{1}{r}\sum_{i=1}^{r}k(x_{i},x_{i})$

where $x_{i}\in X_{i}$. Assuming a finite set of standard basis we can
construct covariance matrix $M_{K}$ associated with the covariance operator
$K$ defined as follows

$M_{K}[i,j]=\langle Ke_{i},e_{j}\rangle$

where $i,j\in\\{1,2,\dots r\\}$. Following the same procedure in Eqn. 5, we
can show that $M_{K}[i,j]\approx\frac{1}{r}k(x_{i},x_{j})$ where $x_{i}\in
X_{i}$ and $x_{i}\in X_{j}$.

We present a pseudo-code for calculating $\text{F}^{2}\text{ID}$ in Algorithm
1. We assume that the feature functions are defined on the interval $[0,1]$.
The $F_{t}=\\{f_{t}^{1},f_{t}^{2},...,f_{t}^{n}\\}$ and
$F_{g}=\\{f_{g}^{1},f_{g}^{2},...,f_{g}^{n}\\}$ are the set of $n$ feature
functions of the real and generated motion respectively where
$f:[0,1]\rightarrow\mathbb{R}$. The feature functions are extracted from the
bottleneck layer of the auto-encoder neural operator. The auto-encoder neural
operator follows the architecture of PaCMO generator without the skip
connections. Each of the feature functions $f^{i}$ for $i\in\\{1,2,...,n\\}$
for both real and generated is represented using a finite set of basis as
following

$f^{i}(x)=\sum_{j=1}^{r^{i}}F^{i}[j]e^{i}_{j}$

where $r^{i}$ is resolution (discretization grid size) of the feature function
$f^{i}$ which depends on the input resolution feed to the auto-encoder neural
operator. The $e^{i}_{j}$ s are the corresponding basis function which depend
on $r^{i}$ and $F^{i}$ s are the coefficients. We model the the features from
real data by Gaussian process $GP(m_{1},k_{1})$ where

$m_{1}(x)=\mathbb{E}[f_{t}]\approx\frac{1}{n}\sum_{i}^{n}f_{t}^{i}(x)$

and

$\displaystyle k_{1}(x,x^{\prime})$
$\displaystyle=\mathbb{E}[\big{(}f_{t}(x)-m_{1}(x)\big{)}\big{(}f_{t}(x^{\prime})-m_{1}(x^{\prime})\big{)}]$
$\displaystyle\approx\frac{1}{n}\sum_{i}^{n}(f^{i}_{t}(x)-m_{1}(x))(f^{i}_{t}(x^{\prime})-m_{1}(x^{\prime}))$

Here the expectations are taken over the distributions of feature functions
for real data and approximated by the the set $F_{t}$. In the same way we
model the feature functions of the generated data by $GP(m_{2},k_{2})$. To
approximate the 2-Wasserstein metric between these Gaussian processes we fix a
resolution $r$. And the matrix operations are computed using Scipy [48].

###  A.2 More Qualitative Results

Figure 4: Each rows represents a complete dyadic motion with the condition
motion (in Red and Green) and motion generated by PaCMO (in Magenta and Blue).