# A Causal Convolutional Neural Network for Motion Modeling and Synthesis

Shuaiying Hou
Zhejiang University
China
<EMAIL_ADDRESS>
&Weiwei Xu
Zhejiang University
China
<EMAIL_ADDRESS>
&Jinxiang Chai
Texa A&M University
USA
<EMAIL_ADDRESS>&Congyi Wang
Xmov
China
<EMAIL_ADDRESS>&Wenlin Zhuang
Southeast University
China
<EMAIL_ADDRESS>&Yu Chen
Xmov
China
<EMAIL_ADDRESS>&Hujun Bao
Zhejiang University
China
<EMAIL_ADDRESS>&Yangang Wang
Southeast University
China
<EMAIL_ADDRESS>corresponding author

###### Abstract

We propose a novel deep generative model based on causal convolutions for
multi-subject motion modeling and synthesis, which is inspired by the success
of WaveNet in multi-subject speech synthesis. However, it is nontrivial to
adapt WaveNet to handle high-dimensional and physically constrained motion
data. To this end, we add an encoder and a decoder to the WaveNet to translate
the motion data into features and back to the predicted motions. We also add
1D convolution layers to take skeleton configuration as an input to model
skeleton variations across different subjects. As a result, our network can
scale up well to large-scale motion data sets across multiple subjects and
support various applications, such as random and controllable motion
synthesis, motion denoising, and motion completion, in a unified way. Complex
motions, such as punching, kicking and, kicking while punching, are also well
handled. Moreover, our network can synthesize motions for novel skeletons not
in the training dataset. After fine-tuning the network with a few motion data
of the novel skeleton, it is able to capture the personalized style implied in
the motion and generate high-quality motions for the skeleton. Thus, it has
the potential to be used as a pre-trained network in few-shot learning for
motion modeling and synthesis. Experimental results show that our model can
effectively handle the variation of skeleton configurations, and it runs fast
to synthesize different types of motions on-line. We also perform user studies
to verify that the quality of motions generated by our network is superior to
the motions of state-of-the-art human motion synthesis methods.

_K_ eywords Deep learning $\cdot$ Temporal convolutional neural network
$\cdot$ Motion synthesis and control $\cdot$ Optimization $\cdot$ Motion
denoising $\cdot$ Motion completion

Figure 1: Our causal convolutional neural network can scale up well to a
large-scale motion dataset across multiple subjects and support a variety of
applications, as listed in this figure, in a unified way with a single set of
network parameters. Left: Examples of motion capture data. Middle: Training of
our network. Right: Applications.

##  1 Introduction

It is a challenging task to learn a powerful generative motion model from
prerecorded human motion data because human motion is intrinsically governed
by highly nonlinear dynamical systems. An appealing solution for generative
motion models should scale up well to motion datasets across multiple
subjects. In addition, it should be accurate and compact, efficient for
runtime evaluation, and amenable to various forms of applications, such as
motion synthesis with or without control inputs, motion prediction, motion
denoising, and motion completion.

Recent deep learning-based motion synthesis algorithms show great potentials
in resolving these issues. The deep neural network with nonlinear activation
functions can well model nonlinear dynamics and generate different motions
without the motion capture data used for network training to save the memory
footprint. Autoregressive models, such as restricted Boltzmann machines (RBMs)
and recurrent neural networks (RNNs) [1, 2, 3], have been applied to motion
synthesis by predicting the possibility of motions in the future. However, to
avoid causal error accumulation in such models, careful training strategies
must be employed. Conditioned on control signals, convolutional neural
networks (CNN), phase-functioned neural network (PFNN), Long-short term memory
(LSTM) networks, and variational autoencoders have been applied to generate
controllable motions to interact with the environment for a specific person
[4, 5, 6, 7]. Despite significant progress in deep learning-based motion
modeling, there is no accurate generative model that can scale up well to
motion datasets across multiple subjects and support different applications in
a unified way.

This paper proposes a novel causal convolutional neural network (CCNet) to
address the aforementioned issues in generative motion modeling and synthesis.
The network architecture is inspired by the success of causal-convolution-
based WaveNet [8] in multi-subject speech synthesis. The causal convolution is
appealing to generative human motion modeling since it can explicitly model
the correlation among a long-range temporal window, which is demonstrated to
be more effective than the hidden states used in RNNs and their variations in
speech synthesis experiments. However, it is a non-trivial task to apply
WaveNet to motion synthesis because motion data is a high dimensional signal,
and one needs to pay attention to foot contact to synthesize plausible
motions. To this end, we adapt the network architecture of the WaveNet by
adding an encoder and a decoder to translate the motion data into features and
back to the predicted motions. Moreover, we add 1D convolution layers to allow
the CCNet to take skeleton configurations as an input, which is critical for
the network to handle the skeleton variations of different subjects. The
output of the CCNet is the probabilistic density function (PDF) of the motion
at the next time step that is conditioned on the motions at previous time
steps, control signals, and the skeleton configuration. Consequently, with a
meticulously-designed training strategy, our CCNet can capture personalized
style variation of different skeletons and effectively support random and
controllable motion synthesis in a unified way using a single set of network
parameters.

The CCNet possesses the desirable properties of the generative motion model.
It is a compact model of size ~4.5M bytes. Combined with a Gaussian loss that
penalizes the deviation of joint angles, positions, and velocities
simultaneously in training, the drifting or freezing issues that are
frequently encountered in RNN models can be effectively mitigated. Thus, the
CCNet can be applied to random motion synthesis, i.e., synthesizing long-time
motion sequences of different subjects without control signals. This renders
it suitable for motion generation, denoising, and completion applications. The
CCNet can also be applied to the controllable motion synthesis. After training
on the motion capture (mocap) data across multi-subjects, we allow the user to
control the motion synthesis result through various control signals, such as
heading direction, velocity, and motion type. The skeleton is also called
control signals hereafter and processed in the same way as other control
signals in the CCNet. Moreover, the CCNet can synthesize motions for novel
skeletons not in the training dataset. After fine-tuning the network with a
few motion data of the novel skeleton, it is able to capture the personalized
style implied in the motion and generate high-quality motions for the
skeleton. Hence, the CCNet has the potential to be used as a pre-trained
network in few-shot learning for motion modeling and synthesis.

We have built a large-scale motion database of 12 subjects with a variety of
motion types and transitional clips between different types of motions.
Complex motions, such as punching, kicking, and kicking while punching, are
also included. The database has a total of 486,282 frames and will be made
public with our code. Experimental results show that, with such a database and
data augmentation, the CCNet can be efficiently trained, and it can generate
high-quality motions fast (~65fps) in the inference. We show its advantages in
the applications of motion processing and controllable motion synthesis for
different subjects and styles. User studies verify that the quality of motions
generated by our network is superior to the motions of state-of-the-art human
motion modeling and synthesis methods.

##  2 Related Work

Human motion modeling and synthesis is a long-standing problem in the area of
computer graphics. In the following, we focus our review on data-driven human
motion modeling, as well as their applications in human motion synthesis and
processing.

Motion Control and Synthesis. Data-driven motion synthesis is built
successfully on the interpolation of motions in a database. For example, Rose
et al. [9] classified the motion database into verbs and adverbs. Then human
motions are interpolated by the constructed radial basis as well as low-order
polynomials. However, interpolation-based methods can not adapt to the rich
repertoire of human behaviors. Generative statistical models, which describe
human movements by hidden parameters and the associated probability
distributions, became the mainstream in the previous decades. Tanco et al.
[10] learned a statistical model from the data set, and motion is then
interpolated by giving the start and end frames, as well as a few keyframes
with the learned statistical model. Other varieties of different statistical
models, including Hidden Markov Models (HMMs) [11, 12], spatial-causal dynamic
models [13, 14], and low-dimensional statistical models [15, 16] are developed
one after the other for human pose analysis and synthesis. Furthermore, motion
graphs [17, 18] and their various extensions [19, 20] are proposed to extend
the statistical models for representing complex human motions. These directed
graphs also provide benefits for the interactive editing and control for
complex human motions. In [21], a generative motion graphs named MotionGraph++
is proposed to process motions at semantic and kinematic levels.

In this paper, we propose an end-to-end deep learning framework for human
motion modeling and synthesis. Our deep-learning model is much more compact
than motion graph++. Besides, our model has a much better generalization
ability than motion graph++. Our experiments show that it can model not only a
rich set of human actions and their transitions but also delicate motion
variations across multiple subjects, a capability that has not been
demonstrated in previous work.

Methods | Generative Model | Scalability | Applications |  Motion Quality
---|---|---|---|---
I | II | I | II | III | IV | V
| MotionGraph[22]
---
[23][17]
$\times$ | $\times$ | $\times$ | $\checkmark$ | ✓ | $\times$ | $\times$ | $\times$ | ✓
MotionGraph++[21] | o | o | $\times$ | ✓ | ✓ | $\times$ | $\times$ | $\times$ | ✓
| ERD-LSTM[2] [6]
---
[3] [24]
✓ | ✓ | o | ✓ | o | o | ✓ | ✓ | o
DAE-LSTM[25] | ✓ | ✓ | o | o | ✓ | o | o | o | ✓
PFNN[5] | ✓ | ✓ | ✓ | - | ✓ | - | - | - | ✓
CCNet | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 1: Comparisons between the CCNet and state-of-the-art deep-learning-
based models in motion modeling and synthesis. Columns I, II of "generative
model" are the model size and motion variation during synthesis respectively;
"scalability" represents multiple subjects; and Columns I, II, III, IV, V for
"applications" are random motion synthesis without control signals, motion
synthesis with control signals, motion denoising, motion completion and motion
prediction. ✓means good, $\times$ means poor and o means ok, namely between
poor and good. -: Not implemented. Since PFNN[5] is designed to predict the
pose for the next frame using phase values, not the probability distribution
of the pose, it’s not suitable for random motion synthesis. For fair
comparisons, we only extend PFNN to controllable motion synthesis for multi-
subjects by adding skeleton configuration as an additional input. We also
extend LSTM networks to support multi-subject motion modeling (see Sec. 6 for
details).

Motion Style. Our method has superior performance for generating different
motion styles associated with different people, even when they possess the
same type of human motions. It is noted that motion style and motion type are
clearly differentiated in this paper. For example, a fat man and a thin man
would have different walking styles for the same motion type, i.e., walking.
The critical challenge is how to model the motion style. In the work of [12],
the motion style is parameterized to learn a statistical model, and different
types of motion can be synthesized from the learned model according to the
input style parameters. Arikan et al. [19] proposed categorizing the human
motions into different motion types, such as turning left and turning right,
and then generating motion sequences according to the input motion types
through a dynamic programming search algorithm. In the work of [26], the
motion style for a single person has been proposed for addressing the problem
of unlabeled heterogeneous motions. However, all the methods mentioned above
only focus on the motion styles for a specific person. None of them can model
the variations of human motion styles across different people. Similar to
[27], our deep generative model can handle personalized style variations
across multiple subjects. However, unlike [27], which can only model
personalized style variations for a particular human motion such as walking or
running, our generative model can scale up well to a rich repertoire of human
motions as well as their transitions. Aberman et al. [28] handled the skeleton
variations by representing skeletons as the static offsets in some arbitrary
initial pose together with the dynamic motions in a tree graph structure.
However, they focused on motion retargetting while we are interested in motion
synthesis.

Deep Learning based Motion Synthesis. In recent years, deep learning has
gained lots of attention in computer vision and computer graphics. Like many
other tasks, such as image segmentation, classification, and object
recognition, human motion synthesis has also benefited from the rapid progress
of deep learning. It provides a remarkable tool for directly learning a
compact, low-dimensional motion space from a dataset without any motion
feature designations. By comparison, traditional successful generative
statistical models for human motion synthesis heavily rely on the human-made
ad-hoc motion features [16, 29, 20]. Holden et al. [30] proposed a
convolutional auto-encoder to learn compact motion representation, termed as
the motion manifolds, for human motion synthesis. Such motion representation
can be utilized to fill in the missing data and perform the motion
interpolation in the manifold space.

In [4], user-friendly high-level parameters for motion control, such as
character motion trajectory and foot contact information, are investigated for
synthesizing the human motions. Phase variable for cyclic motions is
explicitly used as an input to control the weights in the network [5]. In
contrast, the hidden state in LSTM can model the causal dependence of the
motion implicitly. Thus motion synthesis with multi-objective control can be
realized without the phase information [6]. Mixture-of-expert (MoE)
architecture [31, 32, 33] are used to ease the burden of phase labeling for
motions and improve the capacity of the networks for motion synthesis. Ling et
al. [7] leveraged MoE as the decoder of motion VAEs to model the distribution
of possible next poses. Starke et al. [34] added local motion phase feature to
MoE to learn asynchronous movements of each bone and its interaction with
external objects and environments. Reinforcement learning has been widely
applied [35, 36, 37, 38, 39] to train physics-based motion controllers. Peng
et al. [40] adopted a two-level hierarchical control policy with high-level
environment information to make the character be aware of the surroundings.
Won et al. [33] clustered the reference library of motions generated by the
kinematic controller to construct experts and then combined these experts by
deep reinforcement learning. The adversarial training strategy is adopted in
[41, 36] to improve the quality of generated motions. Recently, mapping the
features extracted from music [42] or language [43] to the motion space is
utilized to generate character motions synchronizing with such multi-modal
inputs.

Enormous neural networks are proposed to address long-term motion prediction
problem, which include recurrent neural networks (RNNs) [2, 44, 45, 46], fully
connected networks [47, 48], reinforcement learning [49, 50], graph networks
[51, 52], and generative adversarial networks (GAN) [53]. To better model the
randomness of human motions, Aliakbarian et al. [54] combined the root of
variations with previous pose information to force the network to learn the
motion variations. At the same time, Zhao et al. [55] exploited Bayesian
Hierarchical Hidden semi-Markov Model(BH-HSMM) as generator and discriminators
for adversarial learning. To solve the error accumulation problem for long-
term motion prediction, practical strategies, including adding residual blocks
and introducing sampling in training, are applied to improve RNN [3], and the
auto-condition scheme is adopted in RNN in the work of [56]. QuaterNet [57]
conducts extensive experiments to demonstrate that the quaternion
representation is beneficial to improve the quality of synthesized motions.

Despite significant progress in deep-learning-based motion modeling and
synthesis, constructing a generative model capable of accurately modeling
motion data sets across different subjects remains challenging. The CCNet is
more appealing to human motion modeling because the explicit causal
convolution adopted in CCNet has a larger and more efficient receptive field
than widely used networks such as RNNs or their variations. As shown in our
comparisons (see Sec. 6), in the case of multiple subjects, the CCNet can
capture personalized style variation and produce superior results than
alternative deep learning models in terms of both motion synthesis quality and
motion control accuracy, a capability that has not been demonstrated in
previous work. Finally, as shown in Tab. 1, our model is more flexible and
powerful for motion synthesis and processing. In contrast, traditional motion
graph techniques [22, 23, 17, 21] are hard to scale up to handle large scale
motion data, and the required memory footprint is usually large. For fair
comparisons, ERD-LSTM models [2, 3, 24, 6], DAE-LSTM models [25] and PFNN [5]
are extended to model multi-subject motion data by taking skeleton
configuration as an input (see Sec. 6 for details).

##  3 Overview

The overall framework of our system is illustrated in Fig. 1. Given the
processed motion data (Sec.4), we train a CCNet to predict the PDF of the
future pose conditioned on the poses of past frames and optional control
signals, where the details of control signals are discussed in (sec. 4.2). The
designed CCNet has three types of functional blocks, i.e., encoder, separate
residual block, and decoder, and it outputs the mean and variance of the PDF
(Sec. 5.1). A variety of motion synthesis applications, such as motion
denoising, motion completion, and motion control, can be realized by this
unified generative model with a single set of network parameters. We also test
how the network generalizes to novel skeletons not in the training dataset
(Sec. 6).

During training, we use Gaussian loss, foot contact loss, and smoothness loss
to learn the network parameters (sec. 5.2). Noises are added to the sampled
training motion data such that the trained network is robust to the
accumulated error in the motion synthesis and can produce high-quality, non-
freezing motions. The slight foot sliding in the generated motion is removed
by an inverse kinematic (IK) algorithm according to the predicted foot contact
label by the CCNet. To ease the interactive control, we also train the
proposed network to output the direction and velocity control signal for the
next frame.

##  4 Data Processing and Representation

We build a human motion database of 12 different subjects using the mocap
technique. Three of the subjects are female, and the rest of them are male.
The database includes 10 types of motions: walking, running, jumping with the
left foot, jumping with the right foot, jumping with both feet, back walking,
zombie-walking, kicking, punching and kicking while punching. All subjects are
asked to perform the first 7 types of motions, and 5 of them are asked to
perform the last 3 types of complex motions additionally. The motion recording
speed is 120fps, and the recording time for each subject is within 2 hours.
Thus, there are around 80,000 frames of motion data for each subject. During
recording, we ask each subject to perform two types of motions in one motion
sequence to facilitate the learning of transition between different motion
types. Afterwards, We down-sample the recorded motion data to 60fps and obtain
a total of 486,282 frames to be used as our training and validation datasets.
The validation dataset is formed by randomly selecting one motion sequence of
each subject. Moreover, all the motion sequences of subject 7 are removed from
the training dataset and only present in the validation dataset, which is used
to test how our network can handle the skeleton variation after training on
multi-subject motion data. As a result, the validation dataset contains 41
motion sequences and a total of 88,649 frames. The rest motion data is used in
the training dataset.

###  4.1 Motion representation

The character skeleton in the mocap data is modeled as an articulated figure
with rigid links connected by ball-and-socket joints. The motion at each frame
is recorded as the translation and rotation at the root joint and the relative
rotations at other joints. However, such motion representation is a relatively
local feature since most rotations at ball-and-socket joints are relative to
their parent joint. Thus, we add 3D joint positions and the joints’ angular
and linear velocities into the representation to better model the global
influence of joint rotations on rigid links’ positions and orientations.
Overall, the motion information at $n_{th}$ frame is represented as
$x_{n}=\\{x_{n}^{e},x_{n}^{\omega},x_{n}^{p},x_{n}^{v},x_{n}^{f}\\}$, where
$x_{n}^{e}$ denotes the vector of relative joint rotations represented using
exponential coordinates [58], $x_{n}^{\omega}$ the vector of relative angular
velocities of the joints, $x_{n}^{p}$ the 3D joint positions, and $x_{n}^{v}$
denotes the vector of joint linear velocities. The foot contact information at
$n_{th}$ frame is represented as a 2-dimensional binary vector $x_{n}^{f}$.

Before converting the mocap data into our representation, we first align each
recorded motion clip by translating its first frame to the origin of the
global coordinate system on the XOZ plane and setting the root’s rotation
around the global Y-axis to be zero. For the $n_{th}$ frame in the clip, we
first rotate the root orientation represented in the global coordinate system
of frame $n-1$ to a coordinate system whose Y-axis is $\\{0,1,0\\}$. We then
represent the root position and orientation of $n_{th}$ frame to the rotated
coordinate system at frame $n-1$. However, we still represent the $y$
coordinate of the root joint in the global coordinate system to emphasize this
quantity in network training. The rotations for non-root joints remain the
same with the motion capture data. The linear and angular velocities are
computed by subtracting the corresponding joint positions and rotations in
exponential coordinates at frame $n$ and $n-1$ and representing the difference
vectors in the rotated coordinate system of frame $n-1$. The motion alignment
makes our motion representation invariant to translation and facing
orientation in the plane, which means that no matter where the global root
position and orientation of frame $n-1$ are, the aligned motion representation
remains the same as long as the relative motion is the same.

The foot contact information $x_{n}^{f}$ is used to alleviate the foot sliding
of the generated motions. It can be directly computed from the motion data. We
first compute the distance between the current and the previous frame and
their heights for the left and right toe joints. The height of the joint is
just the y coordinate in our global coordinate system. If the distance and
position y are less than the threshold $\delta_{d}=5mm$ and $\delta_{y}=80mm$
respectively, we set the foot contact label to be 1, otherwise 0.

###  4.2 Control Signal Representation

For the motion data at $n_{th}$ frame, our system allows the user to input
three types of control signals, which control the motions at future frames.
The control signal is denoted by $c_{n}=\\{c_{n}^{d},c_{n}^{t},c_{n}^{s}\\}$,
where $c_{n}^{d}$ denotes the direction and velocity control, $c_{n}^{t}$ the
motion type control, and $c_{n}^{s}$ the skeleton configuration signal used to
differentiate subjects. The control signals used in training are extracted
from the motion capture data.

Direction and velocity control. The control signal $c_{n}^{d}$ is a
12-dimensional vector formed by sparsely sampling, in the causal domain, the
points on the motion trajectory starting from $n_{th}$ in a motion clip. Thus,
this control signal also implicitly implies the velocity information. It is
inspired by the trajectory used in [5], but we only use the future trajectory.
Specifically, starting from the $n_{th}$ frame in the clip, a motion sequence
lasting for 1 second consisting of future frames is extracted, which is 60
frames in our 60fps motion capture setting. Second, we compute the 3D
positions of root joints for all the extracted frames and then represent them
in the $n_{th}$ frame’s coordinate system. All the relative root positions are
projected onto XOZ plane to obtain a future motion trajectory. Finally, the
trajectory is uniformly down-sampled in the causal domain (every 10 frames) to
6 2D points to form the $c_{n}^{d}$. This procedure is repeated for all the
frames in the motion database.

The direction control signal itself contains the velocity information because
it is the heading trajectory in the next one second of future frames. Thus, it
is not necessary to compute another velocity control signal for each frame.
When the user needs to control the velocity of the motion, we allow the user
to input the trajectory length for the future 1-second motion. It will be used
to update the 6 points in $c_{n}^{d}$.

Motion type. Since our database consists of ten types of motions, the motion
type control signal $c_{n}^{t}$ is a 10-dimensional vector using one-hot
coding. We manually label frames in the captured motion to obtain the motion
type signal for training. However, it is difficult to give an exact type label
for those transitional frames between two types of motions. Thus, when the
transition happens, we treat the right-foot-touching-the-ground as the ending
state of the first type of motion and the left-foot-leaving-the-ground as the
starting state of the second type of motion. According to the frame index, we
linearly interpolate the first type label vectors and the second type label
vectors and assign the interpolation results as the motion type signal for the
frames between the ending states and the starting states to the transitional
frames.

Skeleton configuration. Since our database consists of different skeletons,
the skeleton configuration is necessary to help the network to discern the
skeleton variations across different persons. We transform the skeleton’s
chain structure at T-pose into a control signal $c_{n}^{s}$ that consists of
the following components:

$c_{n}^{s}=\\{(h_{r},t_{1}^{x},t_{1}^{y},t_{1}^{z},...,t_{m}^{x},t_{m}^{y},t_{m}^{z}\\},$
(1)

where $h_{r}$ is the height of the root joint, and the 3D positions of non-
root joints, i.e.,
$\\{t_{1}^{x},t_{1}^{x},t_{1}^{y},t_{1}^{z},...,t_{m}^{x},t_{m}^{y},t_{m}^{z}\\}$,
are set to be relative to the root. The number of joints $m$ is set to be 27,
which is the number of non-finger joints in our database. The 27 non-finger
joints’ relative 3D positions and the root joint’s height form the 82
dimensions $c_{n}^{s}$. Note that the skeleton configuration is input to the
network all the time, but other control signals are optional in the motion
synthesis.

Figure 2: The network architecture of the CCNet. It consists of an encoder,
separate residual blocks, and a decoder. The kernel size of Conv1D is set to
1, and the kernel size of CausalConv1D is set to 2 in all the blocks. Given
the input frames, it outputs the Gaussian distribution of a future frame,
direction control signal, and foot contact label.

##  5 CCNet

In this section, we first describe the network structure of CCNet and then
proceed to its training details.

###  5.1 The Network Structure

The network structure of our CCNet $F$ is illustrated in Fig. 2 (detailed
network parameters are reported in the supplementary material). It models the
PDF of the predicted motion for $n_{th}$ frame with the following formula:

$\displaystyle p(x_{n}|X,c_{n-1})=\mathcal{\psi}(X,c_{n-1})=\hskip 200.0pt$
(2) $\displaystyle\hskip
10.0pt\mathcal{\psi}_{D}\big{(}{\mathcal{\psi}_{R}}^{0}(\mathcal{\psi}_{E}(X),c_{n-1})+\sum_{i=1}^{19}\mathcal{\psi}_{R}^{i}(\mathcal{\psi}_{R}^{i-1,..,0}(\mathcal{\psi}_{E}(X)),c_{n-1})\big{)},$

where $X=\\{x_{n-1},...,x_{n-l-1}\\}$ and $c_{n-1}$ are the motion data of
past $l$ frames and the control signals of $(n-1)_{th}$ frame . The dropout
layer before the encoder, denoted by $D$, is used to resolve the possible
over-fitting issue, and its drop probability is set to be 0.5.

The encoder $\mathcal{\psi}_{E}$, the separate residual blocks (SRB)
$\mathcal{\psi}_{R}^{i}$, and the decoder $\mathcal{\psi}_{D}$ are the
functional blocks in the network. There are a total of 20 SRBs in our network.
The superscript $\\{i-1,..,0\\}$ in Eq. 2 indicates that the SRBs in the
network are recursively executed, and each $i_{th}$ block takes the output of
$(i-1)_{th}$ block and the control signals as its inputs. The first separate
residual block $\mathcal{\psi}_{R}^{0}$ takes the output from the encoder and
the control signals as inputs. The output PDF $p(x_{n})$ is set to be the
Gaussian $\mathcal{N}(\hat{\mu}_{n},\hat{\sigma}_{n})$, where its mean
$\hat{\mu}_{n}$ and standard deviation $\hat{\sigma}_{n}$ are the output of
the decoder $\mathcal{\psi}_{D}$. Since the decoder $\mathcal{\psi}_{D}$ might
output a negative standard deviation value after convolution, we compute the
final standard deviation values as $\hat{\sigma}_{n}=e^{\sigma_{n}}$, where
$\sigma_{n}$ is the direct output of the $\mathcal{\psi}_{D}$.

Encoder. The motion representation $X$ of past $l$ frames are first input to
an encoder $\mathcal{\psi}_{E}$ to map the data into features, and the encoder
is of a simple "Conv1D-ReLU" structure where the size of 1D convolution kernel
is $1$ and use ReLU as the activation function. Note that the kernel of size
$1$ makes sure that the feature of the motion at each input frame is
independent.

Separate residual blocks: The core component of the CCNet is the set of
separate residual blocks $\mathcal{\psi}_{R}^{i}$. It is similar to the
residual blocks used in WaveNets [8], which use dilated causal convolution to
guarantee the input motion data’s ordering. The control signals are also
inputted to the residual blocks through convolution layers with kernel size
$1$. The difference is that we use two separate dilated causal convolution
layers and 1D convolution layers to compute separate features for the gated
activation. This is designed to disentangle the information to increase the
capacity of our network. Moreover, the features from the motion data and
control signals are fused through summation. This enables us to switch on/off
the control signals online during the training and inferring. The dilated
Causal Convolution is implemented as in [8]. Specifically, zeros are padded
before the feature of the $\mathcal{\psi}_{E}(x_{n-l-1})$ so that the output
of the convolution of a frame $i$ only depends on the frames before it. The
padding size can be easily computed as $(k-1)*d$, where $k$ is the kernel size
and $d$ the dilation size. The kernel size of the causal convolution and
dilation size in SRBs are set to be 2. As a result, the causal receptive field
of the CCNet is 41 frames, which can be computed using the following formula:

$F=(k-1)+\sum_{i=0}^{{\color[rgb]{0,0,0}19}}(k-1)*2$ (3)

where $k$ is $2$, the kernel size of the dilated Causal Convolution used in
our CCNet.

Fig. 3 shows the details on how the control signals are input to the separate
residual blocks: each type of control signal is input to its own Conv1D layer,
and the kernel size of Conv1D is 1. The input motion feature (channel number:
32) for $\text{SRB}^{i}$ is the feature $O_{r}^{i-1}$ output by
$\text{SRB}^{i-1}$. The output feature $O_{s}^{i}$ (dimensionality of the
output features: 512) is sent to the decoder.

Decoder. The decoder $\mathcal{\psi}_{D}$ maps the summed features from the
SRBs to the PDF of the predicted motion. It is of a simple "ReLU-Conv1D"
structure, where the convolution kernel size is also set to be $1$.

###  5.2 Training Loss

The training loss consists of four terms, a Gaussian loss $L_{G}$, a motion
smoothness loss $L_{s}$, a foot contact label loss $L_{f}$ and a direction
control loss $L_{d}$. It can be formulated into:

$L=L_{G}+\lambda_{1}*L_{s}+\lambda_{2}*L_{f}+\lambda_{3}*L_{d}$ (4)

where the weight $\lambda_{1}$ is set to be $10.0$, $\lambda_{2}$ be $2.0$ and
$\lambda_{3}$ be $1.0$ in all our experiments.

The first term $L_{G}$ is the Gaussian loss. This term follows the Gaussian
mixture loss in [2], while we only use one mode and set the covariance matrix
to be diagonal to reduce the number of parameters. It can be written into:

$L_{G}=-ln(p(x_{n}|\hat{\mu}_{n},\hat{\sigma}_{n})),$ (5)

where $x_{n}$ is the motion representation extracted at $n_{th}$ frame. Thus,
this term enforces the network to output the values of mean $\hat{\mu}_{n}$
and standard deviation $\hat{\sigma}_{n}$ so that the captured motion data is
of high probability. The binary foot contact label in $x_{n}$ is handled in
$L_{f}$ and thus not included in this term. We add a constraint in our
implementation to ensure the standard deviation $\hat{\sigma}_{n}$ is greater
than 1e-4 by a clipping operation, and we observe that the standard deviation
output by the trained CCNet is usually between 1e-4 and 1e-3. Consequently, in
motion synthesis, we can sample a motion according to the Gaussian
distribution to enrich the variation of the synthesized motion. Note that the
Gaussian function is used to maximize the probability of the motion
representation vector of the ground-truth mocap data during training. Thus,
the joint positions and linear velocities included in this term can help to
model the correlations between the rotational degrees of freedom of different
joints since such quantities are affected by all the parent joints on the
kinematic chain connected to the joints.

Figure 3: The detailed architecture of the separate residual block. The
numbers besides $O_{r}^{i-1}$, $O_{r}^{i}$ and $O_{s}^{i}$ indicate the number
of channels.

The second term is the smoothness term to prevent the sudden change of the
motion among neighboring frames, which is as follows:

$L_{s}=\sum_{n=2}^{N}(\hat{\mu}_{n-2}+\hat{\mu}_{n}-2\hat{\mu}_{n-1}).$ (6)

The smoothness loss is only optimized for the mean of predicted Gaussian
distributions since the motion generated by the network is usually close to
the mean at each frame. This term is a soft constraint to prevent the sudden
change of accelerations at joints and make the synthesized motion smoother.

The third term is used to train the network to predict whether the foot is in
contact with the supporting plane at $n_{th}$ frame. Specifically, we adopt
binary cross entropy(BCE) loss function to compute this term:

$L_{f}=BCE(x_{n}^{f},\hat{x}_{n}^{f}),$ (7)

where $x_{n}^{f}$ is the ground-truth foot contact label from the data, and
$\hat{x}_{n}^{f}$ is the network prediction. The foot contact label is used to
trigger IK algorithms to remove the foot sliding in the synthesized motion.

The last term is used when the direction and velocity controls are switched
on. It can be simply written as:

$L_{d}=\|\hat{c}_{n}^{d}-{c}_{n}^{d}\|^{2}+\|\hat{c}_{n}^{v}-{c}_{n}^{v}\|^{2}$
(8)

where ${c}_{n}^{d}$ and ${c}_{n}^{v}$ are the direction and velocity control
signals computed from the motion data, and $\hat{c}_{n}^{d}$ and
$\hat{c}_{n}^{v}$ are the predicted values. This term is useful in interactive
motion control applications when control signals might be input by the user
occasionally. In this case, the predicted control signal values will be fed
into the network to continue the motion synthesis.

###  5.3 Training Details

We train our CCNet using the RMSProp optimizer [59]. The initial learning rate
is 1e-4 and will be decayed by multiplying it by 0.5 every 500 epochs. The
loss curve usually converges around 1000 epochs. The batch size is set to be
256, and each sample in the batch is a motion sequence of 240 consecutive
frames. There are two steps to generate 240 samples in a batch: 1) randomly
select a motion clip from the database and then the starting frame index in
the clip. 2) Sample 240 frames in the clip repeatedly using a one frame
interval, i.e., the starting frame index, $F_{s+1}$, of the next 240 frames
sequence is equal to $F_{s}+1$. For an input sequence,
$X=\\{x_{0},x_{1},...,x_{n-1}\\}$, the CCNet can produce the output
$Y=\\{y_{1},y_{2},...,y_{n}\\}$ due to the guaranteed causal ordering in all
the dilated causal convolutional operations in CCNet. Thus, we can compute the
training loss for all the output motions at different frames, which helps the
CCNet learn to handle the input motions of different lengths.

(a) skeletons of different subjects

(b) meshes of different subjects

Figure 4: Skeletons and meshes in our database. The skeletons of subjects
0$\sim$11 are from left to right, and their heights are 160cm, 170cm, 168cm,
183cm, 172cm, 165cm, 195cm, 170cm, 178cm, 166cm, 155cm, 154cm, respectively.
Subjects 2, 9, and 10 are female.

Data augmentation. We add additional independent identically distributed
Gaussian noises to each sampled motion representation vector of training
motion data to train the network to handle accumulated errors in the motion
synthesis. The mean and standard deviation of the noise is selected to be $0$
and $0.03$.

##  6 Experiments

We have implemented our algorithm using Pytorch 1.6 on a desktop PC with
Intel(R) Xeon(R) Gold 5120 CPU, 128G RAM, and one Tesla V100 SXM2 32GB
graphics card. The trained CCNet has 1.16M parameters, resulting in a model of
size ~4.5M Bytes. The skeleton information is always input to the CCNet in
both random and controllable motion synthesis since they are required to
differentiate between subjects in the experiments. Although the network can
generate high-quality motion, slight foot sliding might still occur. If not
mentioned, the inverse kinematics algorithm is adopted to completely remove
the foot sliding in the generated motion according to the predicted foot
contact label. Besides, we denote the initial frames input to the CCNet to
begin the motion generation as seed frames hereafter.

(a) Random motion synthesis

(b) Noisy motion

(c) Denoised motion

(d) Incomplete motion

(e) Completed motion

Figure 5: Random motion synthesis results. (a) Random motion generation result
for subject 3 (the fourth skeleton shown in Fig. 4). Different colors of the
clothing indicate the different random motions generated by the trained CCNet.
(b) and (c) A motion denoising result for subject 7 (the eighth skeleton shown
in Fig. 4). (d) and (e) A motion completion result for subject 5 (the sixth
skeleton shown in Fig. 4).

Baseline networks: We compare the CCNet to state-of-the-art motion synthesis
networks that are listed below:

  * •

ERD-4LR: the encoder-recurrent-decoder network structure in [2]. We implement
the network using 4 LSTM layers as in [6].

  * •

DAE-LSTM: the network structure in [25] that uses a dropout autoencoder to
filter the predicted poses output by an LSTM network.

  * •

PFNN: the network structure in [5] that adopts a cyclic function to compute
the neural network’s weights by taking the motion phase as an input.

To test these three network structures’ performance in multi-subject motion
synthesis, we add parameters at their first layer to accept as input the
skeleton configuration and the same set of control signals. For clarity, we
refer to the modified networks for random motion synthesis as ERD-4LR-rand and
DAE-LSTM-rand and the modified networks for controllable motion synthesis as
ERD-4LR-cond and DAE-LSTM-cond. Since PFNN is mainly designed for controllable
motion synthesis, not for a generative model, we only compare the CCNet with
PFNN on controllable motion synthesis. We refer to the modified PFNN as PFNN-
cond. Please refer to the supplementary_material.pdf in "other supplementary
materials" for the detailed network parameters of the modified networks.

We separately train these five networks, namely ERD-4LR-rand, DAE-LSTM-rand,
ERD-4LR-cond, DAE-LSTM-cond, and PFNN-cond, and our CCNet, on the multi-
subject motion dataset. For all the networks, the initial learning rate is set
to 1e-4, and it will be decayed by multiplying it by 0.5 every epoch. The
batch size is 256, and each sample in the batch is a motion sequence of 240
consecutive frames. We train these networks for 2000 epochs. The rest training
settings for baseline networks remain the same as reported in their papers.
Please refer to [2], [6], [25] and [5] for the detailed settings. Finally, we
choose the best-performed network snapshots that achieve the lowest validation
loss as the final networks used in the comparisons. Specifically, the network
snapshots are selected as follows: CCNet, ERD-4LR-rand, and ERD-4LR-cond:
1510th epoch, the PFNN-cond network: 1045th epoch, DAE-LSTM-rand: 1650th
epoch, and DAE-LSTM-cond: 1120th epoch.

###  6.1 Random motion synthesis

Random motion generation: In this experiment, the skeleton configuration and
seed frames are fed to the CCNet to generate high-quality motions. However, we
do not input control signals, such as direction, velocity, and motion type, to
the CCNet. Also, we use 120 seed frames to facilitate the comparisons since
such length is chosen in ERD-4LR and DAE-LSTM [6, 25]. As illustrated in Fig.
5a, for the fourth skeleton shown in Fig. 4, five motion sequences are
generated by sampling the pose at frame $n$ using the predicted Gaussian
distribution. The sampled pose is fed back to the network to generate future
frames. The random motion generation can be used in motion prediction given a
long motion sequence as an input, which is useful in on-line pose detection in
computer vision or RGBD-based motion capture [60].

We also feed ERD-4LR-rand and DAE-LSTM-rand with the same 120 seed frames and
the skeleton configuration to synthesize random motions for fair comparisons.
As a result, we found that the CCNet, ERD-4LR-rand, and DAE-LSTM-rand can all
synthesize long period motion sequences with more than 20,000 frames. The
random motion generated by ERD-4LR-rand is also of good quality but a little
bit less smooth than the motion generated by the CCNet. Comparisons conducted
in the user study (Sec. 6.4) also show that the quality of the random motions
generated by the CCNet is superior.

Motion denoising and completion: To test how the CCNet perform in motion
denoising, we randomly select a motion sequence $X=\\{x_{0},x_{1},..,x_{k}\\}$
of a subject and then add independent identically distributed Gaussian noise
(mean 0, standard deviation 0.01~0.1) to the motion data $\hat{X}$. The
network takes $\hat{X}$ as an input and outputs the denoised motion $Y$.
However, the denoised pose is not fed back into the network, which is
different from the random motion generation. For those frames with indices
less than the casual receptive length, 41 frames in our network, they are
denoised according to all the frames before them since the CCNet is trained to
handle motions of different lengths. Fig. 5b and 5c illustrate a denoising
result. The standard deviation of the noise for this experiment is set to
0.08. Before denoising, the right hand and right toe’s trajectories are very
jerky, and the foot is underneath the ground at some frames. It can be seen
that these artifacts are significantly reduced in the denoised motion. To
compare the CCNet to baseline networks in the quality of denoised motions, we
add Gaussian noise to the test data with standard deviations 0.03, 0.05, and
0.1, respectively, and then use CCNet, DAE-LSTM-rand, and ERD-4LR-rand to
denoise the noisy motion data. The error between the ground truth motion and
the denoising result is computed as the Euclidean distance between their
motion representation vectors as described in Sec. 4. As shown in Tab. 2, the
error of the denoised motion generated by the CCNet is less than the motions
denoised by DAE-LSTM-rand and ERD-4LR-rand.

Noise STD | Denoising Errors (mean±std)
---|---
ERD-4LR-rand | DAE-LSTM-rand | CCNet
0.03 | 0.768±0.357 | 0.687±0.384 | 0.528±0.126
0.05 | 0.768±0.357 | 0.687±0.384 | 0.539±0.125
0.1 | 0.77±0.361 | 0.687±0.384 | 0.584±0.117
Table 2: Motion denoising comparison. STD: standard deviation. IK is disabled
in this experiment. The errors of the motions denoised by the CCNet are less
than the errors of the motions denoised by ERD-4LR-rand and DAE-LSTM-rand.

The procedure of motion completion is similar to motion denoising. In the
experimental results illustrated in Fig. 5d and 5e, we first select a
700-frame motion sequence containing walking and jumping with both feet, and
then set the rotations of joints of the right legs of 30% frames to 0. The
CCNet takes the incomplete motion as the inputs and outputs a completed
naturally looking motion.

(a)  (b)

Figure 6: Trajectory-following results of two subjects. (a) A synthesized
motion transiting from walking to running then to zombie walking for subject
10 (the eleventh skeleton shown in Fig. 4). (b) A synthesized motion
transiting from jumping with the right foot to jumping with the left foot then
to jumping with both feet for subject 11 (the twelfth skeleton shown in Fig.
4). We use different colors to represent different motion types at the
different part of trajectories as follows: red-> walking, magenta->running,
green->jumping with the left foot, cyan->jumping with the right foot, green-
yellow->jumping with both feet, blue->back walking, pink->zombie walking,
orange->kicking, purple->punching, and brown->kicking while punching.

(a)

(b)

Figure 7: Our CCNet can synthesize (a) motions heading along a complex
trajectory and (b) Complex motions for subject 5 (the eighth skeleton shown in
Fig. 4).

###  6.2 Controllable motion synthesis

Motion control using user-specified trajectories: Synthesizing different types
of motions along a specified trajectory is a desirable function in motion
planning. We allow users to specify a motion trajectory $J$ on the XOZ plane
with additional velocity and motion type information and then map the
trajectory information into the control signals $c_{n}^{d}$ and $c_{n}^{t}$
supported in our system. Specifically, the trajectory $J$ is represented as
$J=\\{\\{J_{i},t_{i},v_{i}\\},i=1,..,k\\}$, where $J_{i}$ is the $i_{th}$ part
of the whole trajectory represented as an ordered densely sampled 2D points,
the motion type information $t_{i}$, and a scalar velocity value $v_{i}$ are
also associated to the $J_{i}$. For two adjacent parts of the trajectory with
different motion types, we set up 20 transitional frames with interpolated
motion type control signal (see Sec. 4 for the details of motion type
interpolation).

Suppose we already synthesize frame $n$, and its root position projected on
the XOZ plane, denoted by $t_{n}^{p}$, might deviate from the input
trajectory. We first find the closest point $t^{J}$ on the $J$, and then
extract the part of trajectory $\hat{J}^{E}$ lasting for one second starting
from $t^{J}$ using the specified velocities. We then linearly interpolate
between $\hat{J}^{E}$ and the line connecting $t_{n}^{p}$ and the endpoints of
the expected trajectory $\hat{J}^{E}$ to obtain a blended trajectory
$\hat{J}^{b}$. The blending weight for the $\hat{J}^{E}$ starts with 0 and
increases towards 1 according to the time parameter. Afterward, 6 2D points
are uniformly sampled in the causal domain from $\hat{J}^{b}$ to be input as
the $c_{n}^{d}$. All the sampled points are represented into the root
orientation at frame $n$ as described in Sec. 4.

As shown in Fig. 6, our system can synthesize motions following two user-
specified trajectories. Different trajectory colors indicate different types
of motion. Fig. 7a illustrates that the synthesized motions can well follow a
trajectory with large curvatures and frequent change of motion types. In Fig.
7b, we show that the CCNet can generate complex motions, such as kicking and
punching motions present in our training dataset, when the user specifies
these two motion types along a trajectory.

Figure 8: The user interface for interactive control. The green dots on the
ground represent the direction control signal. IK is disabled.

(a) CCNet

(b) ERD-4LR-cond

(c) PFNN-cond

Figure 9: Comparisons against ERD-4LR-cond and PFNN-cond. The character starts
by jumping with the left foot, then changes to jumping with the right foot
till the end. The total errors between the synthesized trajectories (the
yellow lines) and the input trajectories (the green lines) of ERD-4LR-cond,
PFNN-cond and the CCNet are 177.143cm, 156.604cm and 27.043cm, respectively.
IK is disabled in this experiment.

Interactive control: The CCNet can be easily integrated into interactive
applications, and we demonstrate this capability by developing a demo that
allows the user to control the direction, velocity, and motion type through a
keyboard. Direction and velocity signals are used to generate future motion
trajectory $c_{n}^{d}$ online similar to PFNN [5]. The PyTorch implementation
is exported to C++ through the LibTorch API to ease the implementation of this
demo.

Specifically, to control the motion type, the user can use the number keys
from 1 to 5 to select among 5 motion types: walking, running, jumping with the
left foot, jumping with the right foot, and jumping with both feet. Once a
key(for instance, 2) is pressed down, we update the motion type label by
interpolating the new type label with the previous one in 20 frames, which
means the character can transition from the previous motion type to the new
one more smoothly. The user can also control the velocity by pressing up and
down keys and heading direction of the character by pressing the left and
right keys. Once the left key is pressed, the trajectory will be turned left.
It is achieved by first computing a small offset vector $o_{n}=[1,0]*h*0.015$,
where $h$ is the root’s height. This offset will be added to the $c_{n}^{d}$
by $o_{n}*w_{i}$, where $w_{i}=i/5,i=0.5$. Thus, the offset will be added to
the 6 points in the predicted control signal $\hat{c}_{n}^{d}$ through the
corresponding $w_{i}$. The distance between the 2D points in the updated
$\hat{c}_{n}^{d}$ is then adjusted according to the user-specified velocity
$v_{u}$. Since the $\hat{c}_{n}^{d}$ represents the future motion trajectory
within one second, we can adjust the distance between its 2D points by
multiplying it by the ratio between the current scalar velocity of the
character $v_{cur}$ (computed by the length of the 2D points in $c_{n}^{d}$)
and $v_{u}$, which is $v_{u}/v_{cur}$. The velocity is changed from the
current velocity to the user-specified velocity with 20 frames interval. Fig.
8 illustrates the user interface used in interactive control.

Figure 10: Trajectory-following results generated by the CCNet for a unseen skeleton subject 7_b. The skeleton of subject 7_b is generated by scaling the lower body of subject 7’s skeleton by 0.8. Please see the accompanying video from 2m 44s to 3m 3s for the demo. Models | Relative Pose Difference (mean±std)
---|---
CCNet on entire dataset | no fine-tuning | 0.083±0.0925
fine-tuning with walking data | 0.0536±0.0365
fin-tuning with walking and running data | 0.0483±0.0544
CCNet on dataset1 | no fine-tuning | 0.0861±0.155
fine-tuning with walking data | 0.0543±0.0646
fin-tuning with walking and running data | 0.0525±0.0594
CCNet on dataset2 | no fine-tuning | 0.0914±0.108
fine-tuning with walking data | 0.0598±0.077
fine-tuning with walking and running data | 0.059±0.0713
Table 3: The influence of the fine-tuning of CCNet with partial motion data of
an unseen skeleton subject 7. Dataset1 contains motions of subjects 1, 3, 4, 8
and dataset2 contains motions of subjects 0, 5, 6, 11. Fine-tuning with
walking and running mocap data of subject 7 with our entire training dataset
(third row) achieves the lowest relative pose difference. IK is disabled in
this experiment.

Comparisons: We first compare the CCNet with baseline models on how accurate
the generated motion is with respect to the user-specified trajectory. Thus,
we leverage the average distance between the user-specified trajectory and the
root trajectory on the XOZ plane as the criterion. In this accuracy
experiment, with six different trajectories manually specified by users, we
extract the direction control signals and randomly assign motion type
information to trajectory segments. Afterward, we synthesize motions using the
first 120 frames of the 33 locomotion sequences in the validation dataset as
the seed frames for each specified trajectory and get a total of 198
controllable motion synthesis results. The trajectory distance is computed by
summing the closest distance between a projected root position and a target
trajectory at each frame. The mean and standard deviation of the averaged
trajectory distance is as follows: 27.878±8.516 for the CCNet, 158.67±30.94
for PFNN-cond, and 171.973±31.862 for ERD-4LR-cond. Fig. 9 shows an example.
The result of the CCNet is more accurate than the baseline models. We will
show the six trajectories in the supplementary material.

We also compare the motion quality in the case of controllable motion
synthesis through a user study (see Sec. 6.4 for details). The user study
results also verify that the CCNet can generate controllable motions of better
quality in the setting of multi-subjects.

###  6.3 Generalization to Unseen Skeletons

After training with multi-subject motion data, the CCNet can generate motions
for skeletons not in the training dataset. As illustrated in Fig. 4, 5b, 5c,
and 7a, we have applied the trained CCNet to automatically generate denoised
and controllable motions for the skeleton of subject 7 not in the training
dataset. We also test the generalization ability of the CCNet by applying it
to a particularly designed skeleton generated by scaling the skeleton of
subject 7. Fig. 10 illustrates that the CCNet can generalize well to the new
skeleton. Since there is no mocap data for it, we utilize motion retargetting
[61] algorithm to generate 120 seed frames for the new skeleton. Besides, we
use ERD-4LR-cond and PFNN-cond to generate motions for the skeleton. The
results show that motions generated by both of them contain big sudden changes
between seed frames and generated frames, which is inferior to the motion
generated by the CCNet.

Moreover, the CCNet can be used as a pre-trained network in few-shot learning
for motion modeling and synthesis. Given a few motion data of a novel
skeleton, it can learn to capture the personalized style implied in the motion
for the skeleton. Tab.3 shows that, after finetuning the network on the
walking and running motion of subject 7, the relative pose difference,
$rel_{p}$, for all mocap data of this subject in the validation dataset can be
significantly reduced. We compute relative pose difference as
$rel_{p}=\frac{1}{N}\sum_{n=0}^{N}(\|\hat{x}_{n}-x_{n}\|_{2}/\|x_{n}\|_{2})$,
where $N$ is the number of frames, and $\hat{x}_{n}$ and $x_{n}$ are the
motion representation vectors of motions generated by the finetuned CCNet and
the corresponding mocap data. The ability of generalization to new skeletons
is crucial since it can save efforts to capture a large number of high-quality
mocap data for a new skeleton in the motion synthesis applications.

To evaluate how the number of subjects in the dataset influences the
generalization ability of the CCNet, we intentionally put the motions of
subjects 1, 3, 4 and 8 into dataset1 and the rest motions of subjects 0, 5, 6
and 11 to dataset2. Tab. 3 lists the statistics of $rel_{p}$ of the motion
generated by the CCNet finetuned on dataset1 (CCNet on dataset1) and dataset2
(CCNet on dataset2). Since the heights of subjects 1, 3, 4 and 8 are closer to
subject 7’s height, the $rel_{p}$ of CCNet on dataset1 is less than that of
dataset2, but still larger than that of CCNet trained with all mocap data in
the training dataset. Thus, to improve the generalization ability of the CCNet
to new skeletons, it is better to construct a database with more subjects for
the network to learn how to handle skeleton variations.

###  6.4 User Study

(a)

(b)

(c)

Figure 11: Foot-ground penetrations in the motions generated by DAE-LSTM-rand,
ERD-4LR-cond and PFNN-cond. (a) The 611th frame generated by DAE-LSTM-rand for
subject 9 in random motion synthesis. (b) The 450th frame generated by
ERD-4LR-cond for subject 7 in controllable motion synthesis. (c) The 666th
frame generated by PFNN-cond for subject 8 in controllable motion synthesis.
DAE-LSTM-rand, ERD-4LR-cond and PFNN-cond can not effectively differentiate
styles of different skeletons and lead to the foot-ground penetrations, as
indicated by red rectangles. Furthermore, the motions generated by the CCNet
is smoother.

VS | RAND | CONTR
---|---|---
| ERD-4LR-rand
---
(mean: 3.065, std: 1.39)
| DAE-LSTM-rand
---
(mean: 1.75, std: 0.968)
| ERD-4LR-cond
---
(mean: 2.75, std: 1.199)
| DAE-LSTM-cond
---
(mean: 0, std: 0)
| PFNN-cond
---
(mean: 2.375, std: 0.992)
CCNet (RAND: mean: 6.25, std: 1.09 CONTR: mean: 6.625 std: 1.615) |  P-value: 9.1671e-8 t-value: -6.9878 |  P-value: 6.1188e-13 t-value: -11.9588 |  P-value: 2.7365e-8 t-value: -7.4383 |  P-value: 1.7796e-16 t-value: -16.3353 |  P-value: 1.0127e-9 t-value: -8.7165

Table 4: T-test of user-study in the cases of random and controllable motion
synthesis (confidence interval=0.95). VS: performing t-test between the
results of the CCNet and all the results of baseline models in the second row.
RAND: random motion synthesis. CONTR: controllable motion synthesis. Mean: the
average number of generated sequences selected by all the participants after
comparing to mocap sequences in the same group. Std: the standard deviation of
the number being selected.

We perform a t-test to verify the hypothesis that the CCNet can generate
motions of better quality than baseline models. To this end, we design the
user study as follows. First, we present 16 participants with all groups of
motion sequences: three groups for random motion synthesis and four groups for
controllable motion synthesis. Each group contains 16 pairs of motion
sequences. One is the mocap sequence, and the other is generated by the CCNet
or one of the baseline models (Please refer to the supplementary_material.pdf
in "other supplementary materials" for the details of motion generation in the
user study). Second, we ask the participants to select which sequence in a
pair is of better motion quality. If the chosen number of CCNet-generated
motion sequences is larger than the chosen number for other baseline models
with statistical significance, we deem that our hypothesis is verified. The
participants include six females and ten males, and all of them have
experience with 3D animation or games. Before the user study, we present a few
mocap sequences to the participants as examples of goog-quality motions.
Besides, if there are sudden changes or foot penetrations into the ground
plane, the sequence should be regarded as the worse one. At last, we get 15
valid questionnaires for random motion synthesis and 16 for controllable
motion synthesis.

The t-test results are shown in Tab. 4. The P-values of the CCNet vs. other
baseline models are all less than the selected threshold (0.05). Therefore,
the motions generated by the CCNet are significantly different from the
motions generated by baseline models. According to the average number of
motion sequences selected by the participants (mean in Tab. 4), the average
number of selected motion sequences of CCNet is larger than that of other
baseline models. It verifies that the CCNet can generate better motions than
state-of-the-art baseline models in different scenarios (The ANOVA test result
in the supplementary material also verifies the statistical significance of
the user study). Furthermore, we prepare another 5-group data that contain
pairs of motion sequences generated by the CCNet and each baseline model. As
listed in Tab. 5, the number of CCNet-generated motion sequences selected by
participants is still larger than that of the sequences generated by baseline
models. Fig. 11 shows examples of generated motion sequences used in this
study. Motion jittering and penetrations into the ground plane frequently
happen for motion sequences generated using ERD-4LR-cond, ERD-4LR-cond, and
PFNN-cond, which indicates that these models can’t handle the skeleton
variations as well as the CCNet. In addition, we observe that DAE-LSTM-cond
fails to generate long-period, controllable motion.

Groups | numbers for CCNet (mean±std)
---|---
ERD-4LR-rand vs. CCNet | 10.94±1.56
DAE-LSTM-rand vs. CCNet | 12.31±2.34
ERD-4LR-cond vs. CCNet | 11.63±2.87
DAE-LSTM-cond vs. CCNet | 16±0.0
PFNN-cond vs. CCNet | 11.445±1.87
Table 5: The average selected number for CCNet-generated motion sequences.
Baseline-X vs. CCNet: a group of 16 pairs of motion sequences generated by a
baseline model and the CCNet. Mean±std: mean and variance of the number of
CCNet-generated motion sequences selected by all the participants.

###  6.5 Evaluation of Network hyper-Parameters and Training Settings

In this section, we report the experiments conducted to figure out the hyper-
parameters and training settings selected for the CCNet, including causal
receptive length (CRL), numbers of consecutive frames of each sample in a
batch (NCF), the length of seed frames, and the choice of smoothness loss
term. We conduct all the experiments on the same training and validation
datasets to verify our choices. Precisely, the chosen hyper-parameters for our
CCNet in the random and controllable motion synthesis experiments above are as
follows: CRL=41 and NCF=240. They are selected to minimize the loss in Eq. 2
computed on mocap data in the validation set. For better visualization, the
loss curves are plotted using the formula ${\log_{10}(loss+320)}$, where the
loss is computed using Eq. 2. A bias $320$ is added to make $loss+320$
positive since the loss value is usually around $-300$.

Figure 12: Loss curves obtained using different hyper-parameters of the CCNet.
Left: Training. Right: Validation. We modify each hyper-parameter, including
CRL, NCF and with/without skeleton configuration (w/_sk or w/o_sk), and
compute the losses on the training and validation datasets respectively.
Figure 13: Ablation study of smoothness loss term. We remove the smoothness
loss term and evaluate the corresponding re-trained model. Left: Training.
Right: Validation.

Causal receptive length: We conduct experiments to choose the causal receptive
length among three choices: 31 (with dilations of SRBs being repeatedly 1, 2),
41 (with dilations of SRBs being 2), and 46 (with dilations of SRBs being
repeatedly 1, 2, 4), numbers of SRBs are fixed at 20 and we keep all the other
settings the same as described in Section. 5.1. When the causal receptive
length is 31 and 46, the network converges to higher loss after training on
the training dataset, as shown by the red dashed lines and green dash-dot
lines in Fig. 12. We choose CRL as 41 that can lead to the lowest loss in this
experiment.

NCF in a batch: We also test the NCF of each sample in a batch, which is set
to be 60, 120, and 240, respectively. These numbers correspond to 1 second, 2
seconds, and 4 seconds long sequences. This can be easily achieved by changing
the frame numbers in a batch and keep the other parameters the same. For the
comparison’s convenience, we keep the numbers of consecutive frames of each
sample in a batch fixed as 240 when computing the loss on the validation
dataset. The cyan dashed lines in Fig. 12 illustrate that both training and
validation loss exploded before converging when NCF is 60. The magenta dashed
lines in Fig. 12 show that the network converges to a higher loss in the case
of NCF=120, comparing to NCF=240. We hypothesize that 60 and 120 consecutive
frames result in inadequate training data in a batch and too noisy gradient
when training the network. Thus, we set NCF to 240 frames in training.

The importance of the skeleton configuration and the smoothness loss term: It
is implemented by disconnecting the 1D convolution module for the skeleton
configurations signal to the network and check whether the loss on the
validation set increases significantly. The black dash-dot lines in Fig. 12
indicate that the network without skeleton configuration converges to a much
higher loss when training on our multi-subject training dataset. However, the
loss explodes at around 700 epochs on the validation dataset. Thus, the
skeleton configuration plays an important role in the network to disambiguate
different subjects’ motions. To verify the importance of smoothness loss term
to the CCNet, we train the network by removing it from the loss terms. Fig. 13
indicates that the loss curves on both training and validation datasets
explode before converging without smoothness term, which means the smoothness
term is essential for the network to prevent the generated motions from big
sudden changes and converge to a better result.

seed frame numbers | Relative Pose Difference (mean/std)
---|---
ERD-4LR-cond | DAE-LSTM-cond | CCNet
1 | 0.214±0.302 | 0.56±0.572 | 0.13±0.179
5 | 0.18±0.145 | 0.325±0.508 | 0.091±0.128
10 | 0.14±0.159 | 0.303±0.49 | 0.084±0.136
30 | 0.145±0.179 | 0.182±0.27 | 0.074±0.128
60 | 0.12±0.138 | 0.178±0.253 | 0.074±0.124
120 | 0.126±0.16 | 0.115±0.144 | 0.074±0.125
Table 6: Ablation study on the length of seed frames in the case of
controllable motion synthesis. We synthesize motion sequences using different
lengths of seed frames to check their influences on the generated motions. IK
is disabled in this ablation study.

Seed frame length: The influence of seed frame length on the quality of
generated motions is measured by computing the relative pose differences
between generated motion and corresponding mocap data. Specifically, we
extract seed frames from mocap data in the validation dataset and then let the
networks predict a frame to be compared with the corresponding mocap frame in
the case of controllable motion synthesis. Tab. 6 shows the computed relative
pose difference $rel_{p}$ (see its definition in Sec. 6.3) for different seed
frame length, such as 1, 5, 10, 30, 60 and 120. The lower difference value
indicates that the generated motion is more similar to mocap data and thus of
better quality. It can be seen that the CCNet is robust to the variation of
the length of seed frames comparing to ERD-4LR-cond and DAE-LSTM-cond, and it
doesn’t need too long seed frames to synthesize high-quality motions. However,
we observe that there are more obvious jitters between the seed frames and the
generated frames in the cases of 1 and 5 seed frame lengths. We hypothesize
that, for such short length seed frames, the CCNet can’t get enough
information to generate smooth motions.

##  7 Conclusion

We have designed a deep generative motion model, i.e., CCNet, based on causal
convolution proposed in WaveNet [8] to synthesize high-quality motions for
multiple subjects. The trained CCNet can also synthesize several types of
complex motions, such as punching, kicking, and kicking while punching,
included in our database. The CCNet can be applied to various applications,
such as random motion generation, motion denoising, motion completion, and
controllable motion synthesis. Moreover, the CCNet can generate motions for
novel skeletons. Given sample motions of a novel skeleton, the pre-trained
CCNet can be fine-tuned to capture the skeleton’s motion style.

Limitation and future work: Currently, the CCNet trained on our database can
not handle arbitrary skeleton variation. For instance, if we scale the lower
body of a skeleton in our database with a ratio less than 0.6, the CCNet
without fine-tuning will generate motions with severe foot-ground penetration
for the scaled skeleton. We hypothesize that it is because that 11 different
skeletons in our training set might not be enough for the network to learn how
to handle the large space of skeleton variation. Therefore, we plan to
increase the number of subjects in the database to investigate the capacity of
the CCNet. In addition, we also plan to increase the number and types of
complex motions in the database to improve the quality of complex motion
generation. Another issue with our CCNet is the number of seed frames required
to initialize the motion synthesis. When seed frame length is set to between 1
and 5, jitters will occur in the generated motion, which might limit the
application of the CCNet to model swift motions. We are also interested in
investigating to take more temporal information as input, for instance, phase-
like information in PFNN or joint accelerations, to mitigate this issue.

## References

  * [1] Graham W Taylor and Geoffrey E Hinton.  Factored conditional restricted boltzmann machines for modeling motion style.  In Proceedings of the 26th annual international conference on machine learning, pages 1025–1032. ACM, 2009.
  * [2] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4346–4354, 2015.
  * [3] J A Martinez, Michael J Black, and Javier Romero.  On human motion prediction using recurrent neural networks.  pages 4674–4683, 2017.
  * [4] Daniel Holden, Jun Saito, and Taku Komura.  A deep learning framework for character motion synthesis and editing.  ACM Transactions on Graphics (TOG), 35(4):138, 2016.
  * [5] Daniel Holden, Taku Komura, and Jun Saito.  Phase-functioned neural networks for character control.  ACM Trans. Graph., 36(4), July 2017.
  * [6] Kyungho Lee, Seyoung Lee, and Jehee Lee.  Interactive character animation by learning multi-objective control.  ACM Trans. Graph., 37(6), December 2018.
  * [7] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel van de Panne.  Character controllers using motion vaes.  39(4), 2020.
  * [8] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio, 2016.
  * [9] Charles Rose, Michael F Cohen, and Bobby Bodenheimer.  Verbs and adverbs: Multidimensional motion interpolation.  IEEE Computer Graphics and Applications, 18(5):32–40, 1998.
  * [10] Luis Molina Tanco and Adrian Hilton.  Realistic synthesis of novel human movements from a database of motion capture examples.  In Proceedings Workshop on Human Motion, pages 137–142. IEEE, 2000\.
  * [11] Richard Bowden.  Learning statistical models of human motion.  In IEEE Workshop on Human Modeling, Analysis and Synthesis, CVPR, volume 2000, 2000.
  * [12] Matthew Brand and Aaron Hertzmann.  Style machines.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 183–192. ACM Press/Addison-Wesley Publishing Co., 2000.
  * [13] Jinxiang Chai and Jessica K Hodgins.  Constraint-based motion optimization using a statistical dynamic model.  In ACM Transactions on Graphics (TOG), volume 26, page 8. ACM, 2007\.
  * [14] Xiaolin Wei, Jianyuan Min, and Jinxiang Chai.  Physically valid statistical models for human motion generation.  ACM Transactions on Graphics (TOG), 30(3):19, 2011.
  * [15] Keith Grochow, Steven L Martin, Aaron Hertzmann, and Zoran Popović.  Style-based inverse kinematics.  In ACM transactions on graphics (TOG), volume 23, pages 522–531. ACM, 2004.
  * [16] Jinxiang Chai and Jessica K Hodgins.  Performance animation from low-dimensional control signals.  ACM Transactions on Graphics (ToG), 24(3):686–696, 2005.
  * [17] Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard.  Interactive control of avatars animated with human motion data.  In ACM Transactions on Graphics (ToG), volume 21, pages 491–500. ACM, 2002.
  * [18] Rachel Heck and Michael Gleicher.  Parametric motion graphs.  In Proceedings of the 2007 symposium on Interactive 3D graphics and games, pages 129–136. ACM, 2007.
  * [19] Okan Arikan, David A Forsyth, and James F O’Brien.  Motion synthesis from annotations.  In ACM Transactions on Graphics (TOG), volume 22, pages 402–408. ACM, 2003.
  * [20] Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović.  Motion fields for interactive character locomotion.  In ACM Transactions on Graphics (TOG), volume 29, page 138. ACM, 2010.
  * [21] Jianyuan Min and Jinxiang Chai.  Motion graphs++: a compact generative model for semantic motion analysis and synthesis.  ACM Transactions on Graphics (TOG), 31(6):153, 2012.
  * [22] Okan Arikan and David A Forsyth.  Interactive motion generation from examples.  In ACM Transactions on Graphics (TOG), volume 21, pages 483–490. ACM, 2002.
  * [23] Lucas Kovar, Michael Gleicher, and Frédéric Pighin.  Motion graphs.  In ACM SIGGRAPH 2008 classes, page 51. ACM, 2008.
  * [24] Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li.  Auto-conditioned recurrent networks for extended complex human motion synthesis.  2018\.
  * [25] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges.  Learning human motion models for long-term predictions.  In 2017 International Conference on 3D Vision (3DV), pages 458–466. IEEE, 2017.
  * [26] Shihong Xia, Congyi Wang, Jinxiang Chai, and Jessica Hodgins.  Realtime style transfer for unlabeled heterogeneous human motion.  ACM Transactions on Graphics (TOG), 34(4):119, 2015.
  * [27] Jianyuan Min, Huajun Liu, and Jinxiang Chai.  Synthesis and editing of personalized stylistic human motion.  In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games, pages 39–46, 2010.
  * [28] Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen.  Skeleton-aware networks for deep motion retargeting.  ACM Transactions on Graphics (TOG), 39(4):62, 2020.
  * [29] Jack M Wang, David J Fleet, and Aaron Hertzmann.  Gaussian process dynamical models for human motion.  IEEE transactions on pattern analysis and machine intelligence, 30(2):283–298, 2007.
  * [30] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce.  Learning motion manifolds with convolutional autoencoders.  In SIGGRAPH Asia 2015 Technical Briefs, page 18. ACM, 2015.
  * [31] He Zhang, Sebastian Starke, Taku Komura, and Jun Saito.  Mode-adaptive neural networks for quadruped motion control.  ACM Trans. Graph., 37(4), July 2018.
  * [32] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito.  Neural state machine for character-scene interactions.  ACM Trans. Graph., 38(6), November 2019.
  * [33] Jungdam Won, Deepak Gopinath, and Jessica Hodgins.  A scalable approach to control diverse behaviors for physically simulated characters.  ACM Trans. Graph., 39(4), July 2020.
  * [34] Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman.  Local motion phases for learning multi-contact character movements.  ACM Trans. Graph., 39(4), July 2020.
  * [35] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  ACM Transactions on Graphics (TOG), 37(4):143, 2018.
  * [36] Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen.  Carl: Controllable agent with reinforcement learning for quadruped locomotion.  ACM Transactions on Graphics (Proceedings of SIGGRAPH 2020), 39(4), 2020.
  * [37] Libin Liu and Jessica Hodgins.  Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning.  ACM Trans. Graph., 37(4), July 2018.
  * [38] Libin Liu and Jessica Hodgins.  Learning to schedule control fragments for physics-based characters using deep q-learning.  ACM Trans. Graph., 36(3), June 2017.
  * [39] Libin Liu, Michiel van de Panne, and KangKang Yin.  Guided learning of control graphs for physics-based characters.  ACM Transactions on Graphics, 35(3), 2016.
  * [40] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne.  Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning.  ACM Transactions on Graphics (TOG), 36(4):41, 2017.
  * [41] Zhiyong Wang, Jinxiang Chai, and Shihong Xia.  Combining recurrent neural networks and adversarial training for human motion synthesis and control.  IEEE Transactions on Visualization and Computer Graphics, pages 1–1, 2019.
  * [42] Omid Alemi, Jules Françoise, and Philippe Pasquier.  Groovenet: Real-time music-driven dance movement generation using artificial neural networks.  networks, 8(17):26, 2017.
  * [43] Angela S Lin, Lemeng Wu, Rodolfo Corona, Kevin Tai, Qixing Huang, and Raymond J Mooney.  Generating animated videos of human activities from natural language descriptions.  In NeurIPS, 2018.
  * [44] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena.  Structural-rnn: Deep learning on spatio-temporal graphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5308–5317, 2016.
  * [45] Qiang Nie, Ziwei Liu, and Yunhui Liu.  Unsupervised 3d human pose representation with viewpoint and pose disentanglement.  In European Conference on Computer Vision (ECCV), 2020.
  * [46] Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer.  Context-aware human motion prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [47] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom.  Deep representation learning for human motion prediction and classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6158–6166, 2017.
  * [48] Mao Wei, Liu Miaomiao, and Salzemann Mathieu.  History repeats itself: Human motion prediction via motion attention.  In ECCV, 2020.
  * [49] Ye Yuan and Kris Kitani.  Residual force control for agile human behavior imitation and extended motion synthesis.  In Advances in Neural Information Processing Systems, 2020.
  * [50] Jingwei Xu, Huazhe Xu, Bingbing Ni, Xiaokang Yang, Xiaolong Wang, and Trevor Darrell.  Hierarchical style-based networks for motion synthesis.  In eccv, 2020.
  * [51] Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian.  Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [52] Qiongjie Cui, Huaijiang Sun, and Fei Yang.  Learning dynamic relationships for 3d human motion prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [53] Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee.  Neural kinematic networks for unsupervised motion retargetting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8639–8648, 2018.
  * [54] Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould.  A stochastic conditioning scheme for diverse human motion prediction.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [55] Rui Zhao, Hui Su, and Qiang Ji.  Bayesian adversarial human motion synthesis.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [56] Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, and Hao Li.  Auto-conditioned lstm network for extended complex human motion synthesis.  arXiv preprint arXiv:1707.05363, 3, 2017.
  * [57] Dario Pavllo, David Grangier, and Michael Auli.  Quaternet: A quaternion-based recurrent model for human motion.  In BMVC, 2018.
  * [58] Wikipedia.  Quaternion — Wikipedia, the free encyclopedia, 2020.  [Online; accessed 19-November-2020].
  * [59] Alex Graves.  Generating sequences with recurrent neural networks.  Computer Science, 2013.
  * [60] Xiaolin K. Wei, Peizhao Zhang, and Jinxiang Chai.  Accurate realtime full-body motion capture using a single depth camera.  ACM Trans. Graph., 31(6):188:1–188:12, 2012.
  * [61] Samuel R. Buss.  Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods.  Technical report, IEEE Journal of Robotics and Automation, 2004.

##  8 Supplementary Material

###  8.1 Details of Network Parameters

CCNet: The detailed parameters of the separate dilation blocks (SRB) used in
the CCNet are shown in Tab.8 and Tab.9 respectively. The format of the "output
size" column is #channel$\times$#NCF, where #channle indicates the numbers of
feature channels and #NCF indicates the number of consecutive frames (NCF) of
each sample in a batch.

The modification of ERD-4LR and DAE-LSTM: Both original ERD-4LR and DAE-LSTM
networks in Sec.6 of our paper have 1024 hidden units in the linear layers and
512 hidden units in the LSTM layers. To test the performance of these two
types of networks in the multi-subject motion synthesis, we add additional
parameters to let the network accept as inputs the skeleton configuration.
Specifically, we add a Linear-Tanh module (with 1024 hidden units in the
Linear layer) for ERD-4LR and DAE-LSTM to map skeleton configurations to
features and then add them to the feature output by the encoder. The summed
features are fed to LSTM layers for the random motion synthesis. We denote
these two adapted networks for random motion synthesis as _ERD-4LR-rand_ and
_ERD-4LR-rand_. Their network parameters are shown in Tab.10.

Similarly, to extend ERD-4LR and DAE-LSTM to support multi-subject,
controllable motion synthesis, we additionally add a Linear-Tanh module (with
1024 hidden units in the Linear layer) to convert each of the control signals
into features. Similarly, these features are added to the encoder output to
form the input of subsequent LSTM layers. We denote the two adapted networks
for controllable motion synthesis as _ERD-4LR-cond_ and _ERD-4LR-cond_. Their
network parameters are shown in Tab.11.

The modification of PFNN: We also adapt the network architecture of PFNN [5]
to take the skeleton configuration as an input. It is implemented by replacing
terrain data in its original inputs with the skeleton configuration since we
only test the synthesis of motions on a ground plane. The rest inputs of PFNN
remain the same as in [5]. Since PFNN is mainly designed for controllable
motion synthesis, not a generative model, we only compare the CCNet with PFNN
on controllable motion synthesis. The modified PFNN is denoted as _PFNN-cond_
, and its network parameters are shown in Tab.12.

###  8.2 Trajectory-following Error Comparisons

Fig.14 illustrates the six manually specified trajectories used in the
trajectory-following error comparison of controllable motion synthesis (the
"comparisons" paragraph in Sec. 6.2 in our paper). The trajectory-following
error for all the trajectories is visualized in Fig.9.

###  8.3 User Study

Motion generation: To conduct the user study, we first randomly select 16
mocap sequences from the validation dataset, including five motion sequences
of subject 7 and one motion sequence of each of the rest of subjects in our
database. Since subject 7 is not included in the training dataset, we thus
choose more number of its motions to check the motion quality, in this case,
more carefully. Secondly, for the user study on random motion synthesis, we
initialize the CCNet and baseline models using the first 120 frames of each
mocap sequence as seed frames. Consequently, we obtain 16 sequences generated
by CCNet, ERD-4LR-rand, and DAE-LSTM-rand, respectively, to form three groups
of motion pairs. For controllable motion synthesis, seed frames are the same
120 frames of each sequence, and control signals are extracted from the 16
mocap sequences as described in section 4.2. The extracted control signals
guarantee that these networks generate motions with the same motion types as
the mocap sequences. We apply CCNet, ERD-4LR-cond, DAE-LSTM-cond, and PFNN-
cond to generate 16 motion sequences separately. The average length of
selected motion sequences is around 10 seconds, and the length of each
generated motion sequence is chosen to be the same length as the corresponding
mocap sequence. All the motion sequence pairs are present to the participants
in a random order for their evaluation.

We name the videos of the generated motion sequences as follows: "rand0" and
"cond0" are used for the videos of mocap data; "rand1" and "cond1" for the
videos of motions generated by the CCNet; "rand2" and "cond2" for ERD-4LR-rand
and ERD-4LR-cond; "rand3" and "cond3" for DAE-LSTM-rand and DAE-LSTM-cond; and
"cond4" for PFNN-cond.

| Source of Variation | SS | df | MS | F | P-value | F-critical
---|---|---|---|---|---|---|---
Random motion synthesis | Between Groups | 171.375 | 2 | 85.6875 | 59.3792 | 2.3862e-13 | 3.2043
Within Groups | 64.9375 | 45 | 1.4431 | - | - | -
Total | 236.3125 | 47 | - | - | - | -
Controllable motion synthesis | Between Groups | 346.6875 | 3 | 115.5625 | 90.342 | 3.1855e-22 | 2.7581
Within Groups | 76.75 | 60 | 1.2792 | - | - | -
Total | 423.4375 | 63 | - | - | - | -
Table 7: ANOVA-test of user study for confidence interval=0.95 in the cases of
random and controllable motion synthesis. SS: sum-of-squares for between-group
variability. Df: degrees of freedom. MS: mean squares. F: F ratio. -: not
applicable.

ANOVA-test: We also perform an ANOVA-test on the user study results as
illustrated in Tab.7, which also verifies the statistical significance of the
user study.

Block name | Output size | Filter size
---|---|---
CausalConv1 | 32×240 | 1×1, 32, stride 1, dilation 2, padding (2, 0); input: motion frames
cond1_conv1+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: c1
cond2_conv1+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: c2
cond3_conv1+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: c3
CausalConv2 | 32×240 | 1×1, 32, stride 1, dilation 2, padding (2, 0); input: the same as it to CausalConv1
cond1_conv2+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: the same as it to cond1_conv1
cond2_conv2+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: the same as it to cond2_conv1
cond3_conv2+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1; input: the same as it to cond3_conv1
sigmoid | 32×240 | input: the sum of CausalConv1, cond1_conv1+ReLU, cond2_conv1+ReLU and cond3_conv1+ReLU
Tanh | 32×240 | input: the sum of CausalConv2, cond1_conv2+ReLU, cond2_conv2+ReLU and cond3_conv2+ReLU
element-wise multiply | 32×240 | input: sigmoid and Tanh
conv_res | 32×240 |  | 1×1, 32, stride 1, padding 1, dilation 1; input: element-wise multiply
---
output(SRB_resi): the sum of conv_res and the input to CausalConv1
conv_skip | 512×240 |  | 1×1, 512, stride 1, padding 1, dilation 1; input: element-wise multiply
---
output(SRB_skipi): the output of conv_skip

Table 8: Network parameters of a single SRBi. The dilation of SRBi is 2.

Block name | Output size | Filter size
---|---|---
Encoder
conv1+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1
conv2+ReLU | 32×240 | 1×1, 32, stride 1, padding 1, dilation 1
Separate Residual Blocks(SRBs)
SRB0 | SRB_res0: 32×240 | dilation 2; input: conv2, c1, c2 and c3
SRB_skip0: 512×240 | dilation 2; input: conv2, c1, c2 and c3
SRB1 | SRB_res1: 32×240 | dilation 2; input: SRB_res0, c1, c2 and c3
SRB_skip1: 512×240 | dilation 2; input: SRB_res0, c1, c2 and c3
SRB2 | SRB_res2: 32×240 | dilation 2; input: SRB_res1, c1, c2 and c3
SRB_skip2: 512×240 | dilation 2; input: SRB_res1, c1, c2 and c3
SRB3 | SRB_res3: 32×240 | dilation 2; input: SRB_res2, c1, c2 and c3
SRB_skip3: 512×240 | dilation 2; input: SRB_res2, c1, c2 and c3
SRB4 | SRB_res4: 32×240 | dilation 2; input: SRB_res3, c1, c2 and c3
SRB_skip4: 512×240 | dilation 2; input: SRB_res3, c1, c2 and c3
SRB5 | SRB_res5: 32×240 | dilation 2; input: SRB_res4, c1, c2 and c3
SRB_skip5: 512×240 | dilation 2; input: SRB_res4, c1, c2 and c3
SRB6 | SRB_res6: 32×240 | dilation 2; input: SRB_res5, c1, c2 and c3
SRB_skip6: 512×240 | dilation 2; input: SRB_res5, c1, c2 and c3
SRB7 | SRB_res7: 32×240 | dilation 2; input: SRB_res6, c1, c2 and c3
SRB_skip7: 512×240 | dilation 2; input: SRB_res6, c1, c2 and c3
SRB8 | SRB_res8: 32×240 | dilation 2; input: SRB_res7, c1, c2 and c3
SRB_skip8: 512×240 | dilation 2; input: SRB_res7, c1, c2 and c3
SRB9 | SRB_res9: 32×240 | dilation 2; input: SRB_res8, c1, c2 and c3
SRB_skip9: 512×240 | dilation 2; input: SRB_res8, c1, c2 and c3
SRB10 | SRB_res10: 32×240 | dilation 2; input: SRB_res9, c1, c2 and c3
SRB_skip10: 512×240 | dilation 2; input: SRB_res9, c1, c2 and c3
SRB11 | SRB_res11: 32×240 | dilation 2; input: SRB_res10, c1, c2 and c3
SRB_skip11: 512×240 | dilation 2; input: SRB_res10, c1, c2 and c3
SRB12 | SRB_res12: 32×240 | dilation 2; input: SRB_res11, c1, c2 and c3
SRB_skip12: 512×240 | dilation 2; input: SRB_res11, c1, c2 and c3
SRB13 | SRB_res13: 32×240 | dilation 2; input: SRB_res12, c1, c2 and c3
SRB_skip13: 512×240 | dilation 2; input: SRB_res12, c1, c2 and c3
SRB14 | SRB_res14: 512×240 | dilation 2; input: SRB_res13, c1, c2 and c3
Decoder
ReLU+conv3 | 512×240 | 1×1, 512, stride 1, padding 1, dilation 1; input: the sum of SRB_skip0, SRB_skip1, …, SRB_skip14
ReLU+conv4 | 613×240 | 1×1, 613, stride 1, padding 1, dilation 1

Table 9: Network parameters of the CCNet. The input to the CCNet includes 240
frames of motions, the skeleton configuration c1, the direction and velocity
c2, and the motion type c3. The network architecture of the CCNet is inspired
by [8].

Block name | Output size | Input/Output
---|---|---
linear1+Tanh | 1024×240 |  | input: motion frames
---
output: linear1
linear2+Tanh | 1024×240 |  | input: linear1
---
output: linear2
LSTM1 | 512×240 |  | input: linear2
---
output: lstm1
LSTM2 | 512×240 |  | input: lstm1
---
output: lstm2
LSTM3 | 512×240 |  | input: lstm2
---
output: lstm3
LSTM4 | 1024×240 |  | input: lstm3
---
output: lstm4
linear3+Tanh | 1024×240 |  | input: lstm4
---
output: linear3
linear4+Tanh | 613×240 |  | input: linear3
---
output: the predicted motion frames

(a) ERD-4LR-rand.

Block name | Output size | Input/Output
---|---|---
linear1+Tanh | 1024×240 |  | input: motion frames
---
output: linear1
linear2+Tanh | 1024×240 |  | input: linear1
---
output: linear2
cond1_linear+Tanh | 1024×240 |  | input: c1
---
output: cond1_linear
cond2_linear+Tanh | 1024×240 |  | input: c2
---
output: cond2_linear
cond3_linear+Tanh | 1024×240 |  | input: c3
---
output: cond3_linear
LSTM1 | 512×240 |  | input: the sum of the linear2,
---
cond_linear1, cond_linear2 and cond_linear3
output: lstm1
LSTM2 | 512×240 |  | input: lstm1
---
output: lstm2
LSTM3 | 512×240 |  | input: lstm2
---
output: lstm3
LSTM4 | 1024×240 |  | input: lstm3
---
output: lstm4
linear3+Tanh | 1024×240 |  | input: lstm4
---
output: linear3
linear4+Tanh | 613×240 |  | input: linear3
---
output: the predicted motion frames

(b) ERD-4LR-cond.

Table 10: Network parameters of the ERD-4LR-rand and ERD-4LR-cond. The inputs
to the ERD-4LR-rand and ERD-4LR-cond include 240 frames of motions, the
skeleton configuration c1, the direction and velocity c2, and the motion type
c3. The overall network architectures of ERD-4LR-rand and ERD-4LR-cond are the
adaptation of [2] to multi-subject motion synthesis, but implemented with
4-layers LSTM as in [6].

Block name | Output size | Input/Output
---|---|---
dropout+linear1+ReLU | 1024×240 |  | dropout probability: 0.3
---
input: motion frames
output: linear1
linear2+ReLU | 1024×240 |  | input: linear1
---
output: linear2
linear3+ReLU | 1024×240 |  | input: linear2
---
output: linear3
linear4 | 305×240 |  | input: linear3
---
output: linear4
LSTM1 | 512×240 |  | input: linear2,
---
output: lstm1
LSTM2 | 512×240 |  | input: lstm1
---
output: lstm2
LSTM3 | 512×240 |  | input: lstm2
---
output: lstm3
linear5 | 613×240 |  | input: lstm3
---
output: the predicted motion frames

(a) DAE-LSTM-rand.

Block name | Output size | Input/Output
---|---|---
dropout+linear1+ReLU | 1024×240 |  | dropout probability: 0.3
---
input: motion frames
output: linear1
linear2+ReLU | 1024×240 |  | input: linear1
---
output: linear2
linear3+ReLU | 1024×240 |  | input: linear2
---
output: linear3
linear4 | 305×240 |  | input: linear3
---
output: linear4
cond1_linear+Tanh | 1024×240 |  | input: c1
---
output: cond1_linear
cond2_linear+Tanh | 1024×240 |  | input: c2
---
output: cond2_linear
cond3_linear+Tanh | 1024×240 |  | input: c3
---
output: cond3_linear
LSTM1 | 512×240 |  | input: the sum of the linear4,
---
cond_linear1, cond_linear2 and cond_linear3
output: lstm1
LSTM2 | 512×240 |  | input: lstm1
---
output: lstm2
LSTM3 | 512×240 |  | input: lstm2
---
output: lstm3
linear5 | 613×240 |  | input: lstm3
---
output: the predicted motion frames

(b) DAE-LSTM-cond.

Table 11: Network parameters of DAE-LSTM-rand and DAE-LSTM-cond. The inputs to
the DAE-LSTM-rand and DAE-LSTM-cond include 240 frames of motions, the
skeleton configuration c1, the direction and velocity c2, and the motion type
c3. The overall network architectures of DAE-LSTM-rand and DAE-LSTM-cond are
the adaptation of the network in [25] to multi-subject motion synthesis.

Block name | Output size | Input/Output
---|---|---
dropout+pfnn_linear1+ELU | 512×1 |  | dropout probability: 0.7
---
input: the concatenation of
motion frames and control signals
output: pfnn_linear1
dropout+pfnn_linear2+ELU | 512×1 |  | dropout probability: 0.7
---
input: pfnn_linear1
output: pfnn_linear2
dropout+pfnn_linear3 | 437×1 |  | dropout probability: 0.7
---
input: pfnn_linear2
output: the predicted
motion frames and control signals

Table 12: Network parameters of the PFNN-cond. The input to the PFNN-cond is
described in Sec.1 in the supplementary material.

(a) trajectory 1 (b) trajectory 2 (c) trajectory 3 (d) trajectory 4

(e) trajectory 5 (f) trajectory 6

Figure 14: Six trajectories used in trajectory-following comparisons. The
green color indicates the starting point of a trajectory, while the red color
indicates the terminal point.