# MoCapAct: A Multi-Task Dataset for
Simulated Humanoid Control

Nolan Wagener ${}^{\\!1}$ Andrey Kolobov2 Felipe Vieira Frujeri2
Ricky Loynd2 Ching-An Cheng2 Matthew Hausknecht∗2
1Institute for Robotics and Intelligent Machines, Georgia Institute of
Technology
2Microsoft Research Correspondence to<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
###### Abstract

Simulated humanoids are an appealing research domain due to their physical
capabilities. Nonetheless, they are also challenging to control, as a policy
must drive an unstable, discontinuous, and high-dimensional physical system.
One widely studied approach is to utilize motion capture (MoCap) data to teach
the humanoid agent low-level skills (e.g., standing, walking, and running)
that can then be re-used to synthesize high-level behaviors. However, even
with MoCap data, controlling simulated humanoids remains very hard, as MoCap
data offers only kinematic information. Finding physical control inputs to
realize the demonstrated motions requires computationally intensive methods
like reinforcement learning. Thus, despite the publicly available MoCap data,
its utility has been limited to institutions with large-scale compute. In this
work, we dramatically lower the barrier for productive research on this topic
by training and releasing high-quality agents that can track over three hours
of MoCap data for a simulated humanoid in the dm_control physics-based
environment. We release _MoCapAct_ (Motion Capture with Actions), a dataset of
these expert agents and their rollouts, which contain proprioceptive
observations and actions. We demonstrate the utility of MoCapAct by using it
to train a _single_ hierarchical policy capable of tracking the _entire_ MoCap
dataset within dm_control and show the learned low-level component can be re-
used to efficiently learn downstream high-level tasks. Finally, we use
MoCapAct to train an autoregressive GPT model and show that it can control a
simulated humanoid to perform natural motion completion given a motion prompt.
Videos of the results and links to the code and dataset are available at the
project website.

##  1 Introduction

The wide range of human physical capabilities makes simulated humanoids a
compelling platform for studying motor intelligence. Learning and utilization
of motor skills is a prominent research topic in machine learning, with
advances ranging from emergence of learned locomotion skills in traversing an
obstacle course (Heess et al., 2017) to the picking up and carrying of objects
to desired locations (Merel et al., 2020; Peng et al., 2019a) to team
coordination in simulated soccer (Liu et al., 2022). Producing natural and
physically plausible human motion animation (Harvey et al., 2020; Kania et
al., 2021; Yuan and Kitani, 2020) is an active research topic in the game and
movie industries. However, while physical simulation of human capabilities is
a useful research domain, it is also very challenging from a control
perspective. A controller must contend with an unstable, discontinuous, and
high-dimensional system that requires a high degree of coordination to execute
a desired motion.

Tabula rasa learning of complex humanoid behaviors (e.g., navigating through
an obstacle field) is extremely difficult for all known learning approaches.
In light of this challenge, motion capture (MoCap) data has become an
increasingly common aid in humanoid control research (Merel et al., 2017; Peng
et al., 2018). MoCap trajectories contain kinematic information about motion:
they are sequences of configurations and poses that the human body assumes
throughout the motion in question. This data can alleviate the difficulty of
training sophisticated control policies by enabling a simulated humanoid to
learn _low-level_ motor skills from MoCap demonstrations. The low-level skills
can then be re-used for learning advanced, higher-level motions. Datasets such
as CMU MoCap (CMU, 2003), Human3.6M (Ionescu et al., 2013), and LaFAN1 (Harvey
et al., 2020) offer hours of recorded human motion, ranging from simple
locomotion demonstrations to interactions with other humans and objects.

However, since MoCap data only offers kinematic information, utilizing it in a
physics simulator requires recovering the actions (e.g., joint torques) that
induce the sequence of kinematic poses in a given MoCap trajectory (i.e.,
track the trajectory). While easier than tabula rasa learning of a high-level
task, finding an action sequence that makes a humanoid track a MoCap sequence
is still non-trivial. For instance, this problem has been tackled with
reinforcement learning (Chentanez et al., 2018; Merel et al., 2019b; Peng et
al., 2018) and adversarial learning (Merel et al., 2017; Wang et al., 2017).
The computational burden of finding these actions scales with the amount of
MoCap data, and training agents to recreate hours of MoCap data requires
significant compute. _As a result, despite the broad availability of MoCap
datasets, their utility—and their potential for enabling research progress on
learning-based humanoid control—has been limited to institutions with large
compute budgets._

Figure 1: The MoCapAct Dataset includes expert policies that are trained to
track individual clips. A dataset of noise-injected rollouts (containing
observations and actions) is then collected from each expert. These rollouts
can subsequently be used to, for instance, train a multi-clip or GPT policy.

To remove this obstacle and facilitate the use of MoCap data in humanoid
control research, we introduce MoCapAct (Motion Capture with Actions, Fig. 1),
a dataset of high-quality MoCap-tracking policies for a MuJoCo-based (Todorov
et al., 2012) simulated humanoid as well as a collection of rollouts from
these expert policies. The policies from MoCapAct can track 3.5 hours of
recorded motion from CMU MoCap (CMU, 2003), one of the largest publicly
available MoCap datasets. We analyze the expert policies of MoCapAct and, to
illustrate MoCapAct’s usefulness for learning diverse motions, use the expert
rollouts to train a single hierarchical policy which is capable of tracking
_all_ of the considered MoCap clips. We then re-use the low-level component of
the policy to efficiently learn downstream tasks via reinforcement learning.
Finally, we use the dataset for generative motion completion by training a GPT
network (Karpathy, 2020) to produce a motion in the MuJoCo simulator given a
motion prompt.

##  2 Related Work

##### MoCap Data

Of the existing datasets featuring motion capture of humans, the largest and
most cited are CMU MoCap (CMU, 2003) and Human3.6M (Ionescu et al., 2013).
These datasets feature tens of hours of human motion capture arranged as a
collection of clips recorded at 30-120Hz. They demonstrate a wide range of
motions, including locomotion (e.g., walking, running, jumping, and turning),
physical activities (e.g., dancing, boxing, and gymnastics), and interactions
with other humans and objects.

##### MoCap Tracking via Reinforcement Learning

To make use of MoCap data for downstream tasks, much of prior work first
learns individual clip-tracking policies. Peng et al. (2018) and Merel et al.
(2019a, b, 2020) use reinforcement learning (RL) to learn the clip-tracking
policies, whereas Merel et al. (2017) use adversarial imitation learning. Upon
learning the tracking policies, there are a variety of ways to utilize them.
Peng et al. (2018) and Merel et al. (2017, 2019a) learn a skill-selecting
policy to dynamically choose a clip-tracking policy to achieve new tasks.
Merel et al. (2019b, 2020) instead opt for a distillation approach, whereby
they collect rollouts from the clip-tracking policies and then train a
hierarchical multi-clip policy via supervised learning on the rollouts. The
low-level policy is then re-used to aid in learning new high-level tasks.

Alternatively, large-scale RL may be used to learn a single policy that covers
the MoCap dataset. Hasenclever et al. (2020) use a distributed RL setup for
the MuJoCo simulator (Todorov et al., 2012), while Peng et al. (2022) use the
GPU-based Isaac simulator (Makoviychuk et al., 2021) to perform RL on a single
machine.

While some prior work has released source code to train individual clip-
tracking policies (Peng et al., 2018; Yuan and Kitani, 2020), their included
catalog of policies is small, and the resources needed to train per-clip
policies scale linearly with the number of MoCap clips. In the process of our
work, we found that we needed about _50 years_ of wall-clock time to train the
policies to track our MoCap corpus using a similar approach to Peng et al.
(2018).

##### Motion Completion

Outside of the constraints of a physics simulator, learning natural
completions of MoCap trajectories (i.e., producing a trajectory given a prompt
trajectory) is the subject of many research papers (Mourot et al., 2022),
typically motivated by the challenging and labor-intensive process of creating
realistic animations for video games and films. Prior work (Aksan et al.,
2021; Harvey et al., 2020; Kania et al., 2021; Mao et al., 2019; Tevet et al.,
2022; Wang et al., 2019) typically trains a model to replicate the kinematic
motion found in a MoCap dataset, which is then evaluated according to how well
the model can predict or synthesize motions given some initial prompt on held-
out trajectories.

The more difficult task of performing motion completion _within a physics
simulator_ is not widely studied. Yuan and Kitani (2020) jointly learn a
kinematic policy and a tracking policy, where the kinematic policy predicts
future kinematic poses given a recent history of observations and the tracking
policy outputs a low-level action to track the predicted poses.

##  3 The dm_control Humanoid Environment

Figure 2: The humanoid displaying a variety of motions from the CMU MoCap
dataset.

Our simulated humanoid of interest is the “CMU Humanoid” (Fig. 2) from the
dm_control package (Tunyasuvunakool et al., 2020), which contains 56 joints
and is designed to be similar to an average human body. The humanoid contains
a rich and customizable observation space, from proprioceptive observations
like joint positions and velocities, actuator states, and touch sensor
measurements to high-dimensional observations like images from an egocentric
camera. The action $a$ is the desired joint angles of the humanoid, which are
then converted to joint torques via some pre-defined PD controllers. The
humanoid operates in the MuJoCo simulator (Todorov et al., 2012).

The dm_control package contains a variety of tools for the humanoid. The
package comes with pre-defined tasks like navigation through an obstacle field
(Heess et al., 2017), maze navigation (Merel et al., 2019a), and soccer (Liu
et al., 2022), and a user may create custom tasks with the package’s API. The
dm_control package also integrates 3.5 hours of motion sequences from the CMU
Motion Capture Dataset (CMU, 2003), including clips of locomotion (standing,
walking, turning, running, jumping, etc.), acrobatics, and arm movements. Each
clip $C$ is a reference state sequence
$(\hat{s}^{C}_{0},\hat{s}^{C}_{1},\dots,\hat{s}^{C}_{T_{C}-1})$, where $T_{C}$
is the clip length and each $\hat{s}^{C}_{t}$ contains kinematic information
like joint angles, joint velocities, and humanoid pose.

As discussed in Section 2, training a control policy to work on all of the
included clips requires large-scale solutions. For example, Hasenclever et al.
(2020) rely on a distributed RL approach that uses about ten billion
environment interactions collected by 4000 parallel actor processes running
for multiple days. To our knowledge, there are no agents publicly available
that can track all the MoCap data within dm_control. We address this gap by
releasing a dataset of high-quality experts and their rollouts for the “CMU
Humanoid” in the dm_control package.

##  4 MoCapAct Dataset

The MoCapAct dataset (Fig. 1) consists of:

  * •

experts each trained to track an individual snippet from the MoCap dataset
(Section 4.1) and

  * •

HDF5 files containing rollouts from the aforementioned experts (Section 4.2).

We include documentation of the MoCapAct dataset in Appendix A.

###  4.1 Clip Snippet Experts

Our expert training scheme largely follows that of Merel et al. (2019a, b) and
Peng et al. (2018), which we now summarize.

##### Training

We split each clip in the MoCap dataset into 4–6 second snippets with 1-second
overlaps. With 836 clips in the MoCap dataset, this clip splitting results in
2589 snippets. For each clip snippet $c$, we train a time-indexed Gaussian
policy $\pi_{c}(a|s,t)$ to track the snippet. We use the same clip-tracking
reward function $r_{c}(s,t)$ as Hasenclever et al. (2020), which encourages
matching the MoCap clip’s joint angles and velocities, positions of various
body parts, and joint orientations. This reward function lies in the interval
$[0,1.4]$. To speed up training, we use the same early episode termination
condition as Hasenclever et al. (2020), which activates if the humanoid
deviates too far from the snippet. To help exploration, the initial state of
an episode is generated by randomly sampling a time step from the given
snippet. The Gaussian policy $\pi_{c}$ uses a mean parameterized by a neural
network as well as a fixed standard deviation of $0.1$ for each action to
induce robustness and to prepare for the noisy rollouts (Section 4.2). We use
the Stable-Baselines3 (Raffin et al., 2021) implementation of PPO (Schulman et
al., 2017) to train the experts. Our training took about _50 years_ of wall-
clock time. We give hyperparameters and training details in Section B.1.

##### Results

To account for the snippets having different lengths and for the episode
initialization scheme used in training, we report our evaluations in a length-
normalized fashion.111We point out that PPO uses the original unnormalized
reward for policy optimization. For a snippet $c$ (with length $T_{c}$) and
some policy $\pi$, recall that we initialize the humanoid at some randomly
chosen time step $t_{0}$ from $c$ and then generate the trajectory $\tau$ by
rolling out $\pi$ from $t_{0}$ until either the end of the snippet or early
termination. Let $R(\tau)$ and $L(\tau)$ denote the accumulated reward and the
length of the trajectory $\tau$, respectively. We define the normalized
episode reward and normalized episode length of $\tau$ as
$\frac{R(\tau)}{T_{c}-t_{0}}$ and $\frac{L(\tau)}{T_{c}-t_{0}}$, respectively.
One consequence of this definition is that trajectories that are terminated
early in a snippet yield smaller normalized episode rewards and lengths. Next,
we define the average normalized episode reward and average normalized episode
length of policy $\pi$ on snippet $c$ as
$\hat{R}_{c}(\pi)=\mathbb{E}_{t_{0}\sim
c}\mathbb{E}_{\tau\sim\pi|t_{0}}\mathopen{}\mathclose{{}\left[\frac{R(\tau)}{T_{c}-t_{0}}}\right]$
and $\hat{L}_{c}(\pi)=\mathbb{E}_{t_{0}\sim
c}\mathbb{E}_{\tau\sim\pi|t_{0}}\mathopen{}\mathclose{{}\left[\frac{L(\tau)}{T_{c}-t_{0}}}\right]$,
respectively. For example, if $\pi$ always successfully tracks some MoCap
snippet from any $t_{0}$ to the end of the snippet, $\pi$ has an average
normalized episode length of $1$ on snippet $c$.

Table 1: Snippet expert results on the MoCap snippets within dm_control. We disable the Gaussian noise for $\pi_{c}$ when computing these results. | Mean | Standard deviation | Median | Minimum | Maximum
---|---|---|---|---|---
Average normalized episode reward | $0.816$ | $0.153$ | $0.777$ | $0.217$ | $1.233$
Average normalized episode length | $0.997$ | $0.022$ | $1.000$ | $0.424$ | $1.000$
Figure 3: Clip expert results on the MoCap snippets within dm_control.

Overall, the clip experts reliably track the overwhelming majority of the
MoCap snippets (Table 1 and Fig. 3). Averaged over all the snippets, the
experts have a per-joint mean angle error of $0.062$ radians. We find that
$80\%$ of the trained experts have an average normalized episode length of at
least $0.999$. We also observe there is a bimodal structure to the reward
distribution in Fig. 3, which is due to many clips having artifacts like
jittery limbs and extremities clipping through the ground. These artifacts
limit the extent to which the humanoid can track the clip. Among the handful
of experts with very low reward (between $0.2$ and $0.5$), we find that the
corresponding clips are erroneously sped up, making them impossible to track
in the simulator.

Figure 4: Visualizations of clip experts. The top two rows show episodes
(first: walking, second: cartwheel) where the expert (bronze humanoid) closely
tracks the corresponding MoCap clip (grey humanoid). The bottom row shows a
clip where the expert and MoCap clip differ in behavior. The MoCap clip
demonstrates a 360-degree jump, whereas the expert jumps without spinning.

The experts produce motion that is generally indistinguishable from the MoCap
reference (Fig. 4), from simple walking behaviors seen in the top row to
highly coordinated motions like the cartwheel in the middle row. On some
clips, the expert deviates from the clip because the demonstrated motion is
too highly dynamic, such as the 360-degree jump in the bottom row. Instead,
the expert typically learns some other behavior that keeps the episode from
terminating early, which in this case is jumping without spinning. We also
point out that, in these failure modes, the humanoid still tracks some
portions of the reference, such as hand positions and orientations. Yuan and
Kitani (2020) rectify similar tracking issues by augmenting the action space
with external forces on certain parts of the humanoid body, but we do not
explore this avenue since the issue only affects a small number of clips. We
encourage the reader to visit the project website to see videos of the clip
experts.

###  4.2 Expert Rollouts

Following Merel et al. (2019b), we roll out the experts on their respective
snippets and collect data from the rollouts into a dataset $\mathcal{D}$. In
order to obtain a broad state coverage from the experts, we repeatedly roll
out the _stochastic_ experts (i.e., with Gaussian noise injected into the
actions) starting from different initial states. This injected noise helps the
dataset cover states that a policy learned by imitating the dataset would
visit, therefore mitigating the distribution shift issue for the learned
policy (Laskey et al., 2017; Merel et al., 2019b).

For each clip snippet $c$, we denote the corresponding expert policy as
${\pi_{c}(a|s,t)=\mathcal{N}(a;\mu_{c}(s,t),0.1^{2}I)}$, where $\mu_{c}(s,t)$
is the mean of the expert’s action distribution. We initialize the humanoid at
some point in the snippet $c$ (half of the time at the beginning of the
snippet and otherwise at some random point in the snippet). We then roll out
$\pi_{c}$ until either the end of the snippet or early termination using the
scheme from Section 4.1. At every time step $t$ in the rollout, we log the
humanoid state $s_{t}$, the target reference poses
$s_{t}^{\mathrm{ref}}=(\hat{s}_{t+1}^{c},\dots,\hat{s}_{t+5}^{c})$ from the
next five steps of the MoCap snippet, the expert’s sampled action $a_{t}$, the
expert’s mean action $\bar{a}_{t}=\mu_{c}(s_{t},t)$, the observed snippet
reward $r_{c}(s_{t},t)$, the estimated value $\hat{V}^{\pi_{c}}(s_{t})$, and
the estimated advantage $\hat{A}^{\pi_{c}}(s_{t},a_{t})$ into HDF5 files.

We release two versions of the rollout dataset:

  * •

a “large” $600$-gigabyte collection at 200 rollouts per snippet with a total
of 67 million environment transitions (corresponding to $620$ hours in the
simulator) and

  * •

a “small” $50$-gigabyte collection at 20 rollouts per snippet with a total of
5.5 million environment transitions (corresponding to $51$ hours in the
simulator).

In our application of MoCapAct (Section 5), we use the “large” version of the
dataset. We do observe, though, that the multi-clip policy results (Section
5.1) are similar when using either dataset.

##  5 Applications

(a) Multi-clip tracking policy.

(b) GPT policy.

Figure 5: Policies used in the applications.

We train two policies (Fig. 5) using our dataset:

  1. 1.

A hierarchical policy which can track all the MoCap snippets and be re-used
for learning new high-level tasks (Section 5.1).

  2. 2.

An autoregressive GPT model which generates motion from a given prompt
(Section 5.2).

###  5.1 Multi-Clip Tracking Policy

We first show the MoCapAct dataset can reproduce the results in Merel et al.
(2019b) by learning a single policy that tracks the entire MoCap dataset
within dm_control. Our policy architecture (Fig. 5(a)) follows the same
encoder-decoder scheme as Merel et al. (2019b), who introduce a “motor
intention” $z_{t}$ which acts as a low-dimensional embedding of the MoCap
reference $s_{t}^{\mathrm{ref}}$. The intention $z_{t}$ is then decoded into
an action $a_{t}$. In other words, the policy $\pi$ is factored into an
encoder $\pi_{\mathrm{enc}}$ and a decoder $\pi_{\mathrm{dec}}$. The encoder
$\pi_{\mathrm{enc}}(z_{t}|s_{t},s_{t}^{\mathrm{ref}},z_{t-1})$ compresses the
MoCap reference $s_{t}^{\mathrm{ref}}$ into an intention $z_{t}$ and may use
the current humanoid state $s_{t}$ and previous intention $z_{t-1}$ in
predicting the current intention. Furthermore, the encoder outputs an
intention which is _stochastic_ , which models ambiguity in the MoCap
reference and allows for the high-level behavior to be specified more
coarsely. The decoder $\pi_{\mathrm{dec}}(a_{t}|s_{t},z_{t})$ translates the
sampled intention $z_{t}$ into an action $a_{t}$ with the aid of the state
$s_{t}$ as an additional input.

####  5.1.1 Training

In our implementation, the encoder outputs the mean and diagonal covariance of
a Gaussian distribution over a 60-dimensional motor intention $z_{t}$. The
decoder outputs the mean of a Gaussian distribution over actions with a
standard deviation of $0.1$ for each action. In training, we maximize a
variant of the multi-step imitation learning objective from Merel et al.
(2019b):

$\mathbb{E}_{\vbox{\Let@\restore@math@cr\default@tag\halign{\hfil$\m@th\scriptstyle#$&$\m@th\scriptstyle{}#$\hfil\cr(s_{1:T},s_{1:T}^{\mathrm{ref}},\bar{a}_{1:T},c)&\sim\mathcal{D},\\\
z_{0:T}&\sim\pi_\mathrm{enc}\crcr}}}\mathopen{}\mathclose{{}\left[\sum_{t=1}^{T}\mathopen{}\mathclose{{}\left[w_{c}(s_{t},\bar{a}_{t})\log\pi_{\mathrm{dec}}(\bar{a}_{t}|s_{t},z_{t})-\beta\,\mathrm{KL}(\pi_{\mathrm{enc}}(z_{t}|s_{t},s_{t}^{\mathrm{ref}},z_{t-1})\,\|\,p(z_{t}|z_{t-1}))}\right]}\right],$

where $T$ is the sequence length, $w_{c}$ is a clip-dependent data-weighting
function, $p(z_{t}|z_{t-1})$ is an autoregressive prior, and $\beta$ is a
hyperparameter.

The weighting function $w_{c}$ allows for some data points to be considered
more heavily, which may be useful given the spectrum of expert performance.
Letting $\lambda$ be a hyperparameter, we consider the following four
weighting schemes:

  * •

Behavioral cloning (BC): $w_{c}(s,a)=1$. This scheme is commonly used in
imitation learning and treats every data point equally.

  * •

Clip-weighted regression (CWR):
$w_{c}(s,a)=\exp(\hat{R}_{c}(\pi_{c})/\lambda)$. This scheme upweights data
from snippets where the experts have higher average normalized rewards.

  * •

Advantage-weighted regression (AWR) (Peng et al., 2019b):
$w_{c}(s,a)=\exp(\hat{A}^{\pi_{c}}(s,a)/\lambda)$. This scheme upweights
actions that perform better than the expert’s average return.

  * •

Reward-weighted regression (RWR) (Peters and Schaal, 2007):
${w_{c}(s,a)=\exp(\hat{Q}^{\pi_{c}}(s,a)/\lambda)}$, where
$\hat{Q}^{\pi_{c}}(s,a)=\hat{V}^{\pi_{c}}(s)+\hat{A}^{\pi_{c}}(s,a)$. This
scheme upweights state-actions which have higher returns, which typically
happens with good experts at earlier time steps in the corresponding snippet.

The KL divergence term encourages the decoder to follow a simple random walk.
In this case, the prior has the form
$p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\alpha z_{t-1},\sigma^{2}I)$, where
$\alpha\in[0,1]$ is a hyperparameter and ${\sigma=\sqrt{1-\alpha^{2}}}$. This
prior in turn encourages the marginals to be a spherical Gaussian, i.e.,
${p(z_{t})=\mathcal{N}(z_{t};0,I)}$. Furthermore, the regularization
introduces a bottleneck (Alemi et al., 2017) that limits the information the
intention $z_{t}$ can provide about the state $s_{t}$ and MoCap reference
$s_{t}^{\mathrm{ref}}$. This forces the encoder to only encode high-level
information about the reference (e.g., direction of motion of leg) while
excluding fine-grained details (e.g., precise velocity of each joint in leg).

In our experiments, we found that the training takes about three hours on a
single-GPU machine. More training details are available in Section B.2.

##### Results

All four regression approaches yield broadly good results (Table 2), achieving
$80\%$ to $84\%$ of the experts’ performance on the MoCap dataset (cf. Table
1). We also see that every weighted regression scheme gives some improvement
over the unweighted approach. AWR only gives $1\%$ improvement over BC, likely
because the experts are already near-optimal and the dataset lacks sufficient
state-action coverage to reliably contain advantageous actions. CWR gives a
$3\%$ improvement over BC, which arises from the objective placing more
emphasis on data coming from high-reward clips. Finally, RWR gives a $5\%$
improvement over BC, which comes from increased weight on earlier time steps
in high-reward clips. This is a sensible weighting scheme since executing a
skill requires taking correct actions at earlier time steps before completing
the skill at later time steps. As a point of comparison to prior work, the
RWR-trained policy achieves an average reward-per-step (i.e.,
$\mathbb{E}[R(\tau)/L(\tau)]$) of $0.67$ on the “Locomotion” subset of the
MoCap data, which is $96\%$ of the reward-per-step achieved by the large-scale
RL approach of Hasenclever et al. (2020). We also find that the RWR-trained
policy has a per-joint mean angle error of $0.085$ radians.

Table 2: Multi-clip results on the MoCap snippets, showing the mean and standard deviation over three seeds. For evaluation, we disable the Gaussian noise for $\pi_{\mathrm{dec}}$ but keep the stochasticity for $\pi_{\mathrm{enc}}$. | BC | CWR | AWR | RWR
---|---|---|---|---
Avg. normalized episode reward | $0.654\pm 0.005$ | $0.671\pm 0.003$ | $0.661\pm 0.003$ | $\mathbf{0.688\boldsymbol{\pm}0.002}$
Avg. normalized episode length | $0.855\pm 0.004$ | $0.858\pm 0.003$ | $0.861\pm 0.001$ | $\mathbf{0.868\boldsymbol{\pm}0.002}$

(a) Multi-clip policy’s performance on training and validation sets.

(b) Performance of multi-clip policy relative to expert policies.

Figure 6: Performance of RWR-trained multi-clip policy.

To assess the generalization of the multi-clip policy, we train the policy
using RWR on a subset of MoCapAct covering $90\%$ of the MoCap clips. We treat
the remaining $10\%$ of the clips as a validation set when evaluating the
multi-clip policy. We find that the multi-clip policy performs similarly on
the training set and validation set clips (Fig. 6(a)), with the validation set
performance even being slightly higher than the training set performance (mean
of $0.699$ vs. $0.674$). This is likely because the clips in the validation
set are slightly easier.

To account for the reward scale of the clips, we also report the multi-clip
policy’s performance relative to the clip experts (Fig. 6(b)). Again, the
training set and validation set relative performances are very similar, though
now the multi-clip policy has a small relative performance drop in the
validation set (mean of $0.797$ vs. $0.815$). We also observe that the multi-
clip policy outperforms the clip experts on $13\%$ of the MoCap snippets.

We encourage the reader to visit the project website to see videos of the
multi-clip policy.

####  5.1.2 Re-Use for Reinforcement Learning

We re-use the decoder $\pi_{\mathrm{dec}}$ from an RWR-trained multi-clip
policy for reinforcement learning to constrain the behaviors of the humanoid
and speed up learning. In particular, we study two tasks that require adept
locomotion skills:

  1. 1.

A sparse-reward go-to-target task where the agent receives a non-zero reward
only if the humanoid is sufficiently close to the target. The target relocates
once the humanoid stands on it for a few time steps.

  2. 2.

A velocity control task where shaped rewards encourage the humanoid to go at a
given speed in a given direction. The desired speed and direction change
randomly every few seconds.

We treat $\pi_{\mathrm{dec}}$ as part of the environment and the motor
intention $z$ as the action. We thus learn a new high-level policy
$\pi_{\mathrm{task}}(z|s)$ that steers the low-level policy to maximize the
task reward.

Given the tasks are locomotion-driven, we also consider a more specialized
decoder with a 20-dimensional intention which is trained solely on locomotion
clips from MoCapAct (called the “Locomotion” subset) to see if further
restricting the learned skills offers any more speedup. As a baseline, we also
perform RL without a low-level policy.

Table 3: Returns for the transfer tasks, showing the mean and standard deviation over five seeds. | General low-level policy | Locomotion low-level policy | No low-level policy
---|---|---|---
Go-to-target | $96.3\pm 2.8$ | $66.1\pm 32.8$ | $7.5\pm 1.1$
Velocity control | $1074\pm 55$ | $884\pm 81$ | $1157\pm 89$
Figure 7: Training curves for transfer tasks. All experiments use five seeds.

We find that re-using a low-level policy drastically speeds up learning and
usually produces higher returns (Table 3 and Fig. 7). For the go-to-target
task, the locomotion-based low-level policy induces faster training than the
more general low-level policy, though it does converge to lesser performance
and on one out of five seeds converges to a very low reward. This performance
gap is likely a combination of the lower dimensionality of the locomotion
policy restricting the degree of control by the high-level policy and the
“Locomotion” subset excluding some useful behaviors, a result also found by
Hasenclever et al. (2020). The baseline without the low-level policy fails to
learn the task. For the velocity control task, the locomotion-based policy
induces slightly faster learning than the general policy but again results in
lower reward. The baseline without the low-level policy learns the task more
slowly, though it does achieve high reward eventually.

In both tasks, we find that including a pretrained low-level policy produces
much more realistic gaits. The humanoid efficiently runs from target to target
in the go-to-target task and smoothly changes speeds and direction of motion
in the velocity control task. On the other hand, the baseline approach
produces incredibly unusual motions. In the go-to-target task, the humanoid
convulses and contorts itself towards the first target before falling to the
ground. In the velocity control task, the humanoid rapidly taps the feet to
propel the body at the desired velocity. We encourage the reader to visit the
project website to see videos of the RL results.

###  5.2 Motion Completion with GPT

We also train a GPT model (Radford et al., 2019) based on the minGPT
implementation (Karpathy, 2020) to generate motion. Starting with a motion
prompt (sequence of humanoid observations generated by a clip expert), the GPT
policy (Fig. 5(b)) autoregressively predicts actions from the context of
recent humanoid observations. We train the GPT by sampling 32-step sequences
(corresponding to $1$ second of motion) of humanoid observations
$s_{(t-31):t}$ and expert’s mean actions $\bar{a}_{(t-31):t}$ from the
MoCapAct dataset $\mathcal{D}$ and performing supervised learning using the
mean squared error loss on the predicted action sequence.

To roll out the policy, we provide the GPT policy with a 32-step prompt from a
clip expert and let GPT roll out thereafter. The episode either terminates
after 500 steps (about $15$ seconds) or if a body part other than the feet
touches the ground (e.g., humanoid falling over). On many clip snippets, the
GPT model is able to control the humanoid for several seconds past the end of
the prompt (Table 4 and Fig. 8(a)), with similar lengths on the training set
and a held-out validation set of prompts. We also observe that on many clips
the GPT can control the humanoid for several times longer than the length of
the corresponding clip snippet (Table 4 and Fig. 8(b)).

Table 4: Motion completion statistics on the MoCap snippets. | Mean | Standard deviation | Median | Minimum | Maximum
---|---|---|---|---|---
Episode length (seconds) | $5.47$ | $3.47$ | $4.38$ | $0.23$ | $15.00$
Relative episode length | $1.15$ | $0.94$ | $0.87$ | $0.05$ | $7.63$

(a) Absolute episode lengths of GPT.

(b) Relative episode lengths of GPT.

Figure 8: Episode lengths of GPT on MoCap snippets.

(a) Locomotion clip where behaviors align.

(b) Locomotion clip where behaviors differ.

(c) Non-locomotion clip where behaviors differ.

Figure 9: PCA projections of action sequences of length $32$ from experts and
GPT.

To visualize the rollouts, we perform principal component analysis (PCA) on
action sequences of length $32$ applied by GPT and the snippet expert used to
generate the motion prompt (Fig. 9). Qualitatively, we find that GPT usually
repeats motions demonstrated in locomotion prompts, such as the running motion
corresponding to Fig. 9(a). Occasionally, GPT will produce a different motion
than the underlying clip, usually due to ambiguity in the prompt. For example,
in Fig. 9(b), GPT has the humanoid repeatedly step backwards, whereas the
expert takes repeated side steps. In Fig. 9(c), the GPT policy performs an
entirely different arm-waving motion than that of the expert. We encourage the
reader to visit the project website to see videos of GPT motion completion.

##  6 Discussion

We presented a dataset of high-quality MoCap-tracking policies and their
rollouts for the dm_control humanoid environment. From these rollouts, we
trained multi-clip tracking policies that can be re-used for new high-level
tasks and GPT policies which can generate humanoid motion when given a prompt.
We have open sourced our dataset, models, and code under permissive licenses.

We do point out that our models and data are only applicable to the dm_control
environment, which uses MuJoCo as the backend simulator. We also point out
that all considered clips only occur on flat ground and do not include any
human or object interaction. Though this seems to limit the environments and
tasks where this dataset is applicable, the dm_control package
(Tunyasuvunakool et al., 2020) has tools to change the terrain, add more MoCap
clips, and add objects (e.g., balls) to the environment. Indeed, prior work
has used custom clips which include extra objects (Merel et al., 2020; Liu et
al., 2022). While the dataset and domain may raise concerns on automation, we
believe the considered simulated domain is limited enough to not be of ethical
import.

This work significantly lowers the barrier of entry for simulated humanoid
control, which promises to be a rich field for studying multi-task learning
and motor intelligence. In addition to the showcases presented, we believe
this dataset can be used in training other policy architectures like decision
and trajectory transformers (Chen et al., 2021; Janner et al., 2021) or in
setups like offline reinforcement learning (Fu et al., 2020; Levine et al.,
2020) as the dataset allows research groups to bypass the time- and energy-
consuming process of learning low-level motor skills from MoCap data.

## Acknowledgments and Disclosure of Funding

We thank Leonard Hasenclever for providing helpful information used in
DeepMind’s prior work on humanoid control. We also thank Byron Boots for
suggesting to use PCA projections for visualization. Finally, we thank the
reviewers for their invaluable feedback.

The data used in this project was obtained from mocap.cs.cmu.edu. The database
was created with funding from NSF EIA-0196217.

## References

  * Aksan et al. (2021) E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges.  A Spatio-Temporal Transformer for 3D Human Motion Prediction.  In _2021 International Conference on 3D Vision (3DV)_ , pages 565–574. IEEE, 2021.
  * Alemi et al. (2017) A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy.  Deep Variational Information Bottleneck.  In _International Conference on Learning Representations_ , 2017.
  * Bohez et al. (2022) S. Bohez, S. Tunyasuvunakool, P. Brakel, F. Sadeghi, L. Hasenclever, Y. Tassa, E. Parisotto, J. Humplik, T. Haarnoja, R. Hafner, M. Wulfmeier, M. Neunert, B. Moran, N. Siegel, A. Huber, F. Romano, N. Batchelor, F. Casarini, J. Merel, R. Hadsell, and N. Heess.  Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors.  _arXiv preprint arXiv:2203.17138_ , 2022.
  * Chen et al. (2021) L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch.  Decision Transformer: Reinforcement Learning via Sequence Modeling.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Chentanez et al. (2018) N. Chentanez, M. Müller, M. Macklin, V. Makoviychuk, and S. Jeschke.  Physics-Based Motion Capture Imitation With Deep Reinforcement Learning.  In _Proceedings of the 11th Annual International Conference on Motion, Interaction, and Games_ , pages 1–10, 2018.
  * CMU (2003) CMU.  Carnegie Mellon University Graphics Lab Motion Capture Database.  http://mocap.cs.cmu.edu, 2003.
  * Falcon (2019) W. Falcon.  PyTorch Lightning.  https://github.com/PyTorchLightning/pytorch-lightning, 2019.
  * Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4RL: Datasets for Deep Data-Driven Reinforcement Learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Harvey et al. (2020) F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal.  Robust Motion In-Betweening.  _ACM Transactions on Graphics (TOG)_ , 39(4):60–1, 2020.
  * Hasenclever et al. (2020) L. Hasenclever, F. Pardo, R. Hadsell, N. Heess, and J. Merel.  CoMic: Complementary Task Learning & Mimicry for Reusable Skills.  In _International Conference on Machine Learning_ , pages 4105–4115. PMLR, 2020.
  * Heess et al. (2017) N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver.  Emergence of Locomotion Behaviours in Rich Environments.  _arXiv preprint arXiv:1707.02286_ , 2017.
  * Ionescu et al. (2013) C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.  Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1325–1339, 2013.
  * Janner et al. (2021) M. Janner, Q. Li, and S. Levine.  Offline Reinforcement Learning as One Big Sequence Modeling Problem.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Kania et al. (2021) K. Kania, M. Kowalski, and T. Trzciński.  TrajeVAE: Controllable Human Motion Generation from Trajectories.  _arXiv preprint arXiv:2104.00351_ , 2021.
  * Karpathy (2020) A. Karpathy.  minGPT.  https://github.com/karpathy/minGPT, 2020.
  * Kingma and Ba (2015) D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  In _International Conference on Learning Representations_ , 2015.
  * Laskey et al. (2017) M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg.  DART: Noise Injection for Robust Imitation Learning.  In _Conference on Robot Learning_ , pages 143–156. PMLR, 2017.
  * Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Liu et al. (2022) S. Liu, G. Lever, Z. Wang, J. Merel, S. M. A. Eslami, D. Hennes, W. M. Czarnecki, Y. Tassa, S. Omidshafiei, A. Abdolmaleki, N. Y. Siegel, L. Hasenclever, L. Marris, S. Tunyasuvunakool, H. F. Song, M. Wulfmeier, P. Muller, T. Haarnoja, B. D. Tracey, K. Tuyls, T. Graepel, and N. Heess.  From Motor Control to Team Play in Simulated Humanoid Football.  _Science Robotics_ , 7(69), 2022.
  * Makoviychuk et al. (2021) V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State.  Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.  In _Neural Information Processing Systems Datasets and Benchmarks Track_ , 2021.
  * Mao et al. (2019) W. Mao, M. Liu, M. Salzmann, and H. Li.  Learning Trajectory Dependencies for Human Motion Prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9489–9497, 2019.
  * Merel et al. (2017) J. Merel, Y. Tassa, D. TB, S. Srinivasan, J. Lemmon, Z. Wang, G. Wayne, and N. Heess.  Learning Human Behaviors from Motion Capture by Adversarial Imitation.  _arXiv preprint arXiv:1707.02201_ , 2017.
  * Merel et al. (2019a) J. Merel, A. Ahuja, V. Pham, S. Tunyasuvunakool, S. Liu, D. Tirumala, N. Heess, and G. Wayne.  Hierarchical Visuomotor Control of Humanoids.  In _International Conference on Learning Representations_ , 2019a.
  * Merel et al. (2019b) J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess.  Neural Probabilistic Motor Primitives for Humanoid Control.  In _International Conference on Learning Representations_ , 2019b.
  * Merel et al. (2020) J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess.  Catch & Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks.  _ACM Transactions on Graphics (TOG)_ , 39(4):39–1, 2020.
  * Mourot et al. (2022) L. Mourot, L. Hoyet, F. Le Clerc, F. Schnitzler, and P. Hellier.  A Survey on Deep Learning for Skeleton-Based Human Animation.  In _Computer Graphics Forum_ , volume 41, pages 122–157. Wiley Online Library, 2022.
  * Peng et al. (2018) X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne.  DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–14, 2018.
  * Peng et al. (2019a) X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine.  MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies.  _Advances in Neural Information Processing Systems_ , 32, 2019a.
  * Peng et al. (2019b) X. B. Peng, A. Kumar, G. Zhang, and S. Levine.  Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning.  _arXiv preprint arXiv:1910.00177_ , 2019b.
  * Peng et al. (2022) X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler.  ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters.  _ACM Transactions On Graphics (TOG)_ , 41(4):1–17, 2022.
  * Peters and Schaal (2007) J. Peters and S. Schaal.  Reinforcement Learning by Reward-Weighted Regression for Operational Space Control.  In _Proceedings of the 24th International Conference on Machine Learning_ , pages 745–750, 2007.
  * Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.  Language Models Are Unsupervised Multitask Learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Raffin et al. (2021) A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann.  Stable-Baselines3: Reliable Reinforcement Learning Implementations.  _Journal of Machine Learning Research_ , 2021.
  * Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal Policy Optimization Algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Tevet et al. (2022) G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or.  MotionCLIP: Exposing Human Motion Generation to CLIP Space.  In _European Conference on Computer Vision_. Springer, 2022.
  * Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa.  MuJoCo: A Physics Engine for Model-Based Control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.
  * Tunyasuvunakool et al. (2020) S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa.  dm_control: Software and Tasks for Continuous Control.  _Software Impacts_ , 6:100022, 2020.
  * Wang et al. (2019) B. Wang, E. Adeli, H.-k. Chiu, D.-A. Huang, and J. C. Niebles.  Imitation Learning for Human Pose Prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7124–7133, 2019.
  * Wang et al. (2017) Z. Wang, J. S. Merel, S. E. Reed, N. de Freitas, G. Wayne, and N. Heess.  Robust Imitation of Diverse Behaviors.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Yuan and Kitani (2020) Y. Yuan and K. Kitani.  Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis.  _Advances in Neural Information Processing Systems_ , 33:21763–21774, 2020.

This appendix provides the following sections:

  * •

documentation of the dataset (Appendix A),

  * •

training details (Appendix B), and

  * •

extra results (Appendix C).

##  Appendix A Dataset Documentation

###  A.1 Clip Snippet Experts

We signify a clip snippet expert by the snippet it is tracking. We denote a
snippet by the clip ID, its start step, and its end step. For example,
CMU_006_12-151-336 is the snippet corresponding to the clip CMU_006_12 with
start step $151$ and end step $336$. Taking CMU_006_12-151-336 as an example
expert, the file hierarchy for the snippet expert is:
.1 CMU_006_12-151-336. .2 clip_info.jsonContains clip ID, start step, and end
step.. .2 eval_rsi/model. .3 best_model.zipContains policy parameters and
hyperparameters.. .3 vecnormalize.pklUsed to get normalizer for observation
and reward.. The expert policy can be loaded using Stable-Baselines3’s
functionality.

###  A.2 Expert Rollouts

The expert rollouts consist of a collection of HDF5 files, one file per clip.
An HDF5 file contains expert rollouts for each constituent snippet as well as
miscellaneous information and statistics. To facilitate efficient loading of
the observations, we concatenate all the proprioceptive observations (joint
angles, joint velocities, actuator activations, etc.) from an episode into a
single numerical array and provide indices for the constituent observations in
the observable_indices group.

Taking CMU_009_12.hdf5 (which contains three snippets) as an example, we have
the following HDF5 hierarchy:
.1 CMU_009_12.hdf5. .2 n_rsi_rollouts$R$, number of rollouts from random time
steps in snippet.. .2 n_start_rollouts$S$, number of rollouts from start of
snippet.. .2 ref_stepsIndices of MoCap reference relative to current time
step. Here, $(1,2,3,4,5)$.. .2 observable_indices. .3 walker. .4
actuator_activation$(0,1,\ldots,54,55)$. .4
appendages_pos$(56,57,\ldots,69,70)$. .4 body_height$(71)$. .4 $\vdots$. .4
world_zaxis$(2865,2866,2867)$
. .2 statsStatistics computed over the entire dataset.. .3 act_meanMean of the
experts’ sampled actions.. .3 act_varVariance of the experts’ sampled
actions.. .3 mean_act_meanMean of the experts’ mean actions.. .3
mean_act_varVariance of the experts’ mean actions.. .3 proprio_meanMean of the
proprioceptive observations.. .3 proprio_varVariance of the proprioceptive
observations.. .3 count Number of observations in dataset.
. .2 CMU_009_12-0-198Rollouts for the snippet CMU_009_12-0-198.. .2
CMU_009_12-165-363Rollouts for the snippet CMU_009_12-165-363.. .2
CMU_009_12-330-529Rollouts for the snippet CMU_009_12-330-529..

Each snippet group contains $R+S$ rollouts. The first $S$ episodes correspond
to episodes intialized from the start of the snippet and the last $R$ episodes
to episodes initialized at random points in the snippet. We now uncollapse the
CMU_009_12-0-198 group within the HDF5 file to reveal the rollout structure:
.1 CMU_009_12-0-198. .2 early_termination$(R+S)$-boolean array indicating
which episodes terminated early.
. .2 rsi_metricsMetrics for episodes that initialize at random points in
snippet.. .3 episode_returns$R$-array of episode returns.. .3
episode_lengths$R$-array of episode lengths.. .3 norm_episode_returns$R$-array
of normalized episode rewards.. .3 norm_episode_lengths$R$-array of normalized
episode lengths.
. .2 start_metricsMetrics for episodes that initialize at start in snippet.
. .2 0First episode, of length $T$.. .3 observations. .4
proprioceptive$(T+1)$-array of proprioceptive observations.. .4
walker/body_camera$(T+1)$-array of images from body camera (not included).. .4
walker/egocentric_camera$(T+1)$-array of images from egocentric camera (not
included).
. .3 actions$T$-array of sampled actions executed in environment.. .3
mean_actions$T$-array of corresponding mean actions.. .3 rewards$T$-array of
rewards from environment.. .3 values$T$-array computed using the policy’s
value network.. .3 advantages$T$-array computed using generalized advantage
estimation.
. .2 1Second episode.. .2 $\vdots$. .2 $R+S-1$$(R+S)^{\mathrm{th}}$ episode..
To keep the dataset size manageable, we do _not_ include image observations in
the dataset. We do provide code to log them when rolling out the experts for
generating the dataset.

###  A.3 Hosting Plan

The link to the dataset can be found on the project website. We provide a
“large” rollout dataset where $R=S=100$ with size $600~{}\mathrm{GB}$ and a
“small” rollout dataset where $R=S=10$ with size $50~{}\mathrm{GB}$. The
dataset website also includes the policies we trained in Section 5, i.e., the
multi-clip tracking policies, RL-trained task policies, and the GPT policy. We
also provide a Python script to download individual experts and rollouts from
the dataset.

##  Appendix B Training Details

###  B.1 Clip Snippet Experts

####  B.1.1 MoCap Snippets

(a) Lengths of the MoCap clips.

(b) Lengths of the snippets generated from the clips.

Figure 10: Lengths of clips and snippets.

The MoCap dataset has a wide spread in clip length (Fig. 10(a)), with the
longest clip being $6371$ time steps ($191$ seconds). Training clip experts to
track long clips is potentially slow and laborious, so we follow Merel et al.
[2019b] by dividing clips longer than $210$ time steps ($6.3$ seconds) into
short snippets. In particular, we divide the clip into uniformly-sized
snippets with an overlap of $33$ time steps ($1$ second) such that the longest
snippet has at most $210$ time steps. This yields a snippet dataset with a
much tighter range of snippet lengths (Fig. 10(b)). We do _not_ divide the
clips from the “Get Up” subset of the MoCap dataset since they contain
involved motions of getting up from the ground.

####  B.1.2 Expert Training Details

Table 5: Hyperparameters for clip snippet expert training. Total environment steps |  $150$ million
---|---
Environment steps per policy update | $8192$
PPO epochs | $10$
PPO minibatch size | $512$
PPO clipping parameter $\varepsilon$ | $0.25$
GAE parameter $\lambda$ | $0.95$
Discount factor $\gamma$ | $0.95$
$\ell_{2}$ gradient norm clipping value | $1$
Adam step size |  $110-5$ for first $50\mathrm{M}$ env. steps $610-6$ for next $50\mathrm{M}$ env. steps $310-6$ for last $50\mathrm{M}$ env. steps

We use the Stable-Baselines3 [Raffin et al., 2021] implementation of PPO
[Schulman et al., 2017] to optimize each expert. Each expert is a neural
network with three hidden layers, 1024 neurons in each hidden layer, and the
$\tanh$ activation. At the start of each episode, we randomly select a time
step from the corresponding clip snippet (excluding the last 10 time steps
from the snippet) and initialize the humanoid to match the clip features at
the corresponding step. We evaluate the policy every 1 million environment
steps using 1000 episodes under the same initialization scheme (but now
excluding the last 30 time steps of the snippet) and the same action noise of
$0.1$ as for training rollouts. We also end the training if the average
normalized episode length is at least $0.98$ and the average normalized
episode reward does not improve by more than $1\%$ from the current best
reward after 10 million environment steps. We normalize the observation and
reward using running statistics from the environment. We give the other
relevant hyperparameters in Table 5. For details of the reward function and
early termination of an episode, we refer the reader to the appendix of
Hasenclever et al. [2020].

We ran the training on a mix of Azure Standard_H8 (8 CPUs), Standard_H16 (16
CPUs), Standard_NC6s_v2 (6 CPUs and 1 P100 GPU), and Standard_ND6s (6 CPUs and
1 P40 GPU) VMs.

The observables for the clip expert are: joints_pos, joints_vel,
sensors_velocimeter, sensors_gyro, end_effectors_pos, world_zaxis,
actuator_activation, sensors_touch, sensors_torque, time_in_clip.

###  B.2 Multi-Clip Tracking Policy

Table 6: Hyperparameters for multi-clip tracking policy training. Adam step size | $510-4$
---|---
Minibatch sequence length $T$ | $30$
Minibatch size | $256$
$\ell_{2}$ gradient norm clipping value | $1$
KL divergence weight $\beta$ | $0.1$
Autoregressive parameter $\alpha$ | $0$
Weighting temperature $\lambda$ |  CWR: $0.2$ AWR: $8$ RWR: $4$

We train the multi-clip policy
$\pi(a_{t},z_{t}|s_{t},s_{t}^{\mathrm{ref}},z_{t-1})=\pi_{\mathrm{enc}}(z_{t}|s_{t},s_{t}^{\mathrm{ref}},z_{t-1})\pi_{\mathrm{dec}}(a_{t}|s_{t},z_{t})$
by optimizing the following imitation objective:

$\mathbb{E}_{\vbox{\Let@\restore@math@cr\default@tag\halign{\hfil$\m@th\scriptstyle#$&$\m@th\scriptstyle{}#$\hfil\cr(s_{1:T},s_{1:T}^{\mathrm{ref}},\bar{a}_{1:T},c)&\sim\mathcal{D},\\\
z_{0:T}&\sim\pi_\mathrm{enc}\crcr}}}\mathopen{}\mathclose{{}\left[\sum_{t=1}^{T}\mathopen{}\mathclose{{}\left[w_{c}(s_{t},\bar{a}_{t})\log\pi_{\mathrm{dec}}(\bar{a}_{t}|s_{t},z_{t})-\beta\,\mathrm{KL}(\pi_{\mathrm{enc}}(z_{t}|s_{t},s_{t}^{\mathrm{ref}},z_{t-1})\,\|\,p(z_{t}|z_{t-1}))}\right]}\right],$

where $p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\alpha z_{t-1},(1-\alpha^{2})I)$ for
some $\alpha\in[0,1]$. We do this (for each data point in a minibatch) by
sampling $z_{0}\sim\mathcal{N}(0,I)$, sampling a $T$-step data sequence (of
humanoid states $s_{1:T}$, MoCap references $s_{1:T}^{\mathrm{ref}}$, and
expert’s mean actions $\bar{a}_{1:T}$) from the dataset $\mathcal{D}$,
unrolling the recurrent policy through the sampled sequence, performing
backpropagation through time on the objective function, and finally updating
the network using the Adam optimizer [Kingma and Ba, 2015]. To speed up
training, we normalize the humanoid state $s_{t}$ and MoCap reference
$s_{t}^{\mathrm{ref}}$ using the corresponding mean and standard deviation
computed over the entire dataset. For the weighted schemes, we multiply the
weight $w_{c}$ by a constant that ensures the average data weight is $1$ so
that the KL regularization term maintains the same relative weight. For all
schemes, we also sample data from shorter clips at a higher rate to ensure the
rollout data from the clips is uniformly even. This gives about $1\%$
improvement in policy evaluation compared to vanilla sampling.

We use PyTorch Lightning [Falcon, 2019] to train the multi-clip policy. The
encoder and decoder are both neural networks with $1024$ neurons per hidden
layer and use layer norm and the $\mathrm{ELU}$ activation. The encoder has
two hidden layers, while the decoder has three hidden layers. We ran the
training on Azure Standard_ND24s VMs, each equipped with 24 CPUs and 4 P40
GPUs. We periodically evaluate the multi-clip policy by running 1000 episodes
on the set of MoCap snippets following the same reference state initialization
scheme as in the rest of the paper. We found we only need to train the policy
for about $50\,000$ steps (about $10\%$ of an epoch) before plateauing on
policy evaluation (Section C.2.1). We give the other relevant hyperparameters
in Table 6.

The observables for the policy are:

  * •

Encoder: joints_pos, joints_vel, sensors_velocimeter, sensors_gyro,
end_effectors_pos, world_zaxis, actuator_activation, sensors_touch,
sensors_torque, body_height, reference_rel_bodies_pos_local,
reference_rel_bodies_quats

  * •

Decoder: joints_pos, joints_vel, sensors_velocimeter, sensors_gyro,
end_effectors_pos, world_zaxis, actuator_activation, sensors_touch,
sensors_torque

###  B.3 Transfer for Reinforcement Learning

####  B.3.1 Go-to-Target Task

This task matches that of Hasenclever et al. [2020], which we refer the reader
to for details.

####  B.3.2 Velocity Control

In this task, a target speed $s^{*}\in[0,4.5]$ and direction
$\psi^{*}\in[0,2\pi)$ are randomly sampled every $10$ seconds. Defining the
target velocity as $v_{t}^{*}=(s^{*}\cos\psi^{*},s^{*}\sin\psi^{*})$ and the
humanoid’s current velocity as $v_{t}$, the reward is defined as the product
of a speed factor and direction factor:

$r_{t}=\exp\mathopen{}\mathclose{{}\left(-\mathopen{}\mathclose{{}\left(\frac{\|v_{t}\|-\|v_{t}^{*}\|}{\eta}}\right)^{2}}\right)\mathopen{}\mathclose{{}\left(\frac{1+\mathrm{score}(v_{t},v_{t}^{*})}{2}}\right)^{k},$

where $\mathrm{score}(v_{t},v_{t}^{*})=v_{t}\cdot
v_{t}^{*}/\|v_{t}\|\|v_{t}^{*}\|$ gives the cosine of the angle between the
two velocity vectors. In our experiments, we set $\eta=0.75$ and $k=7$. We
also experimented with the velocity error reward used by Bohez et al. [2022]
but found that our reward was easier to optimize. We terminate the episode
either after 2000 time steps ($60$ seconds) or if any body part other than the
feet touches the ground.

####  B.3.3 Hyperparameters

Table 7: Hyperparameters for RL transfer tasks. Total environment steps |  $150$ million
---|---
Environment steps per policy update | $16\,384$
PPO epochs | $10$
PPO minibatch size | $1024$
PPO clipping parameter $\varepsilon$ | $0.2$
KL divergence threshold for early stopping | $0.3$
Entropy bonus coefficient |  General low-level policy: $110-4$ Locomotion low-level policy: $110-3$ No low-level policy: $110-4$
GAE parameter $\lambda$ | $0.95$
Discount factor $\gamma$ | $0.99$
$\ell_{2}$ gradient norm clipping value | $1$
Adam step size | $510-5$
Number of actors | $32$
Initial standard deviation for task policy |  With low-level policy: $2.5$ Without low-level policy: $0.5$
Maximum per-element action magnitude for task policy |  With low-level policy: $3$ Without low-level policy: $1$

Like the snippet experts, we train the task policies using the Stable-
Baselines3 implementation of PPO. Each task policy is a neural network with
three hidden layers, 1024 neurons per hidden layer, and the $\tanh$
activation. We ran the training on Azure Standard_ND6s (6 CPUs and 1 NVIDIA
P40 GPU) VMs. We give other hyperparameters in Table 7.

###  B.4 Motion Completion with GPT

Table 8: Hyperparameters for GPT training. Adam step size | $310-6$
---|---
Minibatch size | $256$
$\ell_{2}$ gradient norm clipping value | $1$
Attention dropout probability | $0.1$
Embedding dropout probability | $0.1$
Residual dropout probability | $0.1$
Block size | $32$
Embedding size | $768$
Attention heads | $8$
Number of layers | $8$
Weight decay | $0.1$

We train a variant of minGPT [Karpathy, 2020] that we adapted to accept
continuous inputs and output continuous actions. This particular model has 57
million parameters and was trained with a context length of 32 time steps,
corresponding to roughly one second of motion. Similar to the multi-clip
policy (Section B.2), we sample state $s_{(t-31):t}$ and mean-action
$\bar{a}_{(t-31):t}$ sequences of length $32$ from the MoCapAct dataset
$\mathcal{D}$. To speed up training, we normalize the humanoid state $s_{t}$
using the corresponding mean and standard deviation computed over the entire
dataset. We use the mean squared error loss on the sequence of predicted
actions from the GPT. We trained GPT using PyTorch Lightning [Falcon, 2019] on
Azure Standard_NC24s_v3, each equipped with 24 CPUs and 4 V100 GPUs, for 2
million steps, corresponding to one week of wall-clock time. We give the other
relevant hyperparameters in Table 8.

The observables for the GPT policy are: joints_pos, joints_vel,
sensors_velocimeter, sensors_gyro, end_effectors_pos, world_zaxis,
actuator_activation, sensors_touch, sensors_torque, body_height. Importantly,
GPT is not given any reference data from the MoCap clip, so any motion
generation was done only on the basis of the historical context provided.

##  Appendix C More Results

###  C.1 Clip Snippet Experts

####  C.1.1 Training Curves

Figure 11: Snippet expert training curves on MoCap dataset.

We give the learning curves for the experts in Fig. 11. In particular, we plot
the quantiles $0,0.1,\ldots,0.9,1$ to visualize how the _distribution_ of
experts improves over the course of training. Overall, we see reliable
improvement of the experts with convergence at about 100 million environment
transitions.

####  C.1.2 Expert Performance vs. Snippet Length

Figure 12: Scatter plot of experts’ performance versus the snippet length.
Here, the Gaussian noise of the experts is disabled. The performance appears
to be independent of snippet length.

Here, we study whether longer snippets are “harder” to track by the expert.
Fig. 12 shows scatter plots of the experts’ normalized episode reward and
length as a function of the snippet length. Overall, the snippet length does
not appear to affect the experts’ performance as indicated by the fitted
curves being relatively flat.

####  C.1.3 Noisy Expert Evaluations

Table 9: Clip expert results on the MoCap snippets within dm_control using the stochastic $\pi_{c}$. | Mean | Standard deviation | Median | Minimum | Maximum
---|---|---|---|---|---
Average normalized episode reward | $0.689$ | $0.092$ | $0.690$ | $0.179$ | $0.876$
Average normalized episode length | $0.984$ | $0.029$ | $0.990$ | $0.403$ | $1.000$

(a) Episode rewards and lengths of the noisy experts.

(b) Performance ratio of noisy expert to deterministic expert.

Figure 13: Noisy expert results on the MoCap snippets within dm_control. The
noisy experts incur a small performance drop from their deterministic
counterparts.

Because the MoCapAct dataset is formed from noisy rollouts of the experts, it
is sensible to assess the performance of the experts when rolled out with
noise. Table 9 and Fig. 13(a) show that the experts still have strong
performance. We point out the noisy experts on average attain $85\%$ of the
performance of the deterministic experts (Fig. 13(b)).

Figure 14: Scatter plot of noisy experts’ performance versus the snippet
length. There is a minor decrease in performance as the snippet length
increases.

From the scatter plot of the noisy experts (Fig. 14), we see a minor decrease
in reward and episode length as the snippet gets longer. This is probably due
to longer snippets giving more time steps for the noise to destabilize the
humanoid.

###  C.2 Multi-Clip Tracking Policy

####  C.2.1 Training Curves

Figure 15: Multi-clip policy training curves on MoCap snippets.

Fig. 15 shows the reward curves for the four weighting schemes. Overall, the
reward plateaus after about $50\,000$ iterations for each scheme, and reward-
weighted regression performs markedly better than the other three schemes.

####  C.2.2 Autoregressive Parameter $\alpha$

Figure 16: Comparison of multi-clip policy’s performance when varying the
autoregressive parameter $\alpha$ for the prior distribution
$p(z_{t}|z_{t-1})$. Here, we use the RWR-weighting scheme. Performance is
broadly similar for both values of $\alpha$.

Merel et al. [2019b] found that using an autoregressive parameter of
$\alpha=0.95$ gave $50\%$ improvement in policy performance over $\alpha=0$.
Interestingly, in our experiments we found that the performance gap is much
smaller (Fig. 16), with $\alpha=0.95$ only giving $3\%$ improvement.
Accordingly, we set ${\alpha=0}$ for our experiments (corresponding to a
temporally independent prior of ${p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};0,I)}$)
so that we could better control the size of the intentions $z_{t}$ generated
by our reference encoder.

####  C.2.3 Scatter Plots on Snippets and Clips

Figure 17: Scatter plot of the multi-clip policy’s performance versus the
snippet length. Here, the Gaussian noise of the policy is disabled. Longer
snippets tend to result in lower episode lengths.

Fig. 17 shows the scatter plot of the multi-clip policy on all of the MoCap
snippets. Compared to the noisy experts (Section C.1.3), we see a more
noticeable decline in episode length on long snippets. Intuitively, this is
because longer snippets allow for more opportunities for the multi-clip policy
to make an episode-ending mistake. The normalized reward, on the other hand,
does not give any meaningful trends.

Figure 18: Scatter plot of the multi-clip policy’s performance versus the clip
length. Here, the Gaussian noise of the low-level policy is disabled. Longer
clips tend to result in lower episode rewards and lengths.

One appealing feature of the multi-clip policy is the ability to roll out the
policy on entire clips. This also allows us to discover whether the multi-clip
policy has learned to “stitch” together the overlapping snippets from the
dataset. Fig. 18 shows that while there are long clips that the policy can
reliably track, the overall trend is that longer clips result in lower reward
and episode length. Intuitively, many clips in the MoCap dataset correspond to
locomotion behaviors, which gives many opportunities for the multi-clip policy
to make episode-terminating mistakes. Usually, these mistakes correspond to
the humanoid legs colliding or one of the feet making bad contact with the
ground, both of which cause the humanoid to fall over. The fragility on longer
clips points to a shortcoming of MoCapAct: the rollouts only cover (at most) a
6-second window. Because of this, the multi-clip policy is not trained on
states that would be encountered deep into a rollout (e.g., 30 seconds into a
rollout), which limits the multi-clip policy’s performance on many longer
clips. Long clips that have high rewards and episode lengths usually have the
humanoid standing for long periods of time while doing various arm motions.
Here, the motions are much simpler since the humanoid merely needs to maintain
balance while standing still.