# M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension
and Generation

Mingshuang Luo1,2,3, Ruibing Hou1, Hong Chang1,3, Zimo Liu2, Yaowei Wang2,
Shiguang Shan1,3
1Key Laboratory of Intelligent Information Processing of Chinese Academy of
Sciences (CAS),
Institute of Computing Technology, CAS, China
2Peng Cheng Laboratory, China
3University of Chinese Academy of Sciences, China

###### Abstract

This paper presents M3GPT, an advanced Multimodal, Multitask framework for
Motion comprehension and generation. M3GPT operates on three fundamental
principles. The first focuses on creating a unified representation space for
various motion-relevant modalities. We employ discrete vector quantization for
multimodal control and generation signals, such as text, music and
motion/dance, enabling seamless integration into a large language model (LLM)
with a single vocabulary. The second involves modeling model generation
directly in the raw motion space. This strategy circumvents the information
loss associated with discrete tokenizer, resulting in more detailed and
comprehensive model generation. Third, M3GPT learns to model the connections
and synergies among various motion-relevant tasks. Text, the most familiar and
well-understood modality for LLMs, is utilized as a bridge to establish
connections between different motion tasks, facilitating mutual reinforcement.
To our knowledge, M3GPT is the first model capable of comprehending and
generating motions based on multiple signals. Extensive experiments highlight
M3GPT’s superior performance across various motion-relevant tasks and its
powerful zero-shot generalization capabilities for extremely challenging
tasks.

Figure 1: M3GPT can handle core motion comprehension and generation tasks,
including text-to-motion, motion-to-text, music-to-dance, dance-to-music,
motion prediction, and motion in-between. The motion sequences within the
dashed-line areas are masked in the input.

##  1 Introduction

Motion comprehension and generation in multimodality are crucial for diverse
applications, including AR/VR creation, video games, and virtual reality.
Numerous studies [13, 43, 55, 44] focus on motion comprehension, including
captioning 3D human motions and generating music from 3D human dances111In
this paper, the term ”motion” generally includes ”dance.” We distinguish them
when referring to specific tasks or scenes, such as text-to-motion, and music-
to-dance.. Recent advancements in AI [12, 37, 35, 40] have paved the way for
motion generation, allowing for various control signals including textual
descriptions, music pieces, and human poses. A significant shortcoming of most
existing works is their focus on single-modality control signals, overlooking
the potential for multimodal information integration. More importantly, the
comprehension and generation of motions are predominantly studied in
isolation. In reality, human motion cognition and communication indispensably
require seamless transitions between any motion-relevant modalities.
Therefore, it is vital to develop a unified framework for motion comprehension
and generation that can efficiently utilize multiple signals simultaneously.

Recent works [10, 53, 54, 50] have shown success in developing a unified
multitask motion framework which integrates text-driven and audio-driven
motion generation through a single architecture. Employing a large language
model (LLM), [52] adeptly handles multimodal control signals, such as text and
single-frame pose, to generate consecutive motions. Despite their promising
performance in motion generation, these approaches often fall short in
comprehending motion. MotionGPT [16], a recent innovation, constructs a
unified motion-language model to generate plausible human motions and natural
language descriptions through prompt instructions. However, MotionGPT focuses
solely on a single non-motion modality, i.e., text. While aligning motion with
one additional modality is relatively straightforward, integrating three or
more modalities within a single framework and achieving bidirectional
alignment among them to cover a broad range of modalities for motion
comprehension and generation presents a formidable challenge.

Two main challenges need to be solved for building a unified multimodal
framework for motion comprehension and generation. The first is how to create
a unified representation space across different motion-relevant modalities.
MotionGPT [16] and SpeechGPT [46] separately treat motion and speech as
specific language for seamlessly integrating with text. Inspired by these
efforts [16, 46], we view both motion and music as distinct forms of language,
facilitating better associations with text via LLMs. Specifically, akin to
language, we compress raw motion and music into a sequence of discrete
semantic tokens. By encoding motion, music, and language within a single
vocabulary, we can build a unified representation space across these different
modalities. The second is how to model the connections and synergies among
various motion tasks. Different motion-relevant tasks are interconnected and
can mutually enhance each other. Since text is the most familiar and well-
understood modality for LLMs, we propose employing text as a bridge to
establish connections between different motion tasks. Specifically, to better
learn the complex music-to-dance task where both input and output modalities
are unfamiliar to LLMs, we introduce two auxiliary tasks: music-to-text and
text-to-dance, aimed at aligning music and dance modalities with the
structured text embedding space. This strategy enables us to establish
connections and synergies between music-to-dance and text-to-motion tasks,
facilitating the alignment and collaboration of text, music, and motion/dance
modalities across different tasks.

In this work, we propose a uniform Multimodal, Multitask framework for Motion
comprehension and generation, namely M3GPT, that leverages the strong language
generation capability of LLMs for unifying various motion-relevant tasks, as
depicted in Fig. 1. M3GPT comprises three tires. Firstly, M3GPT is equipped
with multimodal tokenizers capable of compressing raw multimodal data,
including motion, music, and text, into a sequence of discrete semantic
tokens. These discrete representations allow the core LLM to unify motion
comprehension and generation in an autoregressive manner, operating at the
discrete semantic representation space. Secondly, different from [16, 52] that
solely optimize LLM in discrete semantic space, we jointly train LLM and
motion de-tokenizer, optimizing LLM in both discrete semantic space and raw
continuous motion space. This operation enables the motion-space error signals
from de-tokenizer to backpropagate to LLM, enhancing LLM’s ability to generate
the details of motion. Thirdly, we construct paired text descriptions for
music, and design two auxiliary music-to-text and text-to-dance tasks, which
aid in aligning music and dance with the text embedding space. Also, we build
up a shared tokenizer for motion and dance data to project them into a shared
semantic space. These auxiliary tasks and shared tokenizer establish
connections between music-to-dance and text-to-motion, enabling mutual
reinforcement.

We employ a multimodal pre-training + instruction-tuning pipeline to train
M3GPT, enhancing inter-modal alignment and effectively aligning them with
human intent. To our knowledge, M3GPT is the first approach to integrate six
core tasks of motion comprehension and generation—text-to-motion, motion-to-
text, music-to-dance, dance-to-music, motion prediction, and motion in-
between—into a uniform framework. Extensive experiments demonstrate that M3GPT
achieves competitive performance across multiple motion-relevant tasks.
Additionally, through qualitative results, we demonstrate that M3GPT possesses
powerful zero-shot generalization capabilities, e.g., long-term dance
generation and music-text conditioned dance generation.

Methods | T2M | M2T | A2D | D2A | M2M | Random M | Random T | Random A
---|---|---|---|---|---|---|---|---
TM2D[10] | ✔ | ✗ | ✔ | ✗ | ✗ | ✔ | ✗ | ✗
UDE[53] | ✔ | ✗ | ✔ | ✗ | ✗ | ✔ | ✗ | ✗
MotionGPT[52] | ✔ | ✗ | ✗ | ✗ | ✔ | ✔ | ✗ | ✗
MotionGPT[16] | ✔ | ✔ | ✗ | ✗ | ✔ | ✔ | ✔ | ✗
M3GPT (Ours) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔

Table 1: Comparison of recent multimodal, multitask methods across various
motion comprehension and generation tasks. T2M: text-to-motion; M2T: motion-
to-text; A2D: music-to-dance; D2A: dance-to-music; M2M: motion-to-motion that
includes motion prediction and motion in-between. Random M, Random T, and
Random A represent the unconstrained generation of motion, text, and
music333Note: in this paper, the term "audio" specifically refers to "music".
This designation is adopted to avoid confusion between the initial letter "M"
shared by both "music" and "motion," which could lead to ambiguity when these
modalities are represented by their initials., respectively.

##  2 Related Work

Motion comprehension and Generation. Motion comprehension involves two core
tasks: motion-to-text and dance-to-music. Motion-to-text aims to describe
human motion with natural language [31]. For example, recurrent networks have
been used in [31] to accomplish this task. Dance-to-music involves creating a
piece of music from a given dance [15, 21, 55]. For example, Zhun et al. [55]
utilizes a generative adversarial network to generate music from dance videos.
On the other hand, motion generation involves generating diverse human motions
using multimodal inputs, such as text [37, 48, 13, 4, 49], music [21, 14, 44,
35] and incomplete motion [25, 1, 2]. Text-to-motion is one of the most
important motion generation tasks. Recent works typically map text to motion
using different architectures: diffusion model [49] and autoregressive
transformer model [13]. Music-to-dance focuses on generating dance movements
from music. For example, [35] predicts discrete token sequences conditioned on
music, which are then used to regenerate the dance sequence. Motion Completion
generates motion conditioning on partial motions, such as motion prediction
[25, 1] and motion-in-between [28, 36]. Although these methods have shown
promising results in various human motion tasks, most are limited to handling
a single task. Until recently, some works [10, 16, 52, 53] attempt to
integrate two or more tasks into a unified model, as shown in Tab. 3. However,
these works either lack the ability of motion comprehension [53, 52] or fail
to handle music modality [10, 16]. In this work, we propose a unified motion
comprehension and generation framework that can handle multiple control
signals simultaneously.

Language Models and Multimodal. Large language models (LLMs) enabled by
extensive datasets and model size, such as T5 [33], Flan-T5 [6], LLaMA [38],
LLaMA-2 [39] and Vicuna [5], have demonstrated impressive comprehension and
generation capabilities. Researchers have leveraged the capabilities of LLMs
to handle multimodal tasks, expanding them to multimodal large language models
(MLLMs). For example, AnyGPT [45] employs LLaMA-2 [39] to construct an any-to-
any multimodal language model. NExT-GPT [42] employs Vicuna [5] with
multimodal adaptors and diffusion decoders to perform tasks across arbitrary
combinations of text, images, videos, and audio. Recently, the works [16, 52]
attempt to use LLMs for motion-related tasks. [52] uses LLaMA [38] to build a
general-purpose motion generator, which, however, lacks the ability to
comprehend motion. [16] leverages T5 to construct a unified motion-language
model, but cannot deal with music modality.

##  3 Method

To enhance the comprehension and generation of motion-relevant modalities, we
propose a unified multimodal framework named M3GPT. As illustrated in Fig. 2,
M3GPT consists of multimodal tokenizers responsible for compressing raw motion
and music data into discrete tokens (Sec. 3.1), and a motion-aware language
model that learns to understand and generate motion tokens from LLMs by
corresponding text and music (Sec. 3.2). To address motion-relevant tasks, we
employ a three-stage training scheme encompassing multimodal tokenizers
training, modality-alignment pre-training, and instruction tuning (Sec. 3.3).
During the inference process, multimodal tokens are decoded back into their
original representations by associated de-tokenizers (decoders of multimodal
tokenizers), enabling various motion-relevant tasks to be executed via
instructions (Sec. 3.4).

Figure 2: An overview of the M3GPT framework. M3GPT consists of multimodal
tokenizers and a motion-aware language model. The training process of M3GPT
consists of three stages: multimodal tokenizers training, modality-alignment
pre-training, and instruction tuning.

###  3.1 Multimodal tokenizers

As shown in Fig. 2, Multimodal tokenizers aim to discretize continuous human
motion and music into language-like tokens, allowing the three modalities to
be unified within a single language model.

3D Human Motion Tokenizer. To represent motion in discrete semantic tokens, we
build a 3D human motion tokenizer based on Vector Quantized Variational
Autoencoders (VQ-VAE) following [10, 53, 16, 52]. The motion tokenizer
consists of a motion encoder $\mathcal{E}_{m}$ and a motion decoder
$\mathcal{D}_{m}$, along with a codebook
$\mathcal{B}_{m}=\left\\{b^{1},b^{2},\dots,b^{N_{m}}\right\\}$ containing
$N_{m}$ discrete semantic vectors. Notably, to facilitate mutual enhancement
between motion and dance data, we employ a shared tokenizer for both motions
and dances, projecting them into a consistent and shared semantic space.
Formally, given a 3D motion sequence $\boldsymbol{m}\in\mathbb{R}^{T_{m}\times
d_{m}}$, where $T_{m}$ is the time length and $d_{m}$ is the dimensionality of
each frame’s pose, the motion encoder $\mathcal{E}_{m}$ that consists of
several 1-D convolutional layers projects $\boldsymbol{m}$ to a latent
embeddings $\boldsymbol{z}\in\mathbb{R}^{L_{m}\times d}$. Here, $L_{m}$ is the
time interval after downsampling and $d$ is the latent dimension. Next, we
transform $\boldsymbol{z}$ into a collection of codebook entries through
discrete quantization. Specifically, the quantization process replaces each
item of $\boldsymbol{z}$ with its nearest embedding in the codebook
$\mathcal{B}_{m}$, obtaining the quantized latent vectors
$\boldsymbol{e}\in\mathbb{R}^{L_{m}\times d}$ as follows:

$\boldsymbol{e}=\mathop{\arg\min}\limits_{b^{k}\in\mathcal{B}_{m}}\left\|\boldsymbol{z}-b^{k}\right\|_{2}.$
(1)

The motion decoder $\mathcal{D}_{m}$, which consists of several 1-D
deconvolutional layers, projects the quantized embeddings back to raw motion
space as $\boldsymbol{\hat{m}}=\mathcal{D}_{m}\left(\boldsymbol{e}\right)$.
Following [16, 52], the motion tokenizer can be trained by the reconstruction
loss, embedding loss and commitment loss as follows:

$\mathcal{L}_{vq}=\|\boldsymbol{\hat{m}}-\boldsymbol{m}\|_{1}+\|\mathrm{sg}\left[\boldsymbol{z}\right]-\boldsymbol{e}\|_{2}^{2}+\beta\|\boldsymbol{z}-\mathrm{sg}\left[\boldsymbol{e}\right]\|_{2}^{2}~{}.$
(2)

where $\mathrm{sg}\left[\cdot\right]$ is the stop gradient, and $\beta$ is the
factor that adjusts the weight of the commitment loss.

After training the motion tokenizer, a motion sequence $\boldsymbol{m}$ can be
represented as a sequence of discrete codebook-indices of quantized embedding
vector, namely motion tokens $\boldsymbol{q_{m}}\in\mathbb{R}^{L_{m}}$, as
follows:

$\boldsymbol{q_{m}}=\mathop{\arg\min}\limits_{k\in\left\\{1,\dots,N_{m}\right\\}}\left\|\mathcal{E}_{m}\left(\boldsymbol{m}\right)-b^{k}\right\|_{2}.$
(3)

Music Tokenizer. For the music data, we adopt the VQ-VAE in Jukebox [7] as the
music tokenizer, which consists of a music encoder $\mathcal{E}_{a}$, a music
decoder $\mathcal{D}_{a}$ and a music codebook $\mathcal{B}_{a}$. Notably, the
limited number of music samples in dance datasets makes it inadequate for
training an effective music tokenizer. To leverage the strong representation
ability of the tokenizer trained on the large-scale musical dataset, we use
the pre-trained VQ-VAE from Jukebox [7], which has been trained on a dataset
of 1.2 million songs. Specifically, we first segment each input music sample
into 5-second music segments. Then, for each 5 seconds segment
$\boldsymbol{a}\in\mathbb{R}^{T_{a}\times d_{a}}$, we use the pre-trained
music tokenizer $\left\\{\mathcal{E}_{a},\mathcal{B}_{a}\right\\}$ to encode
$\boldsymbol{a}$ into a sequence of discrete codebook-indices
$\boldsymbol{q_{a}}\in\mathbb{R}^{L_{a}}$ (namely music tokens) following Eq.
3.

###  3.2 Language Model Backbone

Expanding Vocabulary. To incorporate multimodal discrete representations into
a pre-trained LLM, we expand the original text vocabulary $V_{t}$ in LLM with
motion vocabulary $\mathcal{B}_{m}$ and music vocabulary $\mathcal{B}_{a}$,
forming a new unified vocabulary
$V=\left\\{V_{t},\mathcal{B}_{m},\mathcal{B}_{a}\right\\}$. To accommodate the
expanded vocabulary, we extend the corresponding embedding and prediction
layer of LLM, where the newly incorporated parameters are initialized
randomly.

Unified Multimodal Language Model. Equipped with multimodal tokenizers, we can
compress multimodal data into discrete token sequences. To be specific,
employing the trained motion tokenizer and music tokenizer, the input motion
$\boldsymbol{m}\in\mathbb{R}^{T_{m}\times d_{m}}$ and music
$\boldsymbol{a}\in\mathbb{R}^{T_{a}\times d_{a}}$ can be mapped into a
sequence of discrete motion tokens $\boldsymbol{q_{m}}\in\mathbb{R}^{L_{m}}$
and music tokens $\boldsymbol{q_{a}}\in\mathbb{R}^{L_{a}}$. Then equipped with
a unified vocabulary $V$, we can formulate various motion-relevant tasks in a
general format, where both input and output tokens come from the same
vocabulary. These tokens can represent natural language, human motion, music,
or any combination, depending on the specific task to be solved. This
naturally enables the core LLM to unify motion comprehension and generation
tasks in an autoregressive manner.

Following [16], we employ T5 [33] as the language model backbone, which is
pre-trained on 750 GB of text tokens. By leveraging this pre-trained large
language model, we can harness its powerful modeling capabilities and
generalizability to develop a more user-friendly, motion-related human-
computer interaction model.

###  3.3 Training Strategy

The training process is divided into three stages. The first stage is
Multimodal Tokenizers Training, which focuses on learning the motion/music
tokenizer to represent motion/music as discrete tokens. The second stage is
Modality-Alignment Pre-training, which aims to align motion, music, and text
modalities, and facilitate collaboration across different motion-relevant
tasks. The third stage is Instruction Fine-Tuning, aimed at enhancing the
model’s instruction-following capability.

Stage1: Multimodal Tokenizers Training. We first train a motion tokenizer
using the objective defined in Eq. 2. As for the music tokenizer, due to the
limited music samples in existing dance datasets, we directly use the pre-
trained VQ-VAE model from Jukebox [7]. This process allows any motion sequence
and music to be represented as a sequence of tokens, enabling seamless
integration with text within LLM. To ensure the stability of LLM training, the
encoder of motion tokenizer and whole music tokenizer remain unchanged.
Notably, we continue to optimize the decoder of motion tokenizer in subsequent
training stages to further enhance the quality of generated motions.

Stage2: Modality-Alignment Pre-training. To enable LLM to handle discrete
modalities, we utilize paired motion corpus to train LLM in a next-token
prediction task. This process aims to align the text, music, and motion
modalities for unified reasoning in LLM.

  * •

Joint optimization of LLM and motion de-tokenizer. Human motion (especially
dance) encompasses intricate details. Previous works [16, 52] keep the motion
de-tokenizer fixed during training LLM, which hinders LLM’s ability to
perceive the distribution and details of motions. Specifically, in the output
space of LLM, different motion tokens are treated as independent classes;
therefore, the cost of classifying a motion token as semantic-similar token
and semantic-distant token is the same. Apparently, relying solely on LLM’s
autoregressive loss is insufficient for capturing the details of motion. To
address this problem, we jointly optimize LLM and motion de-tokenizer in
stage2 and stage3. This strategy enables the reconstruction error signals in
raw motion space to backpropagate to LLM, enhancing LLM’s ability to generate
the details of motion.

  * •

Synergy learning of multitasks. Although aligning text with one additional
modality is relatively straightforward, integrating multiple modalities (e.g.,
motion, text, and music) within a single framework poses a significant
challenge. Additionally, as noted in [3], multitask joint training usually
achieves inferior performance on each individual task compared to single-task
training. This phenomenon is also observed in our text-to-motion task, as
shown in Tab. 2. We argue that the large modality difference among different
motion-relevant tasks (e.g., music-to-dance and text-to-motion) prevents the
model from effectively establishing connections between these tasks. Thus it
is difficult for the model to identify a common optimization direction that
benefits all tasks.

As ‘text’ serves as a highly semantic descriptor for other modalities and is
the most familiar and well-modeled modality to LLM, we use ‘text’ as a bridge
to align motion, text, and music data, thereby mitigating conflicts in
aligning multiple modalities. Initially, we construct paired textual
descriptions for music samples in the dance datasets. Specifically, we use the
style annotations of the music to create paired texts, such as ‘a person is
dancing Jazz’. Then, we construct two auxiliary tasks using the generated
pairs of music and text, i.e., music-to-text and text-to-dance. Through these
two auxiliary tasks, M3GPT implicitly learns to decompose the complex music-
to-dance task into two simpler tasks music-to-text and text-to-dance.
Additionally, with a shared tokenizer for motion and dance, text-to-dance and
text-to-motion tasks occupy the same matching space, and thus can mutually
reinforce each other. In this way, M3GPT builds the synergies between the two
primary motion generation tasks, music-to-dance and text-to-motion,
facilitating mutual reinforcement, as shown in Tab. 2.

Combining the above analysis, we jointly train LLM and motion de-tokenizer
using a mixture of motion comprehension and generation tasks, along with two
auxiliary music-to-text and text-to-dance tasks. Besides the auxiliary tasks,
we consider $2$ basic motion comprehension tasks, i.e., motion-to-text and
dance-to-music, and $4$ basic motion generation tasks, i.e., text-to-motion,
music-to-dance, motion prediction and motion in-between. Formally, for a
specific task, we denote the source input consisting of a sequence of tokens
as $\boldsymbol{q}_{s}=\left\\{\boldsymbol{q}_{s}^{i}\right\\}_{i=1}^{L_{s}}$,
the target output as
$\boldsymbol{q}_{t}=\left\\{\boldsymbol{q}_{t}^{i}\right\\}_{i=1}^{L_{t}}$,
LLM predicts the probability distribution of potential next token at each step
$p_{\theta}\left(\boldsymbol{q}_{t}^{i}|\boldsymbol{q}_{t}^{<i},\boldsymbol{q}_{s}\right)$
in an autoregressive manner. For motion generation tasks, we add a
reconstruction loss. Specifically, when the output tokens are motion tokens,
we pass them to motion de-tokenizer to generate a motion sequence (denoted as
$\boldsymbol{\hat{m}}$), where a reconstruction loss is then employed for
guidance. Overall, during this training process, the objective is to maximize
the log-likelihood of the data distribution and minimize the reconstruction
error within raw motion space:

$\mathcal{L}=\sum_{i=0}^{L_{t}-1}\log
p_{\theta}\left(\boldsymbol{q}_{t}^{i}|\boldsymbol{q}_{t}^{<i},\boldsymbol{q}_{s}\right)+\lambda\left\|\boldsymbol{\hat{m}}-\boldsymbol{m}\right\|_{1},$
(4)

where $\boldsymbol{m}$ denotes the ground-truth for $\hat{\boldsymbol{m}}$
generated by motion de-tokenizer, and $\lambda$ is a hyper-parameter to adjust
the weight for reconstruction loss.

Stage3: Instruction Fine-Tuning. To enhance the generalization and
instruction-following capability of M3GPT, we construct a multimodal
instruction dataset with resort to GPT-4, building upon existing motion
datasets. Specifically, we define $11$ core tasks, each comprising $200/50/50$
training/validation/test instruction templates. For example, an instruction
for text-to-motion task could be "Create a motion that complements the poetic
elements in <Caption_Placeholder>", with <Caption_Placeholder> standing for
any text sequence; an instruction for music-to-dance could be "Script a dance
that adapts to the tempo shifts in <Audio_Placeholder>", with
<Audio_Placeholder> standing for any music sequence. Further details are
available in Appendix B.4.

###  3.4 Inference M3GPT

During inference, we evaluate M3GPT’s performance across multiple motion-
relevant tasks and datasets (Sec. 4.2 and Appendix (C, D). Also, we consider
two challenging dance generation tasks to evaluate the zero-shot
generalization ability of M3GPT:

(1) Generating long-duration dances from long music. Long-duration dance
generation involves creating uninterrupted, coherent dance sequences based on
a single piece of music. Due to the limitation of computational cost and
memory overload, we train M3GPT on the task of 5-second music-to-dance
generation. Conversely, during inference, we can combine this training task
music-to-dance to generate an initial 5-second dance segment, and an unseen
zero-shot task music+dance-to-dance that recursively generates subsequent
dance segments conditioned on both music and previously generated dance
segments, to perform long-duration and coherent dance generation.

(2) Generating dance controlled by both music and text. Integrating music and
text as control signals in dance generation (music+text-to-dance) augments
music-to-dance task with text modality. This process guides the generated
dances to synchronize with particular actions described in input texts. Thanks
to the tokenizer mechanism, M3GPT can seamlessly combine music and text in
LLM’s input, enabling the integration of text instructions to produce a wide
variety of dance sequences.

##  4 Experiments

Table 2: Evaluation of synergy learning and joint optimization of LLM and
motion de-tokenizer on Text-to-Motion (Motion-X [23]) and Music-to-Dance
(AIST++ [18]). T2M: Text-to-Motion. A2D: Music-to-Dance. T2D: Text-to-Dance.
A2T: Music-to-Text. Trained single task refers to a model trained and tested
on a single task. Pre-trained and Instruction-tuned indicate the model after
pre-training (stage2) and instruction tuning (stage3), followed by direct
testing on each task. The arrows ($\uparrow$) indicate that higher values are
better. The arrows ($\downarrow$) indicate that smaller values are better.
Bold indicates the best result.

Methods |  Re-Optimizing motion de-tokenizer | Text-to-Motion |  | Music-to-Dance |
---|---|---|---|---|---
R TOP1 $\uparrow$ | FID $\downarrow$ | Div$\uparrow$ |  | FIDk $\downarrow$ | Divk $\uparrow$ | BAS $\uparrow$ |
Ground Truth | - | 0.675 | 0.009 | 2.316 |  | 17.10 | 8.19 | 0.2374 |
Trained single task |  | 0.645 | 0.081 | 2.124 |  | 83.33 | 5.18 | 0.1835 |
Trained single task | ✓ | 0.656 | 0.078 | 2.133 |  | 75.47 | 5.57 | 0.1884 |
T2M+A2D |  | 0.564 | 0.094 | 2.080 |  | 51.26 | 6.73 | 0.2037 |
T2M+A2D | ✓ | 0.578 | 0.092 | 2.106 |  | 47.71 | 7.47 | 0.1958 |
T2M+A2D+T2D+A2T |  | 0.617 | 0.093 | 2.110 |  | 42.70 | 7.54 | 0.2084 |
T2M+A2D+T2D+A2T | ✓ | 0.626 | 0.088 | 2.197 |  | 25.24 | 7.63 | 0.2217 |
M3GPT (Pretrained without T2D and A2T) |  | 0.526 | 0.105 | 2.058 |  | 40.71 | 7.47 | 0.2030 |
M3GPT (Pretrained without T2D and A2T) | ✓ | 0.547 | 0.104 | 2.099 |  | 37.14 | 7.61 | 0.2005 |
M3GPT (Pre-trained) |  | 0.598 | 0.089 | 2.218 |  | 32.71 | 7.43 | 0.2090 |
M3GPT (Pre-trained) | ✓ | 0.601 | 0.092 | 2.251 |  | 27.65 | 7.52 | 0.2105 |
M3GPT (Instruction-tuned) |  | 0.606 | 0.091 | 2.251 |  | 28.46 | 7.49 | 0.2052 |
M3GPT (Instruction-tuned) | ✓ | 0.615 | 0.093 | 2.253 |  | 24.34 | 7.50 | 0.2056 |

###  4.1 Experimental setup

Datasets and Preprocessing. We use a large-scale text-to-motion dataset:
Motion-X [23], and two music-to-dance datasets: AIST++ [18] and FineDance
[19]. Notably, the 3D pose annotations differ among these datasets, therefore,
we standardize their processing for uniform usage. Specifically, we select
$22$ joints common to these datasets and preprocess the motion samples
following [11], resulting in motion sequences with identical representation.
Further details on datasets and preprocessing are provided in Appendix B.1.

Evaluation Metrics. Different tasks employ distinct evaluation metrics. We use
the most common evaluation metrics to assess the performance of M3GPT for each
task.  (1) Text-to-Motion. Following [16, 10], we use Frechet Inception
Distance (FID), Diversity (Div), R-Precition that calculates the top-1 motion-
to-text retrieval accuracy (R TOP1). (2) Motion-to-Text. Following [16], we
use linguistic metrics like BLEU, CIDEr, along with R-Precision for evaluating
motion-to-text alignment. (3) Music-to-Dance. Following [20, 44], we use FID,
Diversity and Beat Align Score (BAS) on kinetic features [17] (denoted as "k")
to evaluate the dance generation quality. Notably, as noted in [40], the
geometric features [27] are unreliable as a measure of dance generation
quality. So we only use the kinetic features for evaluation. (4) Dance-to-
Music. Following [44], we use Beats Coverage Scores (BCS), Beats Hit Scores
(BHS), and F1 score to evaluate music generation quality. (5) Motion
Prediction and In-Between. Following [16], we use FID and Diversity to measure
the consistency between the provided pose conditions and generated motion.
More details and results on other evaluation metrics are provided in Appendix
B.2 and C.

Implementation Details. For motion tokenizer, we set the codebook size to
$512$. As for music tokenizer, we use the pre-trained VQ-VAE from Jukebox with
a codebook size of $2048$. In term of temporal downsampling rate, the motion
encoder uses a rate of $4$, while the music encoder uses a rate of $128$. We
utilize T5 base [33] as our language model backbone. For training the motion
tokenizer, we use Adam as the optimizer with a batch size of $1000$ and an
initial learning rate of $10^{-4}$. To train the language model backbone, we
employ the Adafactor_dev optimizer and use CosineAnnealingLR as the learning
rate scheduler. The learning rate is set to $2\times 10^{-4}$ for pre-training
stage, and $10^{-4}$ for instruction fine-tuning stage. For hyperparameter
settings, $\lambda$ in Eq. 4 is set to $0.2$, and $\beta$ in Eq. 2 is set to
$0.02$. All our experiments are conducted on 8 NVIDIA A40 GPUs. Further
details on implementation and hyperparameter analysis are provided in Appendix
E.

Table 3: Comparison results on Motion-X [23] dataset. The evaluation metrics
are computed using the encoder introduced in Appendix A. Empty columns of
previous methods indicate that they can not handle the task. Instruction-tuned
only T2M indicates the model that is initially pre-trained on multiple tasks,
followed by instruction tuning solely on text-to-motion task.

Methods | Text-to-Motion | Motion-to-Text | Motion Prediction | Motion In-between
---|---|---|---|---
R TOP1$\uparrow$ | FID$\downarrow$ | Div$\uparrow$ | R TOP3$\uparrow$ | Bleu@4$\uparrow$ | CIDEr$\uparrow$ | FID$\downarrow$ | Div$\uparrow$ | FID$\downarrow$ | Div$\uparrow$
Real | $\text{0.675}^{\pm\text{0.003}}$ | $\text{0.009}^{\pm\text{0.000}}$ | $\text{2.316}^{\pm\text{0.011}}$ | 0.881 | - | - | 0.009 | 2.316 | 0.009 | 2.316
MLD [4] | $\text{0.612}^{\pm\text{0.003}}$ | $\text{0.122}^{\pm\text{0.008}}$ | $\text{2.267}^{\pm\text{0.018}}$ | - | - | - | - | - | - | -
T2M-GPT [47] | $\text{0.647}^{\pm\text{0.002}}$ | $\text{0.101}^{\pm\text{0.005}}$ | $\underline{\text{2.270}}^{\pm\text{0.033}}$ | - | - | - | - | - | - | -
MotionDiffuse [49] | $\underline{\text{0.659}}^{\pm\text{0.002}}$ | $\textbf{\text{0.075}}^{\pm\text{0.004}}$ | $\text{2.220}^{\pm\text{0.022}}$ | - | - | - | - | - | - | -
TM2T [13] | $\text{0.581}^{\pm\text{0.002}}$ | $\text{0.148}^{\pm\text{0.003}}$ | $\text{2.005}^{\pm\text{0.034}}$ | 0.806 | 12.13 | 20.16 | - | - | - | -
MDM [37] | $\text{0.472}^{\pm\text{0.008}}$ | $\text{0.078}^{\pm\text{0.000}}$ | $\text{2.133}^{\pm\text{0.012}}$ | - | - | - | 1.028 | 1.746 | 0.831 | 1.768
Trained single task | $\text{\text{0.656}}^{\pm\text{0.002}}$ | $\text{0.078}^{\pm\text{0.002}}$ | $\text{2.133}^{\pm\text{0.012}}$ | 0.767 | 10.14 | 22.92 | 0.774 | 1.778 | 0.692 | 1.810
M3GPT (Pre-trained) | $\text{0.601}^{\pm\text{0.002}}$ | $\text{0.092}^{\pm\text{0.002}}$ | $\text{2.251}^{\pm\text{0.012}}$ | 0.834 | 11.00 | 24.12 | 0.707 | 1.874 | 0.604 | 1.879
M3GPT (Instruction-tuned) | $\text{\text{0.615}}^{\pm\text{0.003}}$ | $\text{0.093}^{\pm\text{0.002}}$ | $\text{2.253}^{\pm\text{0.026}}$ | 0.845 | 11.50 | 42.93 | 0.682 | 1.838 | 0.612 | 1.900
M3GPT (Instruction-tuned only T2M) | $\textbf{\text{0.661}}^{\pm\text{0.003}}$ | $\underline{\text{0.076}}^{\pm\text{0.002}}$ | $\textbf{\text{2.273}}^{\pm\text{0.026}}$ | - | - | - | - | - | - | -

Table 4: Comparison results on AIST++ [18] and FineDance [19] datasets.

Methods | Music-to-Dance on AIST++ | Music-to-Dance on FineDance | Dance-to-Music on AIST++
---|---|---|---
FID${}_{k}\downarrow$ | Div${}_{k}\uparrow$ | BAS $\uparrow$ | FID${}_{k}\downarrow$ | Div${}_{k}\uparrow$ | BAS $\uparrow$ | BCS$\uparrow$ | BHS$\uparrow$ | F1$\uparrow$
Real | 17.10 | 10.60 | 0.2374 | - | - | 0.2120 | - | - | -
FACT [18] | 35.35 | 5.94 | 0.2209 | 113.38 | 3.36 | 0.1831 | - | - | -
Bailando [35] | 28.16 | 7.83 | 0.2332 | 82.81 | 7.74 | 0.2029 | - | - | -
EDGE [40] | 42.16 | 3.96 | 0.2334 | 94.34 | 8.13 | 0.2116 | - | - | -
Lodge [20] | 37.09 | 5.58 | 0.2423 | 45.56 | 6.75 | 0.2397 | - | - | -
Foley [9] | - | - | - | - | - | - | 96.4 | 41.0 | 57.5
CMT [8] | - | - | - | - | - | - | 97.1 | 46.2 | 62.6
D2MGAN [55] | - | - | - | - | - | - | 95.6 | 88.7 | 93.1
CDCD [56] | - | - | - | - | - | - | 96.5 | 89.3 | 92.7
LORIS [44] | - | - | - | - | - | - | 98.6 | 90.8 | 94.5
Trained single task | 75.47 | 5.57 | 0.1884 | 128.37 | 6.48 | 0.2036 | 93.9 | 93.6 | 92.8
M3GPT (Pre-trained) | $\underline{\text{27.65}}$ | $\underline{\text{7.52}}$ | 0.2105 | 92.35 | 7.67 | 0.2134 | 93.4 | 93.8 | 94.2
M3GPT (Instruction-tuned) | 24.34 | 7.50 | 0.2056 | 86.47 | 7.75 | 0.2158 | 93.6 | 94.0 | 94.9

###  4.2 Ablation Studies

In this section, we conduct ablation studies to validate the effectiveness of
our method. We use the same model architecture throughout the experiments. The
ablation results are shown in Tab. 2.

Effectiveness of joint optimization of LLM and motion de-tokenizer. Different
from previous works [16, 52] that fix motion de-tokenizer during training LLM,
we jointly optimize LLM and motion de-tokenizer in stage2 and stage3, as
detailed in Sec. 3.3. As shown in Tab. 2, the joint optimization consistently
brings performance gains across various metrics and most settings.
Specifically, it largely enhances the fidelity of generated dances, reflected
in a notable decrease in FIDk score. We also notice a minor increase (less
than $0.003$) in FID of text-to-motion task in M3GPT. The possible reason is
that the motion patterns controlled by text are relatively simple, making LLM
optimized solely in discrete semantic space adequate for text-to-motion.
Conversely, dances involve greater complexity, necessitating the joint
optimization of motion decoder to accurately capture intricate dance movements
without compromising information.

Effectiveness of synergy learning of multitasks. During the training of M3GPT,
we introduce a synergy multitask learning strategy by constructing two
auxiliary tasks: Text-to-Dance (T2D) and Music-to-Text (A2T), as detailed in
Sec. 3.3. As shown in Tab. 2, the inclusion of T2D and A2T consistently brings
performance gains across various metrics on both text-to-motion and music-to-
dance tasks. Specifically, for music-to-dance, the FIDk score is decreased by
nearly $10$ points, indicating that the synergy learning helps generate more
realistic dances. We argue that by incorporating these two auxilary tasks,
M3GPT implicitly learns to decompose the complex music-to-dance into two
simpler tasks, music-to-text and text-to-dance. This way, the text-to-motion
task can assist in learning the music-to-dance task, thereby enhancing its
performance.

###  4.3 Comparisons with State-of-the-arts

In this section, we compare our M3GPT with state-of-the-arts on multiple core
motion-relevant tasks. We respectively report the comparison results on text-
to-motion dataset, Motion-X [23], and music-to-dance datasets, AIST++ [18] and
FineDance [19]. More quantitative and qualitative results are provided in
Appendix C and D. Also, in the supplementary material’s zip file, we provide
the render videos of generated motions/dances and generated music files by our
M3GPT.

Main results on text-to-motion dataset. On the text-to-motion dataset,
Motion-X, we evaluate M3GPT on $4$ tasks, i.e., text-to-motion, motion-to-
text, motion prediction, and motion in-between. The comparison results are
shown in Tab. 3. As shown, M3GPT achieves competitive performance across all
evaluated tasks, highlighting its capability to address diverse motion tasks
in a single model. Also, for text-to-motion task, M3GPT (instruction-tuned
only T2M), which combines multitask pre-training and instruction fine-tuning
solely on T2M task, yields better performance than Trained single task that
only trains the model on T2M task. The results demonstrate that the multitask
pre-training is conductive to enhance the performance of individual task.

Main results on music-to-dance datasets. On the music-to-dance datasets,
AIST++ and FineDance, we evaluate M3GPT on $2$ tasks, i.e., music-to-dance and
dance-to-music. As shown in Tab. 4, in general, the performance of multitask
pre-training and instruction fine-tuning in M3GPT outperforms single-task
training, underscoring the effectiveness of multitask training for dance-
relevant tasks. Also, M3GPT achieves competitive performance on most metrics.
For music-to-dance, the best-performing method on the FineDance dataset is
Lodge [20]. This approach features a specialized two-stage architecture for
generating long-duration dance sequences, progressively refining movements
from coarse to fine granularity using a diffusion model. On AIST++ dataset,
M3GPT reports the best FIDk of $24.34$ for music-to-dance task, the best BHS
and F1 of $94.0$ and $94.9$ for dance-to-music task.

Figure 3: Qualitative results for long-term dance and music-text conditioned
dance generation of M3GPT.

###  4.4 Evaluation on Zero-Shot Tasks

In this section, we explore M3GPT’s capabilities in handling zero-shot tasks,
as mentioned in Sec. 3.4. Fig. 3 (a) shows the long-term dance generation. As
seen, M3GPT can generate a coherent dance sequence by seamlessly integrating
the music-to-dance and zero-shot music+dance-to-dance tasks. Fig. 3 (b) shows
the generated 3D dance with both music and text instruction. We can see that
M3GPT maintains plausible visual results in accordance with input text
instructions (cartweel), underscoring its remarkable zero-shot generalization
capability.

##  5 Conclusion

In this paper, we present M3GPT, a unified framework for comprehending and
generating motion aligned with both text and music modalities. By employing
text as a bridge, we build connections and synergies between different motion-
relevant tasks, facilitating mutual reinforcement. Additionally, the joint
optimization of LLM and motion de-tokenizer further enriches the details of
generated motion, enhancing overall motion generation quality. Our extensive
evaluations across various motion-relevant tasks demonstrate the effectiveness
of M3GPT in both motion comprehension and generation. Besides, M3GPT exhibits
strong zero-shot generalization abilities, enabling it to handle previously
unseen and challenging motion-relevant tasks.

Limitations and Broader Impacts. Although our M3GPT has successfully processed
signals from motion, text, and music modalities for motion comprehension and
generation, it focuses on modeling human body movements, excluding hands and
faces details. Future research can extend the scope of M3GPT to include hands
and facial modeling.

## References

  * Butepage et al. [2017] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom.  Deep representation learning for human motion prediction and classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6158–6166, 2017.
  * Chen et al. [2023a] Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu.  Humanmac: Masked motion completion for human motion prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9544–9555, 2023a.
  * Chen et al. [2022] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton.  A unified sequence interface for vision tasks.  _Advances in Neural Information Processing Systems_ , 35:31333–31346, 2022.
  * Chen et al. [2023b] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu.  Executing your commands via motion diffusion in latent space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18000–18010, 2023b.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023.  _URL https://lmsys. org/blog/2023-03-30-vicuna_ , 3(5), 2023.
  * Chung et al. [2024] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _Journal of Machine Learning Research_ , 25(70):1–53, 2024.
  * Dhariwal et al. [2020] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * Di et al. [2021] Shangzhe Di, Zeren Jiang, Si Liu, Zhaokai Wang, Leyan Zhu, Zexin He, Hongming Liu, and Shuicheng Yan.  Video background music generation with controllable music transformer.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 2037–2045, 2021.
  * Gan et al. [2020] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba.  Foley music: Learning to generate music from videos.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 758–775. Springer, 2020.
  * Gong et al. [2023] Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang.  Tm2d: Bimodality driven 3d dance generation via music-text integration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9942–9952, 2023.
  * Guo et al. [2022a] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5152–5161, 2022a.
  * Guo et al. [2022b] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5152–5161, 2022b.
  * Guo et al. [2022c] Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng.  Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts.  In _European Conference on Computer Vision_ , pages 580–597. Springer, 2022c.
  * Han et al. [2023] Bo Han, Yi Ren, and Yuheng Li.  Dance2midi: Dance-driven multi-instruments music generation.  _arXiv preprint arXiv:2301.09080_ , 2:3, 2023.
  * Huang et al. [2020] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang.  Dance revolution: Long-term dance generation with music via curriculum learning.  _arXiv preprint arXiv:2006.06119_ , 2020.
  * Jiang et al. [2024] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.  Motiongpt: Human motion as a foreign language.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Kensuke Onuma [2008] and Jessica K Hodgins Kensuke Onuma, Christos Faloutsos.  Fmdistance: A fast and effective distance function for motion capture data.  In _Eurographics_ , pages 83–86, 2008.
  * Li et al. [2021] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13401–13412, 2021.
  * Li et al. [2023] Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li.  Finedance: A fine-grained choreography dataset for 3d full body dance generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10234–10243, 2023.
  * Li et al. [2024a] Ronghui Li, YuXiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu, and Xiu Li.  Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives.  _arXiv preprint arXiv:2403.10518_ , 2024a.
  * Li et al. [2024b] Sifei Li, Weiming Dong, Yuxin Zhang, Fan Tang, Chongyang Ma, Oliver Deussen, Tong-Yee Lee, and Changsheng Xu.  Dance-to-music generation with encoder-based textual inversion of diffusion models.  _arXiv preprint arXiv:2401.17800_ , 2024b.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Lin et al. [2023] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang.  Motion-x: A large-scale 3d expressive whole-body human motion dataset.  _Advances in Neural Information Processing Systems_ , 2023.
  * Loper et al. [2023] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pages 851–866. 2023.
  * Martinez et al. [2017] Julieta Martinez, Michael J Black, and Javier Romero.  On human motion prediction using recurrent neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2891–2900, 2017.
  * McFee et al. [2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python.  In _SciPy_ , pages 18–24, 2015.
  * Müller et al. [2005] Meinard Müller, Tido Röder, and Michael Clausen.  Efficient content-based retrieval of motion capture data.  In _ACM SIGGRAPH_ , pages 677–685. 2005.
  * Oreshkin et al. [2023] Boris N Oreshkin, Antonios Valkanas, Félix G Harvey, Louis-Simon Ménard, Florent Bocquelet, and Mark J Coates.  Motion in-betweening via deep $delta$-interpolator.  _IEEE Transactions on Visualization and Computer Graphics_ , 2023.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10975–10985, 2019.
  * Plappert et al. [2018] Matthias Plappert, Christian Mandery, and Tamim Asfour.  Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks.  _Robotics and Autonomous Systems_ , 109:13–26, 2018.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Romero et al. [2022] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  _arXiv preprint arXiv:2201.02610_ , 2022.
  * Siyao et al. [2022] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11050–11059, 2022.
  * Starke et al. [2023] Paul Starke, Sebastian Starke, Taku Komura, and Frank Steinicke.  Motion in-betweening with phase manifolds.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , 6(3):1–17, 2023.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano.  Human motion diffusion model.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Tseng et al. [2023] Jonathan Tseng, Rodrigo Castellon, and Karen Liu.  Edge: Editable dance generation from music.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 448–458, 2023.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.
  * Yamada et al. [2018] Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata.  Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions.  _IEEE Robotics and Automation Letters_ , 3(4):3441–3448, 2018.
  * Yu et al. [2023] Jiashuo Yu, Yaohui Wang, Xinyuan Chen, Xiao Sun, and Yu Qiao.  Long-term rhythmic video soundtracker.  In _International Conference on Machine Learning_ , pages 40339–40353. PMLR, 2023.
  * Zhan et al. [2024] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al.  Anygpt: Unified multimodal llm with discrete sequence modeling.  _arXiv preprint arXiv:2402.12226_ , 2024.
  * Zhang et al. [2023a] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu.  Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.  _arXiv preprint arXiv:2305.11000_ , 2023a.
  * Zhang et al. [2023b] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen.  T2m-gpt: Generating human motion from textual descriptions with discrete representations.  _arXiv preprint arXiv:2301.06052_ , 2023b.
  * Zhang et al. [2023c] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan.  Generating human motion from textual descriptions with discrete representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14730–14740, 2023c.
  * Zhang et al. [2024a] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024a.
  * Zhang et al. [2024b] Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al.  Large motion model for unified multi-modal motion generation.  _arXiv preprint arXiv:2404.01284_ , 2024b.
  * Zhang et al. [2021] Yan Zhang, Michael J Black, and Siyu Tang.  We are more than our joints: Predicting how 3d bodies move.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3372–3382, 2021.
  * Zhang et al. [2024c] Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang.  Motiongpt: Finetuned llms are general-purpose motion generators.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 7368–7376, 2024c.
  * Zhou and Wang [2023] Zixiang Zhou and Baoyuan Wang.  Ude: A unified driving engine for human motion generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5632–5641, 2023.
  * Zhou et al. [2023] Zixiang Zhou, Yu Wan, and Baoyuan Wang.  A unified framework for multimodal, multi-part human motion synthesis.  _arXiv preprint arXiv:2311.16471_ , 2023.
  * Zhu et al. [2022a] Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov.  Quantized gan for complex music generation from dance videos.  In _European Conference on Computer Vision_ , pages 182–199. Springer, 2022a.
  * Zhu et al. [2022b] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan.  Discrete contrastive diffusion for cross-modal music and image generation.  _arXiv preprint arXiv:2206.07771_ , 2022b.

## Appendix

##  Appendix A Text-Motion Alignment Model

Due to the lack of a powerful and publicly available text-motion alignment
model, we independently leverage existing datasets to develop a functional
text-motion alignment model, which is used to evaluate tasks such as text-to-
motion, motion-to-text, motion prediction, and motion in-between. We adopt the
motion encoder architecture $E_{m}$ from [12] and use the pretrained CLIP [32]
ViT-B/32 model for the text encoder $E_{t}$. As depicted in Fig. 4, the
training of the text-motion alignment model is split into two phases: pre-
training the motion auto-encoder and text-motion contrastive learning. During
the pre-training phase, we use motion data from the text-to-motion, and dance
data from the music-to-dance tasks. This stage employs a reconstruction loss
to ensure the model achieves a robust initial state capable of extracting an
expressive motion representation. In the text-motion contrastive learning
phase, we utilize text-motion pair data from the text-to-motion task. We
incorporate an adapter MLP layer into both the motion and text encoders to
align the dimensions of $\mathbf{z}_{m}$ and $\mathbf{z}_{t}$ at 512. This
setup facilitates the alignment of text and motion in the representational
space. The motion reconstruction loss $L_{\mathrm{recon\\_motion}}$ for the
pre-training stage and the contrastive learning loss $L_{\mathrm{infoNCE}}$
for the aligning stage are used to optimize this model, as follows,

$\mathcal{L}_{\mathrm{recon\\_motion}}=\|x-\tilde{x}\|^{2}$ (5)

$\mathcal{L}_{\mathrm{infoNCE}}=-\frac{1}{N}\sum_{i=1}^{K}\log\left(\frac{\exp(\langle
z_{i}^{\prime},z_{i}^{t}\rangle/\tau)}{\sum_{j=1}^{K}\exp(\langle
z_{i}^{\prime},z_{j}^{t}\rangle/\tau)}\right)$ (6)

Figure 4: Pipeline of Text-Motion Alignment Model. The training of the text-
motion alignment model includes two stages: pre-training motion auto-encoder
and text-motion contrastive learning.

##  Appendix B Details for Training and Evaluating

###  B.1 Data Introductions and Preprocessing

We leverage the largest available text-to-motion dataset, Motion-X [23], along
with widely-used music-to-dance datasets, AIST++ [18] and FineDance [19], for
our multitask training regimen. Motion-X is used for text-to-motion, motion-
to-text, motion prediction, and motion in-between tasks, while AIST++ and
FineDance datasets support both music-to-dance and dance-to-music tasks, and
are also adapted for motion prediction and in-between tasks to enhance our
training resources.

Motion-X includes 15.6 million precise 3D whole-body SMPL-X [30] pose
annotations across 81.1K motion sequences with sequence-level semantic text
descriptions. AIST++ contains 1,409 dance motion sequences across 10 dance
genres with SMPL [24] pose annotations, and FineDance provides 7.7 hours of
dance, totaling 831,600 frames with SMPL-H [34] poses at 30 fps across 16
dance genres.

We standardize data across these datasets by selecting the 22 common joints
and normalizing each motion sequence to face the same direction and to run at
30 fps. We use a processing technique consistent with prior research [12, 4,
16] that integrates joint velocities, positions, and rotations for consistent
motion representation, facilitating effective utilization across tasks. For
the music data, we use the Librosa toolkit [26] to load raw .wav data at a
sampling rate of 22050 Hz, processed into features by the Jukebox encoder [7].
To optimize the use of these datasets, we strategically employ data from
specific datasets for different tasks. During training for music and dance
tasks, we randomly select 5-second segments from complete music tracks and
corresponding dance segments as training samples, setting the sample length to
6.25 seconds for motion prediction and in-between tasks with AIST++ and
FineDance. When assessing music-to-dance on the FineDance dataset, we don’t
evaluate all 5-second samples directly. Instead, we generate continuous long
dance sequences using music-to-dance and motion prediction, then segment these
into 30-second samples for evaluation to align with Lodge’s testing
methodology.

###  B.2 Comprehensive Evaluation Metrics

Different tasks utilize specific evaluation metrics. We use the consistent
evaluation metrics following prior research [12, 16, 10, 20, 55, 44].

  * •

Text-to-Motion. We measure the discrepancy between generated and actual motion
features using Frechet Inception Distance (FID), assess diversity (Div) among
all generated motion sequences, and evaluate motion-text semantic correlation
with R-precision. Multimodal Distance (MM Dist) quantifies the disparity
between motions and texts. A specialized model developed for evaluating the
text-to-motion task on the Motion-X dataset with 22 joints is detailed in
Appendix A.

  * •

Motion-to-Text. We use linguistic metrics including BLEU [29], ROUGE [22],
CIDEr [41], and BertScore [51], along with R-Precision and MM Dist to assess
alignment between generated text and motion.

  * •

Music-to-Dance. We employ the evaluation framework recommended by FACT [18]
and Bailando [35], utilizing FID, Diversity, and Beat Align Score (BAS) to
gauge dance quality. In our paper, we use kinetic features to compute FID and
Diversity.

  * •

Dance-to-Music. We use metrics from [44, 55] such as Beats Coverage Scores
(BCS), Beats Hit Scores (BHS), F1 scores, Beats Coverage Std (CSD), and Beats
Hit Std (HSD) to evaluate music generation quality.

  * •

Motion Prediction and Motion In-between. We use Average Displacement Error
(ADE) and Final Displacement Error (FDE) to assess the quality of predicted
motion. The text-motion alignment model aids in evaluating motion prediction
performance.

###  B.3 Distributed Training for Multitasks

We employ a single-node multi-GPU distributed training strategy for M3GPT,
distributing each task across separate GPUs to facilitate multitask training
through shared model parameters. This method allows us to tailor the maximum
token length of the LLM for each task, based on the longest sample sequence
typical for that task. Specifically, we set the maximum LLM token length to
$192$ for tasks such as text-to-motion, motion-to-text, motion prediction, and
motion in-between. For tasks involving the music modality, such as music-to-
dance and dance-to-music, the maximum length is set at $980$. This task-
specific configuration enables us to optimize batch sizes effectively, thus
maximizing GPU utilization.

In our experimental setup, the batch size is set to $40$ for the text-to-
motion, text-to-dance, and motion-to-text tasks, and $4$ for the music-to-
dance, dance-to-music, and music-to-text tasks. For motion prediction and
motion in-between tasks, the batch size is set to $45$. We also establish the
number of iterations for pre-training at 1,000,000, instruction fine-tuning at
200,000. This structured approach ensures that each task is optimally
processed, enhancing the efficiency and effectiveness of our training regimen.

###  B.4 Tasks for Pre-training and Instruction tuning

As shown in Fig. 5, we define 11 core motion-related tasks for the instruction
tuning of M3GPT, including text-to-motion, random-to-motion, motion-to-text,
random-to-text, motion prediction, motion in-between, music-to-dance, dance-
to-music, random-to-music, text-to-dance, and music-to-text. The tasks of
text-to-dance and music-to-text were specifically constructed based on the
music-to-dance datasets. Random-to-X represents the unconstrained generation
of motion, text and music. In Tab. 5, we present a selection of command
templates for each task, where <Motion_Placeholder>, <Caption_Placeholder>,
and <Music_Placeholder> respectively represent the motion sequence (including
dance sequence), textual description, and music segment from the training
data.

Figure 5: Tasks for M3GPT pre-training and instruction tuning. Random
represents the unconstrained generation of motion/text/music in the
corresponding task. Table 5: Examples of instruction templates for each task
when instruction tuning M3GPT.

Task | Instruction template | Output
---|---|---
text-to-motion | Design a motion that illustrates the emotions conveyed by <Caption_Placeholder>. | <Motion_Placeholder>
How could you express the resilience mentioned in <Caption_Placeholder>
through motion?
Can you develop a motion that captures the existential debate in
<Caption_Placeholder>?
random-to-motion | Can you generate a motion randomly? | <Motion_Placeholder>
Please generate a random motion.
Display a motion for me.
motion-to-text | What themes are explored through the motion in <Motion_Placeholder>? | <Caption_Placeholder>
Can you describe the motion <Motion_Placeholder> with texts?
How would you interpret the actions depicted in <Motion_Placeholder>?
random-to-text | Can you generate a text description for motion randomly? | <Caption_Placeholder>
Give me a caption which describes a action.
How can we describe a motion with texts?
motion prediction | What movements would suitably follow the thematic climax of <Motion_Placeholder>? | <Motion_Placeholder>
What steps might deepen the emotional expression seen in <Motion_Placeholder>?
What movements could follow to resolve the suspense built in
<Motion_Placeholder>?
motion in-between | What new character dynamics could the middle section of <Motion_Placeholder> explore? | <Motion_Placeholder>
Infer the type of dramatic climax that the masked section of
<Motion_Placeholder> might contain.
What potential themes of ascent or descent could be explored in the middle of
<Motion_Placeholder>?
music-to-dance | Script a dance that adapts to the tempo shifts in <Music_Placeholder>. | <Motion_Placeholder>
Create a dance that would visually mimic the lyrical journey in
<Music_Placeholder>.
Compose a dance that explores the genre characteristics of
<Music_Placeholder>.
dance-to-music | Can you design a music for this dance <Motion_Placeholder>? | <Music_Placeholder>
Please create a music based on this dance <Motion_Placeholder>.
Arrange a symphony that captures the shifts in <Motion_Placeholder>.
random-to-music | Please generate a music for a dance randomly. | <Music_Placeholder>
Can you generate a music with dance style?
Creat a music for any style dance.
text-to-dance | Please generate a dance based on the caption <Caption_Placeholder>. | <Motion_Placeholder>
Create a dance for the text <Caption_Placeholder>.
Generate a dance that corresponds to the textual description
<Caption_Placeholder>.
music-to-text | Describe the dance movements that correspond to the given music <Music_Placeholder>. | <Caption_Placeholder>
Describe the dance actions that match the provided music <Music_Placeholder>.
Detail the dance steps associated with the specified music
<Music_Placeholder>.

##  Appendix C Quantitative Comparisons with SOTA Methods

In this section, we compare the performance of our M3GPT with existing SOTA
methods on a broader set of metrics for each task across three datasets:
Motion-X [23], AIST++ [18], and FineDance [19]. Tab. 9 shows the comparison of
text-to-motion on Motion-X dataset. Tab. 9 shows the comparison of motion-to-
text on Motion-X dataset. Tab. 9 shows the comparison of music-to-dance on
AIST++ and FineDance datasets. Tab. 9 shows the comparison of dance-to-music
on AIST++ and FineDance datasets. Tab. 10 shows the comparison of motion
prediction and motion in-between on Motion-X dataset. As shown in these
tables, M3GPT can achieve competitive performance with SOTAs across all
evaluated tasks.

Table 6: Comparison of Text-to-Motion on Motion-X dataset [23]. The arrows
($\uparrow$) indicate that higher values are better. The arrows ($\downarrow$)
indicate that smaller values are better. Bold and underline indicate the best
and the second best result.

Methods |  RPrecision $\uparrow$ | FID $\downarrow$ | MM Dist $\downarrow$ | Div $\uparrow$ | MModality $\uparrow$
---|---|---|---|---|---
Top1 | Top2 | Top3
Real | $\text{0.675}^{\pm\text{0.003}}$ | $\text{0.821}^{\pm\text{0.003}}$ | $\text{0.878}^{\pm\text{0.002}}$ | $\text{0.009}^{\pm\text{0.000}}$ | $\text{2.938}^{\pm\text{0.007}}$ | $\text{2.316}^{\pm\text{0.011}}$ | -
MDM [37] | $\text{0.472}^{\pm\text{0.008}}$ | $\text{0.616}^{\pm\text{0.005}}$ | $\text{0.704}^{\pm\text{0.003}}$ | $\text{0.161}^{\pm\text{0.006}}$ | $\text{5.404}^{\pm\text{0.031}}$ | $\text{2.234}^{\pm\text{0.015}}$ | $\underline{\text{2.241}}^{\pm\text{0.043}}$
MLD [4] | $\text{0.612}^{\pm\text{0.003}}$ | $\text{0.743}^{\pm\text{0.002}}$ | $\text{0.808}^{\pm\text{0.004}}$ | $\text{0.122}^{\pm\text{0.008}}$ | $\text{3.117}^{\pm\text{0.035}}$ | $\text{2.267}^{\pm\text{0.018}}$ | $\text{2.210}^{\pm\text{0.055}}$
T2M-GPT [47] | $\text{0.647}^{\pm\text{0.002}}$ | $\text{0.785}^{\pm\text{0.004}}$ | $\text{0.845}^{\pm\text{0.003}}$ | $\text{0.101}^{\pm\text{0.005}}$ | $\text{3.046}^{\pm\text{0.028}}$ | $\underline{\text{2.270}}^{\pm\text{0.033}}$ | $\textbf{2.226}^{\pm\text{0.036}}$
MotionDiffuse [47] | $\underline{\text{0.659}}^{\pm\text{0.002}}$ | $\underline{\text{0.802}}^{\pm\text{0.004}}$ | $\textbf{0.865}^{\pm\text{0.002}}$ | $\textbf{0.075}^{\pm\text{0.004}}$ | $\text{2.944}^{\pm\text{0.004}}$ | $\text{2.220}^{\pm\text{0.022}}$ | $\text{2.102}^{\pm\text{0.036}}$
Trained single task | $\text{0.656}^{\pm\text{0.002}}$ | $\text{0.795}^{\pm\text{0.001}}$ | $\text{0.843}^{\pm\text{0.001}}$ | $\text{0.078}^{\pm\text{0.000}}$ | $\underline{\text{2.942}}^{\pm\text{0.001}}$ | $\text{2.133}^{\pm\text{0.012}}$ | $\text{2.046}^{\pm\text{0.052}}$
M3GPT (Pre-trained) | $\text{0.601}^{\pm\text{0.002}}$ | $\text{0.751}^{\pm\text{0.003}}$ | $\text{0.803}^{\pm\text{0.002}}$ | $\text{0.092}^{\pm\text{0.002}}$ | $\text{2.945}^{\pm\text{0.001}}$ | $\text{2.251}^{\pm\text{0.012}}$ | $\text{2.188}^{\pm\text{0.074}}$
M3GPT (Fine-tuned) | $\text{0.615}^{\pm\text{0.003}}$ | $\text{0.757}^{\pm\text{0.004}}$ | $\text{0.815}^{\pm\text{0.003}}$ | $\text{0.093}^{\pm\text{0.002}}$ | $\text{2.944}^{\pm\text{0.002}}$ | $\text{2.253}^{\pm\text{0.023}}$ | $\text{2.204}^{\pm\text{0.058}}$
M3GPT(Fine-tuned only T2M) | $\textbf{0.661}^{\pm\text{0.003}}$ | $\textbf{0.804}^{\pm\text{0.004}}$ | $\underline{\text{0.861}}^{\pm\text{0.003}}$ | $\underline{\text{0.076}}^{\pm\text{0.002}}$ | $\textbf{2.940}^{\pm\text{0.002}}$ | $\textbf{2.273}^{\pm\text{0.026}}$ | $\text{2.131}^{\pm\text{0.032}}$

Table 7: Comparison of Motion-to-Text on Motion-X [23].

Methods |  RPrecision $\uparrow$ | MM Dist $\downarrow$ | Bleu@1 $\uparrow$ | Bleu@4 $\uparrow$ | Rouge $\uparrow$ | CIDEr $\uparrow$ | BertScore $\uparrow$
---|---|---|---|---|---|---|---
Top1 | Top2 | Top3
Real | 0.681 | 0.824 | 0.881 | 2.897 | - | - | - | - | -
TM2T [13] | 0.574 | 0.726 | 0.806 | 2.988 | 30.54 | 12.13 | 32.52 | 20.16 | 25.37
Trained single task | 0.565 | 0.706 | 0.767 | 3.011 | 31.07 | 10.14 | 31.65 | 22.92 | 28.19
M3GPT (Pre-trained) | 0.627 | 0.773 | 0.834 | 2.946 | 33.31 | 11.00 | 34.10 | 24.12 | 30.96
M3GPT (Fine-tuned) | 0.631 | 0.783 | 0.845 | 2.950 | 34.27 | 11.50 | 34.55 | 42.93 | 31.49

Table 8: Comparison of Music-to-Dance on AIST++ [18] and FineDance [19].

Methods | Music-to-Dance on AIST++ | Music-to-Dance on FineDance
---|---|---
FID${}_{k}\downarrow$ |  Div${}_{k}\uparrow$ |  BAS$\uparrow$ |  FID${}_{k}\downarrow$ |  Div${}_{k}\uparrow$ |  BAS$\uparrow$
Real | 17.10 | 8.19 | 0.2374 | - | 9.73 | 0.2120
FACT [18] | 35.35 | 5.94 | 0.2209 | 113.38 | 3.36 | 0.1831
Bailando [35] | 28.16 | 7.83 | 0.2332 | 82.81 | 7.74 | 0.2029
EDGE [40] | 42.16 | 3.96 | 0.2334 | 94.34 | 8.13 | 0.2116
Lodge [20] | 37.09 | 5.58 | 0.2423 | 45.56 | 6.75 | 0.2397
Trained single task | 75.47 | 5.57 | 0.1884 | 128.37 | 6.48 | 0.2036
M3GPT (Pre-trained) | $\underline{\text{27.65}}$ | $\underline{\text{7.52}}$ | 0.2105 | 92.35 | 7.67 | 0.2134
M3GPT (Fine-tuned) | 24.34 | 7.50 | 0.2056 | 86.47 | 7.75 | 0.2158

Table 9: Comparison of Dance-to-Music on AIST++ [18] and FineDance [19].

Methods | Dance-to-Music on AIST++ | Dance-to-Music on FineDance
---|---|---
BCS $\uparrow$ |  CSD $\downarrow$ |  BHS $\uparrow$ |  HSD $\downarrow$ |  F1 $\uparrow$ |  BCS $\uparrow$ |  CSD $\downarrow$ |  BHS $\uparrow$ |  HSD $\downarrow$ |  F1 $\uparrow$
Foley [9] | 96.4 | 6.9 | 41.0 | 15.0 | 57.5 | - | - | - | - | -
CMT [8] | 97.1 | 6.4 | 46.2 | 18.6 | 62.6 | - | - | - | - | -
D2MGAN [55] | 95.6 | 9.4 | 88.7 | 19.0 | 93.1 | - | - | - | - | -
CDCD [56] | 96.5 | 9.1 | 89.3 | 18.1 | 92.7 | - | - | - | - | -
LORIS [44] | 98.6 | 6.1 | 90.8 | 13.9 | 94.5 | - | - | - | - | -
Trained single task | 93.9 | 9.2 | 93.6 | 12.8 | 92.8 | 84.84 | 21.61 | 51.35 | 27.13 | 63.97
M3GPT (Pre-trained) | 93.4 | 10.9 | 93.8 | 11.5 | 94.2 | 83.16 | 19.95 | 73.65 | 23.90 | 78.12
M3GPT (Fine-tuned) | 93.6 | 10.1 | 94.0 | 10.6 | 94.9 | 84.10 | 18.36 | 74.66 | 23.45 | 78.23

Table 10: Comparison of Motion Prediction and Motion In-between on Motion-X
[23].

Methods | Motion Prediction |  | Motion In-between |
---|---|---|---|---
FID $\downarrow$ |  Diversity $\uparrow$ |  ADE $\downarrow$ |  FDE $\downarrow$ |  |  FID $\downarrow$ |  Diversity $\uparrow$ |  ADE $\downarrow$ |
Ground Truth | 0.009 | 2.316 | - | - |  | 0.009 | 2.316 | - |
MDM [37] | 1.028 | 1.746 | 8.057 | 11.266 |  | 0.831 | 1.768 | 6.542 |
Trained single task | 0.774 | 1.778 | 7.840 | 9.575 |  | 0.692 | 1.810 | 6.690 |
M3GPT-pretrain | 0.707 | 1.874 | 6.954 | 8.684 |  | 0.604 | 1.897 | 5.692 |
M3GPT-finetune | 0.682 | 1.838 | 6.898 | 8.091 |  | 0.612 | 1.900 | 5.649 |

##  Appendix D Qualitative Results

Fig. 6 presents visualizations for a variety of tasks, including text-to-
motion, motion-to-text, motion prediction, motion in-between, music-to-dance,
long-term dance generation, and music-text conditioned dance generation. The
visualization results show that our method can generate realistic results
across various motion-relevant tasks.

Figure 6: The qualitative results for different motion comprehension and
generation tasks.

##  Appendix E Additional Experiments

The performance of text-to-motion based on different $\lambda$. In Tab. 11,
the performance of M3GPT on Motion-X dataset is analyzed across different
values of $\lambda$ for the text-to-motion task. The results indicate that as
$\lambda$ increases, the model’s recall precision (Top1, Top2, Top3) initially
improves but subsequently declines. The optimal performance is achieved at
$\lambda=0.2$, as evidenced by the highest scores in RPrecision and modality
metrics. Further increases in $\lambda$ to 0.3 and 0.4 lead to deteriorating
performance, particularly in FID and RPrecision, suggesting that excessive
$\lambda$ values may result in over-regularization or reduced adaptability of
the model.

Table 11: Hyper-parameter analysis of $\lambda$. Comparison of Text-to-Motion
on Motion-X [23] with different values of $\lambda$. For this ablation study,
M3GPT is trained solely on the text-to-motion task to examine the impact of
$\lambda$. This study is conducted during the pre-training stage.

Methods |  RPrecision $\uparrow$ | FID $\downarrow$ | MM Dist $\downarrow$ | Diversity $\rightarrow$ | MModality $\uparrow$
---|---|---|---|---|---
Top1 | Top2 | Top3
Real | $\text{0.675}^{\pm\text{0.003}}$ | $\text{0.821}^{\pm\text{0.003}}$ | $\text{0.878}^{\pm\text{0.002}}$ | $\text{0.009}^{\pm\text{0.000}}$ | $\text{2.938}^{\pm\text{0.007}}$ | $\text{2.316}^{\pm\text{0.011}}$ | -
$\lambda$=0.0 | $\text{0.645}^{\pm\text{0.002}}$ | $\text{0.778}^{\pm\text{0.003}}$ | $\text{0.826}^{\pm\text{0.002}}$ | $\text{0.081}^{\pm\text{0.002}}$ | $\text{2.944}^{\pm\text{0.002}}$ | $\text{2.124}^{\pm\text{0.011}}$ | $\text{2.025}^{\pm\text{0.021}}$
$\lambda$=0.1 | $\text{0.649}^{\pm\text{0.002}}$ | $\text{0.787}^{\pm\text{0.002}}$ | $\text{0.838}^{\pm\text{0.002}}$ | $\textbf{0.078}^{\pm\text{0.002}}$ | $\text{2.944}^{\pm\text{0.003}}$ | $\text{2.128}^{\pm\text{0.022}}$ | $\text{2.039}^{\pm\text{0.049}}$
$\lambda$=0.2 | $\textbf{0.656}^{\pm\text{0.002}}$ | $\textbf{0.795}^{\pm\text{0.001}}$ | $\textbf{0.843}^{\pm\text{0.001}}$ | $\textbf{0.078}^{\pm\text{0.000}}$ | $\text{2.942}^{\pm\text{0.001}}$ | $\textbf{2.133}^{\pm\text{0.012}}$ | $\textbf{2.046}^{\pm\text{0.052}}$
$\lambda$=0.3 | $\text{0.629}^{\pm\text{0.002}}$ | $\text{0.716}^{\pm\text{0.002}}$ | $\text{0.784}^{\pm\text{0.001}}$ | $\text{0.095}^{\pm\text{0.002}}$ | $\textbf{2.941}^{\pm\text{0.001}}$ | $\text{2.113}^{\pm\text{0.037}}$ | $\text{2.036}^{\pm\text{0.041}}$
$\lambda$=0.4 | $\text{0.573}^{\pm\text{0.001}}$ | $\text{0.725}^{\pm\text{0.002}}$ | $\text{0.793}^{\pm\text{0.002}}$ | $\text{0.102}^{\pm\text{0.002}}$ | $\text{2.942}^{\pm\text{0.002}}$ | $\text{2.090}^{\pm\text{0.007}}$ | $\text{2.030}^{\pm\text{0.086}}$