# Vision-based Human Pose Estimation via Deep Learning: A Survey

Gongjin Lan∗ ID , , Yu Wu, Fei Hu, , Qi Hao∗ ID This work is partially
supported by the National Natural Science Foundation of China (No: 61773197),
the Shenzhen Fundamental Research Program (No: JCYJ20200109141622964), the
Intel ICRI-IACV Research Fund ($CG\\#52514373$). (Corresponding authors:
Gongjin Lan; Qi Hao.)Gongjin Lan, Yu Wu, Qi Hao are with the Department of
Computer Science and Engineering, Southern University of Science and
Technology, Shenzhen, 518055, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>hao.q@sustech.edu.cn)Fei Hu is with the Department of
Electrical and Computer Engineering, University of Alabama, Tuscaloosa, AL
(email<EMAIL_ADDRESS>
###### Abstract

Human Pose Estimation (HPE) has attracted a significant amount of attention
from the computer vision community in the past decades. Moreover, HPE has been
applied to various domains such as human-computer interaction, sports
analysis, and human tracking via images and videos. Recently, deep learning-
based approaches have shown state-of-the-art performance in HPE-based
applications. Although deep learning-based approaches have achieved remarkable
performance in HPE, a comprehensive review of deep learning-based HPE methods
remains lacking in the literature. In this paper, we provide an up-to-date and
in-depth overview of the deep learning approaches in vision-based HPE. We
summarize these methods of 2D and 3D HPE, and their applications, discuss the
challenges and the research trends through bibliometrics and provide
insightful recommendations for future research. This article provides a
meaningful overview as introductory material for beginners to deep learning-
based HPE, as well as supplementary material for advanced researchers.

###### Index Terms:

Human pose estimation, Human performance assessment, Deep learning, Action
recognition, Bibliometric.

##  I Introduction

Human Pose Estimation (HPE) refers to estimating the positions of human joints
and their associations in images or videos, which is a popular research topic
in computer vision. It has been widely applied to various applications, such
as action analysis [1], HCI [2], gaming [3], sport analysis [4, 5], motion
capture [6], computer-generated imagery [7, 8]. Although HPE has been studied
for decades, it is still an open and challenging task since the two main
aspects of the wide diversity of the human body (such as various human poses,
various clothing, environment or illumination conditions) and reconstruction
ambiguity caused by occlusions (particularly the crowd) [9, 10].

The early HPE approaches often use predefined models and statistical learning
to describe the human poses [11, 12]. However, those methods are incapable of
learning from a large amount of data and suffer from limited model
representation capability. In recent years, deep learning-based approaches
yield great improvements in many computer vision tasks such as classification
[13], object detection [14, 15], and HPE [9]. The success of deep learning in
HPE is mainly due to the following three facts: the availability of big data,
superior representation capability of deep neural networks, and high-
performance hardware (e.g., GPU platform). The deep learning-based methods
dramatically outperform the traditional approaches.

Although there are many promising methods in the vision-based HPE via deep
learning models [9, 16, 17], a lack of articles with an up-to-date and in-
depth review of this domain. We emphasize that a comprehensive overview of HPE
should cover both 2D and 3D HPE studies. In this paper, we aim to provide a
complete and solid survey, analyze the research challenges, and point out the
research trends in HPE. In particular, we apply bibliometrics to retrieve
scientific publications for analyzing the research trends in HPE. This paper
comprehensively reviews HPE topics that cover both 2D and 3D HPE studies,
discusses the challenges, observes the trends, and provides detailed
bibliometrics.

###  I-A Related Work

To date, several survey papers have discussed the related studies in HPE. Dang
et al. [18] provided a survey on deep learning-based 2D HPE, including single-
and multi-person pipelines. Poppe et al. [19] presented an overview of the
literature on vision-based human action recognition. Currently, there are many
studies that investigate monocular HPE, and several survey papers that discuss
the studies in monocular HPE. In [20, 21, 22], the studies of monocular HPE
are reviewed comprehensively, particularly the deep learning-based methods.
Dang et al. [23] provided a comprehensive survey on sensor- or vision-based
human activity recognition. Gadhiya et al. [24] analyzed and compared several
prevalent HPE methods. The latest work [25] reviewed the studies of deep
learning-based 3D HPE. Although these survey papers covered HPE-related
topics, they focus on one of the specific topics in HPE. In this paper, we aim
to provide a comprehensive survey on the vision-based HPE based on deep
learning models in terms of 2D and 3D HPE.

###  I-B Contributions

In this paper, we aim to provide a detailed overview of the existing studies
on deep learning-based human pose estimation. This review has three
objectives:

  * •

Delineate the picture of this field from a ‘helicopter view’.

  * •

Clarify main research streams and provide a complete overview of vision- and
deep learning-based HPE.

  * •

Discuss the challenges and the research trends through bibliometrics, and
provide insightful recommendations for future research.

This survey covers both 2D- and 3D-based approaches. Figure 1 shows the
taxonomy of the approaches (e.g., image-based or video-based, 2D HPE or 3D
HPE, monocular or multi-view), applications, trends, and challenges in this
survey. These contributions provide our survey with a more solid, up-to-date,
and in-depth insight than the existing survey papers.

Figure 1: Taxonomy of this survey.

The rest of this paper is organized as follows. In section II, we present the
preliminary knowledge, common datasets, and the metrics for HPE. Image-based
2D HPE is summarized in section III. We address video-based 2D HPE in
subsection III-C. The studies of 3D HPE is addressed in section IV. The
applications of vision-based HPE using deep learning are presented in section
V. Finally, the research trends and challenges are discussed in section VI,
followed by the conclusions in section VII.

##  II Preliminary Knowledge

In this section, we introduce the preliminary knowledge, including essential
concepts to guide the readers on the big picture of HPE and describe the
solution representation and the well-known datasets with the performance
metrics.

2D/3D | Type | Year | Dataset | URL (Open dataset) | Data Scale | #Joints | Metrics
---|---|---|---|---|---|---|---
2D |  Single- Person | 2010 |  LSP [26] | http://sam.johnson.io/research/lsp.html | 2K images | 14 | PCK&PCP
2013 |  FLIC [27] | https://bensapp.github.io/flic-dataset.html | 5K images | 10 | PCK&PCP
2013 |  J-HMDB [28] | http://jhmdb.is.tue.mpg.de/ | 928 video clips | 15 | PCK&PCP
2013 |  PennAction [29] | http://dreamdragon.github.io/PennAction/ | 2326 video clips | 13 | PCK&PCP
Multi- Person | 2014 |  MPII [30] | http://human-pose.mpi-inf.mpg.de/ | 25K images & 40K persons | 16 | mAP
2016 |  COCO [31] | https://cocodataset.org/#home | 330K images & 250K persons | 17 | AP & AR
2018 |  PoseTrack [16] | https://github.com/umariqb/PoseTrack-CVPR2017 | 46K frames & 276K persons | 15 | mAP
2019 |  CrowdPose [32] | https://github.com/Jeff-sjtu/CrowdPose | 20K images & 80K persons | 14 | mAP
3D |  Single- Person | 2014 |  Human3.6M [33] | http://vision.imar.ro/human3.6m/ | 3.6M frames & 4 camera views | 17 | MPJPE
Multi- Person | 2017 |  CMU Panoptic [34] | http://domedb.perception.cs.cmu.edu/ | 1.5M frames & 512 camera views | 15 | MPJPE
2018 |  3DPW [35] | https://virtualhumans.mpi-inf.mpg.de/3DPW/ | 51K frames & 1 camera view | 18 | MPJPE
| 2017 |  MPI-INF-3DHP [36] | https://vcai.mpi-inf.mpg.de/3dhp-dataset/ | 1.3M frames & 16 camera views | 15 | MPJPE
| 2018 |  JTA [37] | https://github.com/fabbrimatteo/JTA-Dataset | 460K frames & 1 camera view | 14 | MPJPE
TABLE I: Illustration of the well-known datasets in 2D and 3D HPE.

###  II-A Deep Learning in HPE

Currently, deep learning-based approaches have become state-of-the-art methods
in HPE. The availability of big datasets, advanced hardware like GPU, and the
surpassing performance of deep neural networks lead to the increasing interest
in deep learning-based HPE. In this subsection, we discuss three topical types
of neural networks in HPE: Convolutional Neural Networks (CNNs), Recurrent
Neural Networks (RNNs), and Graph Convolutional Networks (GCNs).

####  II-A1 Convolutional Neural Networks

In general, a CNN for HPE tasks consists of two parts. The first part commonly
uses off-the-shelf generic pre-trained networks such as ResNet [38] to extract
features, the so-called backbone network. The second part, called the
prediction head, predicts human poses with the extracted features.

The well-known networks like AlexNet [13] and ResNet [38] show remarkable
classification performance on the dataset of ImageNet, and advanced
performance in HPE [39, 40] as well. However, there is a gap between
classification and HPE tasks since their targeted features and prediction
differ from each other. Instead of directly using the backbones from
classification tasks, HPE-specified backbones need to be improved for HPE
tasks. For example, Hourglass [41], Cascaded Pyramid Network (CPN) [42], and
HRNet [10] are proposed to be the backbone for the deep learning-based
approaches in HPE.

For prediction heads, there are mainly two representative solutions in HPE.
The one directly predicts joint coordinates, which is regarded as the
regression paradigm. The other one generates an intermediate heatmap
representation before computing joint coordinates. For the regression
paradigm, fully connected layers are often adopted to regress concrete
keypoint coordinates. For the heatmap prediction paradigm, the operation of
upsampling [42, 41, 43] is generally used to generate higher resolution
heatmaps.

####  II-A2 Recurrent Neural Networks

Recurrent neural networks stake temporal information among sequential inputs
into consideration. They are widely used in video-based HPE by considering
videos as sequential RGB images. We present the general pipeline of this type
of networks in Figure 3. RNN-based methods perform robust and the state-of-
the-art accuracy [44, 45].

####  II-A3 Graph Convolutional Networks

Instead of taking images as input, graph convolutional networks (GCNs) take
graphs as input. As a human skeleton can be naturally represented as a graph,
GCNs-based methods are prevalent in many skeleton-based tasks. In HPE, GCNs
are generally expected to better exploit the relationship among keypoints and
used for the pose refinement [46], joint association [47], 2D-to-3D pose
lifting [48, 49, 50].

###  II-B Pose Representation

Although the natural representation of human poses (keypoint positions) uses
coordinates in the form of ordered tuples, the existing studies have shown a
significant improvement by representing solutions with heatmaps that can be
regarded as a confidence map. Currently, the heatmap representation has become
a prevalent solution representation in HPE. In this subsection, we describe
how heatmap representation works in the single-person and multi-person HPE.

####  II-B1 Heatmaps in Single-person HPE

To enable neural networks for the heatmap prediction of human joints, the
ground truth of a heatmap is essential. An intuitive way of producing the
ground truth is to design a probability heatmap for each keypoint. For
example, for single-person pose estimation tasks, the heatmaps of
$\mathcal{K}$ keypoints can be defined as $\mathcal{K}$ matrices with the
identical size to the input image $x$. The value of position $p$ (noted as
$\mathcal{H}_{k}(p)$) can be generated by a 2D Gaussian distribution centered
at the position of joint $k$ in $x$.

$\small\mathcal{H}_{k}(p)=e^{\frac{\|p-p_{k}^{*}\|_{2}^{2}}{\sigma^{2}}},\forall
k=1,2,\ldots,\mathcal{K}$ (1)

where $p_{k}^{*}$ is the position of joint $k$ in $x$. In a heatmap, the
predicted position (noted as $p_{pred}^{*}$) can be calculated by regression
models as:

$\small p_{pred}^{*}=\sum_{p\in\mathcal{P}}\mathcal{H}_{k}(p)*p,$ (2)

where $\mathcal{P}$ is the set of the possible positions of joint $k$.

####  II-B2 Heatmaps in Multi-person HPE

There are two primary patterns of heatmaps in multi-person scenarios. One is
to generate $\mathcal{N}\times\mathcal{K}$ heatmaps for $\mathcal{N}$ persons
and their $\mathcal{K}$ joints separately. The other is to generate
$\mathcal{K}$ heatmaps for $\mathcal{K}$ joint of all persons. For the former
pattern ($\mathcal{N}\times\mathcal{K}$ heatmaps), each value at the position
$p$ in a heatmap (noted as $\mathcal{H}_{k,n}(p)$) can be calculated as:

$\small\mathcal{H}_{k,n}(p)=e^{\frac{\|p-p_{k,n}^{*}\|_{2}^{2}}{\sigma^{2}}},~{}\forall
k=1,2,\ldots,\mathcal{K},~{}\forall n=1,2,\ldots,\mathcal{N}$ (3)

where $\mathcal{K}$ and $\mathcal{N}$ are the numbers of joints and persons,
respectively. For the latter pattern, a general method to produce the ground
truth heatmaps is aggregating $\mathcal{N}$ single-person heatmaps into one
heatmap by using a $\max$ operator:

$\small\mathcal{H}_{k}(p)=\max_{n}\mathcal{H}_{k,n}(p),\forall
n=1,2,\ldots,\mathcal{N}$ (4)

In summary, Equation 3 and Equation 4 are the two main methods to calculate
the heatmaps in multi-person HPE.

###  II-C Datasets and Metrics

Datasets are critical for training and evaluating neural networks for deep
learning-based HPE. In this subsection, we introduce the popular datasets and
their applicable tasks, then review the evaluation metrics for image-based 2D
HPE, video-based 2D HPE, and 3D HPE.

####  II-C1 Datasets in 2D HPE

Many datasets have been proposed to evaluate the performance of 2D HPE
approaches. Here, we introduce the prevalent datasets used in 2D HPE. Early
datasets like LSP [26], FLIC [27], Penn Action [29], and J-HMDB [28] mainly
focus on single-person scenes with relatively small scales. The recent
datasets such as COCO [31], MPII [30], CrowdPose [32], and PoseTrack [16] are
used for multi-person HPE with larger-scale data. We summarize and provide the
links of the prevalent datasets as well as their scales and the evaluation
metrics in Table I.

####  II-C2 Datasets in 3D HPE

Unlike the datasets in 2D HPE, acquiring accurate 3D annotations for human
joints in 3D HPE often requires a motion capture system that is generally hard
to be installed in the outdoor environment. Most 3D HPE datasets are created
in the indoor environments or simulation, such as CMU Panoptic [34], 3DPW
[35], MPI-INF-3DHP [36], JTA [37]. Here we introduce the well-known datasets
in 3D HPE and summarize their characteristics in Table I.

####  II-C3 Metrics in 2D HPE

In scientific research, we usually need metrics to evaluate how well a method
performs. Here we introduce two common metrics used in HPE. _Percentage of
Correct Keypoints (PCK)_ literally indicates the percentage of correct
detected keypoints, which can be noted as $\mathcal{PCK}=k/\mathcal{N}$, where
k is the number of correct predicted keypoints, $\mathcal{N}$ is the total
number of keypoints. This metric is generally applied to the studies with the
LSP dataset, MPII dataset, and FLIC dataset in the early 2D HPE methods (see
Table II).

Studies | Years | Backbone | Input Size | Highlights |<EMAIL_ADDRESS>| #Params | GFlops
---|---|---|---|---|---|---|---
Toshev & Szegedy [9] | 2014 | AlexNet | $256\times 256$ | Original deep learning-based method for HPE | - | - | -
Tompson et al. [51] | 2014 | AlexNet | $320\times 240$ | Utilization of heatmaps for solution representation | 82.0 | - | -
Wei et al. [52] | 2016 | CPM | $368\times 368$ | A convolutional pose machine | 88.5 | 31.23M | 85.0
Newell et al. [41] | 2016 | Hourglass | $256\times 256$ | Stacked Hourglass Modules | 90.9 | 23.7M | 41.2
Xiao et al. [43] | 2018 | ResNet | $256\times 256$ | A simple yet effective architecture for HPE | 90.2 | 68.64 M | 17.02
Chu et al. [53] | 2017 | Hourglass | $256\times 256$ | Attention mechanism in contextual representations | 91.5 | 58.1M | -
Yang et al. [54] | 2017 | Hourglass | $256\times 256$ | Pyramid Residual Module | 92.0 | 26.9M | 45.9
Sun et al. [10] | 2019 | HRNet | $256\times 256$ | HRnet for high-resolution representations | 92.3 | 28.54M | 10.27
Bulat et al. [55] | 2020 | Hourglass+UNet | $256\times 256$ |  A hybrid structure by combining [41] and U-Net [56] | 94.1 | 8.5M | 9.9
TABLE II: The state-of-the-art approaches for 2D single-person pose
estimation. The<EMAIL_ADDRESS>scores are shown in the last column and defined in
subsubsection II-C3. They are obtained by testing the methods on the MPII
dataset.

_Object Keypoint Similarity (OKS)_ was proposed originally in the COCO
competition [31] as a variable to compute mean Average Precision (mAP). It can
be calculated as follows:

$\small\mathcal{OKS}=\frac{\sum_{i}(-d_{i}^{2}/2s^{2}k_{i}^{2})\delta_{(v_{i}>0)}}{\sum_{i}\delta_{(v_{i}>0)}},\quad
s.t.~{}v_{i}\in\\{0,1,2\\}$ (5)

where $i$ is a joint index, $d_{i}$ is the distance between the predicted
joint and ground truth, $s$ (object scale) and $k_{i}$ are the keypoint
constants given by the COCO dataset, $\delta=1$ when $v_{i}>0$, otherwise
$\delta=1$, and $v_{i}$ is the visibility flag ($v_{i}=0$: not labelled,
$v_{i}=1$: labelled but not visible, and $v_{i}=2$: labelled and visible) of
the ground truth. The average precision (AP) can be calculated with the
$\mathcal{OKS}$ value by:

$\small
AP=\frac{\mathcal{TP}_{(\mathcal{OKS}>td)}}{\mathcal{TP}_{(\mathcal{OKS}>td)}+\mathcal{FP}_{(\mathcal{OKS}\leq
td)}}$ (6)

where $\mathcal{TP}$ and $\mathcal{FP}$ are the numbers of true positive and
false positive respectively, $td$ is a $\mathcal{OKS}$ threshold. The mean AP
(mAP) is the mean of AP over ten $\mathcal{OKS}$ thresholds at (0.50, 0.55, .
. . , 0.90, 0.95) [31], which is a common metric to evaluate 2D multi-person
pose estimation (see Table III).

####  II-C4 Metrics in 3D HPE

Generally, the metrics of PCK can be extended to evaluate 3D HPE. However,
_Mean Per Joint Position Error (MPJPE)_ is currently a popular metric in 3D
HPE (see Table IV). MPJPE calculates the euclidean distance between the
predicted joint coordinates and ground truth joint coordinates, which can be
formulated as:

$\small\text{MPJPE}=\frac{1}{\mathcal{T}\cdot\mathcal{N}}\sum\limits_{t=1}^{\mathcal{T}}\sum\limits_{i=1}^{\mathcal{N}}\left\lVert
J_{i}^{t}-J_{root}^{t}-(\hat{J_{i}}^{t}-\hat{J}_{root}^{t})\right\rVert^{2}$
(7)

Where $i$ and $t$ are the indexes of a joint and a sample respectively;
$J_{i}$ and $\hat{J_{i}}$ refer to the predicted coordinate and ground-truth
coordinate of $i-th$ joint. $J_{root}$ refers to the coordinate of the root
joint, which is usually predefined as the human pelvis.

##  III 2D Human Pose Estimation

The image-based 2D HPE is to estimate human joint positions in images. Early
approaches mainly use model-based methods. Currently, deep learning-based
methods have shown superior performance in 2D HPE [9]. In this section, we
introduce the deep learning-based 2D HPE approaches from three aspects: image-
based single-person pose estimation (SPPE), image-based multi-person pose
estimation (MPPE), and video-based 2D HPE. Finally, we summarize the open-
source codes of the state-of-the-art 2D HPE in Table VI.

###  III-A Image-based 2D Single-Person HPE

The SPPE task is to estimate the pose of a single person in an image. It is a
fundamental task in HPE and is often used as a basic component of other HPE
tasks. For example, the well-known study [9] proposed a cascaded multi-stage
neural network to predict and refine human poses, in which images are cropped
to ensure a single person in each image. To our knowledge, it is the original
work that applies deep neural networks (DNN) for the HPE.

Recent DNN-based approaches dramatically improve the accuracy of SPPE. Many
generic neural networks with efficient architecture show remarkable
performance in various applications [13, 38]. There are mainly two ways to
improve SPPE performance: improving the solution representation and designing
advanced neural networks. In [51, 57, 58], the methods with solution
representation significantly improved the SPPE performance. In addition, well-
designed neural architectures [10, 52] also performed remarkable performance
in HPE. We summarize the state-of-the-art approaches for 2D single-person pose
estimation in Table II.

####  III-A1 Solution Representation

In general, there are mainly two types of solution representations. One is to
represent joint positions by using coordinates. The other one is to represent
joint positions in the form of probability distributions, i.e., heatmaps as
represented in Equation 1. Early studies mainly used a DNN-based regression
from the input images to the estimated coordinates directly. DeepPose [9] is
an early classic study that applied DNNs to represent coordinate regression
from RGB images and multi-stage refinement for the SPPE tasks.

Solution representation of heatmaps significantly improves the performance of
DNN-based methods in HPE [56]. The methods of heatmap generation have been
widely used in HPE tasks since heatmaps were proposed by Tompson et al. [51]
and Jain et al. [57]. Moreover, Zhang et al. [58] improved the heatmap
representation by the distribution-aware coordinate representation of
keypoints (DARK).

####  III-A2 Neural Network Design

How to design the architecture of neural networks in HPE is a crucial topic. A
generic way is to use the existing well-known neural networks (e.g., AlexNet
[13], ResNet [38]) that have already shown superior performance in other
computer vision tasks. For example, Toshev et al. [9] used an AlexNet-like to
regress the coordinates of human joints from images. Xiao et al. [43] used
ResNet as the backbone of neural networks for HPE.

The latest neural network, the transformer that adopts the attention
mechanism, has been applied in HPE with superior performance. In general,
transformers process the features extracted by CNNs and leverage the attention
mechanism to automatically capture long-range relationships among features. In
2021, Mao et al.[59] used ResNet to extract features from images and a
Transformer model to predict keypoint positions. Li et al.[60] utilized vision
transformers to implement regression-based HPE, where two cascaded
transformers are applied to predict the bounding boxes of the person as well
as their keypoints positions. Yang et al. [61] proposed a new network model,
TransPose which introduces the transformer for human pose estimation. The
attention layers built-in transformers enable TransPose to capture long-range
relationships efficiently. In addition, Xu et al. proposed a scalable HPE
backbone, VitPose [62] that employs plain and non-hierarchical vision
transformers as backbones to extract features for a given person instance and
a lightweight decoder for pose estimation. To date, the existing survey papers
have not discussed the studies with transformers in HPE.

Another generic way is to design specific networks for HPE tasks. For example,
Newell et al. [41] designed a stacked hourglass network as the backbone for
exacting multi-scale features automatically. Hourglass Networks are used in
many HPE studies as backbone networks. Newell et al. [63] used the Hourglass
Network backbone to estimate multi-person poses. BlazePose [64] utilized an
Hourglass-based encoder network to accomplish real-time single-person pose
estimation driven by the MediaPipe framework [65], which achieved a speed of
30 frames per second on the Google Pixel 2 mobile phone. Bulat et al. [55]
designed a hybrid network structure with a combination of the hourglass
network and U-Net [56]. Chen et al. [42] presented another remarkable study
that a cascaded pyramid network is proposed to concatenate features among
different scales. In addition, many studies enhanced the model performance by
slightly modifying the units of a backbone. Specifically, Chu et al. [53]
incorporated the attention mechanism into the hourglass network. Yang et al.
[54] proposed a pyramid residual module to replace the residual module in the
hourglass network.

###  III-B Image-based 2D Multi-Person HPE

In general, MPPE is a more challenging task than SPPE because of the high
complexity in terms of solution space and mutual occlusions.

Input Image Detected Bounding Boxes Human Poses Input Image Detected Keypoints
Human Poses

Figure 2: Illustration of the top-down and bottom-up framework. The first row
shows a representative top-down framework with two stages: proposing a
bounding box for each person and estimating human poses. The second row shows
a representative bottom-up framework with two stages: detecting the joints of
persons and grouping the joints into an associated person. The original images
are from COCO dataset [31].

The current solutions of MPPE generally fall into either top-down approaches
or bottom-up approaches:

  * •

Top-down approaches apply a person detector to detect all single-persons,
followed by estimating the joints of each single-person and calculating each
single-person pose separately.

  * •

Bottom-up approaches detect all joints in an image, followed by
associating/grouping the joints into an associated person.

In this subsection, we address the top-down and bottom-up approaches and
discuss their merits and flaws. We illustrate the frameworks of both
approaches in Figure 2 and summarize the recent well-known MPPE approaches in
Table III.

Methods | Studies/Years | Input Size | Backbone |  Detector/ Grouping | Highlights | mAP | #Params | GFlops
---|---|---|---|---|---|---|---|---
Top-down |  [66] / 2017 | - | ResNet | Mask R-CNN |  Detecting persons and keypoints by mask R-CNN | 63.1 | - | -
[39] / 2017 | $320\times 256$ | PyraNet | Faster R-CNN |  Addressing inaccurate bounding box of persons | 72.3 | 28.1M | 26.7
[42] / 2018 | $384\times 288$ | CPN | FPN |  A two-stage cascaded network to refine keypoints | 72.1 | 27.0M | 6.2
[43] / 2018 | $384\times 288$ | ResNet | Faster R-CNN |  A remarkable simple baseline | 72.2 | - | -
[10] / 2019 | $384\times 288$ | HRNet | Faster R-CNN |  HRnet for high-resolution representations | 75.5 | 63.6M | 32.9
[67] / 2019 | $384\times 288$ | MSPN | MegDet |  A multi-stage network | 76.1 | 120M | 19.9
[17] / 2020 | $512\times 384$ | EvoPose2D | Faster R-CNN |  Neural Architecture Search | 76.8 | 14.7M | 17.7
[46] / 2020 | $384\times 384$ | ResNet | FPN |  GCN-based keypoint refinement | 72.9 | 25.2M | 12.9
[59] / 2021 | $384\times 288$ |  ResNet+ Transformer | Faster R-CNN |  Direct Coordinate Regression via Transformer | 72.2 | - | 20.4G
[61] / 2021 | $256\times 192$ | TransPose | Faster R-CNN |  Transformer decoders with CNNs-based extractor | 75.8 | 17.5M | 21.8G
[62] / 2022 | $256\times 192$ | VitPose | Faster R-CNN |  Vision Transformer Backbone Baseline | 79.8 | 632M | -
Bottom-up |  [68] / 2016 | $256\times 256$ | ResNet |  Integer Programming |  ResNet-based detectors and image-conditioned pairwise terms | - | - | -
[40] / 2017 | $256\times 256$ | VGG+CPM |  Part Affinity Fields |  Grouping by body association and Hungarian algo. | 61.8 | 25.94M | 160
[63] / 2017 | $512\times 512$ | Hourglass |  Associative embedding |  pixel-wise joint embedding for grouping | 66.3 | 277.8M | 206.9
[69] / 2019 | $384\times 384$ | Hourglass | $\emptyset$ |  Predicting root and joints position | 66.9 | - | -
[47] / 2020 | $641\times 641$ | ResNet | DGCN |  Graph convolutional network for grouping | 68.8 | 234M | -
[70] / 2020 | $512\times 512$ | Hourglass |  Associative embedding |  Graph Clustering for grouping | 68.3 | - | -
[71] / 2018 | $480\times 480$ | ResNet+FPN |  Person Detection |  Pose Residual Network assigns keypoints to instances | 69.6 | - | -
[72] / 2020 | $640\times 640$ | HigherHRNet |  Associative embedding |  HigherHRNet for the scale variation challenge | 70.5 | 63.8M | 154.3
TABLE III: The state-of-the-art studies for image-based 2D multi-person HPE.
The mAP scores are defined in subsubsection II-C3, are obtained by testing on
the COCO dataset.

####  III-B1 Top-down approaches

The top-down method is an effective, popular method in 2D multi-person pose
estimation tasks. By using a pretrained human detector to crop images in top-
down approaches, the multi-person tasks can be converted into single-person
tasks. The performance of top-down methods can be enhanced with the
improvements of both human detector and single-person pose estimator. He et
al. [66] showed that multi-person pose estimation can be implemented by
extending HPE tasks to the detection tasks. Fang et al. [39] proposed a
regional multi-person pose estimation framework to improve the performance of
the human detector. Furthermore, the improvement of single-person pose
estimators shows benefits for multi-person pose estimation [58, 73, 42].

In general, the top-down approaches show advanced performance on datasets and
can be easily implemented by combining the existing detectors and SPPE models.
However, the computing of top-down approaches is significantly increasing over
the number of detected persons, which limits its real-time performance for
multiple-person scenarios. Therefore, the top-down approaches are hardly
applied to the real-time HPE tasks in complex scenarios, particularly the
crowd scenes.

####  III-B2 Bottom-up approaches

Instead of performing keypoints detection in the proposed bounding boxes, the
bottom-up methods often consist of two parts: keypoints detection and
keypoints grouping. For keypoints detection, body keypoints of all persons in
an image are detected directly by the heatmaps described in Equation 3. for
keypoints grouping, the detected keypoints need to be grouped into the single-
person. The deep neural networks are generally applied to assign keypoints to
the proposed bounding boxes of the detected person. Newell et al. [63]
introduced associative embedding to train neural networks for assigning
keypoints to each person. Jin et al. [70] applied graph neural networks to
group the detected joints.

Moreover, it is shown in [72] that the bottom-up approaches are more robust
than the top-down approaches when applied to crowded scenes, which is crucial
for practical applications. However, the top-down methods achieve better
performance in terms of accuracy, while the computing time inevitably
increases as the number of detected persons increases. By contrast, the
bottom-up approaches take relatively constant computing time for multi-person
HPE [40, 69], which is much less sensitive to the number of targeted persons.
In conclusion, the bottom-up approaches are conducive to real-time multi-
person pose estimation on the low-performance hardware platform.

###  III-C Video-based 2D HPE

The video-based 2D HPE is generally a more complicated task than the image-
based 2D HPE. A pipeline of the generic video-based 2D HPE is shown in Figure
3. Unlike static images, video frames are likely to involve the problem of
image degeneration such as motion blur and video defocus. Although the video-
based 2D HPE performance may degenerate because of motion blur, video defocus,
and temporary occlusions, these video-based approaches generally surpass
image-based approaches by capturing temporal information. The correlations
among video frames can be used to further improve the self-supervised
approaches in HPE.

Figure 3: Illustration of a video-based 2D HPE pipeline where a Neural
Networks-based (NNs-based) module is utilized to extract temporal information.
The original images are from the Penn Action dataset [29].

In video-based 2D HPE, it is costly to manually annotate human joints in each
frame of video, which restricts the obtaining of large-scale datasets for the
video-based HPE. To solve this issue, the temporal correlation and consistency
among the sequences need to be fully investigated. Jain et al. [57] originally
proposed a CNN-based approach to combine RGB images and motion features for
improving both the accuracy and speed of HPE. Specially, the recurrent neural
networks, such as the well-known long short-term memory (LSTM), perform
remarkably to capture temporal consistency among frames [44, 45]. Nie et al.
[74] proposed a dynamic kernel distillation model to distil previous temporal
cues among frames. Although these methods perform state-of-the-art accuracy,
they are supervised learning-based by using large-scale and densely labelled
data in video-based HPE. By contrast, semi-supervised learning is useful for
video-based 2D HPE with sparsely labelled data since labelling large-scale and
densely data is costly and labour-intensive. To this end, Bertasius et al.
[75] proposed PoseWarper propagate pose information in sparsely labelled (each
k frames) videos.

Finally, we collect the open-source implementations of well-known state-of-
the-art 2D HPE works, as shown in Table VI. We summarize the prevalent real-
time open-source implementations for practical applications (e.g., OpenPose,
AlphaPose, a lightweight version of OpenPose, and BlazePose [64] driven by the
framework of MediaPipe [65] ), which generally offer open-source codes to
users.

###  III-D Robustness Analysis

Robustness is a crucial property that should be considered in deep learning-
based methods. We review the studies of robustness analysis for HPE in this
subsection. Currently, robustness analysis is usually applied to 2D HPE but
rarely considered in 3D HPE. Deep learning-based methods are generally
sensitive and vulnerable to the attack of adversarial samples.

Although the robustness of HPE is rarely studied, it is crucial to be
considered in the design and evaluation of HPE methods. In [41], it is
demonstrated that the PCK of the Hourglass network dramatically decrease from
$89.4$ to $0.57$ in the testing with adversarial samples. Wang et al. [76]
proposed the new datasets COCO-C, MPIIC, and OCHuman-C, which were
reconstructed on the basis of COCO [31], MPII [30], and OCHuman [77] for
evaluating the robustness of HPE methods. Shah et al. [78] comprehensively
investigated the adversarial robustness of HPE methods. The experimental
results show that 1) heatmap-based methods perform more robust than
regression-based methods, and 2) different body joints generally perform
different robustness to the attacks of adversarial samples. For example, the
head and neck exhibited prominent robustness, while the joints of the hips,
knees, and ankles are sensitive to disturbances. These works revealed the
robustness of the existing deep learning-based HPE methods.

##  IV 3D Human Pose Estimation

The 3D HPE is to predict the 3D positions of human joints. It is a challenging
task due to the large solution space and inherent ambiguity. Moreover, the
lack of outdoor large-scale 3D HPE datasets challenges the practical
performance of 3D HPE approaches since the existing datasets 3D HPE is mostly
collected by indoor motion capture systems [33, 34]. Early studies either
implemented 3D HPE with model-based approaches [79, 80] or regarded the task
as a regression problem which can be solved by optimization algorithms. Since
the DNN-based methods [81, 82] outperform the previous works by automatically
learning representations from large-scale data, deep learning-based methods
have become the most popular methods in 3D HPE as well.

In this section, we review deep learning-based 3D HPE studies as shown in
Table IV. According to the characteristics of inputs, we categorize the 3D HPE
into three types: monocular 3D HPE, multi-view 3D HPE, and multimodal 3D HPE.
Finally, we present a collection of the open-source code of the state-of-the-
art 3D HPE approaches in Table VI.

Views | Modality | Studies | Methods | Highlights | MPJPE | Dataset
---|---|---|---|---|---|---
Monocular | Vision |  Ching-Hang et al. [82] | CNN | A matching method to lift poses | 82.72 | Human3.6m
Julieta et al. [83] | CNN | Off-the-shelf detectors & lifting networks | 87.3
Dushyant et al. [84] | CNN | Occlusion-robust pose-maps | 69.9
Zhao et al. [48] | CNN+GCN | An novel SemGCN for 2D-3D lifting. | 60.8
Shichao et al. [85] | CNN |  Evolutionary 2D-3D data augmentation & Cascaded 2D-3D lifting networks | 50.9
Kehong et al. [86] | CNN | Differentiable pose augmentor for 2D-3D lifting | 50.2
Yujun et al. $\cite[cite]{[\@@bibref{}{cai2019exploiting}{}{}]}^{\dagger}$ | CNN+GCN | GCN-based 2D-3D sequence lifting. | 48.8
|  Wenbo et al. $\cite[cite]{[\@@bibref{}{hu2021conditional}{}{}]}^{\dagger}$ | CNN+GCN |  Graph for skeleton representation & GCN for 2D-3D sequence lifting | 41.1 |
|  Vision & IMUs |  Timo et al. [35] | CNN | Video Inertial Poser to fusing images and IMUs | 26 | 3DPW
Multi-view | Vision |  Karim et al. [87] | CNN | An end-to-end DNN triangulation method | 17.7 | Human3.6m
Zhang et al. [88] | CNN | An adaptive multi-view fusion method | 19.5
Yihui et al. [89] | CNN | An epipolar transformer | 26.9
Haibo et al. [90] | CNN | A cross-view fusion network | 31.17
Size et al. [91] | CNN+GCN |  Learnable association matching & graph-based 3D pose refinement | 15.84 | CMU Panoptic
Vision & IMUs |  Trumble et al. [92] | CNN | A two-stream network fuses video and IMUs | 87.3 | Human3.6m
Gilbert et al. [93] | CNN |  Incorporating [92] with enhanced 3D HPE | 71.9
Zhang et al. [94] | CNN | An orientation regularized pictorial model | 24.6
Malleson et al. [95] | CNN | Optimization based on IMUs and poses | 62 | Total Capture
|  Huang et al. [96] | CNN | DeepFuse for vision-IMU data fusion | 28.9
TABLE IV: The state-of-the-art approaches for image-based 3D HPE. The MPJPE
scores that are defined in subsubsection II-C4, can be obtained by testing the
methods on the Human3.6M dataset. The symbol ${\dagger}$ denotes the methods
of using image sequence.

###  IV-A Monocular 3D HPE

The monocular 3D HPE task is to predict the 3D positions of human joints for a
monocular image, which is known to be an ill-posed problem. DNNs have shown
remarkable performance on predicting the depth of monocular images [97] and
the monocular 3D HPE tasks [83, 98]. The monocular 3D HPE methods can be
generally categorized into two types: the single-stage and the two-stage
methods. The basic difference between these two types is that two-stage
methods entail using off-the-shelf 2D predictors with the 2D HPE datasets. The
single-stage methods predict 3D poses from images directly. By contrast, two-
stage methods estimate 2D poses and then lift the 2D poses to 3D poses. In
this subsection, we summarize the state-of-the-art works of both methods for
monocular 3D HPE.

####  IV-A1 Single-stage Approaches

As DNN-based methods can automatically build mappings from 2D images to 3D
poses, the single-stage 3D HPE can be generally viewed as an extension of 2D
HPE. Early works applied the regression paradigm to estimate 3D human pose
directly. Li et al. [81] originally applied an end-to-end approach with the
deep neural network for 3D HPE by the combination of a joint detector and a
joint regressor. Luvizon et al. [99] proposed a multi-task framework to
perform multi-task learning, which predicts 3D poses directly.

Compared to the early studies, there are many current single-stage approaches
that apply heatmap representation to 3D HPE. For example, Pavlakos et al.
[100] proposed a single-stage method that predicted 3D heatmaps in voxel space
and proposed a coarse-to-fine prediction scheme to reduce the large 3D heatmap
cost. To reduce the computational cost of direct predicting 3D voxel heatmaps,
Nibali et al. [101] predicted the marginal 2D heatmaps to produce 3D
coordinates.

For the single-stage methods in multi-person 3D HPE, they can be similarly
categorized into the top-down and bottom-up methods, recalling the Figure 2.
The top-down methods directly predict 3D poses of the detected person in each
proposed bounding box. Bottom-up methods estimate the 3D pose by detecting all
3D joints and grouping all joints into person-specific sets. Typically, Fabbri
et.al [102] proposed a volumetric heatmap autoencoder to estimate 3D joint
locations and employed a distance-based heuristic strategy to associate the
body joints of a person. In summary, the framework of the top-down and bottom-
up methods in monocular multi-person 3D HPE is similar to the frameworks in
Figure 2, while the joint points are distributed in the 3D space.

####  IV-A2 Two-stage Approaches

Although the single-stage methods are efficient, the two-stage methods
generally yield better performance as they could further benefit from 2D
information and large-scale 2D HPE datasets. Many studies utilize off-the-
shelf 2D estimators to produce 2D poses and followed by a 2D-3D lifting
method. For example, Martinez et al. [83] used an off-the-shelf 2D estimator
to produce 2D poses and then employed a simple lifting neural network to
predict 3D poses on the basis of 2D poses. Zheng et al. [103] utilized vision
transformer to implement the 3D lifting from 2D pose sequence. For two-stage
methods, weak-supervised approaches are often used to leverage the 2D-to-3D
constraint [73]. Pavllo et al. [104] fuse 2D poses sequence from videos to
predict accurate 3D poses that exploit unlabeled videos in a semi-supervised
manner.

In multi-person 3D HPE, the two-stage methods also employ the top-down and
bottom-up paradigms yet incorporate the two-stage process. For the top-down
methods, Rogez et al. [105] employed a region proposal network to propose
bounding boxes for targeted persons and developed a pose proposal network to
estimate the human poses in these bounding boxes. Zanfir et al. [106]
performed bottom-up 2D MPPE and then recovered 3D poses from 2D poses by a 3D
pose decoding module.

Many approaches conducted the two-stage approaches of graph neural networks to
the lifting in 3D HPE. For two-stage approaches, a GCN-based lifting network
generally takes the graph from 2D poses as the input to produce 3D poses. Zhao
et al. [48] proposed a semantic graph convolutional network to regress 3D
joint coordinates from the 2D joint coordinates. Hu et al. [49] proposed a
directed graph-based skeleton representation and applied a graph convolutional
network to exploit both spatial and temporal information of image sequences.
The two-stage GCN-based HPE approaches show competitive performance by
considering the graph of the prior knowledge of the human skeleton. However,
GCNs-based approaches generally integrate other DNNs-based models to extract
2D poses in the first stage as the input of GCNs (the second stage), which
limits the usage of GCNs in the HPE field.

###  IV-B Multi-view 3D HPE

The multi-view 3D HPE tasks aim to predict 3D joints position with synchronous
multi-view cameras. Although the classic triangulation method (the details
refer to [107]) can calculate accurate 3D object locations via multi-view
camera systems, the triangulation method performs a drawback of noticeable
sensitivity to inaccurate 2D prediction. Alleviating the error of 2D
prediction is a critical problem of multi-view 3D HPE. Iskakov et al. [87]
proposed an end-to-end DNN-based learnable triangulation method to produce
confidence weights for each view. Qiu et al. [90] proposed a multi-view fusion
layer to improve 2D pose estimation and used a recursive pictorial structure
model to predict 3D pose. He et al. [89] introduced epipolar geometry to
multi-view fusion, which significantly decreased the number of parameters in
the fusion module.

Similar to the multi-person HPE, the existing multi-view 3D HPE can be
categorized into two types: the top-down and bottom-up methods. The top-down
methods predict the 2D poses of each view and match the 2D poses to perform
the 3D reconstruction. Dong et al. [108] proposed a multi-way matching
algorithm that combined both geometric and appearance cues to match the
detected 2D poses across views. Chen et al. [109] utilized the temporal
consistency to match multi-view 2D poses to 3D poses, and retained and updated
the 3D poses by the cross-view multi-person tracking. Huang et al. [110]
proposed a dynamic matching algorithm to match corresponding multi-view 2D
poses from different views for each person and then used point triangulation
to recover 3D poses. Currently, there are a few studies that apply bottom-up
approaches to multi-view 3D HPE. Elmi et al. [111] originally applied a
bottom-up approach to the multi-view 3D HPE. The 2D features of each image
were processed by a backbone network and then aggregated by an un-projection
layer into a 3D input representation. Finally, a sub-voxel joint detection
module and a skeleton decoder module were employed to produce a set of 3D
poses.

###  IV-C Multimodal Learning in 3D HPE

Multimodal learning is a deep learning-based approach that builds models with
multiple modalities. A modality often refers to a sensory modality such as
vision, touch signal, or radio signal. With the fusion of multiple sensors,
multimodal learning approaches are more robust and capable of overcoming the
challenges like occlusions than vision methods [112]. Although researchers
have shown increasing interest in the field of multimodal learning approaches
for HPE, a few studies apply multimodal learning to implement 3D HPE. We
retrieve the related studies of multimodel learning in the Scopus database, as
shown in 4(d).

In this subsection, we review the related deep learning-based approaches that
utilize 2D vision data, depth information, and multimodal information (e.g,
IMUs signal). Marin et al. [113] proposed a deep depth pose model to combine
RGB-D information and a set of predefined 3D poses to predict the 3D joint
positions.

Furthermore, the deep learning-based methods with the data of IMUs and images
have shown remarkable results in 3D HPE (as shown in Table IV). Marcard et al.
[35] adopted IMUs and a mobile camera to estimate 3D human poses, and proposed
the skinned multi-person linear model to produce initial 3D poses and then
associated the 3D poses with 2D detected persons. Huang et al. [96] used the
orientation of body parts, that is captured by IMUs, to refine the image-based
pose estimation. In summary, multimodal HPE generally shows superior
performance compared to vision-only HPE. However, it’s worth noting that the
lack of labeled datasets is a critical problem in multimodal HPE.

##  V Applications

HPE has been used in a variety of applications such as action analysis [1],
HCI [2], gaming [3], sport analysis [4, 5], motion capture [6], computer-
generated imagery [7, 8]. Accurate joint points are semantically informative
and can be utilized in computer vision tasks like action recognition, and
human tracking [99]. In this section, we present the main applications of HPE
(see the summary in Table V).

2D/3D HPE | Applications | Methods |  Remarks
---|---|---|---
2D HPE |  Action Analysis |  $\bullet$ Bottom-up MPPE (OpenPose [40]) + GCN [114] $\bullet$ Top-down MPPE (AlphaPose [39]) + GCN |  HPE provides spatial joint data and implicit temporal correlationship for action analysis.
Character Animation |  $\bullet$ Top-down MPPE [52] \+ Motion Transfer [115] $\bullet$ Bottom-up MPPE [40] \+ Motion Transfer [8] |  Animation by HPE-based motion transferring from human performer to animated character
Sports Analysis |  $\bullet$ Bottom-up MPPE [40] \+ Coaching method [116] $\bullet$ Video-based HPE + Coaching method [4] |  HPE-based sports analysis by comparisons between players and exemplars
Medical & Clinical |  $\bullet$ Top-down MPPE + Exercise Supervision [117] $\bullet$ Bottom-up MPPE + Exercise Supervision [118] |  Joint positions provide rich information for clinical applications like in-bed/sleep monitoring
3D HPE |  Action Analysis |  $\bullet$ Multimodal 3D HPE (Kinect-based) + Random Forest [119] $\bullet$ Multimodal 3D HPE (Kinect-based) + CNN [120] |  3D action analysis can exploit both 2D joint position and extra depth information
HCI |  $\bullet$ Multimodal 3D HPE (Kinect-based) + HCI [2] $\bullet$ Monocular 3D HPE + HCI [3] |  HPE-based HCI is a natural and contactless way
Character Animation |  $\bullet$ Two-stage Monocular 3D HPE + Motion Transfer [121, 6] |  3D character animations benefit from 3D HPE and motion transfer
Sports Analysis |  $\bullet$ Multimodal 3D HPE (Kinect-based) + Coaching Method [122, 5] $\bullet$ Monocular 3D HPE [81] \+ Coaching Method [123] |  3D HPE provides more dimensionality (3D angles between joints) for a coaching system
Medical & Clinical |  $\bullet$ Multimodal 3D HPE (Kinect-based) + Neural Recording [124] $\bullet$ Multimodal 3D HPE (Kinect-based) + Exercise Supervision [125] |  3D HPE-based clinical monitoring and rehabilitation system use human joint and depth information
TABLE V: Summary of the approaches in HPE-based applications.

###  V-A Action Analysis

Action recognition/prediction is mainly a temporal task based on image
sequences. The traditional methods might demand extensive computation and
appear unstable to environmental variations in terms of illuminations, objects
in the background or foreground, body scale, and motion blur [126]. The human
skeleton is naturally a high-level representation which has shown benefits for
action analysis tasks such as action recognition [127] and action detection
[128]. For example, Duan et al. [129] stacked 2D heatmaps from human pose
sequences to 3D heatmap volumes and utilized ResNet layers to predict human
actions from these volumes. Liu et al. [1] used graph convolutional network to
utilize 3D pose sequence for action analysis.

Currently, with the development of sensors (e.g., Kinect [125]) and HPE
algorithms, large-scale and accurate skeleton data for action recognition
[127] becomes accessible. The space correlations and temporality of skeleton
sequences provide informative prior knowledge to yield a robust motion
pattern. On large-scale action recognition dataset NTURGB-D [127], the state-
of-the-art skeleton-based methods [129, 130] achieved more than 95% accuracy,
while image-only methods [99] achieved less than 90% accuracy.

In addition, the skeleton-based action analysis can be used to build smart
surveillance systems. Hbali et al. [119] used HPE and action analysis to build
up an elderly monitoring system for alerting dangerous activities. Guo et al.
[131] used skeleton-based action recognition to identify unsafe behaviours of
construction workers. In conclusion, HPE provides significant spatial joint
information and implicit temporal correlations for skeleton-based action
analysis.

###  V-B Human-Computer Interaction

Human-computer interaction (HCI) has been studied for several decades and
plays an important role in our daily lives. Traditional HCI techniques allow
humans to interact with computers via tangible devices or interfaces such as
mice, keyboards, or touch screens. Compared with traditional interactions,
HPE-based HCI provides a natural and contactless way that is highly suitable
for the difficult pandemic situation of COVID-19, particularly when using
public devices.

HPE-based HCI techniques are widely applied in many applications such as arts,
gaming, and virtual reality. Most vision-only applications consider the
estimation of 2D human poses since currently, 2D HPE approaches can offer more
accurate and prompt predictions than 3D HPE. While 3D HPE often employs depth-
aware sensors (like RGB-D cameras) to produce reliable and informative 3D
poses. The well-known depth-aware distributed Microsoft Kinects [125] is
designed to capture human body movements for gaming and virtual reality
teleconference systems [132]. Kamel et al. [123], Thar et al. [133], and Park
[5] et al. used 3D monocular HPE and a single camera to capture human poses
for producing action evaluation and feedback for Tai Chi, Yoga, and golf,
respectively. In summary, HPE-based HCI is a type of natural and contactless
interface that differs from the traditional HCI.

###  V-C Character Animation

Creating high-quality character animation is important in animation films and
computer games. Traditional methods rely on creating costly and time-consuming
frame-wise animations. HPE-based performance-driven animations accomplish
character motions by transferring motions from a human performer to an
animated character, which is prevalent in the film and game industries
conveniently and economically. For 2D animation applications, Willett et al.
[8] proposed a novel method to jointly yield 2D animation and 2D character
creation by leveraging human pose. In addition, the HPE technique is helpful
in augmented reality. For instance, Weng et al. [115] utilized human skeleton
information via HPE to generate 3D character animation from an image. To
generate 3D animations, a commonly used method is to control all parts of a
character via motion capture [6]. In summary, DL-based HPE approaches provide
an alternative to traditional motion capture for creating both 2D and 3D
animations.

###  V-D Sports Analysis

Sports performance analysis provides statistics/recording for coaches and
players to improve their performance. The automatic sports analysis demands
precise postures and motion of players, which requires accurate markerless
motion capture techniques. The HPE approaches perform effectively in the
analysis of many sporting activities. HPE-based methods can be used to compare
the differences between learners and instructors in various sports. For
example, Park et al. [5] used the HPE approach to analyze activities in golf.
By comparing a user’s swing with the reference swing, the user can examine
his/her evaluation result in head movement, knee alignment, swing rhythm, and
balance of golf swing in their approach. Kamel et al. [123], and Thar et al.
[133] employed HPE to evaluate the difference between a learner’s posture and
an instructor’s posture in Tai Chi and Yoga, respectively. Wang et al. [4]
built up an HPE-based athletic training assistance system to detect bad poses
from a sequence of 2D users’ poses. In addition, HPE-based approach is applied
to capture players’ motion in sports like badminton [116], soccer [134, 135],
and tennis [136, 137]. In conclusion, HPE-based sports analysis mainly relies
on a comparison between a player and an exemplar. 2D HPE is more commonly
adopted than 3D HPE as 3D HPE currently requires either multi-view setups or
depth-aware sensors.

###  V-E Medical and Clinical Applications

Accurate joint positions provide rich information for clinical applications
like in-bed monitoring, sleep laboratories, epilepsy monitoring, and intensive
care units. Gabeiel et al. [124] combined synchronized Kinect v2 and standard
clinical electrocorticography monitors, which record neural activity from the
cortex, to investigate the relationship between human movement behaviours and
neural activity. In rehabilitation medicine, HPE is successfully applied to
improve the rehabilitation of patients. Obdržálek et al. [125] employed HPE
for observation and online feedback when coaching elderly patients with the
daily physical exercise routine. Li et al. [117] introduced an in-home lower-
limb rehabilitation system with an HPE-based model to help patients perform
rehabilitation activities even without the presence of physical therapists. In
[118], Rabbito used OpenPose [40] and motion capture system Vicon
111https://www.vicon.com/ to analyze patient gait for rehabilitation medicine.

In summary, HPE-based clinical applications generally fall into two main
types, in-bed monitoring and rehabilitation training. Clinical environments
are suitable for installing multi-view cameras or depth-aware sensors, which
encourages 3D HPE. HPE-based Rehabilitation training currently is similar to
HPE-based sports coaching, which is mostly based on the comparison between a
user and an exemplar. While traditional motion capture systems are available
in limited environments, HPE-based methods have great potential for providing
more extensive human motion data.

##  VI Research Trends and Challenges

In this section, we aim to point out the observed research trends via
bibliometrics, raise the challenging issues, and provide insightful
recommendations for future research.

###  VI-A Research Trends

In this subsection, we identify the research trends via bibliometrics which
refers to the use of statistical methods to analyze books, articles and other
publications. It is an effective way to measure publications in the scientific
community. However, current survey papers rarely use bibliometrics to analyze
the data of publications, particularly in HPE. This paper applies
bibliometrics to retrieve the publications for discovering and demonstrating
the research trends in HPE.

In general, bibliometric data can be obtained from various databases. In this
paper, we choose the Scopus database 222https://en.wikipedia.org/wiki/Scopus
for literature retrieval. Scopus is the largest abstract/citation database
with peer-reviewed literature, which is released by Elsevier. The resources of
Scopus are more accurate and comprehensive than other alternatives such as
PubMed, Web of Science, and Google Scholar [138]. Importantly, Elsevier
provides a Python library 333https://github.com/pybliometrics-
dev/pybliometrics to retrieve the data for the expected topics from Scopus
database [139]. In this work, we review the articles in vision-based HPE using
deep learning by retrieving "title, abstract, and keywords" in the Scopus
database. We present the trends that are observed from the literature in the
four aspects: multi-person pose estimation, 3D HPE, efficient deep learning-
based HPE, and multi-modal learning. To observe the trends clearly, we
retrieve the data for each year over ten years (2012 - 2022). We retrieve the
number of publications in Scopus data from the four aspects, as shown in
Figure 4, where the red dash lines are the fit of the number of publications
from 2012 to 2021 (NOT 2022).

(a)

(b)

(c)

(d)

Figure 4:  The number of publications on the HPE topics of a) multi-person
pose estimation, b) 3D HPE, c) efficient deep learning-based HPE, d)
multimodal learning HPE from Scopus. Scopus database returned 274, 769, 185,
111 results for a), b), c), and d) respectively until 08/09/2022. To observe
the trends clearly, we retrieve the data of each year over the ten years from
2012 to 2022. The red dashed lines are the fit of the number of publications
from 2012 to 2021 (NOT 2022). The source code to retrieve the data are
available at %\textcolor{red}{https://github.com/wuyuuu/elsevier-search.

####  VI-A1 Multi-Person HPE

Multi-Person HPE has become an important research topic. We retrieve the
literature on this topic by using the search code TITLE-ABS-KEY(("multi-
person" or "crowd") and ("multi-person pose estimation" or "human pose
estimation")). The number of publications over the years (from 2012 to 2022)
is shown in 4(a). It shows a very significant increasing trend in the number
of Multi-Person HPE publications since 2015, while an insignificant drop in
2021.

On one hand, the small number of publications shows that the multi-person pose
estimation in crowd scenes is still rarely investigated. On the other hand,
the increasing number of publications shows the increasing trend in this
topic. This technique has been applied in various applications such as multi-
person gaming and sports analysis. However, as reported in [32], the
performance of the existing methods degrades dramatically as the crowd level
increases. In multi-person HPE, there are still many open issues and
challenges that need to be further studied in the future.

####  VI-A2 3D HPE

Accurately estimating 3D positions for human joints is an important topic in
computer vision. Compared to 2D HPE, 3D HPE could provide extra depth
information which brings broader applications such as markerless motion
capture, video games, and sports analysis [5, 92]. We observe that 3D HPE
attracts more and more research interest recently. We design the code of
TITLE-ABS-KEY(("3D") and ("human pose estimation")) to retrieve the literature
on the 3D HPE-related topics in the Scopus database. The number of
publications over the years (from 2012 to 2022) is shown in 4(b).

We observe an increasing number of 3D HPE publications from 2014 to 2021.
However, the existing 3D HPE methods still need to be investigated for desired
performance because of unsolved problems such as the ill-posed property, the
lack of outdoor environment annotations, and the incapability of performing
real-time inference on edge devices. Therefore, the 3D HPE will be a crucial
research trend in the future.

####  VI-A3 Efficient Deep Learning-based HPE

To apply HPE methods to real applications, the efficiency of HPE approaches is
a crucial research topic. In particular, although deep learning-based methods
have achieved state-of-the-art performance in HPE, the approaches of deep
neural networks may suffer from high computational costs and inference delay.
We design the code of TITLE-ABS-KEY(("efficient" or "real-time") and ("human
pose estimation")) to retrieve literature in efficient HPE. The number of
publications over year (from 2012 to 2022) is shown in 4(c). It shows a
significant increase since 2014 with respect to the number of publications in
efficient HPE. We note that the HPE efficiency problem was generally tackled
by designing efficient networks manually in previous approaches [56, 40],
while current studies [17, 140] introduce neural architecture search (NAS)
into HPE, which might be a typical method in solving the efficiency problem of
neural network in the future.

####  VI-A4 Multimodal Learning for HPE

Multimodal learning extracts features from multiple sensing modalities, which
alleviates the complexity of unimodal methods. Recently, the multimodal
learning-based approaches [94, 35] have been validated for robustness and
accuracy in HPE tasks. We retrieve the literature by using the code of TITLE-
ABS-KEY(("multi-modal" or"multimodal" or "IMUs" or "radio signal") and ("human
pose estimation") and not ("distribution")) to observe and demonstrate the
research trend. The number of publications over years (from 2012 to 2022) is
shown in 4(d). Although the total number of publications implies a lack of
attention in multimodal learning HPE, the steady growth of publications
implies an increasing interest in this topic. The multimodal learning methods
offer solutions and robustness to the vision-based model in addressing
occlusions, which are promising solutions towards the in-the-wild challenge
[35]. The multimodal learning methods for HPE still need to be further
investigated in the future.

###  VI-B Research Challenges

A remarkable HPE approach should consider both high accuracy and high
efficiency. Although many studies have investigated the HPE with prominent
performance, there are many considerable challenges to achieve both goals. An
accurate HPE approach is generally demanded to deal with the challenges of
various occlusions and personal appearances, estimate the depth of monocular
image, and remain robust to image degeneration. In addition, the current
biased datasets (e.g., incomprehensive outdoor 3D annotations, relatively rare
uncommon poses) is a prevalent challenge for the practical accuracy of HPE in
real-world scenarios. Moreover, implementing efficient deep learning-based HPE
on resource-limited devices is a notorious challenge. The current deep
learning-based HPE generally take a lot of computing time because of the
large-scale networks. In particular, the computing time significantly creases
with the creasing number of people for the crowd scenes. Therefore, a number-
robust HPE algorithm is critical for implementing an efficient HPE. In this
subsection, we summarize and discuss the challenges in HPE from two aspects:
accuracy and efficiency.

####  VI-B1 Challenges in Accurate HPE

##### Diverse Human Poses and Appearances

A fundamental challenge in developing accurate HPE comes from the diversity of
human poses [12]. Despite the vast range of human appearances, the human body
entails high degree of freedom, demanding an advanced presentation ability for
data-driven approaches in HPE. Additionally, image degeneration like motion
blur and image defocus exists in video-based data [44], which also hinders HPE
approaches from achieving remarkable performance.

##### Occlusions

Although current HPE methods perform outstanding performance on many public
datasets, a well-known issue is noticeable performance degeneration caused by
occlusions and highly deformable human body [30, 39, 141]. Self-occlusions and
mutual occlusions could prompt the occlusions and environmental truncation,
while mutual occlusions can occur extensively in crowd scenarios [32] to cause
the performance to decline dramatically. The highly deformable human body can
cause ambiguity in small-scale human instances [141] or the specific human
poses as well. Thus, designing the powerful HPE method for occlusion scenarios
with the capability of utilizing global information and prior knowledge would
be a challenging issue.

##### Incomprehensive Datasets

To apply HPE approaches to practical applications, challenges could come from
the gap between current incomprehensive datasets and real-world applications.
For example, uncommon poses like falling down are less likely to appear in
datasets, and the outdoors 3D HPE dataset is relatively rare. This gap leads
to an imbalanced learning problem, which could hinder the applications of 3D
HPE in the real world. Although current approaches can leverage semi-
supervised learning [104] or synthetic dataset [37] to enrich the datasets,
the semi-supervised methods still need a lot of quality training data [98].
Current datasets lack realistic simulation of lighting effects, clothing
meshes, and environment interactions [142]. Therefore, it is difficult to
transfer the trained neural networks from the simulation to real-world
applications. Developing a remarkable HPE on current incomprehensive datasets
is still challenging, particularly for deployment in complex environments.

Types | Studies |  URL (Open source code) |  Remarks
---|---|---|---
2D/3D HPE | MMPose |  https://github.com/open-mmlab/mmpose |  Well-known platform
2D HPE |  Associative embedding [63] |  https://github.com/princeton-vl/pose-ae-train |  A SOTA grouping for bottom-up approaches
Hourglass [41] |  https://github.com/princeton-vl/pose-hg-demo |  Effective yet simple backbone
OpenPose [40] |  https://github.com/CMU-Perceptual-Computing-Lab/openpose |  Real-time & bottom-up
AlphaPose [39] |  https://github.com/MVIG-SJTU/AlphaPose |  Real-time & top-down
Higher-HRNet [72] |  https://github.com/HRNet/HigherHRNet-Human-Pose-Estimation |  A SOTA bottom-up approach
HRNet [10] |  https://github.com/HRNet/HRNet-Human-Pose-Estimation |  A SOTA top-down approach
RLE [143] |  https://github.com/Jeff-sjtu/res-loglikelihood-regression |  A SOTA regression-based HPE
UDP-POSE [144] |  https://github.com/HuangJunJie2017/UDP-Pose |  1st in ICCV 2019 COCO keypoint challenge
DARK [58] |  https://ilovepose.github.io/coco/ |  2nd in ICCV 2019 COCO keypoint challenge
Lite-HRNet [145] |  https://github.com/HRNet/Lite-HRNet |  Lightweight HRNet-based model
Lightweight OpenPose [146] |  https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch |  Real-time on CPU
BlazePose [64] |  https://google.github.io/mediapipe/solutions/pose.html |  Real-time 2D single-person HPE driven by MediaPipe [65]
PRTR [60] |  https://github.com/mlpc-ucsd/PRTR |  2D Pose Regression transformer
|  DCPose [147] |  https://github.com/Pose-Group/DCPose |  1st in PoseTrack2017 & PoseTrack2018
3D HPE |  Epipolar transformer [89] |  https://github.com/yihui-he/epipolar-transformers |  A SOTA multi-view approach
Learnable triangulation [87] |  https://saic-violet.github.io/learnable-triangulation/ |  A SOTA multi-view approach
SMAP [148] |  https://github.com/zju3dv/SMAP |  SOTA single-view multi-person
DOPE [149] |  https://github.com/naver/dope |  Real-time whole-body 3D HPE
VNect [150] |  http://gvv.mpi-inf.mpg.de/projects/VNect/ |  Real-time single-view approach
Synthetic occlusion [151] |  https://github.com/isarandi/synthetic-occlusion |  1st place in ECCV2018 3D HPE Challenge
Integral regression [152] |  https://github.com/JimmySuen/integral-human-pose |  2nd place in ECCV2018 3D HPE Challenge
PoseFormer [103] |  https://github.com/zczcwh/PoseFormer |  3D pose transformer
Top-down & bottom-up Integration [153] |  https://github.com/3dpose/3D-Multi-Person-Pose |  A SOTA monocular multi-person 3D HPE
Normalizing flows [154] |  https://github.com/twehrbein/Probabilistic-Monocular-3D-Human-Pose-Estimation-with-Normalizing-Flows |  A SOTA monocular 3D HPE
PoseAug [86] |  https://github.com/jfzhang95/PoseAug |  A data augmentation framework for 3D HPE
TABLE VI: The open-source code of the state-of-the-art HPE methods.

####  VI-B2 Challenges in Efficient HPE

##### Computation-intensive Neural Networks

Eventually, the HPE approach needs to be implemented for applications in the
real world. However, the state-of-the-art neural networks [10, 148] are
generally hard to be implemented on mobile devices or embedded devices as
their enormous computational cost. Thus, it is crucial to design lightweight
neural networks for efficient HPE. The existing methods of designing
lightweight neural networks are mainly manual design and heuristic design
(e.g., NAS [140]). However, the method of manual design is hard to balance the
accuracy and network size [145]. NAS-based methods [140, 17] generally need
various computational cost even weeks of CPU time. Therefore, developing
lightweight neural networks for HPE is still a challenging task.

##### Time-Consuming MPPE

Currently, MPPE algorithms consume increasing computation time over the
increasing number of targeted persons. Top-down approaches estimate the pose
of each detected person after the person detection stage. Bottom-up approaches
[63, 40] predict similarity values between keypoints and employ a matching
algorithm (e.g., Hungarian algorithm) for grouping keypoints. Note that top-
down and bottom-up approaches are two-stage methods due to top-down approaches
require an extra detection stage and bottom-up approaches require an extra
grouping stage, besides estimating keypoint locations.

Compared to two-stage methods, single-stage methods generally perform superior
in terms of computational cost. A promising single-stage method [69] predicts
the locations of persons and keypoints’ offsets to each location. However, the
single-stage methods are generally not as competitive as the state-of-the-art
two-stage methods [72, 10] in terms of accuracy. How to develop a desired
efficient MPPE is still an interesting challenge, particularly for
applications in the real world.

##  VII Conclusions

In this paper, we presented an up-to-date and in-depth overview of the deep
learning approaches in vision-based HPE. We systematically introduced the
preliminary knowledge in HPE and reviewed the HPE approaches in two
categories: 2D-based approaches and 3D-based approaches. We discussed a number
of interesting applications of deep learning-based HPE. Finally, we pointed
out the research trends via bibliometrics, raised the challenging issues, and
provided insightful recommendations for future research. To help readers to
reproduce the state-of-the-art methods, we summarized the open-source codes of
the well-known studies for 2D and 3D deep learning-based HPE in Table VI,
which could help readers easily implement their HPE tasks. This article
provides a meaningful overview as introductory material for beginners to deep
learning-based HPE, as well as supplementary material for advanced
researchers.

## References

  * [1] Z. Liu, H. Zhang, Z. Chen, Z. Wang, and W. Ouyang, “Disentangling and unifying graph convolutions for skeleton-based action recognition,” in _CVPR 2020_ , pp. 143–152.
  * [2] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognition in parts from single depth images,” in _CVPR 2011_ , pp. 1297–1304.
  * [3] S.-R. Ke, L. Zhu, J.-N. Hwang, H.-I. Pai, K.-M. Lan, and C.-P. Liao, “Real-time 3D human pose estimation from monocular view with applications to event detection and video gaming,” in _2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance_. IEEE, 2010, pp. 489–496.
  * [4] J. Wang, K. Qiu, H. Peng, J. Fu, and J. Zhu, “AI coach: Deep human pose estimation and analysis for personalized athletic training assistance,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 374–382.
  * [5] S. Park, J. Yong Chang, H. Jeong, J.-H. Lee, and J.-Y. Park, “Accurate and efficient 3D human pose estimation algorithm using single depth images for pose analysis in golf,” in _CVPR Workshops 2017_ , pp. 49–57.
  * [6] Z. Yang, W. Zhu, W. Wu, C. Qian, Q. Zhou, B. Zhou, and C. C. Loy, “TransMoMo: Invariance-driven unsupervised video motion retargeting,” in _CVPR 2020_ , pp. 5306–5315.
  * [7] A. Hornung, E. Dekkers, and L. Kobbelt, “Character animation from 2D pictures and 3D motion data,” _ACM Transactions on Graphics_ , vol. 26, no. 1, pp. 1–es, 2007.
  * [8] N. S. Willett, H. V. Shin, Z. Jin, W. Li, and A. Finkelstein, “Pose2Pose: pose selection and transfer for 2D character animation,” in _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , 2020, pp. 88–99.
  * [9] A. Toshev and C. Szegedy, “DeepPose: Human pose estimation via deep neural networks,” in _CVPR 2014_ , pp. 1653–1660.
  * [10] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in _CVPR 2019_ , pp. 5693–5703.
  * [11] D. Hogg, “Model-based vision: a program to see a walking person,” _Image and Vision computing_ , vol. 1, no. 1, pp. 5–20, 1983.
  * [12] T. B. Moeslund and E. Granum, “A survey of computer vision-based human motion capture,” _Computer vision and image understanding_ , vol. 81, no. 3, pp. 231–268, 2001.
  * [13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [14] G. Lan, J. Benito-Picazo, D. M. Roijers, E. Domínguez, and A. Eiben, “Real-time robot vision on low-performance computing hardware,” in _2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV)_. IEEE, 2018, pp. 1959–1965.
  * [15] G. Lan, L. De Vries, and S. Wang, “Evolving efficient deep neural networks for real-time object recognition,” in _2019 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2019, pp. 2571–2578.
  * [16] M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall, and B. Schiele, “PoseTrack: A benchmark for human pose estimation and tracking,” in _CVPR 2018_ , pp. 5167–5176.
  * [17] W. McNally, K. Vats, A. Wong, and J. McPhee, “EvoPose2D: Pushing the boundaries of 2d human pose estimation using accelerated neuroevolution with weight transfer,” _IEEE Access_ , vol. 9, pp. 139 403–139 414, 2021.
  * [18] Q. Dang, J. Yin, B. Wang, and W. Zheng, “Deep learning based 2D human pose estimation: A survey,” _Tsinghua Science and Technology_ , vol. 24, no. 6, pp. 663–676, 2019.
  * [19] R. Poppe, “A survey on vision-based human action recognition,” _Image and vision computing_ , vol. 28, no. 6, pp. 976–990, 2010.
  * [20] Y. Chen, Y. Tian, and M. He, “Monocular human pose estimation: A survey of deep learning-based methods,” _Computer Vision and Image Understanding_ , vol. 192, p. 102897, 2020.
  * [21] W. Liu and T. Mei, “Recent advances of monocular 2D and 3D human pose estimation: A deep learning perspective,” _ACM Computing Surveys (CSUR)_ , 2022.
  * [22] W. Gong, X. Zhang, J. Gonzàlez, A. Sobral, T. Bouwmans, C. Tu, and E.-h. Zahzah, “Human pose estimation from monocular images: A comprehensive survey,” _Sensors_ , vol. 16, no. 12, p. 1966, 2016.
  * [23] L. M. Dang, K. Min, H. Wang, M. J. Piran, C. H. Lee, and H. Moon, “Sensor-based and vision-based human activity recognition: A comprehensive survey,” _Pattern Recognition_ , vol. 108, p. 107561, 2020.
  * [24] R. Gadhiya and N. Kalani, “Analysis of deep learning based pose estimation techniques for locating landmarks on human body parts,” in _2021 International Conference on Circuits, Controls and Communications (CCUBE)_. IEEE, 2021, pp. 1–4.
  * [25] J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, and L. Shao, “Deep 3D human pose estimation: A review,” _Computer Vision and Image Understanding_ , p. 103225, 2021.
  * [26] S. Johnson and M. Everingham, “Clustered pose and nonlinear appearance models for human pose estimation,” in _Proceedings of the British Machine Vision Conference_ , 2010.
  * [27] B. Sapp and B. Taskar, “MODEC: Multimodal decomposable models for human pose estimation,” in _CVPR 2013_.
  * [28] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, “Towards understanding action recognition,” in _ICCV 2013_ , pp. 3192–3199.
  * [29] W. Zhang, M. Zhu, and K. G. Derpanis, “From actemes to action: A strongly-supervised representation for detailed action understanding,” in _ICCV 2013_ , pp. 2248–2255.
  * [30] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2D human pose estimation: New benchmark and state of the art analysis,” in _CVPR 2014_ , pp. 3686–3693.
  * [31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _ECCV 2014_ , pp. 740–755.
  * [32] J. Li, C. Wang, H. Zhu, Y. Mao, H.-S. Fang, and C. Lu, “CrowdPose: Efficient crowded scenes pose estimation and a new benchmark,” in _CVPR 2019_ , pp. 10 863–10 872.
  * [33] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 7, pp. 1325–1339, jul 2014.
  * [34] H. Joo, T. Simon, X. Li, H. Liu, L. Tan, L. Gui, S. Banerjee, T. Godisart, B. Nabbe, I. Matthews _et al._ , “Panoptic studio: A massively multiview system for social interaction capture,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 1, pp. 190–204, 2017.
  * [35] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll, “Recovering accurate 3D human pose in the wild using IMUs and a moving camera,” in _ECCV 2018_ , pp. 601–617.
  * [36] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt, “Monocular 3D human pose estimation in the wild using improved cnn supervision,” in _2017 international conference on 3D vision_. IEEE, 2017, pp. 506–516.
  * [37] M. Fabbri, F. Lanzi, S. Calderara, A. Palazzi, R. Vezzani, and R. Cucchiara, “Learning to detect and track visible and occluded body joints in a virtual world,” in _ECCV 2018_ , pp. 430–446.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR 2016_ , June.
  * [39] H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu, “RMPE: Regional multi-person pose estimation,” in _ICCV 2017_ , pp. 2334–2343.
  * [40] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2D pose estimation using part affinity fields,” in _CVPR 2017_ , pp. 7291–7299.
  * [41] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in _European conference on computer vision_. Springer, 2016, pp. 483–499.
  * [42] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun, “Cascaded pyramid network for multi-person pose estimation,” in _CVPR 2018_ , pp. 7103–7112.
  * [43] B. Xiao, H. Wu, and Y. Wei, “Simple baselines for human pose estimation and tracking,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 466–481.
  * [44] Y. Luo, J. Ren, Z. Wang, W. Sun, J. Pan, J. Liu, J. Pang, and L. Lin, “LSTM pose machines,” in _CVPR 2018_ , pp. 5207–5215.
  * [45] B. Artacho and A. Savakis, “UniPose: Unified human pose estimation in single images and videos,” in _CVPR 2020_ , pp. 7035–7044.
  * [46] Y. Bin, Z.-M. Chen, X.-S. Wei, X. Chen, C. Gao, and N. Sang, “Structure-aware human pose estimation with graph convolutional networks,” _Pattern Recognition_ , vol. 106, p. 107410, 2020.
  * [47] Z. Qiu, K. Qiu, J. Fu, and D. Fu, “DGCN: Dynamic graph convolutional network for efficient multi-person pose estimation,” in _AAAI 2020_ , vol. 34, no. 07, pp. 11 924–11 931.
  * [48] L. Zhao, X. Peng, Y. Tian, M. Kapadia, and D. N. Metaxas, “Semantic graph convolutional networks for 3D human pose regression,” in _CVPR 2019_ , pp. 3425–3435.
  * [49] W. Hu, C. Zhang, F. Zhan, L. Zhang, and T.-T. Wong, “Conditional directed graph convolution for 3D human pose estimation,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 602–611.
  * [50] Y. Cai, L. Ge, J. Liu, J. Cai, T.-J. Cham, J. Yuan, and N. M. Thalmann, “Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks,” in _ICCV 2019_ , pp. 2272–2281.
  * [51] J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler, “Joint training of a convolutional network and a graphical model for human pose estimation,” in _Advances in neural information processing systems_ , 2014, pp. 1799–1807.
  * [52] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in _CVPR 2016_ , pp. 4724–4732.
  * [53] X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang, “Multi-context attention for human pose estimation,” in _CVPR 2017_ , pp. 1831–1840.
  * [54] W. Yang, S. Li, W. Ouyang, H. Li, and X. Wang, “Learning feature pyramids for human pose estimation,” in _ICCV 2017_ , pp. 1281–1290.
  * [55] A. Bulat, J. Kossaifi, G. Tzimiropoulos, and M. Pantic, “Toward fast and accurate human pose estimation via soft-gated skip connections,” in _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition_ , 2020, pp. 101–108.
  * [56] U. Rafi, B. Leibe, J. Gall, and I. Kostrikov, “An efficient convolutional network for human pose estimation.” in _BMVC_ , vol. 1, 2016, p. 2.
  * [57] A. Jain, J. Tompson, Y. LeCun, and C. Bregler, “Modeep: A deep learning framework using motion features for human pose estimation,” in _Asian conference on computer vision_. Springer, 2014, pp. 302–315.
  * [58] F. Zhang, X. Zhu, H. Dai, M. Ye, and C. Zhu, “Distribution-aware coordinate representation for human pose estimation,” in _CVPR 2020_ , pp. 7093–7102.
  * [59] W. Mao, Y. Ge, C. Shen, Z. Tian, X. Wang, and Z. Wang, “Tfpose: Direct human pose estimation with transformers,” _arXiv preprint arXiv:2103.15320_ , 2021\.
  * [60] K. Li, S. Wang, X. Zhang, Y. Xu, W. Xu, and Z. Tu, “Pose recognition with cascade transformers,” in _CVPR 2021_ , pp. 1944–1953.
  * [61] S. Yang, Z. Quan, M. Nie, and W. Yang, “Transpose: Keypoint localization via transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 802–11 812.
  * [62] Y. Xu, J. Zhang, Q. Zhang, and D. Tao, “ViTPose: Simple vision transformer baselines for human pose estimation,” _arXiv preprint arXiv:2204.12484_ , 2022.
  * [63] A. Newell, Z. Huang, and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in _Advances in neural information processing systems_ , 2017, pp. 2277–2287.
  * [64] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, and M. Grundmann, “Blazepose: On-device real-time body pose tracking,” _arXiv preprint arXiv:2006.10204_ , 2020.
  * [65] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee _et al._ , “Mediapipe: A framework for building perception pipelines,” _arXiv preprint arXiv:1906.08172_ , 2019\.
  * [66] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _ICCV 2017_ , pp. 2961–2969.
  * [67] W. Li, Z. Wang, B. Yin, Q. Peng, Y. Du, T. Xiao, G. Yu, H. Lu, Y. Wei, and J. Sun, “Rethinking on multi-stage networks for human pose estimation,” _arXiv preprint arXiv:1901.00148_ , 2019.
  * [68] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele, “Deepercut: A deeper, stronger, and faster multi-person pose estimation model,” in _European Conference on Computer Vision_. Springer, 2016, pp. 34–50.
  * [69] X. Nie, J. Feng, J. Zhang, and S. Yan, “Single-stage multi-person pose machines,” in _ICCV 2019_ , pp. 6951–6960.
  * [70] S. Jin, W. Liu, E. Xie, W. Wang, C. Qian, W. Ouyang, and P. Luo, “Differentiable hierarchical graph grouping for multi-person pose estimation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 718–734.
  * [71] M. Kocabas, S. Karagoz, and E. Akbas, “Multiposenet: Fast multi-person pose estimation using pose residual network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 417–433.
  * [72] B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang, and L. Zhang, “HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation,” in _CVPR 2020_ , pp. 5386–5395.
  * [73] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy, “Towards accurate multi-person pose estimation in the wild,” in _CVPR 2017_ , pp. 4903–4911.
  * [74] X. Nie, Y. Li, L. Luo, N. Zhang, and J. Feng, “Dynamic kernel distillation for efficient pose estimation in videos,” in _ICCV 2019_ , pp. 6942–6950.
  * [75] G. Bertasius, C. Feichtenhofer, D. Tran, J. Shi, and L. Torresani, “Learning temporal pose estimation from sparsely-labeled videos,” in _Advances in neural information processing systems_ , 2019, pp. 3027–3038.
  * [76] J. Wang, S. Jin, W. Liu, W. Liu, C. Qian, and P. Luo, “When human pose estimation meets robustness: Adversarial algorithms and benchmarks,” in _CVPR 2021_ , pp. 11 855–11 864.
  * [77] S.-H. Zhang, R. Li, X. Dong, P. Rosin, Z. Cai, X. Han, D. Yang, H. Huang, and S.-M. Hu, “Pose2Seg: Detection free human instance segmentation,” in _CVPR 2019_ , pp. 889–898.
  * [78] N. Jain, S. Shah, A. Kumar, and A. Jain, “On the robustness of human pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 29–38.
  * [79] M. Andriluka, S. Roth, and B. Schiele, “Discriminative appearance models for pictorial structures,” _International journal of computer vision_ , vol. 99, no. 3, pp. 259–280, 2012.
  * [80] M. Bergtholdt, J. Kappes, S. Schmidt, and C. Schnörr, “A study of parts-based object class detection using complete graphs,” _International journal of computer vision_ , vol. 87, no. 1-2, p. 93, 2010\.
  * [81] S. Li and A. B. Chan, “3D human pose estimation from monocular images with deep convolutional neural network,” in _Asian Conference on Computer Vision_. Springer, 2014, pp. 332–347.
  * [82] C. Chen and D. Ramanan, “3D human pose estimation = 2D pose estimation + matching,” in _CVPR 2017_ , pp. 5759–5767.
  * [83] J. Martinez, R. Hossain, J. Romero, and J. J. Little, “A simple yet effective baseline for 3D human pose estimation,” in _ICCV 2017_ , pp. 2659–2668.
  * [84] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Sridhar, G. Pons-Moll, and C. Theobalt, “Single-shot multi-person 3D pose estimation from monocular RGB,” in _2018 International Conference on 3D Vision_. IEEE Computer Society, 2018, pp. 120–130.
  * [85] S. Li, L. Ke, K. Pratama, Y.-W. Tai, C.-K. Tang, and K.-T. Cheng, “Cascaded deep monocular 3D human pose estimation with evolutionary training data,” in _CVPR 2020_ , pp. 6173–6183.
  * [86] K. Gong, J. Zhang, and J. Feng, “PoseAug: A differentiable pose augmentation framework for 3D human pose estimation,” in _CVPR 2021_ , pp. 8575–8584.
  * [87] K. Iskakov, E. Burkov, V. S. Lempitsky, and Y. Malkov, “Learnable triangulation of human pose,” in _ICCV 2019_ , pp. 7717–7726.
  * [88] Z. Zhang, C. Wang, W. Qiu, W. Qin, and W. Zeng, “AdaFuse: Adaptive multiview fusion for accurate human pose estimation in the wild,” _International Journal of Computer Vision_ , pp. 1–16, 2020.
  * [89] Y. He, R. Yan, K. Fragkiadaki, and S. Yu, “Epipolar transformers,” in _CVPR 2020_ , pp. 7776–7785.
  * [90] H. Qiu, C. Wang, J. Wang, N. Wang, and W. Zeng, “Cross view fusion for 3D human pose estimation,” in _ICCV 2019_. IEEE, pp. 4341–4350.
  * [91] S. Wu, S. Jin, W. Liu, L. Bai, C. Qian, D. Liu, and W. Ouyang, “Graph-based 3D multi-person pose estimation using multi-view images,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 148–11 157.
  * [92] M. Trumble, A. Gilbert, C. Malleson, A. Hilton, and J. P. Collomosse, “Total capture: 3D human pose estimation fusing video and inertial sensors.” in _BMVC_ , vol. 2, no. 5, 2017, pp. 1–13.
  * [93] A. Gilbert, M. Trumble, C. Malleson, A. Hilton, and J. Collomosse, “Fusing visual and inertial sensors with semantics for 3D human pose estimation,” _International Journal of Computer Vision_ , vol. 127, no. 4, pp. 381–397, 2019.
  * [94] Z. Zhang, C. Wang, W. Qin, and W. Zeng, “Fusing wearable IMUs with multi-view images for human pose estimation: A geometric approach,” in _CVPR 2020_ , pp. 2200–2209.
  * [95] C. Malleson, A. Gilbert, M. Trumble, J. Collomosse, A. Hilton, and M. Volino, “Real-time full-body motion capture from video and IMUs,” in _2017 International Conference on 3D Vision (3DV)_. IEEE, 2017, pp. 449–457.
  * [96] F. Huang, A. Zeng, M. Liu, Q. Lai, and Q. Xu, “DeepFuse: An imu-aware network for real-time 3D human pose estimation from multi-view image,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 429–438.
  * [97] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in _CVPR 2018_ , pp. 2002–2011.
  * [98] H. Rhodin, M. Salzmann, and P. Fua, “Unsupervised geometry-aware representation for 3D human pose estimation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 750–767.
  * [99] D. C. Luvizon, D. Picard, and H. Tabia, “2D/3D pose estimation and action recognition using multitask deep learning,” in _CVPR 2018_ , pp. 5137–5146.
  * [100] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis, “Coarse-to-fine volumetric prediction for single-image 3D human pose,” in _CVPR 2017_ , pp. 7025–7034.
  * [101] A. Nibali, Z. He, S. Morgan, and L. Prendergast, “3D human pose estimation with 2D marginal heatmaps,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1477–1485.
  * [102] M. Fabbri, F. Lanzi, S. Calderara, S. Alletto, and R. Cucchiara, “Compressed volumetric heatmaps for multi-person 3D pose estimation,” in _CVPR 2020_ , pp. 7204–7213.
  * [103] C. Zheng, S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding, “3D human pose estimation with spatial and temporal transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 656–11 665.
  * [104] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli, “3D human pose estimation in video with temporal convolutions and semi-supervised training,” in _CVPR 2019_ , pp. 7753–7762.
  * [105] G. Rogez, P. Weinzaepfel, and C. Schmid, “LCR-NET++: Multi-person 2D and 3D pose detection in natural images,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 5, pp. 1146–1161, 2019.
  * [106] A. Zanfir, E. Marinoiu, M. Zanfir, A.-I. Popa, and C. Sminchisescu, “Deep network for the integrated 3D sensing of multiple people in natural images,” _Advances in Neural Information Processing Systems_ , vol. 31, pp. 8410–8419, 2018.
  * [107] A. Harltey and A. Zisserman, _Multiple view geometry in computer vision (2. ed.)_. Cambridge University Press, 2006.
  * [108] J. Dong, W. Jiang, Q. Huang, H. Bao, and X. Zhou, “Fast and robust multi-person 3D pose estimation from multiple views,” in _CVPR 2019_ , pp. 7792–7801.
  * [109] L. Chen, H. Ai, R. Chen, Z. Zhuang, and S. Liu, “Cross-view tracking for multi-human 3D pose estimation at over 100 FPS,” in _CVPR 2020_ , pp. 3279–3288.
  * [110] C. Huang, S. Jiang, Y. Li, Z. Zhang, J. Traish, C. Deng, S. Ferguson, and R. Y. Da Xu, “End-to-end dynamic matching network for multi-view multi-person 3D pose estimation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 477–493.
  * [111] A. Elmi, D. Mazzini, and P. Tortella, “Light3DPose: Real-time multi-person 3D pose estimation from multiple views,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 2755–2762.
  * [112] M. Zhao, T. Li, M. Abu Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi, “Through-wall human pose estimation using radio signals,” in _CVPR 2018_ , pp. 7356–7365.
  * [113] M. J. Marin-Jimenez, F. J. Romero-Ramirez, R. Muñoz-Salinas, and R. Medina-Carnicer, “3D human pose estimation from depth maps using a deep combination of poses,” _Journal of Visual Communication and Image Representation_ , vol. 55, pp. 627–639, 2018.
  * [114] S. Kim, K. Yun, J. Park, and J. Y. Choi, “Skeleton-based action recognition of people handling objects,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 61–70.
  * [115] C.-Y. Weng, B. Curless, and I. Kemelmacher-Shlizerman, “Photo wake-up: 3D character animation from a single photo,” in _CVPR 2019_ , pp. 5908–5917.
  * [116] Y. Su and Z. Liu, “Position detection for badminton tactical analysis based on multi-person pose estimation,” in _2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)_. IEEE, 2018, pp. 379–383.
  * [117] Y. Li, C. Wang, Y. Cao, B. Liu, J. Tan, and Y. Luo, “Human pose estimation based in-home lower body rehabilitation system,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [118] R. Rabbito, “Using deep learning-based pose estimation algorithms for markerless gait analysis in rehabilitation medicine,” Master’s thesis, Politecnico di Torino, 2021.
  * [119] Y. Hbali, S. Hbali, L. Ballihi, and M. Sadgal, “Skeleton-based human activity recognition for elderly monitoring systems,” _IET Computer Vision_ , vol. 12, no. 1, pp. 16–26, 2017.
  * [120] T.-H. Tsai and C.-W. Hsu, “Implementation of fall detection system based on 3D skeleton for deep learning technique,” _IEEE Access_ , vol. 7, pp. 153 049–153 059, 2019.
  * [121] L. Kumarapu and P. Mukherjee, “Animepose: Multi-person 3d pose estimation and animation,” _Pattern Recognition Letters_ , vol. 147, pp. 16–24, 2021\.
  * [122] A. Elaoud, W. Barhoumi, E. Zagrouba, and B. Agrebi, “Skeleton-based comparison of throwing motion for handball players,” _Journal of Ambient Intelligence and Humanized Computing_ , vol. 11, no. 1, pp. 419–431, 2020.
  * [123] A. Kamel, B. Liu, P. Li, and B. Sheng, “An investigation of 3D human pose estimation for learning tai chi: A human factor perspective,” _International Journal of Human–Computer Interaction_ , vol. 35, no. 4-5, pp. 427–439, 2019.
  * [124] P. Gabriel, W. K. Doyle, O. Devinsky, D. Friedman, T. Thesen, and V. Gilja, “Neural correlates to automatic behavior estimations from RGB-D video in epilepsy unit,” in _2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2016, pp. 3402–3405.
  * [125] Š. Obdržálek, G. Kurillo, F. Ofli, R. Bajcsy, E. Seto, H. Jimison, and M. Pavel, “Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population,” in _2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2012, pp. 1188–1193.
  * [126] B. Ren, M. Liu, R. Ding, and H. Liu, “A survey on 3D skeleton-based action recognition using learning method,” _arXiv preprint arXiv:2002.05907_ , 2020\.
  * [127] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot, “NTU RGB+ D 120: A large-scale benchmark for 3D human activity understanding,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 10, pp. 2684–2701, 2019.
  * [128] B. Li, H. Chen, Y. Chen, Y. Dai, and M. He, “Skeleton boxes: Solving skeleton based action detection with a single deep convolutional neural network,” in _2017 IEEE International Conference on Multimedia & Expo Workshops_. IEEE, 2017, pp. 613–616.
  * [129] H. Duan, Y. Zhao, K. Chen, D. Lin, and B. Dai, “Revisiting skeleton-based action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2969–2978.
  * [130] S. Das, S. Sharma, R. Dai, F. Bremond, and M. Thonnat, “VPN: Learning video-pose embedding for activities of daily living,” in _European Conference on Computer Vision_. Springer, 2020, pp. 72–90.
  * [131] H. Guo, Y. Yu, Q. Ding, and M. Skitmore, “Image-and-skeleton-based parameterized approach to real-time identification of construction workers’ unsafe behaviors,” _Journal of Construction Engineering and Management_ , vol. 144, no. 6, p. 04018042, 2018.
  * [132] G. Lan, Z. Luo, and Q. Hao, “Development of a virtual reality teleconference system using distributed depth sensors,” in _2016 2nd IEEE International Conference on Computer and Communications (ICCC)_. IEEE, 2016, pp. 975–978.
  * [133] M. C. Thar, K. Z. N. Winn, and N. Funabiki, “A proposal of yoga pose assessment method using pose detection for self-learning,” in _2019 International Conference on Advanced Information Technologies (ICAIT)_. IEEE, 2019, pp. 137–142.
  * [134] R. Afrouzian, H. Seyedarabi, and S. Kasaei, “Pose estimation of soccer players using multiple uncalibrated cameras,” _Multimedia Tools and Applications_ , vol. 75, no. 12, pp. 6809–6827, 2016.
  * [135] L. Bridgeman, M. Volino, J.-Y. Guillemaut, and A. Hilton, “Multi-person 3D pose estimation and tracking in sports,” in _CVPR Workshops 2019_ , pp. 2487–2496.
  * [136] R. Kurose, M. Hayashi, T. Ishii, and Y. Aoki, “Player pose analysis in tennis video based on pose estimation,” in _2018 International Workshop on Advanced Image Technology (IWAIT)_. IEEE, 2018, pp. 1–4.
  * [137] E. Wu and H. Koike, “FuturePong: Real-time table tennis trajectory forecasting using pose prediction network,” in _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–8.
  * [138] A. Martín-Martín, E. Orduna-Malea, M. Thelwall, and E. D. López-Cózar, “Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories,” _Journal of Informetrics_ , vol. 12, no. 4, pp. 1160–1177, 2018.
  * [139] M. E. Rose and J. R. Kitchin, “Pybliometrics: Scriptable bibliometrics using a python interface to scopus,” _SoftwareX_ , vol. 10, p. 100263, 2019.
  * [140] L. Xu, Y. Guan, S. Jin, W. Liu, C. Qian, P. Luo, W. Ouyang, and X. Wang, “ViPNAS: Efficient video pose estimation via neural architecture search,” in _CVPR 2021_ , pp. 16 072–16 081.
  * [141] B. Cheng, B. Xiao, J. Wang, H. Shi, T. S. Huang, and L. Zhang, “HigherHRNet: Scale-aware representation learning for bottom-up human pose estimation,” in _CVPR 2020_.
  * [142] C. Doersch and A. Zisserman, “Sim2real transfer learning for 3D human pose estimation: motion to the rescue,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 12 949–12 961.
  * [143] J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu, and C. Lu, “Human pose regression with residual log-likelihood estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 025–11 034.
  * [144] J. Huang, Z. Zhu, F. Guo, and G. Huang, “The devil is in the details: Delving into unbiased data processing for human pose estimation,” in _CVPR 2020_ , pp. 5700–5709.
  * [145] C. Yu, B. Xiao, C. Gao, L. Yuan, L. Zhang, N. Sang, and J. Wang, “Lite-HRNet: A lightweight high-resolution network,” in _CVPR 2021_ , pp. 10 440–10 450.
  * [146] D. Osokin, “Real-time 2D multi-person pose estimation on CPU: Lightweight OpenPose,” _arXiv preprint arXiv:1811.12004_ , 2018.
  * [147] Z. Liu, H. Chen, R. Feng, S. Wu, S. Ji, B. Yang, and X. Wang, “Deep dual consecutive network for human pose estimation,” in _CVPR 2021_ , pp. 525–534.
  * [148] J. Zhen, Q. Fang, J. Sun, W. Liu, W. Jiang, H. Bao, and X. Zhou, “Smap: Single-shot multi-person absolute 3D pose estimation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 550–566.
  * [149] P. Weinzaepfel, R. Brégier, H. Combaluzier, V. Leroy, and G. Rogez, “Dope: Distillation of part experts for whole-body 3D pose estimation in the wild,” in _European Conference on Computer Vision_. Springer, 2020, pp. 380–397.
  * [150] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt, “VNect: Real-time 3D human pose estimation with a single RGB camera,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 1–14, 2017.
  * [151] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe, “Synthetic occlusion augmentation with volumetric heatmaps for the 2018 eccv posetrack challenge on 3D human pose estimation,” _arXiv preprint arXiv:1809.04987_ , 2018\.
  * [152] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei, “Integral human pose regression,” in _ECCV 2018_ , pp. 529–545.
  * [153] Y. Cheng, B. Wang, B. Yang, and R. T. Tan, “Monocular 3d multi-person pose estimation by integrating top-down and bottom-up networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7649–7659.
  * [154] T. Wehrbein, M. Rudolph, B. Rosenhahn, and B. Wandt, “Probabilistic monocular 3D human pose estimation with normalizing flows,” _arXiv preprint arXiv:2107.13788_ , 2021.