# Deep Reinforcement Learning with Symmetric Prior for Predictive Power
Allocation to Mobile Users

Jianyu Zhao, Chenyang Yang
Beihang University, Beijing, China
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep reinforcement learning has been applied for a variety of wireless tasks,
which is however known with high training and inference complexity. In this
paper, we resort to deep deterministic policy gradient (DDPG) algorithm to
optimize predictive power allocation among $K$ mobile users requesting video
streaming, which minimizes the energy consumption of the network under the no-
stalling constraint of each user. To reduce the sampling complexity and model
size of the DDPG, we exploit a kind of symmetric prior inherent in the actor
and critic networks: permutation invariant and equivariant properties, to
design the neural networks. Our analysis shows that the free model parameters
of the DDPG can be compressed by $2/K^{2}$. Simulation results demonstrate
that the episodes required by the learning model with the symmetric prior to
achieve the same performance as the vanilla policy reduces by about one third
when $K=10$.

##  I Introduction

Deep reinforcement learning (DRL) has a wide range of applications in wireless
tasks [1, 2, 3, 4], aimed to make decision for resource management in an on-
line, end-to-end, model-free or distributed manner.

One of the important applications is predictive resource allocation, which has
been shown to provide a remarkable gain in terms of boosting the performance
of mobile networks by optimizing radio resources based on future information
[5, 6, 7]. Most existing works optimize the resource allocation policy either
assuming perfect future information [7] or using the predicted information
with machine learning [5, 6]. By resorting to reinforcement learning, the
policy can be optimized directly from current and past observations, as
illustrated in [8] by designing an energy-saving policy for video transmission
under the quality-of-service (QoS) constraint of a mobile user.

Nonetheless, DRL-based solutions are known with high sample complexity, i.e.,
the number of episodes required to achieve a desirable performance is
prohibitively large. For example, the policy derived in [8] converges to the
optimal policy after $5\times 10^{4}$ episodes, which is about 2000 hours with
each episode nearly 150 s! This is unacceptable for the practical use of most
wireless applications.

One possible approach to improve sample efficiency is to introduce inductive
bias into the structure of deep neural networks (DNNs) [9]. By exploiting the
priori knowledge for the input-output relation underlying a task, the
hypothesis space for searching the model parameters of a DNN can be reduced,
such that fewer samples are required for training. As a large class of priori
knowledge, permutation invariance (PI) or permutation equivalence (PE) has
been embedded into DNN by parameter sharing to reduce the sample complexity
[10]. A natural question is: can we harness this class of priors for DRL to
reduce the required episodes?

In this paper, we make an attempt to reduce the sample complexity of DRL. In
particular, we resort to deep deterministic policy gradient (DDPG) algorithm
[11] to optimize predictive power allocation for video streaming that
minimizes the average energy consumed by base stations (BSs) to ensure the QoS
of every mobile user. By revealing and leveraging the PI and PE properties
inherent in the actor and critic networks of DDPG, we design a PE/PI-DDPG by
introducing parameter sharing into fully-connected DNNs (FC-DNNS). Our results
show that the numbers of episodes required for convergence and the free model
parameters for training the PE/PI-DDPG are much less than the FC-DNN based
DDPG when the number of users is large.

##  II System Model and Problem Formulation

Consider a learning-enabled cellular network, where $M$ BSs connected with a
central unit (CU) serve $K$ mobile users. The CU monitors and records the
status of each user via the BSs, learns a resource allocation policy for the
users, and controls the BSs to execute the policy by sending instructions. The
users move across multiple cells during video streaming. We assume that each
user is associated with the BS that has the strongest large-scale channel
gain.

Each video is divided into $N_{v}$ segments. The playback duration of each
segment is divided into $N_{f}$ time frames, each with duration $\Delta T$.
Assume that the large-scale channel gains are constant in each frame but may
change among frames. Each frame is further divided into $N_{s}$ time slots,
each with duration $\tau$, i.e., $\tau=\Delta T/N_{s}$. Assume that the small-
scale channel gains remain constant in each time slot and are independently
and identically distributed among time slots.

Denote the large-scale channel gain from the $k$th user to its associated BS
in the $t$th frame as $\alpha_{t}^{k}$, and the small-scale channel gain from
the user to the BS in the $j$th time slot of the $t$th frame as $g_{tj}^{k}$.
When the users are served with orthogonal frequency division access, the data
rate of the $k$th user in the $j$th time slot of the $t$th frame can be
expressed as
$R_{tj}^{k}=W^{k}\log_{2}\left(1+\frac{\alpha_{t}^{k}g_{tj}^{k}}{\sigma_{0}^{2}}p_{tj}^{k}\right)$,
where $W^{k}$ is the bandwidth for the $k$th user, $p_{tj}^{k}$ is the
transmit power allocated to the $k$th user in the $j$th time slot of the $t$th
frame, and $\sigma_{0}^{2}$ is the noise power.

To avoid stalling, each video segment should be delivered to the buffer of
each user before playback. Denote $S_{n}^{k}$ (in bits) as the size of the
$n$th segment in the video requested by the $k$th user, then the QoS
constraint of the user can be expressed as
$\sum_{n=1}^{l}\sum_{t=(n-1)N_{f}+1}^{nN_{f}}\sum_{j=1}^{N_{s}}\tau
R_{tj}^{k}\geq\sum_{n=2}^{l+1}S_{n}^{k}$, $l=1,\cdots,N_{v}-1$. The total
energy consumed at the BSs by delivering the video to the $k$th user in the
$t$th time frame is $E_{t}^{k}=\frac{1}{\rho}\sum_{j=1}^{N_{s}}\tau
p_{tj}^{k}+\Delta TP_{c}$, where $\rho$ reflects the impact of power
amplifier, cooling and power supply, and $P_{c}$ is the power for operating
the baseband and radio frequency circuits.

We optimize power allocation among the users to minimize the average total
energy consumed at the BSs required to ensure the QoS of every mobile user,
i.e.,

$\displaystyle\min_{\\{p_{tj}^{k}\\}}~{}$
$\displaystyle\mathbb{E}\left[\sum_{k=1}^{K}\left(\sum_{t=1}^{N_{v}N_{f}}\left(\frac{1}{\rho}\sum_{j=1}^{N_{s}}\tau
p_{tj}^{k}+\Delta TP_{c}\right)\right)\right]$ (1a) $\displaystyle s.t.~{}$
$\displaystyle\sum_{n=1}^{l}\sum_{t=(n-1)N_{f}+1}^{nN_{f}}\sum_{j=1}^{N_{s}}\tau
R_{tj}^{k}\geq\sum_{n=2}^{l+1}S_{n}^{k},$ (1b)
$\displaystyle~{}l=1,\cdots,N_{v}-1,k=1,\cdots,K$
$\displaystyle\sum_{k=1}^{K}p_{tj}^{k}I(m,k)\leq P_{\max},\quad m=1,\cdots,M$
(1c)

where the average is taken over both large-scale and small-scale channel
gains, $I(m,k)$ is an indicator function with $I(m,k)$ = 1 if the $k$th user
is associated with the $m$th BS, and $I(m,k)$ = 0 otherwise.

At the time instance when a user initiates video streaming, i.e., the start of
the first time slot in the first frame, the future values of $\alpha_{t}^{k}$
and $g_{tj}^{k},t=1,\cdots,N_{v}N_{f},j=1,\cdots,N_{v}N_{f}Ns$ are unknown. To
find the solution of the problem in (1), we resort to DRL to make the
prediction and optimization simultaneously from observations in an end-to-end
manner.

##  III DRL-based Predictive Power Allocation

In a standard RL framework, the agent observes a state $s_{t}$ from
environment at time step $t$ and selects an action $a_{t}$ based on a policy
$\pi$. One time step later, the agent receives a reward $r_{t+1}$ as the
consequence of the action, and observes a new state $s_{t+1}$. The goal of the
agent is to find a policy $\pi^{*}$ that maximizes the expected return
starting from $s$, i.e., the state-value function
$V_{\pi}\left(s\right)=\mathbb{E}\left[\sum_{t=0}^{T}\gamma^{t}r\left(s_{t},\pi\left(s_{t}\right)\right)|s_{0}=s\right]$,
where $\gamma$ denotes the discount factor.

###  III-A Formulating Problem (1) in RL Framework

The power allocation optimization can be formulated as the following RL
problem, where the CU serves as the agent.

####  III-A1 Action

A straightforward formulation is to regard $p_{tj}^{k}$ as the action, which
however is harder to learn and incurs large signaling overhead between the CU
and BSs [8]. As analyzed in [8], optimizing $p_{tj}^{k}$ is equivalent to
optimize the average rate
$\bar{R}_{t}^{k}\triangleq\mathbb{E}_{g}[R_{tj}^{k}]$. In particular, a BS can
adjust the transmit power in the $j$th time slot of the $t$th frame with the
average rate according to
$p_{tj}^{k}=p^{opt}\left(\alpha_{t}^{k},g_{ti}^{k},\bar{R}_{t}^{k}\right)$,
which can be obtained from a water-filling power allocation policy and the
relation between $\bar{R}_{t}^{k}$ and the water-level. Therefore, the action
vector is the average rate of all users in each frame,

$\bm{a}_{t}=\left[\bar{R}_{t}^{1},\bar{R}_{t}^{2},\cdots,\bar{R}_{t}^{K}\right]^{\mathsf{T}}$
(2)

where $[\cdot]^{\mathsf{T}}$ denotes transpose. Then, the duration of a time
step in our RL formulation is equal to the frame duration.

####  III-A2 State

To ensure the QoS and maximal power constraints, the buffer status and the
associated BS of each user should be considered. Denote $B_{t}^{k}$ and
$M_{t}^{k}$ as the amount of data remaining in the buffer of the $k$th user
and the index of the BS the $k$th user associated with in time step $t$,
respectively. Denote $l_{t}^{k}$ as the index of the frame of the video
segment the $k$th user playback at time step $t$. Since $l_{t}^{k}$ reflects
the playback progress of the current segment of the user and affects the
transition of $B_{t}^{k}$ to $B_{t+1}^{k}$, it should be included into the
state. Since the fraction of a video having been downloaded affects the
termination of an episode, the ratio of the accumulatively downloaded bits to
the whole video of the $k$th user,
$\eta_{t}^{k}=\frac{\sum_{i=1}^{t}\sum_{j=1}^{N_{s}}\tau
R_{ij}^{k}}{\sum_{n=1}^{N_{s}}S_{n}^{k}}$, is useful for the agent to make the
optimization. Since the average energy consumption for video streaming depends
on the large-scale channel gain, $\alpha_{t}^{k}$ should be an element of the
state. To help the agent make the prediction, the large-scale channel gains in
the past $N_{t}$ time steps are also included in the state. Since a user may
have the same large-scale channel gain at different cells,
$\bm{\alpha}_{t}^{k}\triangleq[\alpha_{1,t}^{k},\cdots,\alpha_{N_{b},t}^{k}]$
should be in the state, where $\alpha_{i,t}^{k}$ is the large-scale channel
gain between the $k$th user and the $i$th neighbouring BS and
$\alpha_{1,t}^{k}=\alpha_{t}^{k}$. Then, the state vector of the $k$th user
can be expressed as
$\bm{s}_{t}^{k}\triangleq[B_{t}^{k},l_{t}^{k},\eta_{t}^{k},M_{t}^{k},\bm{\alpha}_{t-N_{t}}^{k},\cdots,\bm{\alpha}_{t}^{k}]$.
Finally, the state matrix is

$\footnotesize{\begin{matrix}\bm{s}_{t}\\!=\\!\begin{pmatrix}\bm{s}_{t}^{1}\\\
\vdots\\\
\bm{s}_{t}^{K}\end{pmatrix}\\!=\\!\begin{pmatrix}B_{t}^{1}&l_{t}^{1}&\eta_{t}^{1}&M_{t}^{1}&\bm{\alpha}_{t-N_{t}}^{1}&\\!\cdots\\!&\bm{\alpha}_{t}^{1}\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\quad&\vdots\\\
B_{t}^{K}&l_{t}^{K}&\eta_{t}^{K}&M_{t}^{K}&\bm{\alpha}_{t-N_{t}}^{K}&\\!\cdots\\!&\bm{\alpha}_{t}^{K}\\\
\end{pmatrix}\end{matrix}}$ (3)

which has $K$ rows and $4+\left(N_{t}+1\right)N_{b}$ columns.

####  III-A3 Reward

To ensure the constraints in (1b) and (1c), one can employ the safe layer
method [12], where at time step $t=lN_{f}$, $l=1,\cdots,N_{v}-1$, action
$\bm{a}_{t}$ is transformed into $\tilde{\bm{a}}_{t}$ by solving the following
optimization problem,

$\displaystyle\min_{\tilde{\bm{a}}_{t}}~{}$
$\displaystyle\left\|\bm{a}_{t}-\tilde{\bm{a}}_{t}\right\|_{2}$ (4a)
$\displaystyle s.t.~{}$
$\displaystyle\bm{a}_{1}+\cdots+\bm{a}_{t-1}+\bm{a}_{t}\geq\sum_{n=2}^{l+1}\bm{S}_{n}$
(4b)
$\displaystyle\sum_{k=1}^{K}p^{opt}\left(\alpha_{lN_{f}}^{k},g_{lN_{f}i}^{k},\bar{R}_{t}^{k}\right)I(m,k)\leq
P_{\max}$ (4c) $\displaystyle m=1,\cdots,M$

where $\bm{S}_{n}\triangleq\left(S_{n}^{1},\cdots,S_{n}^{K}\right)$. However,
this optimization problem may not have feasible solution for the problem at
hand. For example, if users are not in good channel conditions such that the
amount of data in the buffer is much less than the amount of data required for
playback at time step $t$, then the QoS cannot be ensured even when $P_{\max}$
is used. To circumvent this difficulty, we introduce the safe layer only to
ensure one constraint.

When solving the problem in (4) only with the constraint in (4b) at time step
$t$, both $\sum_{n=2}^{l+1}\bm{S}_{n}$ and $\sum_{i=1}^{t-1}\bm{a}_{t}$ are
fixed constants, thereby the solution can directly be obtained as
$\tilde{\bm{a}}_{t}=\sum_{n=2}^{l+1}\bm{S}_{n}-\sum_{i=1}^{t-1}\bm{a}_{t}$.
However, the problem only with the constraint in (4c) does not have a simple
solution due to the coupling of the users. Hence, we introduce a safe layer to
satisfy the QoS constraint and impose a penalty on the reward when the maximal
power constraint is not satisfied.

The reward for the agent is designed as

$\small r_{t}=-\sum_{k=1}^{K}\sum_{j=1}^{N_{s}}\tau
p_{tj}^{k}-\lambda\sum_{m=1}^{M}\sum_{k=1}^{K}\left(\sum_{j=1}^{N_{s}}\tau
p_{tj}^{k}I(m,k)-P_{\max}\right)^{+}$ (5)

where $\left(x\right)^{+}=\max\left\\{x,0\right\\}$, $\lambda$ is the penalty
coefficient, and $-\sum_{k=1}^{K}\sum_{j=1}^{N_{s}}\tau p_{tj}^{k}$ is the
total transmit energy consumed by all users in the $t$th time step.

###  III-B Transmission Policy Based on DDPG

Since the state matrix lies in continuous space, we resort to DDPG [11] to
solve the problem (1). DDPG maintains two DNNs stored at the CU, namely actor
network $\mu(\mathbf{s};\bm{\theta}_{\mu})$ and critic network
$Q(\mathbf{s},\mathbf{a};\bm{\theta}_{Q})$. The actor network learns the
_policy function_ (i.e., the mapping from the state matrix to the action
vector), whose output is then used to compute $p_{tj}^{k}$. The critic network
learns the _action-value function_ (i.e.,
$Q^{\pi}(\mathbf{s},\mathbf{a})\triangleq\mathbb{E}\left[\sum_{t=0}^{T}\gamma^{t}r\left(s_{t},a_{t}\right)|s_{t}=\mathbf{s},a_{t}=\mathbf{a},\pi\right]$).

During the interactions with the environment, the CU collects the experience
$\mathbf{e}_{t}=[\mathbf{s}_{t},\mathbf{a}_{t},r_{t},\mathbf{s}_{t+1}]$ from
the BSs in a database as
$\mathcal{D}_{\mathcal{B}}=\\{\mathbf{e}_{1},\cdots,\mathbf{e}_{t}\\}$. At
each iteration, a mini-batch of experience is sampled from the database
$\mathcal{D}_{\mathcal{B}}$ to update the model parameters, i.e., _experience
replay_ [13].

The model parameters of the critic network are updated with gradient descent
as

$\bm{\theta}_{Q}\leftarrow\bm{\theta}_{Q}-\frac{\delta_{Q}}{|\mathcal{B}|}\nabla_{\bm{\theta}_{Q}}\sum_{j\in\mathcal{B}}\left[y_{j}-Q(\mathbf{s}_{j},a_{j};\bm{\theta}_{Q})\right]^{2}$
(6)

where $y_{j}=r_{j}+\gamma
Q^{\prime}(\mathbf{s}_{j+1},\mu^{\prime}(\mathbf{s}_{j+1};\bm{\theta}_{\mu}^{\prime});\bm{\theta}_{Q}^{\prime})$,
$\delta_{Q}$ is the learning rate of critic network,
$Q^{\prime}(\mathbf{s},a;\bm{\theta}_{Q}^{\prime})$ and
$\mu^{\prime}(\mathbf{s};\bm{\theta}_{\mu}^{\prime})$ are the target critic
network and target actor network, respectively, which have the same structure
as $Q(\mathbf{s},\mathbf{a};\bm{\theta}_{Q})$ and
$\mu(\mathbf{s};\bm{\theta}_{\mu})$, and are updated by
$\bm{\theta}_{Q}^{\prime}\leftarrow\bm{\omega}\theta_{Q}+(1-\omega)\bm{\theta}_{Q}^{\prime}$
and
$\bm{\theta}_{\mu}^{\prime}\leftarrow\bm{\omega}\theta_{\mu}+(1-\omega)\bm{\theta}_{\mu}^{\prime}$
with very small value of $\omega$ to stabilize the learning procedure [11].

The model parameters of the actor network are updated using the sampled policy
gradient as

$\bm{\theta}_{\mu}\leftarrow\bm{\theta}_{\mu}+\frac{\delta_{\mu}}{|\mathcal{B}|}\nabla_{a}Q(\mathbf{s},a;\bm{\theta}_{Q})|_{\mathbf{s}=\mathbf{s}_{i},a=\mu(s_{i};\bm{\theta}_{\mu})}\nabla_{\bm{\theta}_{\mu}}\mu(\mathbf{s}|\bm{\theta}_{\mu})|_{\mathbf{s}=\mathbf{s}_{i}}$
(7)

where $\delta_{\mu}$ is the learning rate of the actor network.

We add a noise term $n_{t}$ sampled from a noise process $\mathcal{N}$ to the
output of the actor network, i.e.,
$a_{t}=\mu(\mathbf{s}_{t};\bm{\theta}_{\mu})+n_{t}$, to facilitate the
exploration.

##  IV PE/PI-DDPG based Predictive Power Allocation

In this section, we show that the policy function and the action-value
function exhibit a kind of symmetric properties. We then design a PE/PI-DDPG,
where the actor and critic networks are constructed by exploiting the
properties.

###  IV-A Permutation Equivariant and Invariant Properties

We first define the PE and PI properties to be used in the sequel. Consider
arbitrary permutation matrix $\bm{\Pi}$. A multivariate function
$\bm{y}=f(\bm{X})$ is one-dimension (1D)-permutation equivariant to $\bm{X}$
if $\bm{\Pi}\bm{y}=f(\bm{\Pi}\bm{X})$, where $\bm{X}$ is a matric and $\bm{y}$
is a vector. A function $y=f(\bm{X})$ is 1D-permutation invariant to $\bm{X}$
if $y=f(\bm{\Pi}\bm{X})$, where $y$ is a scalar.

The policy function $\mathbf{a}=\tilde{\mu}(\mathbf{s})$ maps the state of all
users $\bm{s}=[\bm{s}^{1},\bm{s}^{2},\cdots,\bm{s}^{K}]^{\mathsf{T}}$ into the
average rates of all users, i.e.,
$\bm{a}=\left[\bar{R}^{1},\bar{R}^{2},\cdots,\bar{R}^{K}\right]^{\mathsf{T}}$.
When the order of users in the state changes (say $\bm{s}^{1}$ swaps with
$\bm{s}^{2}$), the order of average rates changes in the same way
($\bar{R}^{1}$ swaps with $\bar{R}^{2}$) but the policy function remains
unchanged, i.e., $\bm{\Pi}\mathbf{a}=\tilde{\mu}(\bm{\Pi}\mathbf{s})$. Hence,
the policy function is 1D-permutation equivariant to the state.

The action-value function $q=\tilde{Q}(\mathbf{s},\mathbf{a})$ maps the state
and action of all users into the expected return $q$. By stacking the state
and action according to the users’ indexes,

${\begin{matrix}\bm{(}\mathbf{s},\mathbf{a})=\begin{pmatrix}B^{1}&l^{1}&\eta^{1}&M^{1}&\cdots&\bar{R}^{1}\\\
\vdots&\vdots&\vdots&\vdots&\quad&\vdots\\\
B^{K}&l^{K}&\eta^{K}&M^{K}&\cdots&\bar{R}^{K}\\\ \end{pmatrix}\end{matrix}}$
(8)

we can see that the expected return will remain unchanged if the order of the
users changes. Hence, the action-value function is 1D-permutation invariant to
$(\mathbf{s},\mathbf{a})$.

###  IV-B DDPG with PE/PI-DNNs

By introducing parameter sharing into the FC-DNN, permutation equivariant DNN
(PE-DNN) and permutation invariant DNN (PI-DNN) can be used to approximate the
1D-PE and the 1D-PI functions, respectively.

With the input as $\bm{X}\in\mathbb{R}^{n\times D}$, the weight matrix and the
bias vector of the $i$th layer of PE-DNN can be respectively constructed as
[10]

${\begin{matrix}\bm{W}_{i}=\begin{pmatrix}U_{i}&V_{i}&\cdots&V_{i}\\\
V_{i}&U_{i}&\cdots&V_{i}\\\ \vdots&\vdots&\ddots&\vdots\\\
V_{i}&V_{i}&\cdots&U_{i}\\\ \end{pmatrix},&\bm{b}_{i}=\begin{pmatrix}P_{i}\\\
P_{i}\\\ \vdots\\\ P_{i}\\\ \end{pmatrix}\end{matrix}}$ (9)

where $U_{i}$ and $V_{i}$ are sub-matrices, $P_{i}$ is a subvector. The number
of columns of $U_{i}$ and $V_{i}$ is equal to $D$. The numbers of sub-matrices
in each row and each column in weight matrix $\bm{W}_{i}$ are equal, which is
identical to the number of subvectors in bias $\bm{b}_{i}$, and all are equal
to $n$.

For PI-DNN with input $\bm{X}\in\mathbb{R}^{n\times D}$, the bias of the
output layer is the same as FC-DNN and the weight matrix of the output layer
can be constructed as

${\begin{matrix}\bm{W}_{i}=\begin{pmatrix}A_{i}&A_{i}&\cdots&A_{i}\end{pmatrix}\end{matrix}}$
(10)

where $A_{i}$ is a sub-matrix with the number of columns equal to $D$, and the
number of the sub-matrices in the weight matrix $\bm{W}_{i}$ is equal to $n$.
The input and all hidden layers are constructed the same as PE-DNN.

By designing the actor network as a PENN and the critic network as a PINN, the
free model parameters that need to be trained can be computed as follows.
Since the number of free parameters in the bias is far less than that in the
weight matrix, we only count the parameters in the weight matrix.

For notational simplicity, suppose that the number of layers is $H$ and the
number of neurons in each hidden layer is $d$ for both actor and critic
networks.

For the actor network, the dimension of the input is
$K\times\left(4+\left(N_{t}+1\right)N_{b}\right)$, i.e., $n=K$ and
$D=4+\left(N_{t}+1\right)N_{b}$, the dimension of the output is $K\times 1$.
The free parameters in the weight matrix are in two sub-matrices, hence the
number of parameters in the $i$th hidden layer can be expressed as
$\frac{2d^{2}}{K^{2}}$, and the number of parameters in the input layer and
the output layer can be expressed as
$\frac{2dK\left(4+\left(N_{t}+1\right)N_{b}\right)}{K^{2}}$ and
$\frac{2dK}{K^{2}}$, respectively. Consequently, the total number of model
parameters can be obtained as
$\frac{2\left(H-2\right)d^{2}+2dK\left(4+\left(N_{t}+1\right)N_{b}\right)+2dK}{K^{2}}$.

For the critic network, the dimension of the input is
$K\times\left(5+\left(N_{t}+1\right)N_{b}\right)$, i.e., $n=K$ and
$D=5+\left(N_{t}+1\right)N_{b}$, the output is a scalar. The free parameters
of the output layer in the weight matrix are in one sub-matrix, hence the
number of parameters can be expressed as $\frac{d}{K}$, and the number of free
parameters in the $i$th layer is the same as PE-DNN, for $1\leq i\leq H-1$.
Therefore, the total number of free parameters can be obtained as
$\frac{2\left(H-2\right)d^{2}+2dK\left(5+\left(N_{t}+1\right)N_{b}\right)+dK}{K^{2}}$.

If FC-DNNs are used, then the number of free parameters of the actor network
is $\left(H-2\right)d^{2}+dK\left(4+\left(N_{t}+1\right)N_{b}\right)+dK$, and
the number of free parameters of the critic network is
$\small\left(H-2\right)d^{2}+dK\left(5+\left(N_{t}+1\right)N_{b}\right)+d$.
Hence, the actor and critic networks with parameter sharing can reduce the
model parameters by
$\frac{K^{2}\left(\left(H-2\right)d^{2}+dK\left(4+\left(N_{t}+1\right)N_{b}\right)+dK\right)}{2\left(H-2\right)d^{2}+2dK\left(4+\left(N_{t}+1\right)N_{b}\right)+2dK}$
and
$\frac{K^{2}\left(\left(H-2\right)d^{2}+dK\left(5+\left(N_{t}+1\right)N_{b}\right)+d\right)}{2\left(H-2\right)d^{2}+2dK\left(5+\left(N_{t}+1\right)N_{b}\right)+dK}$
times with respect to the FC-DNN, respectively. When the width of the hidden
layer $d$ is much larger than the dimensions of the input and the output of
the actor and critic networks, we can obtain

$\footnotesize\begin{split}\frac{K^{2}\left(\left(H-2\right)d^{2}+dK\left(4+\left(N_{t}+1\right)N_{b}\right)+dK\right)}{2\left(H-2\right)d^{2}+2dK\left(4+\left(N_{t}+1\right)N_{b}\right)+2dK}\\\
\approx\frac{K^{2}\left(\left(H-2\right)d^{2}+dK\left(5+\left(N_{t}+1\right)N_{b}\right)+d\right)}{2\left(H-2\right)d^{2}+2dK\left(5+\left(N_{t}+1\right)N_{b}\right)+dK}\approx\frac{K^{2}}{2}\end{split}$
(11)

This indicates that the number of model parameters in PE/PI-DDPG is $2/K^{2}$
of the DDPG based on FC-DNNs.

##  V Simulation Results

In this section, we evaluate the performance of the PE/PI-DDPG by comparing
with the DDPG based on FC-DNNs.

###  V-A Simulation Setup

Figure 1: Simulation scenario.

Consider a cellular network with multiple BSs located along a straight line,
and $K$ users move along a straight road across cells, as shown in Fig. 1. The
inter-BS distances are 500 m, and the minimum distance between the BSs and the
road is 200 m. The maximal transmit power of each BS is 46 dBm. The noise
power is -95 dBm/Hz and the bandwidth for each user is 2 MHz. Since the
circuit energy consumption is identical for all the considered policies, we
only consider transmit energy consumption. The path loss is modeled as
$35.3+37.6\log_{10}(d)$ in dB, where $d$ is the distance between user and BS
in meters. The small-scale channels follow Rayleigh fading. The playback
duration of each video and each segment is 150 s and 10 s, respectively. Each
segment is with size $1$ MBytes. Each time frame is with the duration of
$\Delta T=1$ s, and each time slot is with the duration of $\tau=1$ ms, i.e.,
each frame contains $N_{s}=$ 1000 time slots. The user moves with random
acceleration, where the acceleration in each frame is drawn from the Gaussian
distribution with zero mean and standard deviation 0.5 m/s2. The initial
velocity of users is set as 16 m/s, and the minimal and maximal velocities of
each user are 12 m/s and 20 m/s, respectively.

###  V-B Fine-Tuned Parameters for PE/PI-DDPG

The actor network has four hidden layers each with 600 nodes, and employs a
modified $\tanh$ function $0.5\times(\tanh(x)+1)$ as the activation function
in the output layer to bound the actions, where the upper bound is the average
rate of the user in the best channel conditions on the road when $P_{\max}$ is
used. The critic network first stacks the state and action together and then
goes through four hidden layers each with 600 nodes, and has no activation
function in the output layer. All the hidden layers in the actor and critic
networks use the rectified linear unit (ReLU) as the activation function. We
use Adam [14] for learning the model parameters with a learning rate of 10-4
and 10-3 for the actor and critic networks, respectively. For the critic
network, we include $L_{2}$ weight decay of 10-4 to avoid over-fitting and use
a discount factor of $\gamma=1$.

We set $N_{b}=2$ and $N_{t}=2$ in the state. The update rate for the target
networks is $\omega=10^{-3}$. The replay memory size is
$|\mathcal{D}_{\mathcal{B}}|=10^{6}$, and the mini-batch size for gradient
descent is $|B|$ = 512. The penalty coefficient is set as $0.1$. The noise
term $n_{t}$ follows Gaussian distribution with zero mean and the variance
decreased linearly from $0.3$ to $0$.

###  V-C Performance Evaluation

We compare the PE/PI-DDPG with the DDPG based on FC-DNNs from three aspects in
terms of the total average energy consumption, the sample complexity, and the
number of free parameters in DNNs.

####  V-C1 Average Energy Consumption

To evaluate the performance of the proposed policy, we compare the total
average energy consumption of all users for each video achieved by the DDPG-
based policies and the optimal policy, which is obtained by solving the
problem (1) assuming perfect prediction of large- and small-scale channel
gains.

Figure 2: Return $\sim$ episodes, averaged over 10 Monte Carlo trials (each
with random initial user locations and accelerations) over 400 successive
episodes.

In Fig. 2, we show the learning curves of the PE/PI-DDPG and the DDPG-based on
FC-DNNs (with legend “DDPG”). Since there is no penalty in return after
convergence, the negative of the converged return is the total energy consumed
for all users. From Fig. 2(a), we can see that both DDPG-based policies can
converge to the optimal policy. From Fig. 2(b), we can see that the PE/PI-DDPG
approaches to the optimal policy much faster than the DDPG-based on FC-DNNs.

####  V-C2 Sample Complexity

The sample complexity of DRL is defined as the minimal number of episodes to
achieve an expected performance on the test set.

TABLE I: Sample Complexity with different number of users. $K$ | PE/PI-DDPG | DDPG | Compression ratio
---|---|---|---
| 2
---
39,000 | 39,000 |  100$\%$
5 | 80,000 | 130,000 |  61$\%$
10 | 135,000 | 400,000 |  34$\%$

In Table I, we compare the number of episodes required by PE/PI-DDPG to
achieve the same expected return with the DDPG with FC-DNNs (simply denoted as
“DDPG”). We can see that the compression ratio increases with the number of
users. When $K=10$, the sample complexity of the PE/PI-DDPG is much lower than
the DDPG with FC-DNNs, i.e., the PE/PI-DDPG can converge three times faster.

####  V-C3 Number of Free Parameters

In Table II, we provide the number of model parameters in the PE/PI-DDPG and
the DDPG based on FC-DNNs (again denoted as “DDPG”). We can see that the
compression ratio is $2/K^{2}$, which coincides with (11). This indicates that
the model size (and hence the memory to store the model for inference) of the
PE/PI-DDPG is small for large number of users.

TABLE II: Number of free parameters with different number of users. $K$ | PE/PI-DDPG | DDPG | Compression Ratio
---|---|---|---
| 2
---
2,907,000 (2.9 M) | 5,814,000 (5.8 M) |  50$\%$
5 | 465,264 (0.46 M) | 5,893,200 (5.9 M) |  8$\%$
10 | 116,376 (0.12 M) | 6,025,200 (6.0 M) |  2$\%$

##  VI Conclusion

In this paper, we strived to reduce the sample complexity and model size of
DRL-based policy by harnessing symmetric priors. We optimized predictive power
allocation for video streaming over wireless networks to minimize the average
energy consumption under the QoS constraint of every mobile user with DDPG. By
exploiting the permutation invariant and equivariant properties in the actor
and critic networks, we constructed the two deep neural networks with
parameter sharing. Simulation results showed that the number of episodes and
the number of free model parameters required by the PI/PE-DDPG to achieve the
same energy consumption of the DDPG based on FC-DNNs reduce quickly and even
dramatically with the number of users. In fact, by designing the actor network
as PI-DNN and the critic network as PE-DNN, the PI/PE-DDPG can adapt to the
change of the number of users in addition to reducing the sample complexity
and model size, which is important for mobile networks but will be evaluated
in future.

## References

  * [1] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” _IEEE Commun. Surveys Tuts._ , vol. 21, no. 4, pp. 3133–3174, Fourth quarter 2019.
  * [2] Z. Zhang, Y. Yang, M. Hua, C. Li, Y. Huang, and L. Yang, “Proactive caching for vehicular multi-view 3D video streaming via deep reinforcement learning,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 5, pp. 2693–2706, May 2019.
  * [3] J. Zhang, Y. Huang, J. Wang, and X. You, “Intelligent beam training for millimeter-wave communications via deep reinforcement learning,” in _Proc. IEEE GLOBECOM_ , 2019.
  * [4] M. Gadaleta, F. Chiariotti, M. Rossi, and A. Zanella, “D-DASH: A deep Q-learning framework for DASH video streaming,” _IEEE Trans. Cogn. Commun. and Netw._ , vol. 3, no. 4, pp. 703–718, 2017.
  * [5] N. Bui and J. Widmer, “Data-driven evaluation of anticipatory networking in LTE networks,” _IEEE Trans. on Mobile Comput._ , vol. 17, no. 10, pp. 2252–2265, Oct. 2018.
  * [6] U. Challita, L. Dong, and W. Saad, “Proactive resource management for LTE in unlicensed spectrum: A deep learning perspective,” _IEEE Trans. Wireless Commun._ , vol. 17, no. 7, pp. 4674–4689, Jul. 2018.
  * [7] C.She and C.Yang, “Energy efficient resource allocation for hybrid services with future channel gains,” _IEEE Trans. Green Commun. and Netw._ , vol. 4, no. 1, pp. 165–179, Mar. 2020.
  * [8] D. Liu, J. Zhao, and C. Yang, “Energy-saving predictive video streaming with deep reinforcement learning,” in _Proc. IEEE GLOBECOM_ , 2019.
  * [9] B. Matthew, R. Sam, X. W. Jane, K.-N. Zeb, B. Charles, and H. Demis, “Reinforcement learning, fast and slow,” _Trends Cogn. Sci._ , vol. 23, no. 5, pp. 408–422, 2019.
  * [10] S. Ravanbakhsh, J. Schneider, and B. Poczos, “Equivariance through parameter-sharing,” in _Proc. JMCR ICML_ , 2017.
  * [11] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in _Proc. ICLR_ , 2015.
  * [12] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa, “Safe exploration in continuous action spaces,” _arXiv preprint_ , 2018\. [Online]. Available: http://arxiv.org/abs/1801.08757
  * [13] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, p. 529, Feb. 2015.
  * [14] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , 2014.


# One-Shot Open-Set Skeleton-Based Action Recognition

Stefano Berti, Andrea Rosasco, Michele Colledanchise, and Lorenzo Natale The
authors are with the Humanoids Sensing and Perception, Istituto Italiano di
Tecnologia. Genoa, Italy. e-mail<EMAIL_ADDRESS>
###### Abstract

Action recognition is a fundamental capability for humanoid robots to interact
and cooperate with humans. This application requires the action recognition
system to be designed so that new actions can be easily added, while “unknown”
actions are identified and ignored. In recent years, deep-learning approaches
represented the principal solution to the Action Recognition problem. However,
most models often require a large dataset of manually-labeled samples. In this
work we target One-Shot deep-learning models, because they can deal with just
a single instance for class. Unfortunately, One-Shot models assume that, at
inference time, the action to recognize falls into the support set and they
fail when the action lies outside the support set. Few-Shot Open-Set
Recognition (FSOSR) solutions attempt to address that flaw, but current
solutions consider only static images and not sequences of images. Static
images remain insufficient to discriminate actions such as sitting-down and
standing-up.

In this paper we propose a novel model that addresses the FSOSR problem with a
One-Shot model that is augmented with a discriminator that rejects “unknown”
actions. This model is useful for applications in humanoid robotics, because
it allows to easily add new classes and determine whether an input sequence is
among the ones that are known to the system. We show how to train the whole
model in an end-to-end fashion and we perform quantitative and qualitative
analyses. Finally, we provide real-world examples.

##  I Introduction

Action Recognition (AR) represents a fundamental task for Human-Robot
Interaction [1, 2, 3], video surveillance [4, 5], and sport-analysis [6, 7],
to mention just a few. Different AR solutions require different data, such as
still images [8] [9], sequences of images [10] [11], and sequences of 2D/3D
skeleton data [12] [13]. The AR-based on sequences of 2D/3D skeleton data has
the advantage to use less computation power and ignores the background,
focusing only on the human pose [14]. To train an AR pipeline, we often need
big labeled datasets, especially for Deep-Learning models. These datasets
require manual collection and annotation, and the resulting model requires a
fine-tuning, or worse, full retraining to simply add or change a class. These
models are badly suited for those applications in robotics in which the robot
is required to learn new actions from a few demonstrations and, perhaps more
importantly, should be able to recognize when a given input belong to one of
the “known” classes. A class of models called Few-Shot Learning (FSL) aims at
solving the problem above by adapting to recognize new classes given few
samples. The FSL models for AR are called Few-Shot Action-Recognition (FSAR)
models [15, 16]. Usually, in a few-shot problem, the model receives as input a
set of few examples for each class called _support set_ and a query; then, the
model outputs the probabilities for the query to belong to a class in the
support set. However, in practical applications, it is not possible to foresee
in advance all the actions humans will perform while interacting with the
robot. In this “Open Set” (OS) scenario, we cannot add all the possible
classes of actions (which are infinite) to the support set and reliably
classify each sample. The OS problem combined with the FSL problem yields the
Few-Shot Open-Set Recognition (FSOSR) [8] problem and its relative application
to AR is called Few-Shot Open-Set Action Recognition (FSOSAR). Typically this
problem is solved by creating a “reject” class, whose instances can be sampled
from the training set and can be inferred from the distances between the query
and the elements of the support set [8] [9]. While the FSOSAR problem has been
investigated for images [8] [9], to the best of our knowledge, there is no
work that addresses the FSOSAR problem for sequences.

Figure 1: An overview of our approach for the One-Shot Three-Way problem. The
query class ”relaxing” is not present in the support set, but the FS part of
the model chooses the ”waving” class as the solution. Our discriminator _Disc_
discards the FS prediction because it provides a low confidence score (0.11).
Thus, our approach selects ”reject” (i.e. “unknown”) as the final class.

In this work, we propose a solution to the FSOSAR problem by defining a
discriminator that assigns a confidence score to the difference between the
features of the query and the features of the predicted class. With episodic
end-to-end learning, we force this difference to have a high norm value when
the difference is high and low otherwise. Moreover, the discriminator forces
this difference to contain information about similarities and differences
between the query prototype and the query-class prototype of the predicted
class. Finally, we propose to use features obtained from a 3D skeleton of the
human, resulting in a simpler model that is less computationally expensive and
easier to train. An overview of our approach is depicted in Figure 1. Our
contributions can be summarized as follows:

  * •

We created a model that deals with Few-Shot Open-Set learning for sequences of
3D skeleton.

  * •

We proposed an episodic end-to-end learning technique that balances the
outcome of the discriminator and allows to jointly train the FSL part and the
OS part of the model.

  * •

We demonstrated qualitatively that the model is capable of differentiating
similar but semantically different actions.

We released the code111Available at https://github.com/stefanoberti/ISBFSAR
and the dataset222Available at https://bit.ly/3OcD9yb in order to make all our
experiments easily reproducible.

##  II Related Work

In this section we give a brief review of various FSL methods, analyzing the
advantages and disadvantages of different types of input data and the main
differences with respect to our approach.

###  II-A Few-Shot Action Recognition (videos)

A recent work [11] proposed to aggregate frame-level features of both the RGB
image and the optical flow across time. For the time-aggregation of the
features, they tested Pooling, Long-Short Term Memory (LSTM), ConvLSTM, and
3D-Convolutions. As a FSL algorithm, they tested Matching Networks,
Prototypical Networks, and Learned Distance Metrics. Another work [10]
proposed to use multiple Temporal Attention Vectors to encode video-wide
temporal information that adapts to the various lengths of frame sequences.
The importance scores are then used to select important temporal information.

The works above use sequences of images as input data that allow taking into
consideration the whole context, like the objects that the human is
interacting with and the surrounding elements of the environment. In contrast,
our work focus on 3D human skeleton data as we do not need to train a
Convolutional Neural Network to extract features from the important parts of
the RGB images. As a result, our model requires much less computational power
to train and fewer samples when the action is mainly discriminated through the
human pose.

###  II-B Few-Shot Action Recognition (skeleton)

A recent work [13] tackles the Skeleton-Based FSAR problem with a variant of
Dynamic Time Warping. It tries to find the best alignment in the temporal and
simulated camera viewpoint spaces between the query and the support frames. A
temporal block encoder expressed the sequences.

Our work also aims at the best temporal alignment between the query and the
support set, but in contrast we do not explicitly simulate different camera
viewpoints since the embedding layer, trained with a dataset where actions are
recorded from different orientations, makes our model implicitly invariant to
skeleton rotations.

###  II-C Few Shot Open-Set Recognition (images)

The FSOSAR problem addresses two problems at the same time: the FSAR problem
and the OS problem. FSOSAR has been studied in the literature by [8] and [9],
but mainly using still images as input data. Some approaches [8] propose to
solve the FSOSR problem with a combination of Meta-Learning, a random
selection of novel classes per episode, a loss that maximizes the posterior
entropy for examples of those classes, and a new metric learning formulation
based on the Mahalanobis distance [17]. Another approach [9] extended the one
above, taking inspiration from the concept of reconstruction-based OS
recognition methods, where an algorithm is trained to reconstruct the training
data and it fails at reconstructing unseen samples. In this work the authors
trained a transformation function based on the fact that the distance between
a query feature and its class prototype is closer than the distance between an
unknown class feature and that prototype. Then they compute the distance
between the transformed query and the transformed elements of the support set
that are not chosen as the class and they check that the difference is under a
certain threshold. In this way, they avoid dependency on pseudo-unseen
examples.

In contrast to these works, our approach considers sequences as input data and
can therefore discriminate ambiguous frames based on previous ones. For
example, by using sequences we can understand if a person is sitting down or
standing up, but that is tricky to understand from a single image.

##  III Background

The Temporal-Relational Cross-Transformer (TRX)[14] is a few-shot method for
action recognition that considers the similarity between all the possible sub-
sequences of length $2$ of the query frames and all the possible sub-sequences
of length $2$ of all the elements in the support set (the model was also
extended to sub-sequences of length ${>2}$, but in this paper we focus only on
pairs of frames). We are interested in the $K$-way $N$-shot problem with $N=1$
(also known as One-Shot). With an attention mechanism, the TRX model
constructs “query pair”-specific class prototypes and selects the class
relative to the element in the support set which has a minimum query-class
difference. Given a query ${Q\triangleq\\{q_{1},...,q_{F}\\}}$, the query
representation for the pair $p$ is defined as:

$Q_{p}\triangleq[\phi(q_{p_{1}})+PE(p_{1}),\phi(q_{p_{2}})+PE(p_{2})]\in
R^{2\times D}$

where $p_{1}$ and $p_{2}$ are ordered indices of frames such that ${1\leq
p_{1}<p_{2}\leq F}$, $F$ is the total number of frames, ${\phi:R^{H\times
W\times 3}\rightarrow R^{D}}$ is a convolutional network that extracts a
D-dimensional embedding from each input frame and $PE$ is the positional
encoding [18]. We define the set of all possible pairs as:

${\Pi\triangleq\\{(n_{1},n_{2})\in\mathbb{N}^{2}:1\leq n_{1}<n_{2}\leq F\\}}$

The support set $S$ contains a support $S^{c}=\\{s^{c}_{1},...,s^{c}_{F}\\}$
for each class $c$. A single frame-pair representation of the video with class
$c$ in the support set with respect to the ordered pair of indices
${m=(m_{1},m_{2})\in\Pi}$ is:

$S^{c}_{m}\triangleq[\phi(s^{c}_{m_{1}})+PE(m_{1}),\phi(s^{c}_{m_{2}})+PE(m_{2})]\in
R^{2\times D}$

and the set of all pair representations in the support set for class c is:

${S^{c}\triangleq\\{S^{c}_{m}:(m\in\Pi)\\}}$

By using three different linear projections called queries
${\Upsilon:R^{D\times 2}\rightarrow R^{D_{\Upsilon}}}$, with key
${\Gamma:R^{D\times 2}\rightarrow R^{D_{\Gamma}}}$ and value
${\Lambda:R^{D\times 2}\rightarrow R^{D_{\Lambda}}}$, we can compute the
query-specific class prototype for the query $Q_{p}$ with respect to the class
$c$:

${t_{p}^{c}\triangleq Softmax(L(\Gamma\cdot S_{m}^{c})\cdot L(\Upsilon\cdot
Q_{p}))\cdot(\Lambda\cdot S_{m}^{c})}$

where ${t_{p}^{c}\in R^{D_{V}}}$ and $L$ is the standard layer normalisation
[19]. Then we compute the distance between the query $Q_{p}$ mapped with
$\Lambda$ and each query-specific class prototype of the support set $S^{c}$:

${T(Q,S^{c})\triangleq\frac{1}{|\Pi|}\sum_{p\in\Pi}||t^{c}_{p}-\Lambda\cdot
Q_{p}||}$

In this way, we can compute a distance between the query and each class of the
support set. The model is then trained with a cross-entropy loss over the
query-class negative distances $T(Q,S^{c})$.

##  IV Problem Formulation

In this section, we formally formulate the problem of FSL and we extend it to
the FSOSR. We pose the problem as a single-instance classification problem,
where each class encodes a different action. Let
${D^{S}\triangleq\\{x_{i}^{S},y_{i}^{S}\\}_{i=1}^{NK}}$ be the support set,
where $x_{i}^{S}$ and $y_{i}^{S}$ represent an instance and its label
respectively, $K$ the number of classes, and $N$ the number of labeled samples
for each class. Given a known query set ${{D^{Kn}\triangleq\\{x_{i}^{Kn}\in
X^{Kn},y_{i}^{Kn}\in Y^{S}\\}_{i=1}^{Q}}}$, where $Q$ is the number of query
samples for each class, the aim of FSL is to infer for each query sample
$x_{i}^{Kn}$ the corresponding class $y_{i}^{Kn}$ that is assumed to always be
contained inside the support labels $Y^{S}$.

In an FSOSR problem, the query set contains also instances of classes that are
not part of the support set. The query set ${D^{Q}\triangleq D^{Kn}\cup
D^{Un}}$ equals the union of both the known query set ${D^{Kn}}$ and the
unknown query set ${D^{Un}\triangleq\\{x_{i}^{Un}\in X^{Un},y_{i}^{Un}\in
Y^{Un}\\}_{i=1}^{N^{Un}}}$, where ${Y^{S}\cap Y^{U}=\emptyset}$ holds and
$N^{Un}$ is the number of unknown instances. Thus, a FSOSR problem requires to
create a classifier $f$ such that given a support set $S$, for each query
sample ${x_{i}}$ and true class ${y_{i}}$, the following holds:

$\centering\bar{f}(x_{i})=\begin{cases}y_{i},&\text{if }y_{i}\in Y^{S}\\\
False,&\text{otherwise}\end{cases}\@add@centering$ (1)

Since previous work focuses on the FSOSR problem for still images [8] [9] and
fail to correctly discriminate some actions as for “standing-up” and “sitting-
down”, we address the problem using sequences of 3D skeletons. We focus on
sequences of 3D skeletons because we are mostly interested in the human pose,
we want to be able to differentiate temporal actions, and because working with
skeleton data requires much less computational power (at least for the
training).

##  V Proposed Solution

In this section, we present a novel model for FSOSR with its relative end-to-
end training technique that allows to increasingly train the discriminator
with a balanced number of positive and negative samples. An overview of our
proposed solution is depicted in Figure 1.

###  V-A Model

Our proposed model is a variation of the TRX [14] model. To adapt it for
sequences of skeleton, we replaced the convolutional neural network
${\phi:R^{H\times W\times 3}\rightarrow R^{D}}$ with an _embedding_ network
${\Psi:R^{J\times 3}\rightarrow R^{D}}$ where $J$ is the number of joints of
the skeleton. We used a Multi-Layer Perceptron [20] as an embedding network,
with a hidden layer of size $J\times 3\times 2$ and an output layer of size
$D$. We also added a ReLU [21] activation function after each layers. Given
different skeleton rotations during training, the embedding function $\Psi$
learns to project similar poses with different rotations into two close
$D$-dimensional vectors.

To extend to the OS this FSAR model, we need to add a “reject” class that
indicates that the query class $y_{i}$ is not contained inside the support set
classes $Y^{S}$. We define ${t^{c}\triangleq stack(\\{t_{p}^{c}:p\in\Pi\\})}$
and ${\tilde{q}\triangleq stack(\\{\Lambda\cdot Q_{p}:p\in\Pi\\})}$, where
$stack$ is a function that concatenates the elements a sequence along a new
dimension, such that ${t^{c}\in\mathbb{R}^{|\Pi|\times D^{\Lambda}}}$
represent the concatenation of the pairs of query-class prototypes and
${\tilde{q}\in\mathbb{R}^{|\Pi|\times D^{\Lambda}}}$ represents the
concatenation of the pairs of query prototypes. We could extend to the OS by
comparing the query-class prototypes $t^{c}$ of each class $c$ with the query
prototypes $\tilde{q}$, but this would lead to many negative samples and an
unbalanced training of the discriminator. Instead, we propose an alternative
approach in which an action is associated to a set of known actions and a
discriminator is trained to accept or reject the best association. We add this
behaviour to our model in the following way: first we get the class $c_{FS}$
that is chosen by the FSL part of the model

$c_{FS}\triangleq\underset{c}{\operatorname{argmin}}\\{T(Q,S^{c})\\}$

, then we give the matrix $\tilde{q}-t^{c_{FS}}$ to a discriminator $Disc$.
The discriminator $Disc$ assigns a score $[0,1]$ to its input that represents
the confidence of associating the query-class features difference with two
semantically close skeleton sequences. We implemented $Disc$ with a Multi-
Layer Perceptron as follows: the first layer
$l^{1}:\mathbb{R}^{D^{\Lambda}}\rightarrow D^{r}$ reduces the dimension of
each element of the sequence, obtaining a matrix with size
$\mathbb{R}^{|\Pi|\times D^{r}}$. Then it flattens the last two-dimension and
uses three layers to progressively reduce the dimension of the array to one
neuron, with a ReLU [21] function between each pair of layers. Finally, we
apply a Sigmoid function to obtain a score in $[0,1]$ that we compare with a
threshold value to either ”accept” or ”reject” the class predicted by the FSL
part of the model. Thus, the output of our model holds the following:

$f(x_{i},S)=\begin{cases}c_{FS},&\text{if }Disc(\tilde{q}-t^{c_{FS}})>\tau\\\
False,&\text{otherwise}\end{cases}$ (2)

where $\tau$ is a threshold that depends on the similarity of the actions we
are considering. Basically, if the actions in the support set are very
similar, a higher threshold value $\tau$ may be needed to correctly reject
unknown queries.

###  V-B Training

The model is trained end-to-end with episodic learning. To train the FSAR part
of the model, we follow the approach of TRX [14]: given a query
$Q_{i}=\\{q_{i1},...,q_{iF}\\}$ with true class $y_{i}\in Y^{S}$ and a support
set $S_{i}=\\{S_{i}^{1},...,S_{i}^{k}\\}$ with $K$ examples (one for each
class $c$), we compute the FS loss with a cross-entropy between the query-
class negative distances and a one-hot target vector for the task $i$ is:

$\ell_{FS}\triangleq\frac{1}{|B|}\sum_{i=0}^{B}CE(stack(\\{-T(Q_{i},S_{i}^{c}):c\in
Y^{S}\\}),\tilde{y}_{i})$

where $B$ is the number of training samples, $CE$ stands for Cross-Entropy and
$\tilde{y}_{i}$ is an one-hot vector with the entry corresponding to $y_{i}$
set to $1$.

To jointly train the discriminator, we need to feed it with positive and
negative samples. To train the model end-to-end we need to consider only true
positive samples, that are samples $Q_{i}$ that were correctly classified by
the FSAR part of the model, and so that $c_{FS,i}=y_{i}$. We define $z$ as the
number of samples correctly classified in a minibatch by the FSAR part of the
model. Now that we have $z$ positive samples, we can obtain the negative
samples by just sampling queries $Q_{i}$ with label $y_{i}$ from the unknown
set $D^{U}$. By sampling $z$ negative samples we can keep the training of the
generator balanced, where the number of samples that it receives at each
training step is ${z\times 2}$ and it increases during training accordingly to
the accuracy of the FSAR part of the model. This also allows us to
backpropagate the OS loss to the FSAR part of the model only where the query
and the most similar element in the support set are similar, providing more
information to the training. We define this loss as:

$\ell_{OS}\triangleq\frac{1}{|B|}\sum_{i=0}^{B}\tilde{\ell}_{OS}(Q_{i},S_{i}^{c})$

where, once defined $\tilde{o}_{i}\triangleq Disc(\tilde{q}_{i}-t_{i}^{c})$:

$\tilde{\ell}_{OS}(Q_{i},S_{i}^{c})=\begin{cases}BCE(\tilde{o}_{i},1),&\text{if
}c_{FS,i}=y_{i}\\\ BCE(\tilde{o}_{i},0),&\text{if }y_{i}\notin Y^{S}\\\
0,&\text{if }y_{i}\in Y_{i}^{S}\wedge c_{FS,i}\neq y_{i}\end{cases}$

where BCE stands for Binary Cross-Entropy. By combining the FS loss and the OS
loss we obtain the total loss:

${\ell_{FSOS}\triangleq\ell_{FS}+\sigma\ell_{OS}}$

where $\sigma$ is a regularizer value.

##  VI Experimental Validation

In this section, we provide the experimental validation of our model. In
particular, we investigate the performance of our model in recognizing new
classes from a small set of examples and the ability of the model to detect
and discard “unkown” actions. We compare our model with the baseline,
providing quantitative and qualitative results. We then show that, for the
classes in the test set, our approach solves the problem defined in Section
IV. The experimental validation characterizes in:

  * •

Quantitative validation: we compared the accuracy of the FSOS task of our
model with respect to a baseline, where the discriminator is not present and a
query is “rejected” if the exponential of the negative query-class distance is
lower than a certain threshold.

  * •

Qualitative validation: we discussed some predictions of our model over the
test set, demonstrating its strength to avoid recognizing wrong actions.

Unfortunately, to the best of our knowledge, there are no other works in the
literature that studied the FSOSAR problem for sequences of 3D skeleton and
therefore we cannot make a fair comparisons with other methods on this task.
Thus, instead, we created a baseline where the discriminator is removed and
the OS loss is computed directly on the negative query-class prototypes
distance without further processing. Since the baseline does not use a
discriminator explicitly trained on the OS task, we expect it to have lower
accuracy.

###  VI-A Comparison with Baseline (Quantitative validation)

We declare a simple baseline called _EXP_ where the discriminator is removed
and the confidence is computed with an exponential function over the maximum
negative distance $\max_{c}\\{-T(Q,S^{c}):c\in Y^{S}\\}$. Indeed, since the
query-class negative distances are bounded in $(-\infty,0]$, its exponential
has value inside $[0,1]$ that can be used as a confidence score to either
confirm or “reject” the class predicted by the FSL part of the model.

As dataset, we consider the NTURGBD-120 dataset [22]. This dataset contains
120 classes involving daily actions, medical actions, and two people
interactions. Each class has about a thousand samples with relative RGB
videos, depth map sequences, 3D skeletal data, and infrared (IR) videos.
However, some samples do not have skeleton data and, as pointed out also in
the literature [23], most of the given skeletons have low quality and are
noisy. Because of this, we decided to re-extract the skeleton from the RGB
videos. Thus, we took $16$ equidistant frames from each video, we extracted
the 3D skeleton with MetrABS [24] and we centered the skeleton with respect to
the pelvis coordinate. The output of the MetrABS heads is already bounded
inside $[-1,1]$, and therefore it did not need normalization. On the other
hand, we decided not to normalize the rotation because we want the embedding
layer to be invariant to the rotation of the skeletons. We removed the 26
classes representing two people interactions since our problem focuses on a
single skeleton at a time. To evaluate the performance of our method, we use
an approach similar to the One-Shot Action Recognition described in the
literature [22], but adapted for the OS task: we randomly select $k$ classes
and for each of them we add in the support set the corresponding exemplar [22]
(one fixed action instance for class), then we compute the metric on all the
elements of the test set. In this way, we have a known set $D^{K}$ that
contains all the samples of the $k$ selected class and an unknown set $D^{U}$
that contains all the samples from the remaining classes. We compute a metric
that we call _FSOS-ACC_ , which is the accuracy between the predicted classes
and the target classes, considering also the “reject” class. We expect our
model to classify as “reject” all the instances in the unknown set $D^{U}n$
and to classify with the correct classes in the support set all the instances
of the known set $D^{K}n$. We performed the comparisons for growing values of
$K$; for each $K$ we train the model $100$ times and we report the mean
accuracy and its standard deviation.

Figure 2: Comparision between the proposed model (_TRX-OS_) and the baseline (_EXP_) by the FSOS-ACC metric on our dataset. The _TRX-OS_ model outperforms the baseline for every $K$. | K=5 | K=10 | K=15
---|---|---|---
TRX-OS | 0.67$\pm$ 0.05 | 0.52$\pm$ 0.04 | 0.43$\pm$ 0.02
EXP | 0.63$\pm$ 0.05 | 0.51$\pm$ 0.04 | 0.41$\pm$ 0.02
RANDOM | 0.17 | 0.09 | 0.06
TABLE I: FSOS-ACC metric for our model (TRX-OS) and the baseline (EXP). We
also show random results for reference.

Figure 2 depicts the results of our comparison, while Table I shows the
numerical values. For higher values of $k$, there is an increasing number of
instances that should be rejected by the discriminator. Moreover, the more
classes there are in the support set, the harder is to match the query with
the true class because it is more probable that some classes are similar. With
$k=5$ we can reject or select the true class the $67\%$ of the times, that
means that our model can be reliably used in real Human-Robot scenarios when
the number of classes is limited. Our model outperforms the baseline for all
the values of $k$. This means that the discriminator learns useful concepts
that are also backpropagated to the FSL part of the model thanks to the Open-
Set loss $\ell_{OS}$.

###  VI-B Discriminator’s Confusion Matrix (Quantitative validation)

Another quantitative analysis refers to the Open-Set Accuracy of our model. In
fact, if we set our support set size to one we can build a confusion matrix on
the output of the discriminator to get insights on which actions are more
difficult to classify. We use the exemplar of one class from the test set as
the support set, then we attempts to classify all the elements of the test set
and we computed the number of instances that are classified as $True$ by our
model for each class. We show a subset of the results of this analysis in
Table II. We expect the elements on the diagonal to have a value close to one
and all the other elements of the matrix to have a value close to zero. Our
model performs well in this task for some classes like _Take off glasses_ and
_Falling_ , but other classes like _Tear up paper_ and _Apply cream on face_
have low performances. This happens mainly for two reasons, as discussed also
in the following Subsection VI-C:

  * •

The skeleton model that we used for training and testing does not track the
hands of the human. As a consequence, it is impossible to differentiate
between actions like _Apply cream on face_ and _Hush (quite gesture)_ because
the focus of these actions is on the hand movement and the fingers position.

  * •

By using skeleton data only, we ignore which object the human is interacting
with. Due to this fact, actions like _Stample book_ and _Tear up paper_ are
very similar.

An important aspect of this analysis is that we set the threshold value $\tau$
of Equation 2 to $0.5$, but different values could be more suitable for
different applications.

###  VI-C Qualitative results

In our applicatio,n when the robot detects an action performed by the human it
triggers his motors and assists the human in the completion of the action.
Since these actions involves also dangerous Human-Robot Interactions (e.g. the
robot passes a box to the human), it is needed that the robot does not
misclassify critical human actions. In the following, we show and discuss some
qualitative examples that justify the existence of our model. We show three
examples in the 3-Way 1-Shot scenario for ease of representation.

|  Take off glasses |  Tear up paper |  Pointing something |  Falling |  Hush (quite) |  Stample book |  Apply cream on face |  Arm circles
---|---|---|---|---|---|---|---|---
Take | 0.91 | 0.01 | 0.03 | 0. | 0.01 | 0. | 0.05 | 0.
Tear | 0. | 0.42 | 0.02 | 0. | 0. | 0.49 | 0.06 | 0.
Pointing | 0. | 0.03 | 0.82 | 0. | 0. | 0. | 0. | 0.
Falling | 0. | 0. | 0. | 0.87 | 0. | 0. | 0. | 0.
Hush | 0.09 | 0. | 0.02 | 0. | 0.74 | 0. | 0.48 | 0.
Stample | 0. | 0.06 | 0. | 0. | 0. | 0.52 | 0. | 0.
Apply | 0.12 | 0. | 0.02 | 0. | 0.48 | 0. | 0.6 | 0.
Arm | 0. | 0.03 | 0.02 | 0. | 0. | 0. | 0. | 0.71
TABLE II: Confusion matrix of our model for the OS task with respect to the test classes. On the rows, we have the classes that were added in the support set, and on the columns, we have the classes that were tested against the exemplar for the class in the support set. Figure 3: First qualitative example visualization. On the left the support set and on the right the known query and the unknown query. | Take (know) | Put (unknown)
---|---|---
Point | 0. | 0.24
Reach | 0. | 0.04
Take | 0.99 | 0.72
OS-Score | 0.87 | 0.09
TABLE III: First qualitative example results. The first three rows shows the
FSL scores for the known/unknown query and all the elements in the support
set, the last row shows the discriminator’s confidence between the
known/unknown query and the element of the support set with highest FSL score.

In the first example, depicted in Figure 3 with results shown in Table III,
both the FS and OS parts of the model correctly classify the known query _Take
off glasses_ , showing that the model effectively learned to deal with
different rotations of the skeleton. The unknown query _Put on headphones_ is
similar to the exemplar of the class _Take off glasses_ in the support set,
but in the first action, the object is brought closer to the head while in the
latter the object is moved away from the head. Indeed, the FS part of the
model chooses the _Take off Glasses_ action because it is the most similar in
the support set, but the discriminator correctly rejects the unknown query,
showing that the discriminator also learned to deal with temporal information.

In the second example, depicted in Figure 4 with results shown in Table IV,
both the FS and OS parts of the model correctly classify the given known query
_Apply cream on face_ as one class in the support set, with a high OS-Score.
This case is not too difficult for the FS part of the model because the _Apply
cream on face_ action is the only one of the support set where the arms reach
the head’s height. For the same reason, the FS part of the model classifies
the known query _Put on headphones_ with high confidence as _Apply cream on
face_ function, but then the discriminator rejects the prediction.

Figure 4: Second qualitative example visualization. On the left the support set and on the right the known query and the unknown query. | Apply (known) | Put (unknown)
---|---|---
Apply | 0.99 | 0.98
Reach | 0.0 | 0.1
Stample | 0. | 0.1
OS-Score | 0.98 | 0.31
TABLE IV: Second qualitative example results. The first three rows shows the FSL scores for the known/unknown query and all the elements in the support set, the last row shows the discriminator’s confidence between the known/unknown query and the element of the support set with highest FSL score. Figure 5: Third qualitative example visualization. On the left the support set and on the right the known query and the unknown query. | Take (known) | Use (unknown)
---|---|---
Drink | 0. | 0.02
Point | 0. | 0.97
Take | 0.99 | 0.01
OS-Score | 0.87 | 0.80
TABLE V: Third qualitative example results. The first three rows shows the FSL
scores for the known/unknown query and all the elements in the support set,
the last row shows the discriminator’s confidence between the known/unknown
query and the element of the support set with highest FSL score.

The third example, depicted in Figure 5 with results shown in Table V, is
interesting because it highlights the limitation of the 3D skeleton data. As
in the other examples, the FSL part of the model chooses the right action in
the support set for the known query _Take off glasses_ and the discriminator
confirms the prediction. Anyway, the unknown query tricks our model. Indeed,
the unknown action _Use a fan_ is performed only with wrist movements and does
not involve movements from the rest of the arm. Since MetrABS does not track
hand movements, the sequence of skeletons extracted from the video does not
describe correctly the action and, just by looking at them, it is easy to
think that the human is extending his arm towards something and nothing else.
Because of this, the FS part of the model chooses _Pointing something_ as the
candidate class and the OS part of the model confirms the wrong prediction
since it does not know about the hand movements.

Figure 6: Support set as a reference for the two unknown queries. Figure 7:
First unknown query: the FS part of the model classifies this query as ”get”
with $54\%$ of confidence, but the query is rejected because the
Discriminator’s output is low (0.07). Figure 8: Second unknown query: the FS
part of the model classifies this query as ”get” with $99\%$ of confidence,
but the query is rejected because the Discriminator’s output is low (0.01).

In Figure 6 we provide examples with data streamed from a camera. The skeleton
are extracted from a input video with MetrABS [24]. In our case, we wanted the
robot to detect when the human is ready to receive a box (i.e. action “get”),
but reject all those cases in which the human is doing different movements. We
can see that the desired behavior is obtained for both the two queries in
Figure 7 and Figure 8: the model associates the actions as closer to “get”,
however, the decision is rejected based on the the OS score, that is low for
both examples. This demonstrates that, without the discriminator and the OS
score, the system would classify all the unknown queries as actions “Get”,
producing unwanted robot behavior.

We showed that our model can efficiently deal with the FSOS problem, avoiding
to incorrect classifying queries with the true class that does not belong to
the support set. Thus, for the classes tested $x_{i}$ and the support set $S$
out proposed approach, encoded by $f(x_{i},S)$ (in Equation (2)) approximates
the function $\bar{f}(x_{i})$ (in Equation (1)) of problem defined in Section
IV, and outperforms the baseline.

##  VII Conclusion

In this paper we introduced a novel model to deal with the Few-Shot Open-Set
Recognition problem for sequences of 3D skeleton data. A discriminator is
added to confirm or “reject” the difference between the query prototype and
the query-class prototypes of the most similar element in the support set. We
explained how to train the discriminator jointly with the FSL part of the
model in such a way to keep the training of the discriminator balanced.
Moreover, the discriminator is trained proportionally with the accuracy of the
FSAR part of the model. We performed a quantitative analysis with a baseline
to demonstrate the importance of the discriminator and qualitative analysis,
demonstrating that our model can differentiate similar skeleton sequences that
perform different actions.

##  VIII Acknowledgment

The paper was supported by the Italian National Institute for Insurance
against Accidents at Work (INAIL) ergoCub Project.

## References

  * [1] Isidoros Rodomagoulakis, Nikolaos Kardaris, Vassilis Pitsikalis, E Mavroudi, Athanasios Katsamanis, Antigoni Tsiami, and Petros Maragos.  Multimodal human action recognition in assistive human-robot interaction.  In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2702–2706. IEEE, 2016.
  * [2] Sharath Chandra Akkaladevi and Christoph Heindl.  Action recognition for human robot interaction in industrial applications.  In 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), pages 94–99. IEEE, 2015.
  * [3] Alina Roitberg, Alexander Perzylo, Nikhil Somani, Manuel Giuliani, Markus Rickert, and Alois Knoll.  Human activity recognition in the context of industrial human-robot interaction.  In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, pages 1–10. IEEE, 2014.
  * [4] Muhammad Attique Khan, Kashif Javed, Sajid Ali Khan, Tanzila Saba, Usman Habib, Junaid Ali Khan, and Aaqif Afzaal Abbasi.  Human action recognition using fusion of multiview and deep features: an application to video surveillance.  Multimedia tools and applications, pages 1–27, 2020.
  * [5] Yamin Han, Peng Zhang, Tao Zhuo, Wei Huang, and Yanning Zhang.  Going deeper with two-stream convnets for action recognition in video surveillance.  Pattern Recognition Letters, 107:83–90, 2018.
  * [6] Guangyu Zhu, Changsheng Xu, Qingming Huang, Wen Gao, and Liyuan Xing.  Player action recognition in broadcast tennis video with applications to semantic analysis of sports game.  In Proceedings of the 14th ACM international conference on Multimedia, pages 431–440, 2006.
  * [7] Khurram Soomro and Amir R Zamir.  Action recognition in realistic sports videos.  In Computer vision in sports, pages 181–208. Springer, 2014.
  * [8] Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos.  Few-shot open-set recognition using meta-learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8798–8807, 2020.
  * [9] Minki Jeong, Seokeon Choi, and Changick Kim.  Few-shot open-set recognition by transformation consistency.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12566–12575, 2021.
  * [10] Yang Bo, Yangdi Lu, and Wenbo He.  Few-shot learning of video action recognition only based on video contents.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 595–604, 2020.
  * [11] Chris Careaga, Brian Hutchinson, Nathan Hodas, and Lawrence Phillips.  Metric-based few-shot learning for video action recognition.  arXiv preprint arXiv:1909.09602, 2019.
  * [12] Bin Ren, Mengyuan Liu, Runwei Ding, and Hong Liu.  A survey on 3d skeleton-based action recognition using learning method.  arXiv preprint arXiv:2002.05907, 2020.
  * [13] Lei Wang, Jun Liu, and Piotr Koniusz.  3d skeleton-based few-shot action recognition with jeanie is not so na$\backslash$” ive.  arXiv preprint arXiv:2112.12668, 2021.
  * [14] Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, and Dima Damen.  Temporal-relational crosstransformers for few-shot action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 475–484, 2021.
  * [15] Yiluan Guo and Ngai-Man Cheung.  Attentive weights generation for few shot learning via information maximization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13499–13508, 2020.
  * [16] Lin Wu, Yang Wang, Hongzhi Yin, Meng Wang, and Ling Shao.  Few-shot deep adversarial learning for video-based person re-identification.  IEEE Transactions on Image Processing, 29:1233–1245, 2019.
  * [17] Prasanta Chandra Mahalanobis.  On test and measures of group divergence: theoretical formulae.  Journal and Proceedings of Asiatic Society of Bengal New series26, 541-588, 1930.
  * [18] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In International conference on machine learning, pages 1243–1252. PMLR, 2017.
  * [19] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [20] Miroslav Kubat.  Neural networks: a comprehensive foundation by simon haykin, macmillan, 1994, isbn 0-02-352781-7.  The Knowledge Engineering Review, 13(4):409–412, 1999.
  * [21] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.
  * [22] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  IEEE transactions on pattern analysis and machine intelligence, 42(10):2684–2701, 2019.
  * [23] Neel Trivedi, Anirudh Thatipelli, and Ravi Kiran Sarvadevabhatla.  Ntu-x: An enhanced large-scale dataset for improving pose-based recognition of subtle human actions.  In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–9, 2021.
  * [24] István Sárándi, Timm Linder, Kai Oliver Arras, and Bastian Leibe.  Metrabs: metric-scale truncation-robust heatmaps for absolute 3d human pose estimation.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(1):16–30, 2020.