[1,2,3]F. Xavier Gaya-Morey These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

1]Group of Computer Graphics, Computer Vision and AI (UGIVIA), Universitat de
les Illes Balears (UIB), Carretera de Valldemossa, km 7.5, Palma, 07122, Illes
Balears, Spain

2]Research Institute of Health Sciences (IUNICS), Universitat de les Illes
Balears (UIB), Carretera de Valldemossa, km 7.5, Palma, 07122, Illes Balears,
Spain

3]Department of Mathematics and Computer Science, Universitat de les Illes
Balears (UIB), Carretera de Valldemossa, km 7.5, Palma, 07122, Illes Balears,
Spain

# Deep Learning for Computer Vision based Activity Recognition and Fall
Detection of the Elderly: a Systematic Review

<EMAIL_ADDRESS>Cristina Manresa-Yee<EMAIL_ADDRESS>José
M. Buades-Rubio<EMAIL_ADDRESS>[  [  [

###### Abstract

As the percentage of elderly people in developed countries increases
worldwide, the healthcare of this collective is a worrying matter, especially
if it includes the preservation of their autonomy. In this direction, many
studies are being published on Ambient Assisted Living (AAL) systems, which
help to reduce the preoccupations raised by the independent living of the
elderly. In this study, a systematic review of the literature is presented on
fall detection and Human Activity Recognition (HAR) for the elderly, as the
two main tasks to solve to guarantee the safety of elderly people living
alone. To address the current tendency to perform these two tasks, the review
focuses on the use of Deep Learning (DL) based approaches on computer vision
data. In addition, different collections of data like DL models, datasets or
hardware (e.g. depth or thermal cameras) are gathered from the reviewed
studies and provided for reference in future studies. Strengths and weaknesses
of existing approaches are also discussed and, based on them, our
recommendations for future works are provided.

###### keywords:

Human activity recognition, Fall detection, Ambient assisted living, Deep
learning, Computer vision, Elderly

##  1 Introduction

While global population continues to grow at a fast rate, life expectancy is
also increasing rapidly, especially in developed countries. According to Bloom
and Luca [1], life expectancy in China and India has risen by nearly 30 years
since 1950. In most developed countries, a 20% of the population is aged 60+,
and will rise to more than 30% in the next four decades.

For this reason, problems related with elderly111In this work, the term
”elderly” has been used to refer to people over 65 years old, for it is the
most extended term for referring to them in the field of Computer Science.
However, in other fields the preferred term can differ, as in the Psychology
field, where the most used term is ”older adults”. care are becoming more
frequent and preoccupying, since more people need to be cared about. One of
the most common dangerous situations this collective has to face is when a
fall occurs. The World Health Organization states some worrying and eye-
opening facts on this matter, like that falls are the second leading cause of
unintentional injury deaths worldwide, that each year an estimated 684,000
individuals die from falls globally and that 37.3 million falls that are
severe enough to require medical attention occur each year, among others [2].
In addition to the harm caused to the elderly when a fall occurs, there are
also economic costs associated with their treatment, which added up in 2010 to
”between 0.85% and 1.5% of the total health care expenditures within the USA,
Australia, EU15 and the United Kingdom” [3].

Automatic detection of the falls of the elderly is possible through data
collected from wearable or environmental devices, such as accelerometers,
gyroscopes, cameras, etc. Not only detection of falls, but also activity
recognition is feasible, which can be of use for multiple purposes: from
automatic life-logging [4] to finding habits to diagnose illnesses based on
the omission or repetition of basic tasks [5].

From the data perspective for fall detection and Human Activity Recognition
(HAR) tasks, vision data from different types of camera is in increasingly use
due to its multiple advantages over wearable devices or other sensors:
detection of multiple events simultaneously, suitability for multiple
subjects, environments and tasks, ease of installation and use, possibility of
verifying the data visually, etc. [6].

From the algorithmic perspective, Deep Learning (DL) has pushed the limits of
what was possible in the domain of digital image processing according to Walsh
et al. [7], becoming the state of the art in many different areas. In recent
years, multiple DL architectures have been designed and tested for computer
vision tasks, posing the question of which one to use for HAR and fall
detection of the older people.

In this work, we perform a Systematic Literature Review (SLR) on DL based HAR
and fall detection from vision data for the elderly care. The review process
is based on the guidelines for performing an SLR in Software Engineering by
Kitchenham and Charters [8], which offer great guidance over the methodology,
structure and good practices. The document is structured as follows: firstly,
the background of the study is approached, offering the insight of previous
reviews on the topic and the definition of the main concepts; secondly, the
review questions are enumerated; next, the review methods are explained in
detail (data sources, search strategy, study selection, quality assessment and
data extraction); then, the resulting list of studies is analyzed in detail to
offer the information needed to answer the review questions; to follow up, in
the discussion, the review questions are answered and the different
discoveries are discussed; finally, the extracted conclusions of the SLR are
presented.

##  2 Summary of previous reviews

Mining the literature, we find several reviews which address fall detection
and HAR, focusing on different aspects or limiting the studies to use some
kind of data. However, most of them (apart from [9]) do not follow a
systematic and rigorous method, which could lead to the introduction of bias
due to researchers’ previous knowledge and opinions [8]. This fact supports
the need of performing the current systematic review on the topic, based on a
predefined search strategy in order to be thorough and fair.

In this section, relevant reviews and surveys of the last ten years are
analyzed and synthesized. The most relevant ones are summarized in Table 1,
however, none of them covers exactly the topics reviewed in this work.

Table 1: Previous reviews on similar topics. Title | Task/s | Ref. | Year
---|---|---|---
In-Home Older Adults’ Activity Pattern Monitoring Using Depth Sensors: A Review | Fall detection, HAR | [10] | 2022
Human Movement Datasets: An Interdisciplinary Scoping Review |  Fall detection, HAR, other | [9] | 2022
Vision-based human fall detection systems using deep learning: A review | Fall detection | [11] | 2022
Human fall detection and activity monitoring: a comparative analysis of vision-based methods for classification and detection techniques |  Fall detection, HAR, other | [12] | 2022
Comprehensive Review of Vision-Based Fall Detection Systems | Fall detection | [13] | 2021
A review on video-based active and assisted living technologies for automated lifelogging |  Fall detection, HAR, Physiological signals | [4] | 2020
A review of state-of-the-art techniques for abnormal human activity recognition | Fall detection, HAR | [14] | 2019
ViFa: an analytical framework for vision-based fall detection in a surveillance environment | Fall detection | [15] | 2019
Vision-based patient monitoring: a comprehensive review of algorithms and technologies |  Fall detection, HAR, other | [16] | 2018

A review about activity pattern monitoring using depth sensors is carried out
in [10], exploring this kind of visual data as a substitute of RGB video or
images to preserve the privacy of the older adults. Studies are grouped by the
type of computing technique used (among which DL can be found) and also used
datasets are analyzed.

A systematic review on datasets used for HAR and fall detection is performed
in [9], resulting in an extensive list of over 700 datasets of multiple
modalities (video, joints, inertial measurement units, etc.). Multiple
taxonomies are created to group the datasets in different ways: by population
groups of interest, by data type, by creation purpose, etc. From all of them,
only four contain elderly people.

In [11], a review on DL based fall detection systems is performed. It offers a
good understanding on how DL is used for this specific task, analyzing
different types of falls, listing popular fall detection datasets, explaining
common metrics for evaluation and reviewing different works on this field,
along with different architectures. Vision data types included are RGB, depth
and thermal. The main drawback of this review related to this systematic work
is the specificity of the reviewed task, which leads to a lack of relevant
studies.

A broader number of tasks are reviewed in [12], detecting not only falls, but
also other useful information by studying different kinds of data extracted
from the video sequences: body shape change, posture, 3D head motion, spatial-
temporal, gait and skeleton. The raw input for all techniques are still RGB,
depth and thermal data. In this case, techniques are not restricted to be
based on DL, but also traditional computer vision techniques are reviewed,
meaning that many of the works will fall out of the scope of the present
systematic review, focused on the DL usage.

Fall detection was already reviewed in [13], where not only DL techniques were
included, in contrast to the aforementioned work [11]. In this case, only
works published in the previous 5 years are reviewed. The most commonly used
processing steps for this task are described and analyzed, along with ML
models, datasets, metrics and tracking techniques. Comparatives are also
performed between different works using the same datasets. The majority of
reviewed works use RGB and depth data, even though a work using IR data is
also included.

A review on fall detection, HAR and physiological signals analysis for Ambient
Assisted Living (AAL) systems is performed in [4]. The search is done on
Google Scholar, between years 2015-2020 (both included) and the search
keywords are provided. This review focuses on analyzing and describing
different works found on the field, rather than on comparing results or
establishing a taxonomy. In addition, privacy and user acceptance from the
different studies are also taken into account.

Table 2: Comparison of previous reviews with ours. By columns, important aspects taken into account in this review, and whether they are addressed or not by each review. From left to right: if the review is systematic; focuses on the exploration of DL solutions; targets elderly people as the users of the system; centers on the use of vision data; explores the Fall Detection and the Human Activity Recognition tasks; explores the use of RGB, depth or infrared data; takes privacy as a critical concern; describes the hardware used in the found studies; and if it studies the deployment of the FD and HAR systems in real environments. |  Systematic | Focus | Task | Data | AAL
---|---|---|---|---|---
|  DL |  Elderly |  Vision |  FD |  HAR |  RGB |  Depth |  IR |  Privacy |  Hardware |  Deployment
Momin et al. [10] | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗
Olugbade et al. [9] | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗
Alam et al. [11] | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗
Rastogi and Singh [12] | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗
Gutiérrez et al. [13] | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗
Climent-Pérez et al. [4] | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗
Dhiman and Vishwakarma [14] | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗
Ezatzadeh and Keyvanpour [15] | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗
Sathyanarayana et al.[16] | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗
This study | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

In [14], the review is centered on abnormal human activity recognition, which
as anomaly detection, refers to rarely occurring activities or non-conforming
patterns in data, including, among others, fall detection. The work offers a
classification of the reviewed studies according to the number of users
targeted at once, the number of dimensions of the visual data (2D or 3D) and
the type of features used: handcrafted or DL based; it also offers useful
tables gathering related studies, and different figures extracted from them,
to illustrate how they perform abnormal HAR.

An analytical framework (ViFa) for comparing different fall detection methods
is proposed in [15]. First, different classifications of the methods are
proposed, based on the input data (monocular, multi-camera or depth), the kind
of features (shape, head motion, etc.) and the classification type (thresholds
or learnt model). Then different criteria are proposed for evaluating the
categorized methods, and finally a qualitative evaluation is performed based
on these criteria. Among the criteria, typical metrics like accuracy can be
found, but also less frequent ones, like the system implementation cost, real-
time execution capabilities and resistance to occlusions and overlapping.

Many different patient monitoring tasks are reviewed in [16]: fall and sleep
apnea detection, and sleep, epilepsy, breathing, facial expression, vital
signs and activity monitoring. Even sub-tasks like pain or depression
detection are included. For each task, the used data, technologies and
techniques are analyzed and discussed, which offers a good overall picture of
patient monitoring algorithms and techniques, at the cost of not diving into
further details for any particular task.

As observed, different aspects of the subject matter addressed in this work
have already been reviewed in previous studies. However, there are distinctive
differences that underscore the necessity of the current work. Table 2 sheds
light upon this by displaying the pivotal aspects considered in the current
review, along with whether they are addressed or not in the aforementioned
reviews.

As shown, the only prior review that centered on the study of DL techniques
was conducted by [11], who did not include HAR in their research, and hence
their review lacks almost half of the studies analyzed in this work.
Conversely, the remaining reviews encompassed also techniques based on the
utilization of handcrafted features or classical vision approaches. By
focusing exclusively on DL-based solutions, our objective is to identify,
categorize, and elucidate the entire spectrum of works built for fall
detection and HAR. Furthermore, there are discernible differences in the
approach when dealing with DL, such as the importance of the training dataset,
architecture, and kind of features extracted and employed. These distinctions
are accounted for as we establish a taxonomy of the identified techniques and
delve into their explanations in detail.

Another noticeable aspect is that HAR was reviewed in less than a half of the
reviews, being the fall detection task the main focus of these. Additionally,
in many cases, these tasks were not the sole focus of the review: as mentioned
earlier, [16] reviewed various patient monitoring tasks, including sleep apnea
detection, and epilepsy, breathing and vital signs monitoring among others.
Similarly, [14] covered other abnormal activities alongside fall detection.
Due to the broader scope of these reviews, they did not uncover as many
studies as this study did in relation to the specific tasks of fall detection
and HAR in the elderly. Furthermore, their analysis did not delve as deeply
into the intricacies of these tasks.

On the other hand, only a limited number of reviews explored the application
of the reviewed systems within Ambient Assisted Living (AAL) systems, along
with the associated implications. Privacy was a major concern only in four
reviews. None of the reviews thoroughly examined the hardware employed, beyond
mentioning the prevalent use of the Kinect camera. Additionally, the effective
deployment of fall detection or HAR systems was not extensively explored in
any of the reviews. Conversely, we attributed great importance to these three
aspects, which play a major role in facilitating the transference to society.

Finally, it is imperative to highlight that, apart from [9], none of the
previous reviews followed a systematic review process. By adhering rigorously
to the systematic review methodology as outlined by [8], our work
distinguishes itself by ensuring a robust and unbiased selection and analysis
of relevant studies. As part of our systematic review methodology, we
conducted a thorough search across various databases, employing well-defined
search strings aligned with our research questions. We carefully assessed the
quality of each study and applied strict exclusion criteria to ensure the
inclusion of only the most relevant and high-quality literature in our review.
This systematic process aims to minimize any potential bias and ensures that
our review is based on a well-rounded selection of literature.

##  3 Review Questions

The review questions of this work are presented in Table 3. Questions RQ1 and
RQ2 are the primary review questions, while the remaining are secondary ones.

Table 3: Primary and secondary research questions used for this SLR. ID | Research Question
---|---
RQ1 |  Which DL techniques are used for human activity recognition and fall detection on elderly people?
RQ1.1 | Which is the most used kind of data?
RQ1.2 | Which are the most used architectures?
RQ1.3 | Which are the most used datasets?
RQ2 | How can these tasks be deployed successfully in a real environment?
RQ2.1 | Which is the most used hardware (cameras, robots, etc.)?
RQ2.2 | How is privacy of the elderly preserved?

In RQ1, the question posed is about the DL technique used to recognize the
activities or detect the fall of an elderly person, while the secondary
questions are about the details of the different aspects when carrying out the
task: the kind of data (RGB, depth, thermal, etc.), the used architecture
(CNN, RNN, etc.), and the dataset. The reason behind this question is to be
able to list the most common characteristics of DL based methods for HAR and
fall detection on the elderly, so as to make easier and faster the developing
of new methods for these tasks.

In RQ2, the question is about the deployment of the different tasks in a real-
case scenario, and the secondary questions focus on two important aspects to
take into account in the deployment: the hardware to use and the privacy of
the users. Works with an implementation in real environments, whether it is
through the use of robots or a camera-based setup, are expected to be found
among the selected studies, so it is desirable to know which are the most used
cameras, robots and other hardware used in the deployment. Moreover, privacy
is a particularly preoccupying aspect to take into account when dealing with
users, and the most extended ways of coping with it are of interest for any
researcher wanting a deployment of a HAR or fall detection system.

##  4 Review Methods

###  4.1 Data sources

For this systematic review, five data sources were selected: SCOPUS, Web of
Science (WOS), IEEE Explore Digital Library, ACM Digital Library and PubMed.

SCOPUS and WOS were selected as generic digital libraries, which contain
knowledge accross multiple areas, whereas the others were chosen for their
more specialized content on a specific area: IEEE is more focused on
engineering and technology, ACM is specialized on computer science and PubMed
is centered on biomedical studies. This selection of databases, with studies
from multiple areas, was made aiming to embrace the maximum amount of related
content.

The distribution of studies found from each source is plotted in Figure 1. As
shown, the majority of studies were collected from ACM and SCOPUS, while very
few (only 108 of a total of 2222) come from PubMed.

Figure 1: Number of publications obtained from each database, before study
selection (2222 in total).

###  4.2 Search strategy

Different query strings were built to match the syntax of each digital
library, keeping the differences at a minimum while using the same synonyms
for the different concepts searched. For each query string, the different
concepts were connected using logical AND, while the synonyms for each concept
where connected with logical OR. To cope with inflection of some keywords, the
”*” operator was used after the root word, allowing any possible end of word.

For the SCOPUS library, the search was limited to only title, abstract or
keywords, since the amount of results obtained otherwise was unfeasible, as
well as the majority being poorly relevant. However, the whole text was
searched for the remaining databases.

The main searched concepts, along with the employed list of synonyms, are the
following:

  * •

Task to perform (activity recognition or fall detection): ”action recognition”
OR ”activit* recognition” OR ”fall* detection” OR ”behaviour recognition” OR
”behaviour detection” OR ”physical activity recognition”

  * •

Ambient Assisted Living: ”monitoring” OR ”assist* living” OR ”AAL” OR ”smart
home” OR ”activit* of daily life” OR ”activit* of daily living” OR ”ADL”

  * •

Target collective (elderly people): ”elder*” OR ”old* people” OR ”senior”

  * •

Kind of data used (Computer Vision): ”vision” OR ”rgb” OR ”video” OR ”image”
OR ”skeleton” OR ”depth” OR ”camera” OR ”gesture”

Since Deep Learning has become popular recently, fewer related studies were
found in the first considered years, as it can be seen in Figure 2. For this
reason, the search was limited to only include studies published in the last
10 years. As shown, most studies found are from the last 5 years (from 2018
and on).

Figure 2: Accumulated publications from 2013 to 2022 (both included) after
study selection.

The results of study collection and duplicate removal can be seen in Figure 3.
As shown, a total of 2222 studies were collected from the different sources
using the aforementioned queries, of which 584 where detected as duplicates
and removed, leaving a total of 1638 studies.

Figure 3: Collected articles after each phase of the systematic search:
collection from the different sources, duplicates removal and study selection,
using inclusion and exclusion criteria.

###  4.3 Study selection

After collecting the studies from the different sources, limiting by year and
removing duplicates, exclusion criteria were applied to exclude non relevant
studies. The exclusion criteria were:

  * •

Deep Learning: studies not using DL were considered not relevant for this
review. Including this concept in the exclusion criteria rather than in the
query strings enabled to find more relevant studies, since there were many
studies not referencing DL directly, but using the name of a specific model
instead.

  * •

Language: not in English or Spanish.

  * •

Data type: data used is not either RGB, depth or IR. Both videos and images
are included. Skeleton data is also included, but only if computed from the
other three kinds of data. Studies where sensory data is used along with
vision data are also include, allowing multimodal approaches.

  * •

Availability: studies not accessible for different reasons: part of payed
content (like book chapters), source website down or retracted content.

  * •

Redundancy: in some cases, a journal article is published extending a work
already presented in a conference. In these cases, the conference proceedings
publications were omitted, since the journal article is an extension of the
same work.

  * •

Task: studies centered on different tasks than HAR or fall detection, like
velocity estimation, gait trend, level of tiredness, etc. Studies which do not
perform HAR or fall detection directly, but present a new dataset for these
tasks are included.

  * •

Target collective: studies which are not centered on elderly people were
excluded. It is not enough to mention this collective as one of the
beneficiaries of the work: the study has to either use data of elderly people
or have them in mind when designing the experiment.

  * •

Works in progress: conference proceedings about works in progress, only
containing the first stages of the study, were excluded. In most cases, they
lacked the experimentation phase.

  * •

Quality: publications with very poor quality (null reproducibility, very
biased decisions, too small datasets, etc.) were excluded . More information
about quality assessment in Section 4.4.

As seen in Figure 3, only 137 studies remain after applying the exclusion
criteria, of which 82 are conference proceedings and 55 are journal articles.
These studies will be presented in more detail, classified and compared in
next sections.

###  4.4 Quality assessment

Since there is no agreed definition of study ”quality”, the proposed
guidelines in [8] were followed, addressing mainly bias and validity of the
studies as a measure of quality. Specifically, the following aspects were
taken into account:

  * •

Reproducibility. Whether the work can be replicated. This can be achieved by
disclosing the dataset used or using external datasets, and by either
publishing the used code for the model or giving enough details to recreate
the model.

  * •

Comparison with other works. Whether the obtained performance of the model is
compared with the state of the art. Note that the comparison should be done in
fair conditions, meaning that the models should be trained and tested on the
same data, since bias can be introduced otherwise.

  * •

Use of external datasets. By testing the model on external datasets, possible
bias from the data can be removed and comparison with other models for the
same task is easier. At the same time, other studies can use the results
without needing to retrain the model on other data.

The aforementioned aspects were included in the list of fields in the data
extraction phase (last 3 fields), which is addressed in Section 4.5. Moreover,
these quality aspects were used also as exclusion criteria, as already
mentioned in Section 4.3.

In addition to these aspects, the type of study, either conference proceedings
or journal article, was also taken into account as a quality indicator, since
journal articles tend to be longer and more mature. As already mentioned,
there are more conference proceedings (82) than journal articles (55).

###  4.5 Data extraction and synthesis

From each study left after applying the exclusion criteria, different data
were extracted in order to summarize the content and to establish taxonomies
for various aspects of interest. All data were gathered in a table where, for
each work, the following fields were extracted:

  * •

Title

  * •

Author/s

  * •

Type (journal article or conference proceedings)

  * •

Publication year

  * •

Task (HAR or fall detection)

  * •

Data type (RGB, depth, IR or skeleton)

  * •

Auxiliary sensor data type (accelerometers, gyroscopes, etc.)

  * •

Camera used

  * •

Dataset (name of external dataset/s or ”custom”)

  * •

DL model/s and task (skeleton joints estimation, feature extraction,
classification, etc.)

  * •

Other ML models or computer vision techniques used

  * •

System integration in a robot (yes/no) and which one

  * •

System integration in a framework (yes/no)

  * •

How is privacy preserved? (depth or IR only, low resolution, etc.)

  * •

Reproducible (yes/no)

  * •

Test with external datasets

  * •

Comparison with other approaches

The full list of relevant studies is shown in Tables 4 and 5, displaying only
basic information of each one. The remaining information will be synthesized
in Section 5 in the form of tables and plots, which allow to see the
distribution of works by used data types, DL model family, dataset, etc. Aside
from this information, especially relevant or interesting aspects of the works
will also be summarized and relevant concepts will be addressed in more
detail.

Table 4: Complete list of relevant studies analyzed in the current systematic review. The task of the studies can either be FD (fall detection), HAR (Human Action Recognition) or FD, HAR (both). CV data column refers to the type of computer vision data used to perform the task/s: RGB, D (depth) or IR (infrared). Data types are separated by a comma if the study requires all of them or by a slash if it only needs one of them (continued on next page). Ref. | Year | Task | CV Data | Ref. | Year | Task | CV Data
---|---|---|---|---|---|---|---
[17] | 2022 | FD | RGB-D | [18] | 2021 | FD | RGB
[19] | 2022 | FD | RGB | [20] | 2021 | FD | RGB
[21] | 2022 | FD | RGB | [22] | 2021 | FD | RGB
[23] | 2022 | FD | RGB | [24] | 2021 | FD | RGB
[25] | 2022 | FD | D | [26] | 2021 | FD | RGB
[27] | 2022 | FD | RGB | [28] | 2021 | FD | RGB
[29] | 2022 | FD | RGB | [30] | 2021 | FD & HAR | IR
[31] | 2022 | FD | RGB | [32] | 2021 | FD & HAR | RGB
[33] | 2022 | FD | RGB | [34] | 2021 | FD & HAR | RGB
[35] | 2022 | FD | RGB/D/IR | [36] | 2021 | FD & HAR | RGB
[37] | 2022 | FD | RGB | [38] | 2021 | FD & HAR | RGB
[39] | 2022 | FD | RGB | [40] | 2021 | FD & HAR | RGB
[41] | 2022 | FD | RGB | [42] | 2021 | HAR | RGB
[43] | 2022 | FD | RGB | [44] | 2021 | HAR | D
[45] | 2022 | FD | RGB | [46] | 2021 | HAR | RGB
[47] | 2022 | FD | RGB | [48] | 2021 | HAR | RGB/D
[49] | 2022 | FD & HAR | RGB | [50] | 2021 | HAR | D
[51] | 2022 | FD & HAR | RGB | [52] | 2021 | HAR | RGB
[53] | 2022 | FD & HAR | RGB | [54] | 2021 | HAR | RGB/D
[55] | 2022 | FD & HAR | RGB | [56] | 2021 | HAR | D
[57] | 2022 | FD & HAR | RGB | [58] | 2021 | HAR | RGB
[59] | 2022 | FD & HAR | RGB | [60] | 2021 | HAR | D
[61] | 2022 | HAR | RGB | [62] | 2021 | HAR | D
[63] | 2022 | HAR | RGB | [64] | 2021 | HAR | RGB
[65] | 2022 | HAR | RGB | [66] | 2021 | HAR | IR
[67] | 2022 | HAR | D | [68] | 2021 | HAR | RGB
[69] | 2022 | HAR | RGB | [70] | 2020 | FD | RGB
[71] | 2022 | HAR | RGB-D | [72] | 2020 | FD | RGB
[73] | 2022 | HAR | RGB | [74] | 2020 | FD | RGB
[75] | 2022 | HAR | RGB | [76] | 2020 | FD | RGB-D
[77] | 2022 | HAR | RGB | [78] | 2020 | FD | RGB
[79] | 2022 | HAR | D | [80] | 2020 | FD | RGB
[81] | 2022 | HAR | RGB | [82] | 2020 | FD | RGB
[83] | 2022 | HAR | D | [84] | 2020 | FD | RGB
[85] | 2022 | HAR | RGB/D | [86] | 2020 | FD | RGB
[87] | 2021 | FD | RGB | [88] | 2020 | FD | RGB-D
[89] | 2021 | FD | RGB | [90] | 2020 | FD | RGB
[91] | 2021 | FD | RGB | [92] | 2020 | FD | RGB
[93] | 2021 | FD | RGB | [94] | 2020 | FD & HAR | RGB
[95] | 2021 | FD | RGB | [96] | 2020 | FD & HAR | IR
[97] | 2021 | FD | RGB | [98] | 2020 | HAR | RGB
[99] | 2021 | FD | RGB | [100] | 2020 | HAR | RGB-D
[101] | 2021 | FD | RGB | [102] | 2020 | HAR | RGB-D
[103] | 2021 | FD | RGB | [104] | 2020 | HAR | RGB
[105] | 2021 | FD | RGB/D | [106] | 2020 | HAR | RGB
[107] | 2021 | FD | RGB | [108] | 2020 | HAR | RGB
[109] | 2021 | FD | RGB | [110] | 2020 | HAR | RGB
[111] | 2021 | FD | RGB | [112] | 2019 | FD | RGB
Table 5: Complete list of relevant studies analyzed in the current systematic review. The task of the studies can either be FD (fall detection), HAR (Human Action Recognition) or FD, HAR (both). CV data column refers to the type of computer vision data used to perform the task/s: RGB, D (depth) or IR (infrared). Data types are separated by a comma if the study requires all of them or by a slash if it only needs one of them (continuation). Ref. | Year | Task | CV Data | Ref. | Year | Task | CV Data
---|---|---|---|---|---|---|---
[113] | 2019 | FD | RGB | [114] | 2019 | HAR | D
[115] | 2019 | FD | RGB | [116] | 2018 | FD | RGB
[117] | 2019 | FD | RGB | [118] | 2018 | FD | RGB
[119] | 2019 | FD | RGB | [120] | 2018 | FD & HAR | IR
[121] | 2019 | FD | RGB | [122] | 2018 | FD & HAR | RGB
[123] | 2019 | FD | RGB, IR | [124] | 2018 | HAR | RGB
[125] | 2019 | FD | RGB | [126] | 2018 | HAR | RGB
[127] | 2019 | FD | RGB-D | [128] | 2018 | HAR | RGB
[129] | 2019 | FD | RGB | [130] | 2017 | FD | RGB
[131] | 2019 | FD | RGB | [132] | 2017 | FD | D
[133] | 2019 | FD | IR | [134] | 2017 | FD | IR
[135] | 2019 | FD | RGB | [136] | 2017 | FD & HAR | RGB-D
[137] | 2019 | HAR | D | [138] | 2017 | FD & HAR | RGB
[139] | 2019 | HAR | RGB | [140] | 2017 | HAR | RGB
[141] | 2019 | HAR | D | [142] | 2017 | HAR | D
[143] | 2019 | HAR | RGB | [144] | 2016 | FD | D
[145] | 2019 | HAR | RGB | [146] | 2016 | FD | RGB
[147] | 2019 | HAR | D | [148] | 2014 | FD | RGB
[149] | 2019 | HAR | RGB | [150] | 2014 | HAR | D
[151] | 2019 | HAR | RGB | [152] | 2014 | HAR | RGB-D
[153] | 2019 | HAR | RGB |  |  |  |

##  5 Results

In this section, tables and charts are provided to summarize the results of
the review. To further improve the visualization and understanding of the
information, different aspects are addressed separately and divided into
subsections, corresponding to the related research questions. Note that
primary research questions will be related to all sections targeting their
secondary review questions.

###  5.1  RQ1: Fall detection and Human Activity Recognition

The two main tasks explored in the review are fall detection and Human
Activity Recognition (HAR), treating fall detection as an especially important
case of HAR. As shown in Figure 4, fall detection has been the most explored
task in the last 10 years (86 studies in total), while there are a smaller
number of studies dealing with HAR (69 studies in total). It can also be
appreciated that it is quite rare to see studies dealing with both tasks at
the same time: only 18 from a total of 137 studies (approximately a 13%).

Figure 4: Distribution of studies by target task: fall detection, HAR or both.

###  5.2 RQ1.1: Input data type

From the collected studies, 3 types of vision data have been considered: RGB,
depth and infrared (IR). The number of occurrences of these can be observed in
Figure 5. It was found that the RGB data are the most extended for fall
detection and HAR on the elderly (114 studies), followed by depth data (31
studies), and finally by IR data (8 studies).

Figure 5: Distribution of studies by data type used. Note that more than one
data type was used in some studies.

There has also been found a notable number of studies which extract skeleton
joints from the vision data to later use them to detect falls or to recognize
different activities. More precisely, a 44% of the studies used skeleton
joints estimation to perform classification, while only a 56% did not, and so
the skeleton joints extraction seems to be a very extended practice.

In some studies, along with vision data, sensor data is used to further
improve the results of the system, by using different models or strategies to
perform the classification and then fusing the results. Five types of sensor
data have been found in the review: Inertial Measurement Unit (IMU)222Studies
not specifying the use of an IMU but using accelerometers and gyroscopes have
also been included in this group., audio, barometer, luminosity and radar. The
studies using these sensors can be seen in Table 6, which add up to a total of
only 11 studies using multimodal data. Note that barometers, radars and
luminosity data are always used together with an IMU.

Table 6: Non computer vision data found in the collected studies, together with the studies using them. Note that they have to be used together with computer vision data in order to be included in the review.  Sensor | Ref. | Year | Authors
---|---|---|---
IMU | [19] | 2022 | Meraikhi and Al-Rajab
| [89] | 2021 | Galvão et al.
| [63] | 2022 | Lin et al.
| [71] | 2022 | Ouyang et al.
| [99] | 2021 | Divya and Sri
| [78] | 2020 | Serpa et al.
| [86] | 2020 | Lv et al.
| [31] | 2022 | Aarthi and Juliet
| [77] | 2022 | Islam et al.
Audio | [137] | 2019 | Siriwardhana et al.
| [58] | 2020 | Iksan et al.
Barometer | [86] | 2020 | Lv et al.
Luminosity | [99] | 2021 | Divya and Sri
Radar | [71] | 2022 | Ouyang et al.

###  5.3 RQ1.2: DL models

Table 7 summarizes all DL models used to perform the tasks of fall detection
and HAR. These models are not always used standalone, but for different
specific tasks: skeleton joints estimation, optical flow computation, feature
extraction, etc. The type of input data used by the models also differs and
needs, in many cases, the computation of a specific type of features. This is
reflected in Figure 6, where a taxonomy of the DL models used in the found
studies according to different aspects is displayed.

Table 7: Complete list of DL models used in the reviewed studies. The first two columns refer to the models’ reference and name, while the last three refer to their usage in the reviewed studies: task/s they are used for, input data they are feed with and the number of studies of the review in which they are found. FD is used for Fall Detection, OF for Optical Flow and OD for Object Detection, for brevity. Different models belonging to the same family (a.k.a. later improvements of the same model) have been grouped; this is the case for R-CNN, LSTM, YOLO, VGG, ResNet, RNN, CNN and GCN models. Model | Studies
---|---
Ref. | Name | Task | Input data | N
[154] | OpenPose | 2D Skeleton | RGB Image | 23
[155] | AlphaPose | 2D Skeleton | RGB Image | 9
[156] | MediaPipe | 2D Skeleton | RGB Image | 4
[157] | PoseNet | 2D Skeleton | RGB Image | 2
[158] | STN | 3D Skeleton | RGB-D Image | 1
[159] | OpenNI | 3D Skeleton | Depth Image | 1
[160] | DeeperCut | 2D Skeleton | RGB Image | 1
[161] | RMPE | 2D Skeleton | RGB Image | 1
[162] | PoseFlow | 2D Skeleton | RGB Image | 1
[163] | Baidu AI | 2D Skeleton | RGB Image | 1
[164] | MobileNet | 2D Skeleton, HAR, FD | RGB Image | 4
[165] | DeepHAR | 2D Skeleton, HAR, FD | RGB Image | 1
[166] | C3D | Features | RGB Video | 1
[167] | DCF-Net | Features | RGB Image | 1
[168] | GoogLeNet | Features | RGB Image | 1
[169] | PCA-Net | Features | RGB Image | 1
[170] | PWC-Net | Features (OF) | RGB Video | 1
[171] | Slowfast | Features | RGB Video | 1
[172] | SqueezeNet | Features | RGB Image | 1
[173] | LiteFlowNet | Features (OF) | RGB Video | 1
[174] | InceptionV3 | Features, HAR | RGB or Depth Image | 2
[175] | R-CNN | Features (OD), FD | RGB Image | 8
[176] | CNN | Features, HAR, FD | Image, Video or features | 56
[177] | LSTM | Features, HAR, FD | Skel. sequence or Video features | 25
[178] | YOLO | Features (OD), HAR, FD | RGB Image | 14
[179] | VGG | Features, HAR, FD | RGB or Depth Image | 11
[180] | ResNet | Features, HAR, FD | RGB or Depth Image or Skel. Pose | 9
[181] | RNN | Features, HAR, FD | Skel. sequence or Video features | 7
[182] | GCN | Features, HAR, FD | Skel. sequence or Video features | 7
[183] | GRU | Features, HAR, FD | Skel. sequence or Video features | 6
[184] | I3D | Features, HAR, FD | RGB Video | 4
[185] | SOM | HAR | Skeleton Pose | 2
[186] | AIA | HAR | Video features and Bound. Boxes | 1
[187] | Transformer | HAR | IR image (8x8) | 1
[188] | Glimpse Clouds | HAR | Skeleton sequence | 1
[189] | iCAN | HAR | Bounding Boxes sequence | 1
[190] | Autoencoder | FD | Different features | 2
[191] | GAN | FD | Different features | 2
[192] | DeepFall | FD | RGB, Depth or IR Video | 1
[193] | DBN | FD | Different features | 1
[29] | FallNet | FD | RGB Video | 1
[194] | Siamese CNNs | FD | RGB or Optical Flow Video | 1
[195] | Sep-TCN | FD | Skeleton sequence | 1
[196] | MLP | HAR, FD | Different features | 5
[197] | AlexNet | HAR, FD | Feature image | 2
[53] | ARFD-Net | HAR, FD | Skeleton sequence | 1
Figure 6: Taxonomy of the DL techniques used in the found studies. The number
of studies where each category was used is displayed in bold. Note that
multiple models were used in many studies, and hence the same study can be
counted in more than one category.

As already mentioned in section 5.2, there is a great amount of studies using
skeleton joints as features for fall detection and HAR, and to estimate these
joints, different DL models are used. The two most used models are: OpenPose
[154] and AlphaPose [155]. OpenPose is based on a non-parametric
representation (which they refer to as Part Affinity Fields) to find all
skeleton joints from all humans in the image at once, while AlphaPose performs
first human detection and then predicts the skeleton joints for each one. From
both, the first one is more used than the second one (23 vs. 9 studies),
although the second one is more recent. Once the skeleton joints have been
extracted, many different models are used for fall detection and HAR using
these data:

  * •

Recurrent Networks: the Long-Short Term Memory (LSTM) [177] and the Gated
Recurrent Unit (GRU) [183] are the most used ones, although others can also be
found (grouped in the table as RNN).

  * •

Graph-Based Network: only one has been found: the Graph-Convolutional Network
(GCN) [182], which treats the skeletons as a graph, rather than as a sequence.

  * •

Convolutional Networks: many are used, like VGG-16 or VGG-19 [179], MobileNet
[164] or AlexNet [197].

There are two models used for optical flow estimation among the studies: PWC-
Net [170] and LiteFlowNet [173], and two used for object (or human) detection:
R-CNN [175] and YOLO [178]. It is much more frequent the use of object
detection (22 studies) than the optical flow computation (2 studies). Object
detection has been found to be used with different purposes in the studies:

  * •

To obtain a sequence of bounding boxes for the objects in the scene, which can
be used as features in next steps.

  * •

To trigger later computation of fall detection or HAR upon detection of human
presence, saving computation time.

  * •

To perform background subtraction, in order to reduce the complexity of the
data by focusing on the target person.

  * •

To get features from the humans in the scene, like the height-to-width ratio,
which are then used for fall detection or HAR in further steps.

DL computed features other than skeleton joints, optical flow and object/human
bounding boxes are mainly computed through convolutional models like VGG-16 or
VGG-19 [179], ResNet [180] or InceptionV3 [174], only to mention a few. Non
DL-based features are also used in the reviewed studies, although less
frequently. Among them we can find: HOG (Histograms of Oriented Gradients),
HOF (Histograms of Optical Flow), LBP (Local Binary Patterns) and the use of
BOVW (Bag of Visual Words). After obtaining the different kinds of features,
multiple DL models are used to perform classification. However, at this stage
it is also frequent to use non-deep machine learning models like the Support
Vector Machine, or even rule-based classification on simple features like the
height-to-width ratio or the spine angle.

From the reviewed studies, there are some which treat the fall detection task
as a normal/abnormal classification task, by modeling normal activities and
treating the falls as abnormal data. To do so, feature extraction is first
performed (either by using pre-trained models to extract spatio-temporal
features from video/image or by using estimated skeleton joints) and then a
model is trained to fit the normal activities. In [144] and [150], Self
Organizing Maps (SOM) are used to model normal activities from simple skeletal
data; a Support Vector Data Description (SVDD) is used in [116] and the MPED-
RNN network is used in [26] also on skeletal data; DeepFall [192] is used in
[35] with different data modalities (RGB, depth and IR); an Autoencoder is
used in [123] and [45] after obtaining spatio-temporal features using other
nets; and Generative Adversarial Networks (GANs) [191] are used in [18] and
[37], using the discriminator as the normal/abnormal classifier.

###  5.4 RQ1.3: Datasets

From the reviewed studies, a list of used datasets for activity recognition
and fall detection has been collected. The list of datasets is summarized in
Table 8. Only publicly available datasets have been included, aiming at
providing a list that can be used for future works on the area. In cases where
datasets do not have an official acronym, a custom one has been used. The
column ”Elderly” refers to whether the dataset contains samples with elderly
people, ”Falls” refers to whether at least one of the classes is for the
falling activity and ”Samples” refers to the number of samples found in the
dataset, whether they are images or videos (note that the length of the videos
can differ greatly). The list has been ordered by the number of studies in
this review using each dataset (in descending order) and then by year (also
descending order).

From the results, the following common characteristics can be observed for the
different aspects:

  * •

Elderly: most datasets do not include elderly people in the samples (only an
11% do).

  * •

Falls: most include falls as a class (57% of the datasets).

  * •

Type: almost all of them offer the samples in video format and not as images
(86% vs. 14%).

  * •

Data types: the RGB images/videos are always provided and, in many cases, also
depth frames and skeleton joints are also provided (in 48% and 38% of the
datasets respectively). The remaining data types are more rarely found:
inertial data is only provided in 13% of the datasets, while IR data and MHV
(Motion History Volumes) only in two and one datasets respectively.

  * •

Samples: the number of samples differs greatly between datasets, varying from
less than 50 to more than 50,000.

  * •

Classes: the number of classes also varies greatly: from 2 classes (usually
fall and not fall) to 700 in the Kinetics 700-2020 dataset [198]. Note that
the datasets which offer that amount of classes are most likely not focused on
AAL, but rather on the activity recognition task.

  * •

Studies: half of the datasets are only used in one study, while only 4 are
used in more than 10 studies.

Table 8: Complete list of publicly available datasets used in the reviewed studies, with their basic specifications. Columns Eld., Cl. and N refer to the presence of elderly people in the datasets, number of classes and number of the reviewed studies in which they appear respectively. Ref. | Dataset | Eld. | Falls | Type | Data types | Samples | Cl. | N
---|---|---|---|---|---|---|---|---
[199] | URFD | No | Yes | Video | RGB-D, Skel., IMU | 140 | 2 | 34
[200] | MultiCam | No | Yes | Video | RGB | 192 | 2 | 16
[201] | UP-FALL | No | Yes | Video | RGB, IR, IMU | 561 | 11 | 15
[202] | NTU RGB+D | No | Yes | Video | RGB-D, Skel. | 56880 | 60 | 13
[136] | FDD-Adhikari | No | No | Image | RGB-D | 21499 | 5 | 8
[203] | Le2i | No | Yes | Video | RGB | 222 | 2 | 6
[204] | FDD-Charfi | No | Yes | Video | RGB | 250 | 2 | 6
[205] | CAD-60 | No | No | Video | RGB-D | 720 | 12 | 4
[100] | PRECIS HAR | No | Yes | Video | RGB-D | 800 | 16 | 3
[206] | UTD-MHAD | No | No | Video | RGB-D, Skel., IMU | 861 | 27 | 3
[207] | MSRDailyActivity3D | No | No | Video | RGB-D, Skel. | 320 | 16 | 3
[208] | HMDB51 | No | Yes | Video | RGB | 6766 | 51 | 3
[209] | KTH | No | No | Video | RGB | 2391 | 6 | 3
[210] | ETRIActivity3D | Yes | Yes | Video | RGB-D, Skel. | 112620 | 55 | 2
[125] | FPDS | No | Yes | Image | RGB | 2064 | 2 | 2
[211] | NTU RGB+D 120 | No | Yes | Video | RGB-D, Skel. | 114480 | 120 | 2
[114] | ToyotaSmartHome | Yes | No | Video | RGB-D, Skel. | 16115 | 31 | 2
[212] | HQFSD | Yes | Yes | Video | RGB | 55 | 2 | 2
[213] | UWA3DII | No | Yes | Video | RGB-D, Skel. | 120 | 30 | 2
[214] | FDD-TST | No | Yes | Video | RGB-D, Skel., IMU | 132 | 8 | 2
[215] | N-UCLA | No | No | Video | RGB-D, Skel. | 100 | 10 | 2
[216] | CAD-120 | No | No | Video | RGB-D | 1200 | 10 | 2
[217] | UCF101 | No | No | Video | RGB | 13320 | 101 | 2
[218] | UTKinect-Action3D | No | No | Video | RGB-D, Skel. | 200 | 10 | 2
[29] | FallAction | No | Yes | Video | RGB | 2000 | 20 | 1
[33] | VWFP | No | Yes | Image | RGB | 6071 | 2 | 1
[35] | MUVIM | Yes | Yes | Video | RGB-D, IR, IMU | 244 | 2 | 1
[219] | FPDS-Elderly | Yes | Yes | Image | RGB | 413 | 2 | 1
[54] | KIST SynADL | No | Yes | Video | RGB-D, Skel. | 462200 | 55 | 1
[198] | Kinetics 700-2020 | No | Yes | Video | RGB | 647907 | 700 | 1
[94] | ALMOND | No | Yes | Video | RGB | 7565 | 22 | 1
[220] | C-MHAD | No | Yes | Video | RGB, IMU | 120 | 7 | 1
[221] | FDD-Chen | No | Yes | Video | RGB | 30 | 2 | 1
[222] | OOPS | No | Yes | Video | RGB | 20338 | 2 | 1
[223] | Kinetics 600 | No | Yes | Video | RGB | 495547 | 600 | 1
[128] | NAD | No | No | Video | RGB | 84 | 7 | 1
[224] | YTBF | No | Yes | Video | RGB-D, Skel. | 606 | 2 | 1
[225] | PKU-MMD | No | Yes | Video | RGB-D, Skel. | 21545 | 51 | 1
[226] | Kinetics 400 | No | No | Video | RGB | 306245 | 400 | 1
[227] | KARD | No | No | Video | RGB-D, Skel. | 540 | 18 | 1
[228] | V-COCO | No | No | Image | RGB | 10346 | 25 | 1
[229] | MMU | No | Yes | Video | RGB | 51 | 2 | 1
[230] | DMLSmartActions | No | Yes | Video | RGB-D, Skel. | 932 | 12 | 1
[231] | Florence3D | No | No | Video | RGB-D, Skel. | 215 | 9 | 1
[232] | ACT42 | No | Yes | Video | RGB-D, Skel. | 6844 | 14 | 1
[233] | BIT-interaction | No | No | Video | RGB | 400 | 8 | 1
[234] | Stanford40 | No | No | Image | RGB | 9532 | 40 | 1
[235] | IXMAS | No | No | Video | RGB, MHV | 330 | 11 | 1

The University of Rzeszow Fall Detection (URFD) [199] is the most used dataset
(34 studies), followed by MultiCam [200] (16 studies), UP-FALL [201] (15
studies) and NTU RGB+D [202] (13 studies). The remaining datasets were used
less than 10 times, with approximately half of the datasets being used in only
one study. URFD is focused on the fall detection problem, thus offering only 2
classes: fall and ADL, offers 70 different sequences from 2 perspectives (140
samples in total) and good variety of data modalities: RGB, depth, skeleton
joints and inertial data. MultiCam does not offer that many data types (only
RGB video), but rather focuses on multiple perspectives for the different
sequences: 24 in total, each one from 8 perspectives (192 samples), also with
only 2 classes: fall or confounding event. In the UP-FALL dataset, data for 17
subjects, 11 activities and 2 perspectives (561 samples) can be found,
allowing both HAR and fall detection, since falls are one of the classes. It
offers RGB video, infrared images and inertial data. NTU RGB+D offers many
more samples: a total of 56880 from 40 subjects performing 60 activities,
using different perspectives, heights and distances for the cameras. Since the
dataset is recorded using Kinect cameras, it contains RGB video, depth images
and skeleton joints. There exists also an extended version of this dataset:
the NTU RGB+D 120 dataset [211], which extends it by adding 60 more classes.
It is only used in two of the reviewed studies.

Although the vast majority of the datasets are obtained by the deployment of
cameras in a real environment, some of them are obtained through a simulation.
More specifically, [33] and [54] offer synthetic images and videos
respectively, rather than real ones. Even though it is easier and cheaper to
obtain a large dataset this way, the suitability of it for a deployment in a
real environment may be compromised.

Some of the reviewed studies did not use datasets in the provided list, but
rather worked with a custom made one. In Figure 7 it can be observed the
proportion of studies using either a custom dataset, an external one or both.
As shown, only 19 studies offer evaluation on both a custom and an external
dataset, while it is more frequent to offer it only on an external dataset (77
studies) rather than only on a custom one (41 studies).

Figure 7: Distribution of studies by dataset used.

###  5.5 RQ2: Framework integration

In fifteen of the reviewed articles, a framework to integrate the HAR or fall
detection task in a real environment was designed, attending to different
aspects: security, use of cloud services, client-server configuration, network
communications, IoT devices, etc. Next, we briefly describe the proposed
frameworks.

In [61], a custom robot is proposed to integrate the HAR task in the
environment, along with other ones like language processing to enable chatbot
functionalities. In [94], a camera system is used instead to get the visual
data and send it to a central server to perform all computation, and then
notifications, reports and alerts are sent to a ”guardian”. In [46], activity
recognition results along with recorded video data are sent to a developed
mobile application, which is used to monitor the users of the system. In
[140], an application implementing the Head-Toes-Knees-Shoulders (HTKS) game
is built to assess cognitive dysfunctions, and a graphical interface is
developed to visualize scores, charts and other information about the game.
[23] centers on the fall detection task, establishing different phases: data
acquisition through a camera system, computation on a central server and alert
to a control center if a fall is detected, in charge of organizing a suitable
response. In [99], a Docker-based system is proposed to control the flow
between the different programs which participate in the fall detection task,
distributing the amount of resources and regulating the communications. Docker
is also used in [103], where the NAO robot is proposed for the data
acquisition and interaction with the user to prevent falling situations. In
[75], recorded video is processed and recognized activities are forwarded
through a gateway to notify doctors or relatives or to update a graphical
interface for monitoring the users. Similarly, in [58] the recordings are sent
to a cloud server, which performs the HAR task and updates a mobile
application. In [55], an intermediate step between the recording and the DL
computation is added, to preprocess the video data and reduce the consumed
bandwidth, which is also the case of [57]. In [126], the Pepper robot is used
to deploy the system for its good acceptance among elderly people, and to
recognize different exercises while interacting with the user. In [39], the
system transfers the recording to a computer for human inspection when an
abnormal state (fall) is detected. In [85], the proposed framework integrates
the recognition of different activities in the monitoring system, and
communicates with a control module which will use the results with different
purposes, like triggering an alarm, and finally a similar framework is
proposed in [47], where only cameras are deployed in the environment, which
send the recordings to the cloud for fall detection and then warning alarms
are triggered when a fall occurs.

###  5.6 RQ2.1: Hardware

To answer the review question RQ2.1, a list containing the hardware used in
the reviewed studies (when available) has been made and is displayed in Table
9. Only hardware related to special cameras (thermal and depth) and to robots
has been taken into account, since there is large variability in the common
RGB cameras and in the hardware used for computation. Moreover, hardware
information relative to datasets not created in the reviewed studies has been
skipped.

Table 9: Hardware (special cameras and robots) used in the collected studies. Type | Model | Brand | Ref. | Year | Authors
---|---|---|---|---|---
Depth Camera | Kinect (or v2) | Microsoft | [136] | 2017 | Adhikari et al.
|  |  | [137] | 2019 | Siriwardhana et al.
|  |  | [150] | 2014 | Jalal et al.
|  |  | [105] | 2021 | Li and Chen
|  |  | [147] | 2019 | Saini et al.
|  |  | [79] | 2022 | Guerra et al.
|  |  | [114] | 2019 | Das et al.
Depth Camera | Astra (or Pro) | Orbbec | [100] | 2020 | Popescu et al.
|  |  | [35] | 2022 | Denkovski et al.
|  |  | [62] | 2021 | Lumetzberger et al.
Depth Camera | XtionLive | ASUS | [144] | 2016 | Parisi and Wermter
Depth Camera | Horizon LiDAR | Livox | [56] | 2021 | Tu et al.
Thermal Camera | MLX90640 | Melexis | [30] | 2021 | Rezaei et al.
|  |  | [96] | 2020 | Tateno et al.
Thermal Camera | FLIR E60 | iTherml | [120] | 2018 | Akula et al.
Thermal Camera | FLIR ONE | iTherml | [35] | 2022 | Denkovski et al.
Thermal Camera | PI450 | Optris | [123] | 2019 | Ma et al.
Thermal Camera | GRID-EYE | Panasonic | [134] | 2017 | Fan et al.
Thermal Camera | AMG8831 | Panasonic | [66] | 2021 | Badarch et al.
Thermal Camera | HTPA32x32d | Heimann Sensor | [133] | 2019 | Rafferty et al.
Robot | Pepper | Aldebaran URG | [143] | 2019 | Nan et al.
|  |  | [126] | 2018 | Costa et al.
|  |  | [110] | 2020 | Lang et al.
Robot | NAO | Aldebaran URG | [144] | 2016 | Parisi and Wermter
|  |  | [103] | 2021 | Killian et al.
Robot | LOLA | Custom made | [125] | 2019 | Maldonado-Bascón et al.
Robot | Dori | Custom made | [61] | 2022 | Kim et al.

To retrieve depth video, the most used camera is the Microsoft Kinect,
followed by the Orbbec Astra. This may be due to the fact that they include
both RGB and depth camera (based on IR camera), allowing 3D skeleton joint
computation with sufficiently accurate results.

There is less consensus in the use of thermal cameras, where many different
models of camera are used. There are also a lot of differences in the data
retrieved, because of the sensors sensitivity to temperature, distance and
resolution.

Only 7 studies have been found deploying HAR or fall detection on an AAL
system using a robot, 3 of them using the Pepper robot, 2 using the NAO robot
and the remaining using custom-made robots.

###  5.7 RQ2.2: Privacy protection

Not all studies lead with the privacy concerns when deploying a HAR or fall
detection system in an AAL system. However, the solution can be found in the
used data type selection, since some of them preserve the privacy better than
others.

In Figure 8, the different privacy protection methods are shown, integrating
the use of a less intrusive data type among them. As seen, from a total of 137
reviewed studies, 61 do not cope with privacy concerns at all, using
unmodified RGB video or images which show the elderly users of the system.

Figure 8: Distribution of studies by method used to preserve privacy. The
total gives 138 studies because in [35] either depth or thermal data can be
used.

There are 3 special cases where RGB data was used but including some kind of
privacy protection: in [123], an IR camera is used to detect the face region
of the frames and remove it from the RGB frames; in [124], blurring of the
frames is performed before further recognition; and in [18], the RGB frames
are modified in such a way that the person can not be identified, but fall
detection can still be applied with good results. Despite this special cases,
the most extended way of preserving the privacy of the users when using RGB
cameras is skeleton joints estimation as the data used for classification.
This is the case of 45 studies, which makes it the most extended privacy
protection technique among all observed.

Another valid approach to preserve privacy is to avoid RGB cameras at all,
either by using only thermal or depth cameras. This is the case of a total of
29 studies, from which the majority of the studies work with depth rather than
with infrared data (22 vs. 7 studies).

##  6 Discussion

In Figure 9, the search process along with the main results found from the
reviewed studies are briefly summarized. In this section, the found results
are used to answer the review questions and the main strengths and weaknesses
of the studies are discussed. A list of our recommendations based on our
findings is also disclosed for use in future works.

Figure 9: Summary of the search process and found results.

###  6.1 RQ1: Which DL techniques are used for human activity recognition and
fall detection on elderly people?

Even though fall detection could be understood as a special case of HAR, the
findings show that the most frequent is to address this tasks separately: only
18 out of 137 studies (see Figure 4) cope with the two tasks. This can be due
to the importance of the fall detection compared to the other activities (e.g.
walking, getting up, etc.), which leads to an unbalanced problem if not dealt
with separately.

To answer the RQ1.1, the most extended vision data type is RGB by far, as
already seen in Figure 5. This may have to do with the ease of obtaining a
common camera compared to a special one, equipped with either depth or
infrared sensors, and other associated benefits, like a lower cost or the
possibility to check data visually in an easier way. Infrared are the least
used kind of cameras. They tend to be placed on the ceiling (top-down
perspective) and to use very low resolutions, which enables the use of simpler
(fewer layers) CNN models [30] [120] [133] and even of non-convolutionals like
the LSTM [66] [96] [134], GRU [134] and Transformer [66]. Depth cameras are
more extended than IR ones, but they are more used to extract skeleton joints
than to perform fall detection and HAR directly on depth images/video.

Most extended architectures, related with RQ1.2, were shown in Section 5.3 and
summarized in Table 7. Moreover, a taxonomy according to different aspects of
the DL models was displayed in 6. Clearly, the most extended architectures are
CNNs, and are used for many different tasks in the reviewed studies: skeleton
joints estimation (most frequent models are OpenPose [154] and AlphaPose
[155]), optical flow computation (PWC-Net [170] and LiteFlowNet [173]), object
detection (R-CNN [175] and YOLO [178]), extraction of spatial features (like
in [97] or [45]), or directly to detect falls or recognize activities from
images (like in [117], [52] or [98]), from video (like in [102] or [81]) or
even from skeletal data (like in [42] or [51]). The other preferred
architecture, from our findings, are RNNs, which are mostly used to work with
skeletal data (like in [158] or [23]), but can also be found after CNNs, so
that CNNs extract spatial features from each frame, and RNNs deal with the
sequence of spatial features (like in [80] or [118]). Among the RNNs, the
preferred model is the LSTM by far, followed by GRU, as seen in Table 7.

Another critical point when performing the fall detection and HAR tasks using
DL is the dataset to learn from (RQ1.3), and the first decision to take is
whether to use an already existing dataset or to create a custom one. Using
only an external one can affect to the applicability of the technique to
specific situations or environments, but it has the advantage of allowing
comparison of different methods on the same data, while using only a custom
one has the opposite effects. In Figure 7, the proportion of each type was
already shown, and 41 can be observed to use only a custom dataset. The main
inconvenient associated with this fact is the external validity of the
findings, since no other state-of-the-art methods are evaluated on the same
data for comparison. Moreover, in most reviewed studies where the only
evaluation was done on a custom dataset, data was not made publicly available
and very poor details were disclosed about them, thus reducing in great
measure the reliability of the obtained results. From our perspective, to
prove the well-functioning of the developed system, these studies should
include an evaluation on external datasets.

By taking a deep look to Table 8, we think that the most used datasets are, in
general, not the most suitable for the fall detection and HAR for the elderly
people. First of all, because they do not contain elderly people at all, which
we think is crucial to guarantee good results in a real deployment. In this
aspect, the ETRIActivity3D [210], ToyotaSmartHome [114], HQFSD [212], MUVIM
[35] and FPDS-Elderly [219] datasets would better fit, although the HQFSD
dataset could be excluded for being synthetic. Another argument to not
consider the most extended datasets is the number of samples, which we find
too poor for the DL models to properly generalize. The two most used datasets
consist of less than 200 samples, while approximately half of the used dataset
contain less than 1,000. Instead, we think that datasets like the
ETRIActivity3D, NTU RGB+D (or NTU RGB+D 120) or ToyotaSmartHome, which all
offer more tan 10,000 samples, would bring better generalization results. A
last note we want to remark on the dataset choosing step is that it is
preferable for a dataset to be constructed for ADL than for general HAR. For
instance, Kinetics (400, 600 or 700) or UCF101 would not be suitable for the
tasks we consider, because they consist of videos collected from the Internet
and may contain cuts and irrelevant activities.

###  6.2 RQ2: How can these tasks be deployed successfully in a real
environment?

A list of studies including fall detection and HAR in a framework has been
given in Section 5.5: a total of 15 studies. In these studies, the most common
is to use a deployment of cameras of different types to collect the data and
send it to a central server to then compute fall detection and HAR.

Answering RQ2.1, the camera used differs greatly in the case of recording RGB
video, but to record depth data, there is a clear preference for the Microsoft
Kinect (7 studies) and the Orbbec Astra (3 studies) cameras, as already shown
in Table 9. On the other hand, there is not a clear preference for thermal
recording, since only one camera is used in more than one study: Melaxis
MLX90640. Therefore, there is a great variability regarding the images taken
with the different cameras: a very broad range of resolutions (from 8 $\times$
8 images with the Panasonic GRID-EYE [134] to 1440 $\times$ 1080 with the
iTherml FLIR ONE [35]), different maximum distances and even different
sensitivity to temperature.

Also in reference to RQ2.1, the most used robots are Pepper and NAO, both from
Aldebaran URG. The two robots include different RGB cameras, microphones,
speakers and other sensors to interact with users in a more human manner. It
is even possible to incorporate new kinds of camera, as shown in [144], where
an ASUS XtionLive is located on the head of the robot to enable depth vision.
However, most probably these two robots are the most extended due to their
totally reprogrammable interface, since it enables to change the software used
for the different tasks in order to better fit different situations.

Another important aspect when deploying a fall detection or HAR system in a
real environment is the privacy of the users (RQ2.2), which is a major concern
in many of the reviewed studies, while it is not for many others: as already
mentioned in Section 5.7, almost half (61 of 137) of the reviewed studies do
not take into account the privacy preservation of the users, either leaving it
for future work or directly omitting it. From our point of view and having
observed the different ways of coping with privacy protection, it is not an
aspect to be skipped or postponed for later, since the main solutions consist
on changing the kind of data used, and thus change radically the possible
solutions. The most straightforward approach is using depth or infrared
imaging, which will guarantee that the users are not recognized in any phase
of the recognition process. However, since this is a very limiting method, a
trade-off between no privacy protection and no RGB recording can be to allow
RGB recording only in the first step of recognition, avoiding the storage of
these data and instead storing (if needed) anonymized features. These kinds of
features can be either modified RGB recordings with privacy protection (like
in [123], [124] and [18]), which allow to store (and thus, check) footage of
real cases but keeping privacy; skeleton joints (the most extended approach,
being used in 44% of the studies, as seen in Section 5.2), which is, at the
same time, a very efficient way to greatly reduce the amount of data to
process; or spatio-temporal features, obtained using different DL models, with
the inconvenient of not being human-interpretable.

###  6.3 Strengths and weaknesses of the reviewed studies

Having reviewed the 137 relevant studies and answered the review questions, in
this subsection we discuss the main strengths and weaknesses we have observed,
which we think can be of use to know at the time of working in new studies on
the topic.

We have found many benefits in the use of skeleton joints: using them instead
of raw image or video data allows a great reduction in the data size, at the
same time of offering user anonymization and still being able to obtain good
results in fall detection and HAR. In addition, there is an increasing number
of ways to obtain human skeletons from RGB or depth data: only in the reviewed
studies, 12 different DL models to estimate skeleton joints from image or
video data were found (shown in Table 7).

The major strength of studies using only depth or infrared data is the privacy
protection they offer, because there is not RGB footage recorded at any time
of the system pipeline. However, the major weaknesses of these studies are
two: having less data to perform detection or recognition (more notable in the
case of IR recordings, where the resolution tends to be much lower) and having
less understandable data, which can bring some problems when the system needs
to be manually checked when errors occur.

From among the reviewed studies, there are 18 which perform both tasks: fall
detection and HAR (refer to Figure 4), which we think is of remarkable
importance, because when performing HAR on elderly people, it will most
possibly be desirable to be able to also detect accidental falls. An important
note on this is that although fall detection can be integrated as another
class when performing HAR, it should be computed separately, because it is
critical. For this reason, most studies including fall detection perform it in
a different way than the recognition of the other classes.

Since fall detection and HAR for the elderly are tasks aiming to help this
collective in an AAL, studies which offer a framework for deploying the system
in a real environment are of especial interest. This is the case of 15
studies, listed in Section 5.5.

Even though the major part of the reviewed studies include an evaluation on
existing datasets for fall detection and HAR, there are 41 which only perform
it on new custom datasets (shown in Figure 7), which limits the reliability of
the results when there is not comparison with other existing techniques or
models. On the other hand, 19 of the studies still make use of a custom
dataset, but also offer an evaluation on external datasets, getting the best
out of both: specialization on the custom data and comparison with other
approaches.

From the data perspective, we have observed two frequent weaknesses: the
number of samples of the datasets used, and the lack of elderly people in
these datasets. As shown in Table 8, the two most used datasets contain less
than two hundred samples which seems too few when using DL to perform the
tasks of fall detection and HAR. Moreover, in the same table it can be seen
that, in the majority of the datasets (89%), elderly people do not participate
in the samples, which can bring problems at the deployment phase, since they
are the target users of the system.

###  6.4 Recommendations for future studies on the topic

Based on the results of this SLR, we list a series of especially important
facts, to our understanding, which should be taken into account when
performing a new study on the topic.

First of all, it is important to assess user’s privacy. As seen, the way to
protect privacy will most probably affect to the kind of data to use, going
from using RGB data to either modified RGB, depth, IR or skeleton data, which
prevent the identification of the users in the footage. For this reason, we
recommend to consider it as a key aspect from the beginning.

To evaluate the proposed model, a publicly available dataset should be used,
to enable comparison with other existing models or techniques. The most
extended ones for fall detection are URFD [199] and MultiCam [200], while for
HAR are UP-FALL [201] and NTU RGB+D [202]. Instead of the mentioned datasets,
however, we encourage the use of ETRIActivity3D [210] or ToyotaSmartHome
[114], which offer a much larger collection of video samples and include
elderly people in the data. Both datasets can be used for HAR, while only
ETRIActivity3D includes falls in the dataset, and both offer different
perspectives from many elderly users, many classes (at least 30) and also
various modalities, including RGB, depth and skeleton joints.

If a custom dataset is provided, we recommend to make it publicly available
for different reasons: it can be of use in future studies, by either directly
using it or merging it with other ones to form a larger dataset; it will
increase the reliability of the experiments on it, by making them replicable;
and it will enable comparison of the proposed model or technique with newer
ones. To collect the custom dataset, the most common is to use a Microsoft
Kinect camera if depth or skeleton data are desired, while there is no common
agreement for infrared recordings.

When deploying the system in a real environment, according to the reviewed
studies, the most usual is to build a camera setup in the environment, which
records the data and sends it to a central server to perform the different
tasks. It is also the most affordable option, depending on the kind of camera,
resolution and processing it is needed by the system. Nevertheless, in case it
is preferred to use an assistive robot, both the NEO and the Pepper robots are
viable solutions that have been used in 5 of the reviewed studies. Their main
advantage is to incorporate cameras, speakers, microphones and other required
components, at the same time of offering many customizable options, which can
make them able to adapt to different projects and environments.

##  7 Conclusions

In this work, an SLR on fall detection and Human Activity Recognition (HAR)
for the elderly has been performed. The study has focused on addressing two
primary review questions related to the implementation of deep learning
techniques for these tasks, using computer vision data, specifically looking
at the various DL architectures, datasets, and data types employed.
Furthermore, the study examined the deployment of these techniques in real-
world environments, considering the hardware used and the privacy of the
users. The findings provide valuable insights into the effective
implementation of deep learning techniques for fall detection and human
activity recognition in elderly care.

We reviewed 137 relevant studies and structured the main findings to make the
available evidence more accessible to readers and researchers. The main
results of this work can be of especial interest for future studies on this
topic given the rising importance and prominence of AAL and DL techniques. Our
main contributions are a summary of the existent methods (Tables 4 and 5), the
datasets used in the reviewed studies which are publicly available (Table 8),
the DL models used along with basic information (Table 7), the depth and
thermal cameras and the robots used in the different studies (Table 9), and a
list of the studies integrating the system in a framework (Section 5.5).

Although privacy is a common concern when deploying camera-based systems, many
of the reviewed studies do not cope with it at all (45% to be precise, as seen
in Figure 8), and more studies assessing elderly users’ perception and
preferences regarding this aspect need to be done. The most extended privacy
protection method, whether if it is chosen bearing the privacy factor in mind
or not, is the use of skeleton joints estimation as data for fall detection
and HAR, being used in 44% of the studies. Other common methods include
avoiding RGB recordings, using instead depth or IR only cameras.

A great variety of DL architectures have been found in the reviewed studies,
but the vast majority are convolutional or recurrent. They are used in
different steps of the fall detection and HAR tasks, like skeleton joints
estimation, optical flow computation or spatio-temporal features extraction
only to mention a few. Convolution-based networks seem to be the preferred
ones for many of the mentioned tasks, like OpenPose [154] for skeleton joints
estimation, as shown in Table 7. However, to cope with skeletal data, RNNs
like the LSTM [177] take the lead.

DL based methods rely on big amounts of data to perform satisfactorily and to
generalize properly, and so we think that the most extended datasets for fall
detection like URFD [199] or MultiCam [200] with under two hundred samples are
not enough, which is also the case of many others of the datasets used in the
reviewed studies. Another important feature the datasets targeting elderly
people should have is, precisely, to include elderly people in the data, which
does not happen in the vast majority of the datasets. With these two factors
in mind, the ETRIActivity3D [210] dataset seems to be the most appropriate one
for the fall detection task, while this one along with the ToyotaSmartHome
[114] dataset seem the best options for the HAR task.

Future work on the topic could include diving deeper into specific DL
architectures, to review the main differences and contributions of the
studies, and even offer an evaluation comparison on different datasets.

## Declarations

Funding Grant PID2019-104829RA-I00 funded by MCIN/AEI/10.13039/ 501100011033,
project EXPLainable Artificial INtelligence systems for health and well-beING
(EXPLAINING). Grant PID2022-136779OB-C32 funded by MCIN/AEI/
10.13039/501100011033 and by ERDF A way of making Europe, project Playful
Experiences with Interactive Social Agents and Robots (PLEISAR): Social
Learning and Intergenerational Communication. F. X. Gaya-Morey was supported
by an FPU scholarship from the Ministry of European Funds, University and
Culture of the Government of the Balearic Islands.

Competing interests The authors have no competing interests to declare that
are relevant to the content of this article.

Authors’ contributions F.Xavier Gaya-Morey: Conceptualization, Methodology,
Validation, Investigation, Writing - Original Draft, Writing - Review &
Editing Preparation, Visualization.

Cristina Manresa-Yee: Conceptualization, Methodology, Writing - Review &
Editing Preparation, Supervision, Project administration, Funding acquisition.

Jose M. Buades-Rubio: Conceptualization, Methodology, Writing - Review &
Editing Preparation, Supervision, Project administration, Funding acquisition.

Ethical and informed consent for data used This article does not contain any
studies with human participants or animals performed by any of the authors.

Data availability and access All relevant data for the study can be found,
structured and organized in the form of tables, throughout this document. No
additional data were generated.

## References

  *   * Bloom and Luca [2016] Bloom, D.E., Luca, D.L.: Chapter 1 - the global demography of aging: Facts, explanations, future. vol. 1, pp. 3–56. North-Holland (2016)
  * WHO [2021] WHO: World Health Organization fact sheets: Falls. https://www.who.int/en/news-room/fact-sheets/detail/falls (2021)
  * Heinrich et al. [2010] Heinrich, S., Rapp, K., Rissmann, U., Becker, C., König, H.-H.: Cost of falls in old age: a systematic review. Osteoporosis International 21, 891–902 (2010)
  * Climent-Pérez et al. [2020] Climent-Pérez, P., Spinsante, S., Mihailidis, A., Florez-Revuelta, F.: A review on video-based active and assisted living technologies for automated lifelogging. Expert Systems with Applications 139 (2020)
  * Khodabandehloo et al. [2021] Khodabandehloo, E., Riboni, D., Alimohammadi, A.: Healthxai: Collaborative and explainable ai for supporting early diagnosis of cognitive decline. Future Generation Computer Systems 116, 168–189 (2021)
  * Nizam and Jamil [2020] Nizam, Y., Jamil, M.M.A.: Classification of daily life activities for human fall detection: A systematic review of the techniques and approaches. Studies in Systems, Decision and Control 273, 137–179 (2020)
  * Walsh et al. [2019] Walsh, J., O’ Mahony, N., Campbell, S., Carvalho, A., Krpalkova, L., Velasco-Hernandez, G., Harapanahalli, S., Riordan, D.: Deep learning vs. traditional computer vision. (2019)
  * Kitchenham and Charters [2007] Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering 2 (2007)
  * Olugbade et al. [2022] Olugbade, T., Bieńkiewicz, M., Barbareschi, G., D’amato, V., Oneto, L., Camurri, A., Holloway, C., Björkman, M., Keller, P., Clayton, M., Williams, A.C.D.C., Gold, N., Becchio, C., Bardy, B., Bianchi-Berthouze, N.: Human movement datasets: An interdisciplinary scoping review. ACM Comput. Surv. 55 (2022)
  * Momin et al. [2022] Momin, M.S., Sufian, A., Barman, D., Dutta, P., Dong, M., Leo, M.: In-home older adults’ activity pattern monitoring using depth sensors: A review. Sensors 22 (2022)
  * Alam et al. [2022] Alam, E., Sufian, A., Dutta, P., Leo, M.: Vision-based human fall detection systems using deep learning: A review. Computers in Biology and Medicine 146, 105626 (2022)
  * Rastogi and Singh [2022] Rastogi, S., Singh, J.: Human fall detection and activity monitoring: a comparative analysis of vision-based methods for classification and detection techniques. Soft Computing 26, 3679–3701 (2022)
  * Gutiérrez et al. [2021] Gutiérrez, J., Rodríguez, V., Martin, S.: Comprehensive review of vision-based fall detection systems. Sensors 21(3) (2021)
  * Dhiman and Vishwakarma [2019] Dhiman, C., Vishwakarma, D.K.: A review of state-of-the-art techniques for abnormal human activity recognition. Engineering Applications of Artificial Intelligence 77, 21–45 (2019)
  * Ezatzadeh and Keyvanpour [2019] Ezatzadeh, S., Keyvanpour, M.R.: Vifa: an analytical framework for vision-based fall detection in a surveillance environment. Multimedia Tools and Applications 78, 25515–25537 (2019)
  * Sathyanarayana et al. [2018] Sathyanarayana, S., Satzoda, R.K., Sathyanarayana, S., Thambipillai, S.: Vision-based patient monitoring: a comprehensive review of algorithms and technologies. Journal of Ambient Intelligence and Humanized Computing 9, 225–251 (2018)
  * Li et al. [2022] Li, S., Man, C., Shen, A., Guan, Z., Mao, W., Luo, S., Zhang, R., Yu, H.: A fall detection network by 2d/3d spatio-temporal joint models with tensor compression on edge. ACM Trans. Embed. Comput. Syst. 21 (2022)
  * Liu et al. [2021] Liu, J., Tan, R., Han, G., Sun, N., Kwong, S.: Privacy-preserving in-home fall detection using visual shielding sensing and private information-embedding. IEEE Transactions on Multimedia 23, 3684–3699 (2021)
  * Meraikhi and Al-Rajab [2022] Meraikhi, S.A., Al-Rajab, M.: A multimodal approach of machine and deep learnings to enhance the fall of elderly people. Journal of Information Technology Management 14, 168–184 (2022)
  * Zherdev et al. [2021] Zherdev, D., Zherdeva, L., Agapov, S., Sapozhnikov, A., Nikonorov, A., Chaplygin, S.: Producing synthetic dataset for human fall detection in ar/vr environments. Applied Sciences (Switzerland) 11 (2021)
  * Chen et al. [2022] Chen, P.-C., Chang, C.-H., Chan, Y.-W., Tasi, Y.-T., Chu, W.C.: An approach to real-time fall detection based on openpose and lstm, pp. 1573–1578 (2022)
  * Feng and Jiang [2021] Feng, X., Jiang, W.: Research on human fall detection based on tiny-yolov3 algorithm, pp. 1326–1330. Association for Computing Machinery, New York, NY, USA (2021)
  * Anwary et al. [2022] Anwary, A.R., Rahman, M.A., Muzahid, A.J.M., Ashraf, A.W.U., Patwary, M., Hussain, A.: Deep learning enabled fall detection exploiting gait analysis. Annu Int Conf IEEE Eng Med Biol Soc 2022, 4683–4686 (2022)
  * Xie et al. [2021] Xie, L., Yang, Y., Zeyu, F., Naqvi, S.M.: Skeleton-based fall events classification with data fusion. Institute of Electrical and Electronics Engineers Inc., Karlsruhe, Germany (2021)
  * Fayad et al. [2022] Fayad, M., Hachani, M.-Y., Mostefaoui, A., Chouali, S., Yahiaoui, R.: Elderly fall detection: A lightweight kinect based deep learning approach, pp. 89–95. Association for Computing Machinery, New York, NY, USA (2022)
  * Fatima et al. [2021] Fatima, M., Yousaf, M.H., Yasin, A., Velastin, S.A.: Unsupervised fall detection approach using human skeletons, pp. 1–6 (2021)
  * Lau et al. [2022] Lau, X.L., Connie, T., Goh, M.K.O., Lau, S.H.: Fall detection and motion analysis using visual approaches. International Journal of Technology 13, 1173–1182 (2022)
  * Berlin and John [2021] Berlin, S.J., John, M.: Vision based human fall detection with siamese convolutional neural networks. Journal of Ambient Intelligence and Humanized Computing (2021)
  * Nigam et al. [2022] Nigam, N., Dutta, T., Verma, D.: Fall-perceived action recognition of persons with neurological disorders using semantic supervision. IEEE Transactions on Cognitive and Developmental Systems, 1 (2022)
  * Rezaei et al. [2021] Rezaei, A.M., Stevens, M.C., Argha, A., Mascheroni, A., Puiatti, A., Lovell, N.H.: An unobtrusive fall detection system using low resolution thermal sensors and convolutional neural networks, pp. 6949–6952 (2021)
  * Aarthi and Juliet [2022] Aarthi, M.S., Juliet, S.: Intelligent fall detection system based on sensor and image data for elderly monitoring, pp. 1259–1265 (2022)
  * Fernando et al. [2021] Fernando, Y.P.N., Gunasekara, K.D.B., Sirikumara, K.P., Galappaththi, U.E., Thilakarathna, T., Kasthurirathna, D.: Computer vision based privacy protected fall detection and behavior monitoring system for the care of the elderly, vol. 2021-September, pp. 1–7. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2021)
  * Carrara et al. [2022] Carrara, F., Pasco, L., Gennaro, C., Falchi, F.: Learning to detect fallen people in virtual worlds, pp. 126–130. Association for Computing Machinery, New York, NY, USA (2022)
  * Tseng et al. [2021] Tseng, H.-T., Hsieh, C.-C., Hsu, T.-Y.: Elder action recognition based on convolutional neural network and long short-term memory, pp. 1–2 (2021)
  * Denkovski et al. [2022] Denkovski, S., Khan, S.S., Malamis, B., Moon, S.Y., Ye, B., Mihailidis, A.: Multi visual modality fall detection dataset. IEEE Access 10, 106422–106435 (2022)
  * Ramirez et al. [2021] Ramirez, H., Velastin, S.A., Meza, I., Fabregas, E., Makris, D., Farias, G.: Fall detection and activity recognition using human skeleton features. IEEE Access 9, 33532–33542 (2021)
  * Galvão et al. [2022] Galvão, Y.M., Portela, L., Barros, P., Araújo Fagundes, R.A., Fernandes, B.J.T.: Onefall-gan: A one-class gan framework applied to fall detection. Engineering Science and Technology, an International Journal (2022)
  * Wang et al. [2021] Wang, X., Talavera, E., Karastoyanova, D., Azzopardi, G.: Fall detection and recognition from egocentric visual data: A case study. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12661 LNCS, 431–443 (2021)
  * Zheng et al. [2022] Zheng, H., Liu, Y., Wu, X., Zhang, Y.: Realization of elderly fall integration monitoring system based on alphapose and yolov4, pp. 604–620. Institute of Electrical and Electronics Engineers Inc., Hangzhou, China (2022)
  * Hasib et al. [2021] Hasib, R., Khan, K.N., Yu, M., Khan, M.S.: Vision-based human posture classification and fall detection using convolutional neural network, pp. 74–79 (2021)
  * Zahan et al. [2022] Zahan, S., Hassan, G.M., Mian, A.: Sdfa: Structure aware discriminative feature aggregation for efficient human fall detection in video. IEEE Transactions on Industrial Informatics, 1–9 (2022)
  * Giannakos et al. [2021] Giannakos, I., Mathe, E., Spyrou, E., Mylonas, P.: A study on the effect of occlusion in human activity recognition, pp. 473–482. Association for Computing Machinery, New York, NY, USA (2021)
  * Rajalaxmi et al. [2022] Rajalaxmi, R.R., Gothai, E., Suganth, V., Vignesh, S., Varun, T.: Vision based fall detection using optimized convolutional neural network, pp. 1–6. Institute of Electrical and Electronics Engineers Inc., Coimbatore, India (2022)
  * Wang and Yang [2021] Wang, J., Yang, W.: Action recognition based on cross spatital temporal graph convolution. Association for Computing Machinery, New York, NY, USA (2021)
  * Anitha and Priya [2022] Anitha, G., Priya, S.B.: Vision based real time monitoring system for elderly fall event detection using deep learning. Computer Systems Science and Engineering 42, 87–103 (2022)
  * Awal et al. [2021] Awal, M.I., Iksan, L.H., Fhamy, R.Z., Basuki, D.K., Sukaridhoto, S., Wada, K.: Action recognition with spatiotemporal analysis and support vector machine for elderly monitoring system, pp. 470–475. Institute of Electrical and Electronics Engineers Inc., Surabaya, Indonesia (2021)
  * Zhang et al. [2022] Zhang, Y., Zheng, X., Liang, W., Zhang, S., Yuan, X.: Visual surveillance for human fall detection in healthcare iot. IEEE Multimedia 29, 36–46 (2022)
  * Byeon et al. [2021] Byeon, Y.-H., Kim, D., Lee, J., Kwak, K.-C.: Body and hand–object roi-based behavior recognition using deep learning. Sensors 21, 1–23 (2021)
  * Inturi et al. [2022] Inturi, A.R., Manikandan, V.M., Garrapally, V.: A novel vision-based fall detection scheme using keypoints of human skeleton with long short-term memory network. Arabian Journal for Science and Engineering (2022)
  * Budisteanu and Mocanu [2021] Budisteanu, E.-A., Mocanu, I.G.: Combining supervised and unsupervised learning algorithms for human activity recognition. SENSORS 21 (2021)
  * Suarez et al. [2022] Suarez, J.J.P., Orillaza, N., Naval, P.: Afar: A real-time vision-based activity monitoring and fall detection framework using 1d convolutional neural networks, pp. 555–559. Association for Computing Machinery, New York, NY, USA (2022)
  * Sivakumar et al. [2021] Sivakumar, M., Iswarya, E., Malusha, K., Priyadharshini, T.Y.: Computer vision based wellness analysis of geriatrics, pp. 1762–1765. Institute of Electrical and Electronics Engineers Inc., Coimbatore, India (2021)
  * Yadav et al. [2022] Yadav, S.K., Luthra, A., Tiwari, K., Pandey, H.M., Akbar, S.A.: Arfdnet: An efficient activity recognition & fall detection system using latent feature pooling. Knowledge-Based Systems 239 (2022)
  * Hwang et al. [2021] Hwang, H., Jang, C., Park, G., Cho, J., Kim, I.-J.: Eldersim: A synthetic data generation platform for human action recognition in eldercare applications. IEEE Access, 1 (2021)
  * Rajavel et al. [2022] Rajavel, R., Ravichandran, S.K., Harimoorthy, K., Nagappan, P., Gobichettipalayam, K.R.: Iot-based smart healthcare video surveillance system using edge computing. Journal of Ambient Intelligence and Humanized Computing 13, 3195–3207 (2022)
  * Tu et al. [2021] Tu, L., Ouyang, X., Zhou, J., He, Y., Xing, G.: Feddl: Federated learning via dynamic layer sharing for human activity recognition, pp. 15–28. Association for Computing Machinery, New York, NY, USA (2021)
  * Wang et al. [2022] Wang, B., Wu, X., Gong, M., Zhao, J., Sun, Y.: Lightweight network based real-time anomaly detection method for caregiving at home, pp. 1323–1328. Institute of Electrical and Electronics Engineers Inc., Hangzhou, China (2022)
  * Iksan et al. [2021] Iksan, L.H., Awal, M.I., Fhamy, R.Z., Pratama, A.A., Basuki, D.K., Sukaridhoto, S.: Implementation of cloud based action recognition backend platform, pp. 1–6 (2021)
  * Xie et al. [2022] Xie, L., Sun, Y., Chambers, J.A., Naqvi, S.M.: Privacy preserving multi-class fall classification based on cascaded learning and noisy labels handling, pp. 1–6. Institute of Electrical and Electronics Engineers Inc., Linköping, Sweden (2022)
  * Tianming et al. [2021] Tianming, Z., Pengbiao, Z., Peng, X., Bintao, W.: Multi-stream cnn-lstm network with partition strategy for human action recognition, pp. 431–435. Association for Computing Machinery, New York, NY, USA (2021)
  * Kim et al. [2022] Kim, J.-W., Choi, Y.-L., Jeong, S.-H., Han, J.: A care robot with ethical sensing system for older adults at home. Sensors (Basel) 22, 7515 (2022)
  * Lumetzberger et al. [2021] Lumetzberger, J., Raoofpour, A., Kampel, M.: Privacy preserving getup detection, pp. 234–243. Association for Computing Machinery, New York, NY, USA (2021)
  * Lin et al. [2022] Lin, F., Wang, Z., Zhao, H., Qiu, S., Shi, X., Wu, L., Gravina, R., Fortino, G.: Adaptive multi-modal fusion framework for activity monitoring of people with mobility disability. IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 26, 4314–4324 (2022)
  * Nambissan et al. [2021] Nambissan, G.S., Mahajan, P., Sharma, S., Gupta, N.: The variegated applications of deep learning techniques in human activity recognition, pp. 223–233. Association for Computing Machinery, New York, NY, USA (2021)
  * He et al. [2022] He, J., Xiang, M., Zhao, X.: An elderly indoor behavior recognition method based on improved slowfast network, vol. 2216. IOP Publishing Ltd, San Francisco, CA, USA (2022)
  * Badarch et al. [2021] Badarch, L., Gochoo, M., Batnasan, G., Alnajjar, F., Tan, T.-H.: Ultra-low resolution infrared sensor-based wireless sensor network for privacy-preserved recognition of daily activities of living. Institute of Electrical and Electronics Engineers Inc., Boston, MA, USA (2021)
  * Patsch et al. [2022] Patsch, C., Zakour, M., Chaudhari, R.: Automatic recognition of human activities combining model-based ai and machine learning, pp. 15–22. SCITEPRESS, Setubal, Portugal (2022)
  * Tan et al. [2021] Tan, T.-H., Hus, J.-H., Liu, S.-H., Huang, Y.-F., Gochoo, M.: Using direct acyclic graphs to enhance skeleton-based action recognition with a linear-map convolution neural network. Sensors 21 (2021)
  * Zhang and Yang [2022] Zhang, C., Yang, X.: Bed-leaving action recognition based on yolov3 and alphapose, pp. 117–123. Association for Computing Machinery, New York, NY, USA (2022)
  * Mobsite et al. [2020] Mobsite, S., Alaoui, N., Boulmalf, M.: A framework for elders fall detection using deep learning, pp. 69–74 (2020)
  * Ouyang et al. [2022] Ouyang, X., Shuai, X., Zhou, J., Shi, I.W., Xie, Z., Xing, G., Huang, J.: Cosmo: Contrastive fusion learning with small data for multimodal human activity recognition, pp. 324–337. Association for Computing Machinery, New York, NY, USA (2022)
  * Han et al. [2020] Han, K., Yang, Q., Huang, Z.: A two-stage fall recognition algorithm based on human posture features. Sensors (Switzerland) 20, 1–21 (2020)
  * Prasad and Ko [2022] Prasad, S.K., Ko, Y.-B.: Deep learning based human activity recognition with improved accuracy, vol. 2022-October, pp. 1492–1495. IEEE Computer Society, Jeju Island, Republic of Korea (2022)
  * Chiang and Zhang [2020] Chiang, J.W.H., Zhang, L.: Deep learning-based fall detection, vol. 12, pp. 891–898. WORLD SCIENTIFIC PUBL CO PTE LTD, Singapore (2020)
  * Achirei et al. [2022] Achirei, S.-D., Heghea, M.-C., Lupu, R.-G., Manta, V.-I.: Human activity recognition for assisted living based on scene understanding. Applied Sciences (Switzerland) 12 (2022)
  * Khraief et al. [2020] Khraief, C., Benzarti, F., Amiri, H.: Elderly fall detection based on multi-stream deep convolutional networks. Multimedia Tools and Applications 79, 19537–19560 (2020)
  * Islam et al. [2022] Islam, M.M., Nooruddin, S., Karray, F.: Multimodal human activity recognition for smart healthcare applications, vol. 2022-October, pp. 196–203. Institute of Electrical and Electronics Engineers Inc., Prague, Czech Republic (2022)
  * Serpa et al. [2020] Serpa, Y.R., Nogueira, M.B., Neto, P.P.M., Rodrigues, M.A.F.: Evaluating pose estimation as a solution to the fall detection problem, pp. 1–7. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2020)
  * Guerra et al. [2022] Guerra, B.M.V., Schmid, M., Beltrami, G., Ramat, S.: Neural networks for automatic posture recognition in ambient-assisted living. Sensors 22 (2022)
  * Berardini et al. [2020] Berardini, D., Moccia, S., Migliorelli, L., Pacifici, I., Massimo, P.D., Paolanti, M., Frontoni, E.: Fall detection for elderly-people monitoring using learned features and recurrent neural networks. Experimental Results 1 (2020)
  * Isoi et al. [2022] Isoi, H., Takefusa, A., Nakada, H., Oguchi, M.: Performance of domain adaptation schemes in video action recognition using synthetic data, pp. 70–79. Association for Computing Machinery, New York, NY, USA (2022)
  * Romaissa et al. [2020] Romaissa, B.D., Mourad, O., Brahim, N., Yazid, B.: Fall detection using body geometry in video sequences, pp. 1–5 (2020)
  * Sun and Chen [2022] Sun, H., Chen, Y.: Real-time elderly monitoring for senior safety by lightweight human action recognition, vol. 2022-May, pp. 1–6. IEEE Computer Society, Lincoln, NE, USA (2022)
  * Wang and Jia [2020] Wang, X., Jia, K.: Human fall detection algorithm based on yolov3, pp. 50–54. Institute of Electrical and Electronics Engineers Inc., Beijing, China (2020)
  * Ji [2022] Ji, Q.: The design of the lightweight smart home system and interaction experience of products for middle-aged and elderly users in smart cities. Comput Intell Neurosci 2022, 1279351 (2022)
  * Lv et al. [2020] Lv, X., Gao, Z., Yuan, C., Li, M., Chen, C.: Hybrid real-time fall detection system based on deep learning and multi-sensor fusion, pp. 386–391. Institute of Electrical and Electronics Engineers Inc., Shenzhen, China (2020)
  * Galvao et al. [2021] Galvao, Y.M., Portela, L., Ferreira, J., Barros, P., Fagundes, O.A.D.A., Fernandes, B.J.T.: A framework for anomaly identification applied on fall detection. IEEE ACCESS 9, 77264–77274 (2021)
  * Kharazian et al. [2020] Kharazian, Z., Rahat, M., Fatemizadeh, E., Nasrabadi, A.M.: Increasing safety at smart elderly homes by human fall detection from video using transfer learning approaches, pp. 2774–2780. Research Publishing Services, Venice, Italy (2020)
  * Galvão et al. [2021] Galvão, Y.M., Ferreira, J., Albuquerque, V.A., Barros, P., Fernandes, B.J.T.: A multimodal approach using deep learning for fall detection. Expert Systems with Applications 168 (2021)
  * Li et al. [2020] Li, J., Xia, S.-T., Ding, Q.: Multi-level recognition on falls from activities of daily living, pp. 464–471. Association for Computing Machinery, New York, NY, USA (2020)
  * Kang et al. [2021] Kang, Y.-K., Kang, H.-Y., Kim, J.-B.: A study of fall detection system using context cognition method, pp. 79–83. Institute of Electrical and Electronics Engineers Inc., Ho Chi Minh City, Vietnam (2021)
  * Chen et al. [2020] Chen, Y., Li, W., Wang, L., Hu, J., Ye, M.: Vision-based fall event detection in complex background using attention guided bi-directional lstm. IEEE Access 8, 161337–161348 (2020)
  * Raj et al. [2021] Raj, A., Singh, D., Prakash, C.: Active human pose estimation for assisted living, pp. 110–116. Association for Computing Machinery, New York, NY, USA (2021)
  * Buzzelli et al. [2020] Buzzelli, M., Albé, A., Ciocca, G.: A vision-based system for monitoring elderly people at home. Applied Sciences (Switzerland) 10 (2020)
  * Yang et al. [2021] Yang, Y., Ren, H., Li, C., Ding, C., Yu, H.: An edge-device based fast fall detection using spatio-temporal optical flow model, pp. 5067–5071. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2021)
  * Tateno et al. [2020] Tateno, S., Meng, F., Qian, R., Li, T.: Human motion detection based on low resolution infrared array sensor, pp. 1016–1021 (2020)
  * Sultana et al. [2021] Sultana, A., Deb, K., Dhar, P.K., Koshiba, T.: Classification of indoor human fall events using deep learning. Entropy (Basel) 23 (2021)
  * Tan et al. [2020] Tan, T.-H., Gochoo, M., Chen, H.-S., Liu, S.-H., Huang, Y.-F.: Activity recognition based on dcnn and kinect rgb images, pp. 1–4 (2020)
  * Divya and Sri [2021] Divya, V., Sri, R.L.: Docker-based intelligent fall detection using edge-fog cloud infrastructure. IEEE Internet of Things Journal 8, 8133–8144 (2021)
  * Popescu et al. [2020] Popescu, A.-C., Mocanu, I., Cramariuc, B.: Fusion mechanisms for human activity recognition using automated machine learning. IEEE ACCESS 8, 143996–144014 (2020)
  * Chen et al. [2021] Chen, Y., Du, R., Luo, K., Xiao, Y.: Fall detection system based on real-time pose estimation and svm, pp. 990–993. Institute of Electrical and Electronics Engineers Inc., Nanchang, China (2021)
  * Mathe et al. [2020] Mathe, E., Tranou, A., Spyrou, E., Perantonis, S.: Human action recognition with deep learning techniques. Association for Computing Machinery, New York, NY, USA (2020)
  * Killian et al. [2021] Killian, L., Julien, M., Kevin, B., Maxime, L., Carolina, B., Mélanie, C., Nathalie, B., Sylvain, G., Sebastien, G.: Fall prevention and detection in smart homes using monocular cameras and an interactive social robot, pp. 7–12. Association for Computing Machinery, New York, NY, USA (2021)
  * Atikuzzaman et al. [2020] Atikuzzaman, M., Rahman, T.R., Wazed, E., Hossain, M.P., Islam, M.Z.: Human activity recognition system from different poses with cnn, pp. 1–5. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2020)
  * Li and Chen [2021] Li, X., Chen, W.: Fall recognition algorithm for the elderly based on home service robot, pp. 329–335. Institute of Electrical and Electronics Engineers Inc., Zhengzhou, China (2021)
  * Gul et al. [2020] Gul, M.A., Yousaf, M.H., Nawaz, S., Rehman, Z.U., Kim, H.: Patient monitoring by abnormal human activity recognition based on cnn architecture. Electronics (Switzerland) 9, 1–14 (2020)
  * Ge et al. [2021] Ge, W., Luo, X., Tao, R., Shi, Y.: Human fall detection algorithm based on mixed attention mechanism, pp. 32–37. Association for Computing Machinery, New York, NY, USA (2021)
  * Jaouedi et al. [2020] Jaouedi, N., Perales, F.J., Buades, J.M., Boujnah, N., Bouhlel, M.S.: Prediction of human activities based on a new structure of skeleton features and deep learning model. Sensors (Switzerland) 20, 1–15 (2020)
  * Pita et al. [2021] Pita, M.S.U., Alon, A.S., Melo, P.M.B., Hernandez, R.M., Magboo, A.I.: Indoor human fall detection using data augmentation-assisted transfer learning in an aging population for smart homecare: A deep convolutional neural network approach, pp. 64–69. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2021)
  * Lang et al. [2020] Lang, X., Feng, Z., Yang, X.: Research on human-robot natural interaction algorithm based on body potential perception, pp. 260–264. Association for Computing Machinery, New York, NY, USA (2020)
  * Vaiyapuri et al. [2021] Vaiyapuri, T., Lydia, E.L., Sikkandar, M.Y., Diaz, V.G., Pustokhina, I.V., Pustokhin, D.A.: Internet of things and deep learning enabled elderly fall detection model for smart homecare. IEEE Access 9, 113879–113888 (2021)
  * Wang et al. [2019a] Wang, H., Gao, Z., Lin, W.: A fall detection system based on convolutional neural networks, pp. 242–246. Association for Computing Machinery, New York, NY, USA (2019)
  * Wang et al. [2019b] Wang, F., Liu, J., Hu, G.D.: A novel indoor human fall detection method based on an end-to-end neural network and bagged tree classifier, pp. 384–389. Association for Computing Machinery, New York, NY, USA (2019)
  * Das et al. [2019] Das, S., Dai, R., Koperski, M., Minciullo, L., Garattoni, L., Bremond, F., Francesca, G.: Toyota smarthome: Real-world activities of daily living, vol. 2019-October, pp. 833–842. Institute of Electrical and Electronics Engineers Inc., Seoul, Korea (South) (2019)
  * Brieva et al. [2019] Brieva, J., Ponce, H., Moya-Albor, E., Martinez-Villasenor, L.: An intelligent human fall detection system using a vision-based strategy, pp. 1–5. Institute of Electrical and Electronics Engineers Inc., Los Alamitos, California, USA (2019)
  * Li et al. [2018] Li, W., Tang, P., Jin, W., Hu, C., He, Z.: Accidental fall detection based on pose analysis and svdd, pp. 9244–9249 (2018)
  * Hassan et al. [2019] Hassan, M.F.A., Hussain, A., Saad, M.H.M., Yusof, Y.: Convolution neural network-based action recognition for fall event detection. International Journal of Advanced Trends in Computer Science and Engineering 8 (2019)
  * Ge et al. [2018] Ge, C., Gu, I.Y.-H., Yang, J.: Co-saliency-enhanced deep recurrent convolutional networks for human fall detection in e-healthcare, pp. 1572–1575 (2018)
  * Mohamed et al. [2019] Mohamed, N.A., Zulkifley, M.A., Kamari, N.A.M.: Convolutional neural networks tracker with deterministic sampling for sudden fall detection, pp. 1–5. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2019)
  * Akula et al. [2018] Akula, A., Shah, A.K., Ghosh, R.: Deep learning approach for human action recognition in infrared images. Cognitive Systems Research 50, 146–154 (2018)
  * Kumar et al. [2019] Kumar, D., Ravikumar, A.K., Dharmalingam, V., Kafle, V.P.: Elderly health monitoring system with fall detection using multi-feature based person tracking, pp. 1–9. Institute of Electrical and Electronics Engineers Inc., Atlanta, GA, USA (2019)
  * Hsieh and Jeng [2018] Hsieh, Y.-Z., Jeng, Y.-L.: Development of home intelligent fall detection iot system based on feedback optical flow convolutional neural network. IEEE ACCESS 6, 6048–6057 (2018)
  * Ma et al. [2019] Ma, C., Shimada, A., Uchiyama, H., Nagahara, H., Taniguchi, R.-i.: Fall detection using optical level anonymous image sensing system. Optics and Laser Technology 110, 44–61 (2019)
  * Dimiccoli et al. [2018] Dimiccoli, M., Mar{́ı}n, J., Thomaz, E.: Mitigating bystander privacy concerns in egocentric activity recognition with deep learning and intentional image degradation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1 (2018)
  * Maldonado-Bascón et al. [2019] Maldonado-Bascón, S., Iglesias-Iglesias, C., Martín-Martín, P., Lafuente-Arroyo, S.: Fallen people detection capabilities using assistive robot. Electronics (Switzerland) 8 (2019)
  * Costa et al. [2018] Costa, A., Martinez-Martin, E., Cazorla, M., Julian, V.: Pharos-physical assistant robot system. Sensors (Basel) 18 (2018)
  * Cameiro et al. [2019] Cameiro, S.A., Silva, G.P.D., Leite, G.V., Moreno, R., Guimaraes, S.J.F., Pedrini, H.: Multi-stream deep convolutional network using high-level features applied to fall detection in video sequences, vol. 2019-June, pp. 293–298. IEEE Computer Society, New York, NY, USA (2019)
  * Chen et al. [2018] Chen, Y., Yu, L., Ota, K., Dong, M.: Robust activity recognition for aging society. IEEE Journal of Biomedical and Health Informatics 22, 1754–1764 (2018)
  * Ferooz et al. [2019] Ferooz, F., Ashraf, M.A., Hussain, W., Butt, A.H., Khan, Y.D.: Person fall recognition by using deep learning: Convolutional neural networks and image category classification using bag of feature, pp. 1–6 (2019)
  * Ge et al. [2017] Ge, C., Gu, I.Y.-H., Yang, J.: Human fall detection using segment-level cnn features and sparse dictionary learning, pp. 1–6 (2017)
  * Safarzadeh et al. [2019] Safarzadeh, M., Alborzi, Y., Ardekany, A.N.: Real -time fall detection and alert system using pose estimation, pp. 508–511 (2019)
  * Hwang et al. [2017] Hwang, S., Ahn, D., Park, H., Park, T.: Maximizing accuracy of fall detection and alert systems based on 3d convolutional neural network: Poster abstract, pp. 343–344. Association for Computing Machinery, New York, NY, USA (2017)
  * Rafferty et al. [2019] Rafferty, J., Medina-Quero, J., Quinn, S., Saunders, C., Ekerete, I., Nugent, C., Synnott, J., Garcia-Constantino, M.: Thermal vision based fall detection via logical and data driven processes, pp. 35–40. Institute of Electrical and Electronics Engineers Inc., Honolulu, HI, USA (2019)
  * Fan et al. [2017] Fan, X., Zhang, H., Leung, C., Shen, Z.: Robust unobtrusive fall detection using infrared array sensors, pp. 194–199 (2017)
  * Huang et al. [2019] Huang, Z., Liu, Y., Fang, Y., Horn, B.K.P.: Video-based fall detection for seniors with human pose estimation. Institute of Electrical and Electronics Engineers Inc., Boston, MA, USA (2019)
  * Adhikari et al. [2017] Adhikari, K., Bouchachia, H., Nait-Charif, H.: Activity recognition for indoor fall detection using convolutional neural network, pp. 81–84. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2017)
  * Siriwardhana et al. [2019] Siriwardhana, C., Madhuranga, D., Madushan, R., Gunasekera, K.: Classification of activities of daily living based on depth sequences and audio, pp. 278–283. Institute of Electrical and Electronics Engineers Inc., Kandy, Sri Lanka (2019)
  * Yu et al. [2017] Yu, M., Gong, L., Kollias, S.: Computer vision based fall detection by a convolutional neural network, pp. 416–420. Association for Computing Machinery, New York, NY, USA (2017)
  * Phyo et al. [2019] Phyo, C.N., Zin, T.T., Tin, P.: Complex human-object interactions analyzer using a dcnn and svm hybrid approach. Applied Sciences (Switzerland) 9 (2019)
  * Gattupalli et al. [2017] Gattupalli, S., Ebert, D., Papakostas, M., Makedon, F., Athitsos, V.: Cognilearn: A deep learning-based interface for cognitive behavior assessment, pp. 577–587. Association for Computing Machinery, New York, NY, USA (2017)
  * Phyo et al. [2019] Phyo, C.N., Zin, T.T., Tin, P.: Deep learning for recognizing human activities using motions of skeletal joints. IEEE Transactions on Consumer Electronics 65, 243–252 (2019)
  * Neili et al. [2017] Neili, S., Gazzah, S., Yacoubi, M.A.E., Amara, N.E.B.: Human posture recognition approach based on convnets and svm classifier, pp. 1–6. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2017)
  * Nan et al. [2019] Nan, M., Ghiță, A.S., Gavril, A.-F., Trascau, M., Sorici, A., Cramariuc, B., Florea, A.M.: Human action recognition for social robots, pp. 675–681 (2019)
  * Parisi and Wermter [2016] Parisi, G.I., Wermter, S.: A neurocognitive robot assistant for robust event detection. Studies in Computational Intelligence 633, 1–27 (2016)
  * Mehr and Polat [2019] Mehr, H.D., Polat, H.: Human activity recognition in smart home with deep learning approach, pp. 149–153. IEEE, New York, NY, USA (2019)
  * Wang et al. [2016] Wang, S., Chen, L., Zhou, Z., Sun, X., Dong, J.: Human fall detection in surveillance video based on pcanet. Multimedia Tools and Applications 75, 11603–11613 (2016)
  * Saini et al. [2019] Saini, R., Kumar, P., Kaur, B., Roy, P.P., Dogra, D.P., Santosh, K.C.: Kinect sensor-based interaction monitoring system using the blstm neural network in healthcare. International Journal of Machine Learning and Cybernetics 10, 2529–2540 (2019)
  * Feng et al. [2014] Feng, P., Yu, M., Naqvi, S.M., Chambers, J.A.: Deep learning for posture analysis in fall detection, pp. 12–17 (2014)
  * Jalal et al. [2019] Jalal, A., Mahmood, M., Hasan, A.S.: Multi-features descriptors for human activity tracking and recognition in indoor-outdoor environments, pp. 371–376 (2019)
  * Jalal et al. [2014] Jalal, A., Kamal, S., Kim, D.: Depth map-based human activity tracking and recognition using body joints features and self-organized map, pp. 1–6. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2014)
  * Ding et al. [2019] Ding, Q., Yang, F., Li, J., Wu, S., Zhao, B., Wang, Z., Xia, S.-T.: Rt-adi: Fast real-time video representation for multi-view human fall detection, pp. 13–18. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA (2019)
  * Iarlori et al. [2014] Iarlori, S., Ferracuti, F., Giantomassi, A., Longhi, S.: Rgb-d video monitoring system to assess the dementia disease state based on recurrent neural networks with parametric bias action recognition and dafs index evaluation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8548 LNCS, 156–163 (2014)
  * Priya et al. [2019] Priya, G.G.L., Jain, M., Santosh, K.C., Mouli, P.V.S.S.R.C.: Temporal super-pixel based convolutional neural network (ts-cnn) for human activity recognition in unconstrained videos. Communications in Computer and Information Science 1035, 255–264 (2019)
  * Cao et al. [2019] Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A.: Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)
  * Fang et al. [2022] Fang, H.-S., Li, J., Tang, H., Xu, C., Zhu, H., Xiu, Y., Li, Y.-L., Lu, C.: Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * Lugaresi et al. [2019] Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M.G., Lee, J., Chang, W.-T., Hua, W., Georg, M., Grundmann, M.: MediaPipe: A Framework for Building Perception Pipelines. arXiv (2019)
  * Papandreou et al. [2018] Papandreou, G., Zhu, T., Chen, L.-C., Gidaris, S., Tompson, J., Murphy, K.: Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In: Computer Vision – ECCV 2018, pp. 282–299. Springer, Cham (2018)
  * Li et al. [2022] Li, S., Man, C., Shen, A., Guan, Z., Mao, W., Luo, S., Zhang, R., Yu, H.: A fall detection network by 2d/3d spatio-temporal joint models with tensor compression on edge. ACM Trans. Embed. Comput. Syst. 21 (2022)
  * OpenNI [2011] OpenNI: OpenNI library for Processing. https://code.google.com/archive/p/simple-openni/ (2011)
  * Insafutdinov et al. [2016] Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schieke, B.: Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In: European Conference on Computer Vision (ECCV) (2016)
  * Fang et al. [2017] Fang, H.-S., Xie, S., Tai, Y.-W., Lu, C.: Rmpe: Regional multi-person pose estimation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2353–2362 (2017)
  * Xiu et al. [2018] Xiu, Y., Li, J., Wang, H., Fang, Y., Lu, C.: Pose flow: Efficient online pose tracking. In: BMVC (2018)
  * [163] BaiduAI: Baidu AI: Human Body Analysis. https://intl.cloud.baidu.com/product/body.html
  * Howard et al. [2017] Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017)
  * Luvizon et al. [2018] Luvizon, D.C., Picard, D., Tabia, H.: 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning. arXiv (2018)
  * Tran et al. [2015] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks, pp. 4489–4497 (2015)
  * Wang et al. [2017] Wang, Q., Gao, J., Xing, J., Zhang, M., Hu, W.: DCFNet: Discriminant Correlation Filters Network for Visual Tracking. arXiv (2017)
  * Szegedy et al. [2015] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)
  * Chan et al. [2015] Chan, T.-H., Jia, K., Gao, S., Lu, J., Zeng, Z., Ma, Y.: Pcanet: A simple deep learning baseline for image classification? IEEE Transactions on Image Processing 24(12), 5017–5032 (2015)
  * Sun et al. [2018] Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume, pp. 8934–8943 (2018)
  * Feichtenhofer et al. [2018] Feichtenhofer, C., Fan, H., Malik, J., He, K.: SlowFast Networks for Video Recognition. arXiv (2018)
  * Iandola et al. [2016] Iandola, F., Han, S., Moskewicz, M., Ashraf, K., Dally, W., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size (2016)
  * Hui et al. [2018] Hui, T.-W., Tang, X., Loy, C.C.: LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation (2018)
  * Szegedy et al. [2016] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)
  * Girshick et al. [2013] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2013)
  * Fukushima [1980] Fukushima, K.: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36(4), 193–202 (1980)
  * Hochreiter and Schmidhuber [1997] Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9(8), 1735–1780 (1997)
  * Redmon et al. [2016] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection, pp. 779–788 (2016)
  * Simonyan and Zisserman [2014] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv 1409.1556 (2014)
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
  * Elman [1990] Elman, J.L.: Finding structure in time. Cognitive Science 14(2), 179–211 (1990)
  * Kipf and Welling [2016] Kipf, T.N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks. arXiv (2016)
  * Cho et al. [2014] Cho, K., Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches (2014)
  * Carreira and Zisserman [2017] Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. arXiv (2017)
  * Kohonen [1990] Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78(9), 1464–1480 (1990)
  * Tang et al. [2020] Tang, J., Xia, J., Mu, X., Pang, B., Lu, C.: Asynchronous interaction aggregation for action detection. In: Computer Vision – ECCV 2020, pp. 71–87. Springer, Cham (2020)
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., Polosukhin, I.: Attention is all you need (2017)
  * Baradel et al. [2018] Baradel, F., Wolf, C., Mille, J., Taylor, G.: Glimpse clouds: Human activity recognition from unstructured feature points, pp. 469–478 (2018)
  * Gao et al. [2018] Gao, C., Zou, Y., Huang, J.-B.: iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection. arXiv (2018)
  * Hinton and Salakhutdinov [2006] Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
  * Goodfellow et al. [2014] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative Adversarial Networks. arXiv (2014)
  * Nogas et al. [2020] Nogas, J., Khan, S., Mihailidis, A.: Deepfall: Non-invasive fall detection with deep spatio-temporal convolutional autoencoders. Journal of Healthcare Informatics Research 4 (2020)
  * Hinton [2009] Hinton, G.E.: Deep belief networks. Scholarpedia 4, 5947 (2009)
  * Koch [2015] Koch, G.R.: Siamese neural networks for one-shot image recognition. (2015)
  * Hampiholi et al. [2020] Hampiholi, B., Jarvers, C., Mader, W., Neumann, H.: Depthwise separable temporal convolutional network for action segmentation. In: 2020 International Conference on 3D Vision (3DV), pp. 633–641 (2020)
  * Rumelhart et al. [1986] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. nature 323(6088), 533–536 (1986)
  * Krizhevsky et al. [2017] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017) https://doi.org/10.1145/3065386
  * Smaira et al. [2020] Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., Zisserman, A.: A Short Note on the Kinetics-700-2020 Human Action Dataset. arXiv (2020)
  * Kwolek and Kepski [2014] Kwolek, B., Kepski, M.: Human fall detection on embedded platform using depth maps and wireless accelerometer. Computer Methods and Programs in Biomedicine 117, 489–501 (2014)
  * Auvinet et al. [2011] Auvinet, E., Rougier, C., Meunier, J., St-Arnaud, A., Rousseau, J.: Multiple cameras fall data set (2011)
  * Martínez-Villaseñor et al. [2019] Martínez-Villaseñor, L., Ponce, H., Brieva, J., Moya-Albor, E., Núñez-Martínez, J., Peñafort-Asturiano, C.: Up-fall detection dataset: A multimodal approach. Sensors 19 (2019)
  * Shahroudy et al. [2016] Shahroudy, A., Liu, J., Ng, T., Wang, G.: Ntu rgb+d: A large scale dataset for 3d human activity analysis. (2016)
  * Charfi et al. [2013] Charfi, I., Miteran, J., Dubois, J., Atri, M., Tourki, R.: Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and adaboost-based classification. Journal of Electronic Imaging 22, 41106 (2013)
  * Charfi et al. [2012] Charfi, I., Miteran, J., Dubois, J., Atri, M., Tourki, R.: Definition and performance evaluation of a robust svm based fall detection solution, pp. 218–224 (2012)
  * Sung et al. [2012] Sung, J., Ponce, C., Selman, B., Saxena, A.: Unstructured human activity detection from rgbd images, pp. 842–849 (2012)
  * Chen et al. [2015] Chen, C., Jafari, R., Kehtarnavaz, N.: Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor, pp. 168–172 (2015)
  * Wang et al. [2012] Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras, pp. 1290–1297 (2012)
  * Kuehne et al. [2011] Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: Hmdb: A large video database for human motion recognition, pp. 2556–2563 (2011)
  * Schuldt et al. [2004] Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local svm approach, vol. 3, pp. 32–363 (2004)
  * Jang et al. [2020] Jang, J., Kim, D., Park, C., Jang, M., Lee, J., Kim, D.: Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly, pp. 10990–10997 (2020)
  * Liu et al. [2019] Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.-Y., Kot, A.: Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–18 (2019)
  * Vanrumste et al. [2016] Vanrumste, B., Debard, G., Croonenborghs, T., Mertes, G., Baldewijns, G.: Bridging the gap between real-life data and simulated data by providing a highly realistic fall dataset for evaluating camera-based fall detection algorithms. Healthcare Technology Letters 3 (2016)
  * Rahmani et al. [2016] Rahmani, H., Mahmood, A., Huynh, D., Mian, A.: Histogram of oriented principal components for cross-view action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2430–2443 (2016)
  * Gasparrini et al. [2016] Gasparrini, S., Cippitelli, E., Gambi, E., Spinsante, S., Wåhslén, J., Orhan, I., Lindh, T.: Proposal and experimental evaluation of fall detection solution based on wearable and depth data fusion 399, 99–108 (2016)
  * wang et al. [2014] wang, J., Nie, X., Xia, Y., Wu, Y., Zhu, S.: Cross-view action modeling, learning, and recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2014)
  * Koppula et al. [2012] Koppula, H., Gupta, R., Saxena, A.: Learning human activities and object affordances from rgb-d videos. The International Journal of Robotics Research 32 (2012)
  * Soomro et al. [2012] Soomro, K., Zamir, A., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR (2012)
  * Xia et al. [2012] Xia, L., Chen, C.-C., Aggarwal, J.K.: View invariant human action recognition using histograms of 3d joints, pp. 20–27 (2012)
  * Maldonado-Bascón et al. [2021] Maldonado-Bascón, S., Iglesias-Iglesias, C., Martín-Martín, P., Lafuente-Arroyo, S.: Elderly Dataset (2021)
  * Wei et al. [2020] Wei, H., Chopada, P., Kehtarnavaz, N.: C-mhad: Continuous multimodal human action dataset of simultaneous video and inertial sensing. Sensors 20 (2020)
  * Chen [2019] Chen, Z.: Fall detection dataset (2019)
  * Epstein et al. [2019] Epstein, D., Chen, B., Vondrick, C.: Oops! Predicting Unintentional Action in Video. arXiv (2019)
  * Carreira et al. [2018] Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A Short Note about Kinetics-600. arXiv (2018)
  * Fan et al. [2017] Fan, Y., Levine, M., Gongjian, W., Qiu, S.: A deep neural network for real-time detection offalling humans in naturally occurring scenes. Neurocomputing 260 (2017)
  * Liu et al. [2017] Liu, C., Hu, Y., Li, Y., Song, S., Liu, J.: PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding. arXiv (2017)
  * Kay et al. [2017] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M., Zisserman, A.: The Kinetics Human Action Video Dataset. arXiv (2017)
  * Gaglio et al. [2015] Gaglio, S., Re, G.L., Morana, M.: Human activity recognition process using 3-d posture data. IEEE Transactions on Human-Machine Systems 45, 586–597 (2015)
  * Gupta and Malik [2015] Gupta, S., Malik, J.: Visual Semantic Role Labeling. arXiv (2015)
  * Chua et al. [2013] Chua, J.-L., Chang, Y., Lim, W.: A simple vision-based fall detection technique for indoor video surveillance. Signal, Image and Video Processing 9 (2013)
  * Amiri et al. [2013] Amiri, S.M., Pourazad, M.T., Nasiopoulos, P., Leung, V.C.M.: Non-intrusive human activity monitoring in a smart home environment, pp. 606–610 (2013)
  * Seidenari et al. [2013] Seidenari, L., Varano, V., Berretti, S., Bimbo, A.D., Pala, P.: Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses, pp. 479–485 (2013)
  * Cheng et al. [2012] Cheng, Z., Qin, L., Ye, Y., Huang, Q., Tian, Q.: Human daily action analysis with multi-view and color-depth data, pp. 52–61. Springer, Berlin, Heidelberg (2012)
  * Kong et al. [2012] Kong, Y., Jia, Y., Fu, Y.: Learning human interaction by interactive phrases, pp. 300–313 (2012)
  * Yao et al. [2011] Yao, B., Jiang, X., Khosla, A., Lin, A., Guibas, L., Li, F.-F.: Human action recognition by learning bases of action attributes and parts, pp. 1331–1338 (2011)