# CrowdMAC: Masked Crowd Density Completion
for Robust Crowd Density Forecasting

Ryo Fujii1 Ryo Hachiuma2 Hideo Saito1
1Keio University 2NVIDIA
{ryo.fujii0112<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

A crowd density forecasting task aims to predict how the crowd density map
will change in the future from observed past crowd density maps. However, the
past crowd density maps are often incomplete due to the miss-detection of
pedestrians, and it is crucial to develop a robust crowd density forecasting
model against the miss-detection. This paper presents a MAsked crowd density
Completion framework for crowd density forecasting (CrowdMAC), which is
simultaneously trained to forecast future crowd density maps from partially
masked past crowd density maps (_i.e_., forecasting maps from past maps with
miss-detection) while reconstructing the masked observation maps (_i.e_.,
imputing past maps with miss-detection). Additionally, we propose Temporal-
Density-aware Masking (TDM), which non-uniformly masks tokens in the observed
crowd density map, considering the sparsity of the crowd density maps and the
informativeness of the subsequent frames for the forecasting task. Moreover,
we introduce multi-task masking to enhance training efficiency. In the
experiments, CrowdMAC achieves state-of-the-art performance on seven large-
scale datasets, including SDD, ETH-UCY, inD, JRDB, VSCrowd, FDST, and croHD.
We also demonstrate the robustness of the proposed method against both
synthetic and realistic miss-detections.

##  1 Introduction

Accurately forecasting a crowd’s movement in a video is crucial for various
applications, such as advancing navigation systems to select an optimal route
for agents (robots) and effectively minimizing the risk of collisions [2, 25].
A crowd’s movement can be naively forecasted by applying the well-studied
trajectory prediction framework [1, 14, 42, 27]. Despite significant attention
and numerous proposed works on the trajectory prediction task, most existing
works [1, 14, 42, 27] rely on the critical assumption that each trajectory is
successfully obtained by two preprocessing (_i.e_., upstream perception)
modules, pedestrian detection [38, 37, 24, 5] and tracking [3, 51, 6, 55].
However, this assumption is not feasible for real-world scenarios since the
complete trajectory for each pedestrian is not always available due to the
miss-detection of pedestrians or incorrect tracking. Using estimated
trajectories as the input for existing trajectory predictors could lead to
significant accumulated errors and pose serious threats to safety [53].

Crowd density forecasting task [34], which directly forecasts future crowd
density maps from the past crowd density maps that are estimated from either
pedestrian detection [38, 37, 24, 5] or crowd density estimation [56, 22, 50,
12] modules, offers the potential as an alternative approach for forecasting
the movements of pedestrians in real-world robotic applications, even though
the movement of each pedestrian is not forecasted. However, the accuracy of
the crowd density forecasting task still highly relies on the obtained past
crowd density maps. Thus, it is crucial to develop a robust crowd density
forecasting model against the error accumulated by the upstream crowd density
estimation or pedestrian detection modules. (_i.e_., miss-detection).

The masked autoencoders (MAE) [16] has garnered significant attention due to
its recent achievements in image-based self-supervised learning. This approach
involves masking a portion of patches in the input data and reconstructing the
missing patches using an autoencoder structure. Inspired by the remarkable
capabilities of MAE in reconstructing masked patches (interpolation) of
spatio-temporal data, we explore its potential for forecasting future patches
(extrapolation) in the crowd density forecasting task. This paper presents
CrowdMAC, which forecasts future crowd maps from observed crowd maps. Unlike
MAE, which masks a subset of patches during training, we mask frames (entire
patches) in the future density maps, and the model is trained to reconstruct
the masked future frames.

As previously mentioned, to robustly forecast the crowd density map with the
miss-detection of the pedestrians, the patches in the input past observed
crowd density maps are masked. During training, the model jointly reconstructs
the masked patches in past observed maps and predicts the patches in future
maps. Through the simultaneous training of reconstruction and future
prediction, the model acquires meaningful representations that enhance its
ability to predict future maps while improving its robustness against miss-
detections.

In addition, drawing inspiration from representation learning approaches that
utilize MAE with non-uniform masking based on token informativeness across
various domain data inputs [20, 44, 33, 30], we propose temporal-density-aware
masking (TDM) strategy that non-uniformly masks observed maps during the
training to facilitate the training and obtain better forecasting performance.
The TDM comprises two components: temporal-aware ratio and density-aware
masking. First, since the last observed frame is the most informative in the
forecasting task compared to the first observed frame, the first module
increases the masking ratio along the time index of the maps. Second, given
the observation that the input crowd density maps are sparse (_e.g_.,
pedestrians may not exist in all pixels in the map) and that reconstructing
these sparse pixels doesn’t aid model training, we propose to use the
accumulated density as empirical semantic richness prior to effectively
guiding the crowd density map masking process, enabling enhanced attention to
semantically rich spatial regions.

Moreover, we adopt multi-task learning using different masks inspired by
MegVit [54], which addresses various video synthesis tasks with a single model
using different masking schemes for efficient training. Specifically, we train
the model with three tasks: future prediction, past prediction, and
interpolation. Training with these tasks enables the model to establish a
robust bidirectional relationship between past and future motion and serves as
a form of regularization, enhancing overall performance.

Our contributions are summarized as follows:

  * •

We present a robust crowd density forecasting approach, CrowdMAC, that
simultaneously reconstructs the masked tokens in the past crowd density maps
and predicts the future crowd density maps during the training.

  * •

We propose temporal-density-aware masking (TDM) strategy that non-uniformly
masks observed maps during the training to facilitate the training and obtain
better forecasting performance.

  * •

We introduce a straightforward yet highly effective multi-task masking scheme
that facilitates the learning of bidirectional motion connections and serves
as a form of regularization to enhance performance.

  * •

We extensively benchmark the performance of the proposed model well-
established trajectory prediction datasets, ETH-UCY [36, 19], SDD [39], inD
[4], and JRDB [31] and crowd analysis datasets, FDST [9], croHD [45], and
VSCrowd [21]. Despite the simplicity of the proposed framework, it outperforms
previous crowd forecasting methods by large margins under two evaluation
settings: ground truth inputs and pedestrian detection inputs. Moreover, our
model surpasses conventional trajectory prediction approaches in both
settings.

  * •

We investigate the robustness of crowd density forecasting approaches against
pedestrian miss-detection, a critical consideration in practical scenarios.
The study validates that the proposed method outperforms previous approaches.

Figure 1: Overview of the CrowdMAC during the training phase. Cube embedding
transforms the crowd density maps into a sequence of tokens. At each training
step, one of the task masks is sampled, and a subset of tokens is masked. The
remaining tokens, along with the space-time position embedding, are fed into a
Transformer encoder and decoder to reconstruct the masked maps. Note that the
multi-task masking is applied only during the training phase, while the only
future prediction task mask is used during the inference phase.

##  2 Related Work

###  2.1 Trajectory Prediction

Human trajectory forecasting aims to infer plausible future positions from
observed trajectories. Deep learning-based trajectory prediction approaches
[1, 14, 35, 52, 18, 27, 14, 41, 18, 17, 28, 42, 11, 13, 29] have become
dominant due to their impressive representational ability. Some studies focus
on better modeling interactions with other agents, such as pedestrians [1, 14,
35, 52] and the environment [18, 27] and some works [14, 41, 18, 17, 28, 42,
13, 29] aim to model the multi-modality of future motions.

However, despite the increasing accuracy of trajectory prediction methods,
they work on trajectory prediction in an idealized setting where ground truth
past trajectories are used as inputs for both training and evaluation without
considering upstream perception errors (_i.e_., pedestrian detection and
tracking error). However, this idealized setting is not feasible in the real
world since the trajectories that contain errors estimated from the upstream
modules are inputted to the trajectory forecasting approaches.

###  2.2 Crowd Density Forecasting

Crowd density forecasting aims to forecast crowd density maps of future
frames. PDFN-ST [34], the patch-based crowd density forecasting model, is
pioneering work in the field of crowd density forecasting. It models the
diverse and complex crowd density dynamics of the scene based on spatially or
spatiotemporally overlapping patches. Crowd density forecasting directly
forecasts future crowd density maps from past crowd density maps. The past
crowd density maps are estimated either from upstream crowd density estimation
or the object detection modules. Thus, similar to the trajectory prediction
method, the crowd density forecasting method highly depends on the accuracy of
the upstream module. Therefore, developing a robust crowd density forecasting
model against errors accumulated by the upstream modules, such as object
detection, is of critical importance and serves as the motivation for this
work. CrowdMAC aims to enhance robustness against the miss-detection of
pedestrians.

###  2.3 Masked Autoencoders (MAE)

Masked autoencoders [16, 10, 49] are a type of denoising autoencoder [48] that
obtain meaningful representations by learning to reconstruct images corrupted
by masking. In particular, the masked autoencoder (MAE) [16] approach
accelerates pre-training to learn representations from images by using an
asymmetric architecture. VideoMAE [46] directly extends MAE and efficiently
reconstructs the pixels of masked video patches with a high masking ratio
through tube masking. Our approach leverages the efficiency of the masked
autoencoder approach and extends it to the crowd density forecasting task. The
original masked autoencoder approaches aim to obtain powerful pretrained
models to improve the accuracy of video-related downstream tasks. On the other
hand, we focus on the interpolation capabilities of the masked autoencoder
itself and apply it to the forecasting task, which is robust to missing inputs
(_i.e_., miss-detection).

Moreover, rather than randomly masking tokens in input images or videos, as
commonly done in MAE or VideoMAE, recent custom masking approaches have been
proposed for various inputs such as images [20], videos [44], 3D LiDAR point
clouds [33], or sequential skeletons [30]. These approaches non-uniformly
sample tokens based on the informativeness of each token. SemMAE [20] utilizes
a masking approach based on semantically segmented parts of objects in images
to obtain a powerful image representation. The informativeness-based masking
strategy enhances the performance of downstream tasks, such as image
recognition or video recognition. Inspired by these approaches, we propose a
temporal-density-aware masking strategy to mask informative tokens in the
input crowd density maps during training for the crowd density forecasting
task.

##  3 CrowdMAC

###  3.1 Problem Definition

The goal of crowd density forecasting is to predict future crowd density maps
over $T_{pred}$ timesteps (_i.e_., future frames),
$C_{pred}=[c_{T_{obs}+1},\ldots,c_{T_{obs}+T_{pred}}]$ from the observed past
crowd density maps over $T_{obs}$ timesteps (_i.e_., past frames),
$C_{obs}=[c_{1},\ldots,c_{T_{obs}}]$. $c_{t}\in[0,1]^{W\times H}$ represents
the crowd density map at time step $t$ with a size of $(W,H)$. The observed
crowd density maps $C_{obs}$ are extracted from each input RGB video frame
$I_{t}$ using the off-the-shelf pedestrian detection [24, 5, 15] or crowd
density estimation [50, 12] approaches.

(a) SDD |  (b) inD |  (c) ETH-UCY
---|---|---
Figure 2: Examples of input crowd density maps in the various trajectory
prediction datasets. The density maps are visualized as a heatmap and
overlayed onto the RGB image for visualization purposes.

###  3.2 Overview

Fig. 1 illustrates the overview of the proposed CrowdMAC. CrowdMAC takes crowd
density maps $C\in\mathbb{R}^{T\times H\times W}$ as input, comprising of
observed crowd density maps $C_{obs}\in\mathbb{R}^{T_{obs}\times H\times W}$
and future crowd density maps $C_{pred}\in\mathbb{R}^{T_{pred}\times H\times
W}$, where $T=T_{obs}+T_{pred}$. It employs the space-time cube embedding [46]
to transform the input maps into a set of token embeddings
$X\in\mathbb{R}^{F\times N_{s}N_{r}}$, where $F$ is the channel dimension of
the tokens, $N_{s}=HW/H_{c}W_{c}$ and $N_{r}=T/T_{c}$ are the numbers of
tokens along the spatial and temporal directions, respectively. $T_{c}$,
$H_{c}$, and $W_{c}$ are the size of each token along temporal, hseven, and
width dimensions, respectively. Then, a subset of tokens is non-uniformly
masked with multi-task masking scheme. The remaining tokens are fed into a
Transformer encoder and decoder [47], along with the space-time position
embedding, to reconstruct the masked maps. The loss function is the mean
squared error (MSE) loss between the masked tokens and the reconstructed ones.

###  3.3 Multi-Task Masking

Instead of training with a single future prediction task, we train the model
on three tasks: future prediction, past prediction, and interpolation. At each
training step, we sample one of the tasks. This multi-task training enables
the model to establish a robust bidirectional relationship between past and
future motion. Additionally, multi-task learning serves as a form of
regularization, mitigating the risk of overfitting. This is achieved by
training the model to perform well across multiple objectives, resulting in
more robust and generalized learning.

For future prediction and past prediction tasks, the proposed temporal-
density-aware masking (TDM) and frame masking are utilized to select tokens to
be masked. For the interpolation task, the TDM is utilized to select tokens to
be masked.

Figure 3: The conceptual figure of the proposed Temporal-density-aware masking
(TDM) module. The number of masked tokens is monotonically increased along
with the time step (Temporal-aware Masking Ratio). For each time index $t$ of
the tokens, the tokens are non-uniformly selected based on the probability of
the accumulated density values of tokens (Density-aware Masking). After the
masking, the tokens colored in gray indicate that the tokens are masked.

###  3.4 Temporal-Density-aware Masking (TDM)

Instead of uniformly masking tokens in frames via random sampling with a fixed
masking ratio, like MAE or VideoMAE, during the training, we propose Temporal-
Density-aware Masking (TDM) that randomly masks informative tokens. The
overview of TDM is visualized in Fig. 3. TDM consists of two modules:
temporal-aware masking (TM) ratio and density-aware masking (DM). TM
calculates the number of masked tokens for each time step from the tokens’
time index, and the sampling probability of each token in each time step is
calculated from DM based on the accumulated density value of each token. The
details of each module are explained in the following sections.

####  3.4.1 Temporal-aware Masking (TM) Ratio

For the future prediction task and interpolation, we assume the latter frame
is more informative than the first frame. Therefore, instead of sampling
tokens randomly with a fixed masking ratio $\gamma$ that computes the number
of the tokens to be masked, the proposed TM ratio $\gamma(t)$ is a
monotonically increasing function of the tokens’ time index of the observation
tokens $t\in\\{1,\ldots,T_{max}\\}$, where $T_{max}=T_{obs}/T_{c}$ for future
prediction task and $T_{max}=T/T_{c}$ for interpolation task. By designing
$\gamma(t)$ to increase along with the time index, the model is trained to
reconstruct more informative tokens in the subsequent frames. Concretely, we
design the $\gamma(t)$ as follows:

$\gamma(t)=1-e^{(-\lambda t/T_{max})},$ (1)

where $\lambda$ is sampled from the uniform distribution $U[0,\lambda_{max}]$
every epoch with hyper-parameter $\lambda_{max}$.

For the past prediction task, we assume the earlier frame is more informative
than the last frame. The TM ratio $\gamma(t)$ is a monotonically decreasing
function of the tokens’ time index of the future tokens
$t\in\\{1,\ldots,T_{max}\\}$, where $T_{max}=T_{pred}/T_{c}$. We design the
$\gamma(t)$ as follows:

$\gamma(t)=1-e^{(-\lambda(T_{max}-t)/T_{max})}.$ (2)

####  3.4.2 Density-aware Masking (DM)

In contrast to RGB images or videos, crowd density maps are often sparse,
meaning that pedestrians do not exist in all the pixels in the maps, and most
pixel values are empirically 0. For example, Fig. 2 visualizes the estimated
crowd density maps from RGB images in the various datasets (_e.g_. SDD [39],
inD [4], and ETH-UCY [36, 19]). We observe that most of the tokens are
sparse111Statistically, $9.6\%$, $5.6\%$, $8.8\%$ pixels in the map have
density values higher than the particular threshold (0.1) on SDD, inD, and
ETH-UCY datasets, respectively. (the accumulated pixel density values of
patches mostly, being 0), and training to reconstruct these tokens through MAE
is not effective [44, 30] (the patch lacks informativeness during the
training). Therefore, we propose to non-uniformly sample the tokens based on
the accumulated density value of each token.

Specifically, we first compute the accumulated density value $d_{i}$ for each
token ${\bm{x}}_{i}\in X$ by taking a sum of the density values in the pixels
that belong to the token as follows:

$d_{i}=\sum_{j\in\Omega_{i}}C_{j},$ (3)

where $\Omega_{i}$ denotes the set of pixels that belong in the $i$-th token.
Then, we calculate the masking probability vector
${\bm{\pi}}^{t}\in\mathbb{R}^{N_{s}}$ for each token’s time index using the
softmax function as follows:

${\bm{\pi}}^{t}={\rm Softmax}({\bm{d}}^{t}/\tau),$ (4)

where ${\bm{d}}^{t}\in\mathbb{R}^{N_{s}}$ is a vector of accumulated densities
for each time index $t$, and $\tau$ is a hyper-parameter that controls the
sharpness of the softmax function. Finally, the indices of the masked tokens
in the observation frames are obtained by sampling a Multinomial distribution
according to the probabilities ${\bm{\pi}}^{t}$ for
$\lfloor\gamma(t)N_{s}\rfloor$ trials without replacement for every time index
$t$.

##  4 Experiments

###  4.1 Datasets

We use a total of seven datasets to investigate the performance of CrowdMAC.
Specifically, we conduct experiments on four datasets for trajectory
prediction: ETH-UCY [36, 19], Stanford Drone Dataset (SDD) [39], Intersection
Drone Dataset (InD)[4], and JackRabbot Dataset (JRDB) [31]. Furthermore, we
evaluate our approach on three datasets for crowd analysis tasks (_e.g_.,
crowd detection, localization, and counting), including Fudan-ShanghaiTech
(FDST) [9], Crowd of Heads Dataset (CroHD) [45], and surveillance-view Video
Crowd dataset (VSCrowd) [21]. As the croHD dataset only provided detection
results from the detector for the test set, we use the detection results for
training and evaluation.

###  4.2 Implementation Details

We employ the the ViT-Small [8] backbone. Following prior work [34], we use
the input size of $80\times 80$ for $20$ frames which consists of $8$ past
frames ($T_{obs}$) and $12$ future frames ($T_{pred}$) in Sec. 3.1. Note that
the $8$ past frames are used as input during the inference. We set the space-
time cube size for the embedding along the spatial dimensions as $W_{c}=8$,
$H_{c}=8$, and along the temporal dimension as $T_{c}=4$. We employ the mean
squared error (MSE) loss between the masked pixels and the reconstructed
pixels for training. We train our model with AdamW optimizer [26] with a base
learning rate $5e^{-4}$ and wseven decay $1e^{-5}$ for $1200$ epochs. We train
the model from scratch for SDD and use the pretrained model from SDD for
initialization when training on other datasets. The hyperparameters were
determined through a standard coarse-to-fine grid search or step-by-step
tuning. We warm up training for $60$ epochs, starting from learning rate
$1e^{-6}$, and decay it to $0$ throughout training using cosine decay. We set
the batch size to $256$ and train the model using two NVIDIA TITAN RTX GPUs.
We employ spatial data augmentation techniques, such as rotation, scaling,
horizontal flip, and vertical flip of the crowd density maps. We empirically
set $\lambda_{max}$ (Eq. 2) of temporal-aware masking as $9$. Also, we set
$\tau$ in density-aware masking (Eq. 4) to $500$.

###  4.3 Evaluation Metrics

Followed by prior work [34], we use Jensen-Shannon (JS) divergence to measure
the performance of the forecasting:

$\mathcal{D}_{JS}(g_{t}||c_{t})=\frac{1}{2}(\mathcal{D}_{KL}(\bar{g}_{t}||\bar{c}_{t})+\mathcal{D}_{KL}(\bar{c}_{t}||\bar{g_{t}})),$
(5)

where $\bar{g}_{t}=g_{t}/\sum_{i,j}g_{t}(i,j)$,
$\bar{c}_{t}=c_{t}/\sum_{i,j}c_{t}(i,j)$ are the predicted and ground truth
normalized density maps, $i$, $j$ are the indices of pixel position, and
$\mathcal{D}_{KL}$ is Kullback-Leibler (KL) divergence:

$\mathcal{D}_{KL}(g_{t}||c_{t})=\frac{1}{WH}\sum_{i,j}{\bar{g}_{t}(i,j)\log(\frac{\bar{g_{t}}(i,j)}{\bar{c}_{t}(i,j)}}).$
(6)

We report the Average JS divergence ($AD_{JS}$) and the Final JS divergence
($FD_{JS}$). $AD_{JS}$ is the divergence between the predicted and the ground
truth map averaged over all the future time steps, while $FD_{JS}$ is the
divergence between the predicted and ground truth map at the final time step.

###  4.4 Comparison Models

Since only one previous work has tackled the crowd density forecasting task,
in addition to the crowd density forecasting method [34], we apply standard
and state-of-the-art trajectory forecasting approaches [27, 40] to address the
crowd density forecasting task following the previous method [34]. We employ
the following methods for comparison:

PDFN-ST [34]: PDFN-ST is a pioneering work that tackles the crowd density
forecasting task by using 3D CNNs to learn local crowd density dynamics in 3D
receptive fields, regarded as spatiotemporal patches.

Y-Net: Y-Net [27] is a heatmap-based model that predicts future human
trajectories by estimating distributions over long-term goals and intermediate
waypoints.

Social-Transmotion: Social-Transmotion [40] is a Transformer-based model for
human trajectory prediction, leveraging diverse visual cues. The model is
designed to predict human behavior by capturing spatiotemporal interactions
between agents.

To evaluate trajectory prediction models using the evaluation metric of crowd
density forecasting, we generate crowd density maps from the predicted
trajectories by trajectory prediction models. Similar to generating ground
truth crowd density maps, we generate crowd density maps by applying a 2D
Gaussian filter to the predicted position for each pedestrian for each frame.

###  4.5 Evaluation Protocols

We evaluate the efficacy of our method under a common setting in the state-of-
the-art trajectory prediction model, which involves observing trajectories
over 8-time steps (equivalent to 3.2 seconds) and subsequently predicting
future trajectories spanning the next 12 time steps (equivalent to 4.8
seconds), employing two distinct protocols.

Table 1: Comparison of the crowd density forecasting and trajectory prediction
approaches using ground truth pedestrian positions (see Sec. 4.5). The lower
metrics ($AD_{JS},FD_{JS}$) are better. As FDST and croHD do not provide
annotations of person identity for the test set, we do not report the results
of the trajectory prediction methods for these datasets.

Dataset | Trajectory Prediction | Crowd Density Forecasting
---|---|---
Y-Net [27] | Social-Trans. [40] | PDFN-ST [34] | Ours
$AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$
SDD [39] | 0.098 | 0.144 | 0.083 | 0.106 | 0.072 | 0.158 | 0.068 | 0.129
ETH-UCY [36, 19] | 0.346 | 0.464 | 0.318 | 0.426 | 0.444 | 0.670 | 0.188 | 0.250
inD [4] | 0.079 | 0.099 | 0.068 | 0.080 | 0.068 | 0.080 | 0.064 | 0.116
JRDB [29] | 0.118 | 0.164 | 0.090 | 0.113 | 0.080 | 0.121 | 0.075 | 0.104
VSCrowd [21] | 0.428 | 0.494 | 0.340 | 0.382 | 0.100 | 0.118 | 0.094 | 0.110
FDST [9] | - | - | - | - | 0.050 | 0.094 | 0.042 | 0.075
croHD [45] | - | - | - | - | 0.039 | 0.049 | 0.038 | 0.046

Ground Truth Input. To evaluate the performance of crowd density forecasting
models independent of the accuracy of the upstream crowd density estimation
module, we conduct experiments using ground truth as input. We train the model
with ground truth maps and evaluate the accuracy with them. Since the ground
truth 2D positions of the pedestrians are annotated in the above trajectory
prediction datasets, we generate ground truth crowd density maps from the
pedestrians’ positions. We apply a 2D Gaussian filter (with a standard
deviation of $3$ px) following the previous approach [34].

Upstream Perception Modules Input. To validate the robustness of our method
against errors (_e.g_. miss-detection of the pedestrians) in upstream
perception modules, we also evaluate the forecasting accuracy in a realistic
setting that uses the estimated results from the upstream perception module as
inputs in the evaluation phase. We train the model on ground truth inputs and
evaluate the accuracy of the crowd forecasting and trajectory prediction
approaches using upstream perception module results. We use the detection and
crowd density estimation for crowd density forecasting approaches and the
detection&tracking for trajectory prediction approaches as upstream modules.
Following PDFN-ST [34], we employ UCY in this protocol for upstream detection
results input setting. As the students001 and uni examples videos in the UNIV
scene are not publicly available, we do not use them in this setting. For the
upstream crowd density estimation results input setting, we choose the dataset
with annotations of pedestrian heads. We use the Faster R-CNN model [38] with
FPN [23] as the detector, along with Deep-SORT [51] as the tracker, pretrained
on the MOT17 [32] dataset. For the detector, we set the confidence threshold
to $0.2$. We utilize the implementation of MMTracking [7]. We use the STEERER
[15], pretrained with the JHU-CROWD++ [43], as a crowd density estimator.

###  4.6 Forecasting Accuracy Comparison

Ground Truth Input. We compare our model with crowd density forecasting and
trajectory prediction models using the ground truth input evaluation protocol.
Tab. 1 summarizes the performance of the methods on trajectory prediction and
crowd analysis datasets. As shown in Tab. 1, our approach outperforms previous
state-of-the-art method PDFN-ST on all trajectory prediction datasets: SDD,
ETH-UCY, inD, and JRDB datasets. For instance, on SDD, we reduce $AD_{JS}$ by
$5.6\%$ (from $0.072$ to $0.068$), and reduce $FD_{JS}$ by $18\%$ (from
$0.158$ to $0.129$) compared to PDFN-ST. Additionally, our CrowdMAC
significantly outperforms the trajectory comparison models. CrowdMAC achieves
a substantial gain on SDD, ETH-UCY, inD, and JRDB on $AD_{JS}$ metric compared
to the Social-Transmotion. Unlike existing trajectory prediction methods, our
approach does not assume complete input trajectories. Therefore, we also
evaluate cases where the observed trajectories are incomplete due to
occlusions or individuals entering the frame partway through the observation
timesteps. Such cases can adversely affect the performance of the trajectory
prediction results. This degrades the performance of trajectory prediction
results.

As shown in Tab. 1, CrowdMAC also outperforms existing both crowd density
forecasting method and trajectory prediction methods on all crowd analysis
datasets: FDST, croHD, and VSCrowd datasets on $AD_{JS}$ and $FD_{JS}$ metric.
The performance of the trajectory prediction model on crowd analysis datasets
is significantly worse than on trajectory prediction datasets. This
discrepancy arises because, unlike trajectory prediction datasets captured
from a high and distant bird’s-eye view, crowd analysis datasets often include
scenes recorded from surveillance cameras that are relatively close to people.
Consequently, many individuals appear partway through the observation frames.
The trajectory prediction model, which assumes that the all pedestrians are
present in all observation frames, fails to handle such cases effectively.

Table 2: Comparison of crowd density forecasting and trajectory prediction
approaches using the upstream detector and tracker perception modules results
input (see Sec. 4.5).

Dataset | Trajectory Prediction | Crowd Density Forecasting
---|---|---
Y-Net [27] | Social-Trans. [40] | PDFN-ST [34] | Ours
$AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$
UNIV [19] | 0.216 | 0.300 | 0.385 | 0.431 | 0.473 | 0.533 | 0.281 | 0.291
ZARA1 [19] | 0.778 | 0.809 | 0.475 | 0.568 | 0.737 | 1.047 | 0.385 | 0.389
ZARA2 [19] | 0.882 | 0.889 | 0.463 | 0.645 | 0.554 | 0.621 | 0.358 | 0.417
VSCrowd [21] | 1.378 | 1.974 | 0.726 | 0.807 | 0.250 | 0.389 | 0.115 | 0.125

Table 3: Comparison of crowd density forecasting and trajectory prediction
approaches using the upstream crowd density estimation perception module
results input (see Sec. 4.5).

Dataset | PDFN-ST [34] | Ours
---|---|---
$AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$
ZARA2 [19] | 0.517 | 0.576 | 0.478 | 0.479
VSCrowd [21] | 0.240 | 0.259 | 0.233 | 0.258
FDST [9] | 0.272 | 0.333 | 0.198 | 0.200

Upstream Perception Modules Input. We validate the robustness of our method
against errors in upstream perception modules using detection results as
input, as shown in Tab. 2, and crowd density estimation results as input, as
shown in Tab. 3. Our CrowdMAC consistently outperforms both the crowd density
forecasting models and the trajectory prediction models in both settings. For
instance, CrowdMAC reduces $AD_{JS}$ by $54\%$ on VSCrowd compared to PDFN-ST
with the detection input setting. CrowdMAC reduces $AD_{JS}$ by $7.5\%$ on
ZARA2, by $2.9\%$ on VSCrowd, and by $17\%$ on FDST compared to PDFN-ST with
the detection input setting. Our CrowdMAC also significantly outperforms
trajectory prediction models. All these results confirm that the proposed
model significantly improves robustness against real data, which is reasonable
as the standard prediction system does not explicitly account for upstream
perception errors.

###  4.7 Robustness against Miss-detection

We explore the robustness of the crowd density forecasting models against the
miss-detection of pedestrians with the synthetically-generated and realistic
miss-detections.

(a) SDD with synthetic miss-detections. |  (b) FDST with realistic miss-detections.
---|---
(c) VSCrowd with realistic miss-detections.
Figure 4: We compare the robustness of the models with synthetic miss-
detections on the SDD and with realistic miss-detections on the FDST and
VSCrowd.

Synthetic Miss-Detection. We evaluate the robustness against synthetically
generated miss-detections. We train the models with ground-truth input, and
synthetic miss-detections are generated by randomly sampling pedestrians in
the frames that will not be used as input during the evaluation. Fig. 4 (a)
summarizes the $AD_{JS}$ metric of the previous approach (PDFN-ST) and
CrowdMAC with various miss-detection ratios. Since the performance drop of our
model is the lower than PDFN-ST among multiple miss-detection ratios, our
model is robust against miss-detections compared to the PDFN-ST.

Realistic Miss-Detection. We further investigate the robustness against miss-
detection using data preprocessed by the pedestrian detection module [38] on
the FDST and VScrowd. We varied the confidence threshold value of the object
detector from $0.2$ (as described in Sec. 4.5) to higher values ($>0.9$) to
generate input data with miss-detections. Increasing the detection confidence
thresholds resulted in more miss-detections of pedestrians (_i.e_., false
negatives in detection). In Fig. 4 (b,c), the $AD_{JS}$ metric of each method
(PDFN-ST and Ours) is presented for various detection confidences. Similar to
Fig. 4 (a), our proposed method exhibits the lower performance drop compared
to the PDFN-ST, highlighting its robustness against realistic miss-detections.

Table 4: Ablation study of temporal-density-aware masking (TDM) on the SDD and
FDST. The TDM consists of a temporal-aware masking (TM) ratio and density-
aware masking (DM).

TM Ratio | DM | SDD | FDST
---|---|---|---
$AD_{JS}$ | $FD_{JS}$ | $AD_{JS}$ | $FD_{JS}$
- | - | 0.072 | 0.132 | 0.046 | 0.082
- | ✓ | 0.070 | 0.130 | 0.043 | 0.075
✓ | - | 0.071 | 0.130 | 0.045 | 0.079
✓ | ✓ | 0.068 | 0.129 | 0.042 | 0.075

###  4.8 Ablation Studies

We perform two ablation studies on the proposed TDM module to assess its
effectiveness. In this section, we use the ground truth input evaluation
protocol as mentioned in Sec. 4.6.

Effect of the Temporal-Density-aware Masking. Tab. 4 shows the effect of each
component (TM, DM), in TDM. It can be seen that both of the components
improved the density forecasting performance.

Effect of multi-task masking. Tab. 5 shows the effect of mul-task masking. The
model trained with a single future prediction task performs considerably worse
compared to the model trained with the multi-task approach. In particular, the
history prediction task significantly contributes to the improved performance.

Choice of Temporal-aware Masking (TM) Ratio Function. We explore the design of
the TM ratio function in Eq. 2. We compare the results with the approaches
that employ different masking ratio functions in Tab. 6. The first row shows
the results of the approach that uses the constant masking ratio during the
training. From the second row to the fifth row show the approaches that employ
various masking ratio functions instead of the function in Eq. 2, such as
concave (square root), linear, convex (square, cubic), respectively. We
observe that our temporal-aware masking ratio function outperforms the
approach with the fixed masking ratio as well as the approaches that employ
the other mask ratio functions.

Figure 5: Qualitative results on the SDD and FDST datasets. The density maps
are visualized as a heatmap and overlayed onto the RGB image for visualization
purposes. Table 5: Ablation study of multi-task masking on the SDD.

Future Pred. | Interpolation | Past Pred. | $AD_{JS}$ | $FD_{JS}$
---|---|---|---|---
✓ | - | - | 0.080 | 0.146
✓ | ✓ | - | 0.070 | 0.129
✓ | - | ✓ | 0.075 | 0.143
✓ | ✓ | ✓ | 0.068 | 0.129

Table 6: Ablation study of temporal-aware masking (TM) ratio function on the
SDD.

TM Ratio Function | $AD_{JS}$ | $FD_{JS}$
---|---|---
Constant | 0.077 | 0.136
Square Root | 0.076 | 0.138
Linear | 0.074 | 0.137
Square | 0.071 | 0.129
Cubic | 0.075 | 0.134
Ours | 0.068 | 0.129

###  4.9 Qualitative Evaluation

Fig. 5 visualizes the qualitative results of the proposed method as well as
the ground-truth density maps on the SDD and FDST datasets. The proposed
method accurately forecasts the crowd density maps.

##  5 Conclusion

In summary, we propose CrowdMAC, a new learning framework for crowd density
forecasting, which operates in a masked completion fashion. CrowdMAC is
simultaneously trained to forecast future crowd density maps from masked past
observation maps and reconstruct the masked observation maps to improve the
robustness of the crowd density forecasting task against the miss-detection of
pedestrians. We further propose Temporal-Density-aware Masking (TDM) that non-
uniformly masks the patches in the observed crowd density maps, facilitating
training and obtaining better forecasting performance. Moreover, we introduce
multi-task masking to enhance the efficiency in training. Empirical results
show that our method outperforms the state-of-the-art crowd density
forecasting methods, as well as trajectory prediction approaches, and achieves
robustness against synthetically generated or realistically occurring miss-
detections.

## References

  * [1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese.  Social LSTM: Human Trajectory Prediction in Crowded Spaces.  In CVPR, 2016.
  * [2] Anoop Aroor, Susan L. Epstein, and Raj Korpan.  Online Learning for Crowd-Sensitive Path Planning.  In AAMAS, 2018.
  * [3] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In ICIP, 2016.
  * [4] Julian Bock, Robert Krajewski, Tobias Moers, Steffen Runde, Lennart Vater, and Lutz Eckstein.  The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections.  In IV, 2020.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-End Object Detection with Transformers.  In ECCV, 2020.
  * [6] Tianlong Chen, Shaojin Ding, Jingyi Xie, Ye Yuan, Wuyang Chen, Yang Yang, Zhou Ren, and Zhangyang Wang.  ABD-Net: Attentive but Diverse Person Re-Identification.  In ICCV, 2019.
  * [7] MMTracking Contributors.  MMTracking: OpenMMLab video perception toolbox and benchmark.  https://github.com/open-mmlab/mmtracking, 2020.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In ICLR, 2021.
  * [9] Yanyan Fang, Bi-Sheng Zhan, Wandi Cai, Shenghua Gao, and Bo Hu.  Locality-Constrained Spatial Transformer Network for Video Crowd Counting.  ICME, 2019.
  * [10] Christoph Feichtenhofer, haoqi fan, Yanghao Li, and Kaiming He.  Masked Autoencoders As Spatiotemporal Learners.  In NeurIPS, 2022.
  * [11] Ryo Fujii, Jayakorn Vongkulbhisal, Ryo Hachiuma, and Hideo Saito.  A Two-Block RNN-Based Trajectory Prediction From Incomplete Trajectory.  IEEE Access, 2021.
  * [12] Junyu Gao, Qi Wang, and Xuelong Li.  PCC Net: Perspective Crowd Counting via Spatial Convolutional Network.  TCSVT, 2020.
  * [13] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu.  Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion.  In CVPR, 2022.
  * [14] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi.  Social GAN: Socially Acceptable Trajectories With Generative Adversarial Networks.  In CVPR, 2018.
  * [15] Tao Han, Lei Bai, Lingbo Liu, and Wanli Ouyang.  STEERER: Resolving Scale Variations for Counting and Localization via Selective Inheritance Learning.  In ICCV, 2023.
  * [16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked Autoencoders Are Scalable Vision Learners.  In CVPR, 2022.
  * [17] Boris Ivanovic and Marco Pavone.  The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs.  In ICCV, 2019.
  * [18] Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, S. Hamid Rezatofighi, and Silvio Savarese.  Social-BiGAT: Multimodal Trajectory Forecasting Using Bicycle-GAN and Graph Attention Networks.  In NeurIPS, 2019.
  * [19] Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese.  Learning an Image-Based Motion Context for Multiple People Tracking.  In CVPR, 2014.
  * [20] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng.  SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders.  In NeurIPS, 2022.
  * [21] Haopeng Li, Lingbo Liu, Kunlin Yang, Shinan Liu, Junyu Gao, Bin Zhao, Rui Zhang, and Jun Hou.  Video crowd localization with multifocus gaussian neighborhood attention and a large-scale benchmark.  TIP, 2022.
  * [22] Yuhong Li, Xiaofan Zhang, and Deming Chen.  CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes.  In CVPR, 2018.
  * [23] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature Pyramid Networks for Object Detection.  In CVPR, 2017.
  * [24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar.  Focal Loss for Dense Object Detection.  In ICCV, 2017.
  * [25] Weizhe Liu, Krzysztof Lis, Mathieu Salzmann, and Pascal Fua.  Geometric and Physical Constraints for Drone-Based Head Plane Crowd Density Estimation.  In IROS, 2019.
  * [26] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [27] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik.  From Goals, Waypoints & Paths to Long Term Human Trajectory Forecasting.  In ICCV, 2021.
  * [28] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon.  It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction.  In ECCV, 2020.
  * [29] Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang.  Leapfrog Diffusion Model for Stochastic Trajectory Prediction.  In CVPR, 2023.
  * [30] Yunyao Mao, Jiajun Deng, Wengang Zhou, Yao Fang, Wanli Ouyang, and Houqiang Li.  Masked motion predictors are strong 3d action representation learners.  In ICCV, 2023.
  * [31] Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese.  Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.  IEEE TPAMI, 2021.
  * [32] Anton Milan, Laura Leal-Taixé, Ian D. Reid, Stefan Roth, and Konrad Schindler.  MOT16: A Benchmark for Multi-Object Tracking.  CoRR, 2016.
  * [33] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai.  Occupancy-MAE: Self-Supervised Pre-Training Large-Scale LiDAR Point Clouds With Masked Occupancy Autoencoders.  IV, 2023.
  * [34] Hiroaki Minoura, Ryo Yonetani, Mai Nishimura, and Yoshitaka Ushiku.  Crowd Density Forecasting by Modeling Patch-Based Dynamics.  RA-L, 2021.
  * [35] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel.  Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction.  In CVPR, 2020.
  * [36] Stefano Pellegrini, Andreas Ess, and Luc Van Gool.  Improving Data Association by Joint Modeling of Pedestrian Trajectories and Groupings.  In ECCV, 2010.
  * [37] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You Only Look Once: Unified, Real-Time Object Detection.  In CVPR, 2016.
  * [38] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In NeurIPS, 2015.
  * [39] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese.  Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes.  In ECCV, 2016.
  * [40] Saeed Saadatnejad, Yang Gao, Kaouther Messaoud, and Alexandre Alahi.  Social-transmotion: Promptable human trajectory prediction.  In ICLR, 2024.
  * [41] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese.  SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints.  In CVPR, 2019.
  * [42] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone.  Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data.  In ECCV, 2020.
  * [43] Vishwanath A. Sindagi, Rajeev Yasarla, and Vishal M. Patel.  JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method.  IEEE TPAMI, 2022.
  * [44] Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H. Li, Mingkui Tan, and Chuang Gan.  Masked Motion Encoding for Self-Supervised Video Representation Learning.  In CVPR, 2023.
  * [45] Ramana Sundararaman, Cedric De Almeida Braga, Eric Marchand, and Julien Pettre.  Tracking Pedestrian Heads in Dense Crowd.  In CVPR, 2021.
  * [46] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training.  In NeurIPS, 2022.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In NeurIPS, 2017.
  * [48] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and Composing Robust Features with Denoising Autoencoders.  In ICML, 2008.
  * [49] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao.  VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking.  In CVPR, 2023.
  * [50] Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan.  Learning From Synthetic Data for Crowd Counting in the Wild.  In CVPR, 2019.
  * [51] Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich.  Simple online and realtime tracking with a deep association metric.  In ICIP, 2017.
  * [52] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen.  GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction With Relational Reasoning.  In CVPR, 2022.
  * [53] Yihong Xu, Loïck Chambon, Éloi Zablocki, Mickaël Chen, Alexandre Alahi, Matthieu Cord, and Patrick Pérez.  Towards motion forecasting with real-world perception inputs: Are end-to-end approaches competitive?  In ICRA, 2024.
  * [54] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang.  MAGVIT: Masked generative video transformer.  In CVPR, 2023.
  * [55] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  ByteTrack: Multi-Object Tracking by Associating Every Detection Box.  In ECCV, 2022.
  * [56] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma.  Single-Image Crowd Counting via Multi-Column Convolutional Neural Network.  In CVPR, 2016.