#  Learning Salient Boundary Feature for Anchor-free
Temporal Action Localization

Chuming Lin∗1 Chengming Xu∗2 Donghao Luo1 Yabiao Wang1 Ying Tai1
Chengjie Wang1 Jilin Li1 Feiyue Huang1 Yanwei Fu2
1Youtu Lab  Tencent  2Fudan University  China
{chuminglin, michaelluo, caseywang, yingtai, jasoncjwang, jerolinli,
<EMAIL_ADDRESS>
{cmxu18<EMAIL_ADDRESS>

###### Abstract

Temporal action localization is an important yet challenging task in video
understanding. Typically, such a task aims at inferring both the action
category and localization of the start and end frame for each action instance
in a long, untrimmed video. While most current models achieve good results by
using pre-defined anchors and numerous actionness, such methods could be
bothered with both large number of outputs and heavy tuning of locations and
sizes corresponding to different anchors. Instead, anchor-free methods is
lighter, getting rid of redundant hyper-parameters, but gains few attention.
In this paper, we propose the first purely anchor-free temporal localization
method, which is both efficient and effective. Our model includes (i) an end-
to-end trainable basic predictor, (ii) a saliency-based refinement module to
gather more valuable boundary features for each proposal with a novel boundary
pooling, and (iii) several consistency constraints to make sure our model can
find the accurate boundary given arbitrary proposals. Extensive experiments
show that our method beats all anchor-based and actionness-guided methods with
a remarkable margin on THUMOS14, achieving state-of-the-art results, and
comparable ones on ActivityNet v1.3. Code is available at
https://github.com/TencentYoutuResearch/ActionDetection-AFSD.

††footnotetext: ∗ indicates equal contributions.††footnotetext: This work was
done when Chengming Xu was an intern at Tencent Youtu Lab. Yanwei Fu is the
corresponding author.

##  1 Introduction

Recently, with the progress of technology, a dramatically increasing number of
videos have been stored and accessible from various daily activities. Temporal
Action Localization (TAL), as a fundamental aspect of video understanding,
thus plays an important role in real life, extending in several practical
applications such as video analysis and summarization, human interaction, etc.
Compared with action recognition that takes medium-range videos as input and
only requires class labels as prediction, TAL is aimed at not only classifying
every activity instance in each video, but also looking for the accurate
temporal locations of them.

Figure 1: Compared with actionness and anchor-based methods, anchor-free
method is more efficient and flexible to produce fewer proposals without any
extra classifier and pre-defined anchors.

Current TAL models mainly focus on learning actionness of each frame [20, 18,
17, 26, 27, 24] or adjustment of pre-defined anchors [38, 23, 22], named
actionness-guided methods and anchor-based methods, as shown in Fig. 1. In
spite of reasonable good results on benchmark datasets, such methods are still
limited to the following points: (1) Both methods will produce a bunch of
redundant proposals. For example, given a video with $T$ frames, we have to
produce $\mathcal{O}(T^{2})$ and $C\cdot T$ proposals for the “actionness-
guided” BSN [20], and “anchor-based” R-C3D [38], individually. Here $C$ is the
number of pre-defined anchors. These proposals lead to prohibitive
computational cost in both calculating the training loss and post-processing
for testing. (2) Actionness-guided methods can solely provide predictions of
temporal boundary, while they have to rely on the extra model such as S-CNN
[33] and P-GCN [41] for classification. Nevertheless, the models of two stages
are isolated and thus incapable of sharing information for the end-to-end
update. (3) Typically, anchor-based methods are very sensitive to some
critical hyper-parameters, such as the number and size of pre-defined anchors;
and it is very non-trivial to tune these hyper-parameters in the real-world
applications.

Alternatively, an efficient recipe for localization is to resort to the
anchor-free method, which does not require pre-defined anchors. Typically,
this type of method only generates one proposal for each temporal location in
the form of a pair of values representing the distance between the start and
end moments to the current location, individually.

In contrast to the existed methods, anchor-free model saves huge amount of
pre-defined anchors, while assembling both boundary regression and
classification in one model, thus being productive. Furthermore, even though
some pilot studies, e.g., Yang et. al. [40] observed relatively weak results
for anchor-free methods, the supporting evidence in object detection [42]
shows that such methods with well designed network structure and training
strategy should, in principle, be comparable with anchor-based ones.

To this end, in this paper we propose a novel purely anchor-free TAL framework
dubbed Anchor-Free Saliency-based Detector (AFSD). Essentially, we first build
a naive anchor-free predictor containing an end-to-end trainable backbone
network, a feature pyramid network and a simple prediction network which
outputs the action class and the temporal distance of the start and end from
each location. To learn a more accurate boundary, we refer to former TAL
methods [20, 18] indicating the importance of boundary or context feature.
These works obtain such features mainly by merging the neighbor of start and
end moments with convolutions or mean pooling. However, we claim that in fact
moment-level feature is more valuable than region-level feature for
distinguishing whether an action starts or ends. As shown in Fig. 2, the
background regions near the start and end moments are showing other irrelevant
scenes, while regions inside the action are almost the same, which cannot
provide any information for judging if the action starts or ends. Such an
example indicates the importance of a moment-level feature.

Figure 2: An action instance of cliff diving. Note that the start and end
moments of the movement are more salient than others, which can bring us
significant information to judge the boundary and completeness of the action.

Therefore, we propose a novel boundary pooling which, instead of aggregating
the whole region, tries to find the most salient moment-level feature for both
start and end regions. We further equip the boundary pooling with a newly
proposed Boundary Consistency Learning (BCL) strategy to regularize the
pooling operation to provide the correct boundary features for each action. In
detail, we employ a modified ground truth signal indicating start and end
moments to guide the model. Then, we rearrange the video clips to help model
discriminate background and action features by self-supervised contrastive
learning. We conduct extensive experiments on THUMOS14 and ActivityNet1.3. On
THUMOS14 our model attains 3.7% improvement on<EMAIL_ADDRESS>against the state-of-
the-art methods. The results on ActivityNet1.3 are also comparable.

In summary, our paper has the following contributions:

  1. 1.

We, for the first time, propose a purely anchor-free temporal action
localization model. This model enjoys not only less hyper-parameter to tune
and less outputs to process, but also better performance, thus making the best
of both worlds.

  2. 2.

To make full use of the anchor-free framework, we discuss the impact of
boundary features and propose novel boundary pooling method whose output is
used along with the coarse proposals to generate fine-grained predictions.
Moreover, we introduce a novel Boundary Consistency Learning strategy which
can constrain the model to learn better boundary feature.

##  2 Related Work

Anchor-based Localization Anchor-based localization models rely on adjusting
the pre-defined anchors. TURN [10] aggregates features from basic video unit
for clip-level features, which are used to classify the activity and regress
the temporal boundary. R-C3D [38] takes the inspiration from faster-RCNN [31],
which utilizes a streamline including proposal generation, proposal-wise
pooling and final prediction. GTAN [23] modifies the pooling procedure,
adopting a weighted average via a learnable Gaussian kernel for each proposal.
Due to the fixed pre-defined anchors, such methods are not flexible when it
comes to various action classes. Different from them, our model does not
require tuning extra hyper-parameters for anchors, thus more efficient.

Actionness-guided Localization Unlike anchor-based methods, actionness-guided
methods mainly focus on evaluating ‘actionness’, which indicates the
probability of a potential action, for each frame or clip in a video. The
actionness is afterwards post-processed to generate action proposals. Zhao et.
al. designed SSN [44] in which course proposals are first divided into three
semantic parts, learned respectively. Next probability of activity and
completeness is predicted and used to merge different proposals. Lin et. al.
proposed BSN [20] which learns to predict start, end and actionness of each
temporal location. The proposals are generated by gathering locations with
high start and end probabilities, with low confidence ones further abandoned
by an evaluation module. They later improved this framework to BMN [18], which
additionally generates a Boundary-Matching confidence map to help get better
proposals. While no pre-defined anchors are required for actionness-guided
methods, such methods are more like enumeration methods where all possible
combinations of temporal locations are considered, thus totally different from
anchor-free localization where boundaries are directly predicted for each time
step.

Anchor-Free Object Detection Analogous to TAL, there is a surge of the usage
of anchor-free methods in object detection. YOLO [30] is the most well-known
anchor-free method, in which a neural network model is directly used to
predict coordinates of bounding boxes from raw images. Such framework is too
simple, thus suffering from poor performance. Following works mainly focus on
improving performance via setting different prediction targets and using more
detailed features. CornerNet [16] lets the model learn to predict top-left and
bottom-right keypoints of each bounding box. FCOS [35] aims to learn the
distance to boundaries of each spatial location and utilizes feature pyramid
for objects with diverse scales. BorderDet [28] modifies the RoI pooling into
BorderAlign to get more powerful proposal-level feature. We take inspiration
from these methods to design a basic anchor-free localizer, along with making
full use of the temporal insights of videos to propose novel refinement
strategy and consistency learning.

Contrastive Learning There has been an increasing interest in contrastive
learning used in unsupervised learning. Compared with the application of
contrastive learning in image understanding [8, 13], fewer contributions of
contrastive learning have been made in video understanding. Guillaume et. al.
[25] proposed a temporal contrastive training strategy for action recognition,
in which an autoregressive model is used to predict future video segment given
enough information, and then compare the prediction with ground truth. Gong
et. al. [12] adopted a contrastive scoring method to evaluate action proposals
in unsupervised temporal localization in the inference phase. Compared with
these works, we make further trial into leveraging contrastive learning to
help train a supervised temporal localization model, which has never been
studied before.

##  3 Method

Denote a video dataset as
$\mathcal{T}=\\{\mathcal{T}^{train},\mathcal{T}^{test}\\}$, each data instance
$\\{X,\Psi_{X}\\}$ contains a video $X=\\{x_{t}\\}_{t=1}^{T}$ with $T$ RGB
frames or optical flows. The corresponding annotation $\Psi_{X}$ can be
depicted as tuples $\\{(\phi_{m},y_{m})\\}_{m=1}^{M_{X}}$ where $M_{X}$ is the
number of action instances in $X$, $\phi_{m}=(\psi_{m},\xi_{m})$ denotes the
start time, end time and $y_{m}$ indicates the action category. Our goal is to
train a model to predict proposals with class scores which could have high
recall and precision with the ground truth on the test set
$\mathcal{T}^{test}$.

Figure 3: The overview of our approach. Given an input video $X$, we employ
I3D model to extract feature and construct 1D temporal pyramid features. Next,
each pyramid feature is utilized to generate coarse proposals via basic
prediction module. Finally, our saliency-based refinement module will adjust
the class score, start and end boundaries and predict the corresponding
quality score for each coarse proposal. Note that our model is a fully end-to-
end method and trained with I3D feature extraction network without any
preprocessing. Figure 4: (a) Saliency-based Refinement Module: utilize coarse
boundaries, FPN feature and frame-level feature to construct salient boundary
feature. (b) Boundary Pooling: search salient moment features in the boundary
regions of the input feature.

Overview We propose a purely anchor-free architecture named AFSD which is
shown in Fig. 3. Concretely, given a video $X$, we first process the video
with a backbone network and a feature pyramid network. Take RGB frames as
example, for each video $X$, we use a Kinetics pre-trained I3D [6] model to
extract a 3D feature $F\in\mathbb{R}^{T^{{}^{\prime}}\times
C^{{}^{\prime}}\times H^{{}^{\prime}}\times W^{{}^{\prime}}}$, where
$T^{{}^{\prime}},C^{{}^{\prime}},H^{{}^{\prime}},W^{{}^{\prime}}$ denote the
time step, channel, height and width individually. This feature is afterwards
flattened along the last three dimensions to a 1D feature sequence. Such a
sequence can contain the temporal and spatial information of whole video. We
then exert a feature pyramid network including several temporal convolutions,
of which the detailed architecture is shown in our supplement, to merge the
spatial dimension and aggregate the temporal dimension in different levels.
The pyramid features are further utilized to generate a coarse proposal
sequence $\\{(\hat{\psi}_{i},\hat{\xi}_{i},\hat{y}_{i}^{C})\\}$ with a basic
anchor-free prediction module (Sec. 3.1), which includes a simple regressor
and classifier. After that for each proposal, the predicted temporal regions
are employed to get the salient boundary features with the boundary pooling
(Sec. 3.2). The boundary features are exploited together with the feature
pyramid to output a fine-grained prediction
$\\{(\Delta\hat{\psi}_{i},\Delta\hat{\xi}_{i},\hat{y}_{i}^{R})\\}$ for both
temporal regression and action classification.

###  3.1 Basic Prediction Module

We first build a basic anchor-free prediction module to get coarse temporal
boundaries. For instance, for the feature of the $l$-th FPN level
$f^{l}\in\mathbb{R}^{T_{l}\times C}$, we first project it to features
$f^{l}_{loc}$ and $f^{l}_{cls}$ embedded in two latent space corresponding to
localization and classification by two branches with two temporal convolutions
respectively. Both of these two features $f^{l}_{loc}$ and $f^{l}_{cls}$ are
processed with one layer of temporal convolution shared among all FPN layers
to get coarse start and end boundary distances
$(\hat{d}_{i}^{s},\hat{d}_{i}^{e})$ and class score $\hat{y}_{i}$ for each
location $i$. Next we can get start and end time for $i$-th time step in
$l$-th level as follow:

$\displaystyle\hat{\psi}_{i}$ $\displaystyle=i*2^{l}-\hat{d}_{i}^{s},$ (1)
$\displaystyle\hat{\xi}_{i}$ $\displaystyle=i*2^{l}+\hat{d}_{i}^{e}.$

$T_{l}$ proposals are generated for $l$-th FPN layer in all. Such a simple
framework can already temporally detect actions in a anchor-free manner, which
enjoys several merits including no requirement for pre-defined anchors and
less but more accurate predictions as discussed in Sec. 1. In the following
sections, we focus on designing appropriate modules and training strategy for
anchor-free TAL methods with better performance.

###  3.2 Saliency-based Refinement Module

As mentioned in Sec. 1, several existing works have shown the importance of
boundary feature in TAL, especially for predicting the temporal distance.
However, since different action instances could have various lengths, it is
hard to attain these boundary information for all proposals via several simple
temporal convolutions because of the limited receptive field. Therefore, we
propose a saliency-based refinement module which is illustrated in Fig. 4(a),
where we utilize the FPN features along with the coarse proposals to help our
model gather boundary features to refine the predictions. For simplicity, we
omit the subscript standing for FPN layers in the following detail. Take the
localization feature $f_{loc}$ for example, first we project it into two
latent spaces sensitive to start and end activities respectively via
convolutional layers:

$\displaystyle f^{s}$
$\displaystyle=\sigma(\textrm{GN}(\text{Conv1}(f_{loc})))\quad\in\mathbb{R}^{T_{l}\times
C},$ (2) $\displaystyle f^{e}$
$\displaystyle=\sigma(\textrm{GN}(\text{Conv2}(f_{loc})))\quad\in\mathbb{R}^{T_{l}\times
C},$

where $\sigma$ and GN denote ReLU and Group Normalization [37]. With the
projection, the model can learn start and end sensitive signals separately,
thus leaving less learning load to the FPN features for better training.

Then, given the coarse boundary results
$\\{(\hat{\psi}_{k},\hat{\xi}_{k})\\}_{k=1}^{T_{l}}$ of the corresponding
$l$-th FPN level, the $k$-th start and end regions $T_{s}^{k}$, $T_{e}^{k}$
are constructed as:

$\displaystyle T_{s}^{k}$
$\displaystyle=\left[\hat{\psi}_{k}-\frac{\hat{w}_{k}}{\delta_{a}},\hat{\psi}_{k}+\frac{\hat{w}_{k}}{\delta_{b}}\right],$
(3) $\displaystyle T_{e}^{k}$
$\displaystyle=\left[\hat{\xi}_{k}-\frac{\hat{w}_{k}}{\delta_{b}},\hat{\xi}_{k}+\frac{\hat{w}_{k}}{\delta_{a}}\right],$

where $\hat{w}_{k}=\hat{\psi}_{k}-\hat{\xi}_{k}$ means the length of proposal,
$\delta_{a},\delta_{b}$ are hyper-parameters controlling the proportion of
selected regions outside and inside the proposal.

Next, an aggregation function $\mathcal{A}$ is applied to $f^{s}$ and $f^{e}$
in start and end regions respectively to collect the relevant boundary
features. Despite a lot of instantiations of $\mathcal{A}$ in former works,
such as mean pooling [10], Gaussian weighted average [23] and directly
gathering and concatenating [17], these methods would introduce useless
knowledge from frames not representing the action boundaries, thus blocking
the model from precise predictions. Hence we propose a novel boundary pooling
method to get the moment-level boundary feature
$\hat{f}^{s},\hat{f}^{e}\in\mathbb{R}^{T_{l}\times C}$ as shown in Fig. 4(b).
The boundary pooling works as following:

$\displaystyle\hat{f}^{s}(k,i)$ $\displaystyle=\max_{j\in
T_{s}^{k}}f^{s}(j,i)\quad i=1,\cdots,C,$ (4) $\displaystyle\hat{f}^{e}(k,i)$
$\displaystyle=\max_{j\in T_{e}^{k}}f^{e}(j,i)\quad i=1,\cdots,C.$

Maximization is utilized aiming to select the largest activated cell, i.e.,
the most salient moment, for each channel along the temporal region. Note that
as FPN goes deeper, the temporal dimension decreases to be too small for
boundary pooling to find the appropriate boundary. Therefore, we add a shared
frame-level feature $f_{frame}$ by applying upsample and several convolutions
to the bottom FPN feature, from which start and end frame-level features
$\tilde{f}^{s},\tilde{f}^{e}$ are extracted for each proposal with the same
projection and pooling procedure as in Eq. 2 and Eq. 4.

After boundary pooling, the refined features are built by concatenating
original features and all boundary features. A temporal convolution is applied
to reduce channels:

$\hat{f}=\text{Conv}(f||\hat{f}^{s}||\hat{f}^{e}||\tilde{f}^{s}||\tilde{f}^{e}),$
(5)

where $\cdot||\cdot$ denotes channel-wise concatenation. These features are
again employed as input of a simple network with temporal convolution to
predict offsets of regression $(\Delta\hat{\psi},\Delta\hat{\xi})$, which can
be added to the coarse predictions to get a fine-grained ones
$(\tilde{\psi},\tilde{\xi})$, and refined class score $\hat{y}^{R}$.

###  3.3 Boundary Consistency Learning

Although the boundary pooling can extract the most salient features, it cannot
guarantee that the pooled features represent the true action boundary. Such a
property is pivotal since if boundary pooling focuses on a background frame,
the model cannot have enough useful information, and thus being misled to
wrong results. To regularize our boundary pooling, we further propose Boundary
Consistency Learning (BCL), which has two components: Activation Guided
Learning and Boundary Contrastive Learning.

Activation Guided Learning  Specifically, we re-scale the sensitive features
$f^{s}$ and $f^{e}$ and take channel-wise mean:

$\tilde{g}^{s}=\mathrm{mean}(\tanh(f^{s})),\tilde{g}^{e}=\mathrm{mean}(\tanh(f^{e})).$
(6)

These two features can be seen as confidence revealing the probability of
occurrence of start or end moment. We can obtain the ground truth
$g^{s},g^{e}\in\mathbb{R}^{T}$ by following definition:

$\displaystyle g^{s}(i)$
$\displaystyle=\mathbb{I}(i\in\mathcal{B}(\psi_{m})\text{ for }\forall
m\in[1,M_{X}]),$ (7) $\displaystyle g^{e}(i)$
$\displaystyle=\mathbb{I}(i\in\mathcal{B}(\xi_{m})\text{ for }\forall
m\in[1,M_{X}]),$

where $\mathcal{B}(\cdot)$ denotes the neighbor and $\mathbb{I}(\cdot)$ is the
indicator function. After that we can calculate the Cross Entropy:

$\ell_{act}=\text{BCE}(g^{s},\tilde{g}^{s})+\text{BCE}(g^{e},\tilde{g}^{e}),$
(8)

where BCE denotes the binary cross entropy. With $g^{s}$ and $g^{e}$ as
guidance, we can constrain the feature to have high activation at the
occurrence and closure of each action.

Boundary Contrastive Learning Consider a video $X$ containing an action
instance $A$ and other background. If we split the action and fill in a random
part of background, we will have two incomplete action fragments $A_{1},A_{2}$
and background $Bg$ between them. Applying boundary pooling to these three
regions leads to three pairs of features
$(f^{s}_{A_{1}},f^{e}_{A_{1}}),(f^{s}_{A_{2}},f^{e}_{A_{2}}),(f^{s}_{Bg},f^{e}_{Bg})$.
In general, since $A_{1}$ and $A_{2}$ are continuous, the sensitive features
$f^{e}_{A_{1}}$ and $f^{s}_{A_{2}}$ should be similar to each other and
distant to $f^{s}_{Bg}$ and $f^{e}_{Bg}$ if we restrict boundary pooling to
only make use of few frames inside the actions (i.e., a large $\delta_{b}$ in
Eq. 3). However, this property could be broken when model is learnt to take
high activations on background. In such situation, $f^{e}_{A_{1}}$ would be
close to $f^{s}_{Bg}$ and $f^{s}_{A_{2}}$ would be close to $f^{e}_{Bg}$.
Therefore, a good way to guarantee appropriate features is to apply the
contrastive learning on these features, enlarging the distance between the
sensitive features of action fragments and background. Formally, it can be
realized by utilizing the following triplet objective function:

$\ell_{trip}=\max(\|f^{e}_{A_{1}}-f^{s}_{A_{2}}\|^{2}-\|f^{e}_{A_{1}}-f_{Bg}\|^{2}+1,0),$
(9)

where $f_{Bg}\in\left\\{f_{Bg}^{s},f_{Bg}^{e}\right\\}$. In practice, we first
count the minimal action length $w_{min}$ in one video. Next if the video has
an action instance whose length is larger than $2\cdot w_{min}$ and a
background clip with length $w_{min}$, we include this video into our
contrastive learning pool and implement the above splitting procedure. In this
way, both split action and background are long enough to be distinguished. In
together, our Boundary Consistency Learning can be summarized into the
following form:

$\ell_{con}=\ell_{act}+\ell_{trip}.$ (10)

###  3.4 Training and Inference

Label Assignment For coarse prediction, we assign each location $i$ as a
positive sample to ground truth $j$ when $\psi_{j}\leq i\leq\xi_{j}$ is
satisfied. For refined prediction, we calculate the temporal IoU (tIoU)
between each coarse boundary prediction and the corresponding ground truth. A
location $i$ is regarded as positive if its tIoU score is greater than 0.5.
Denote $N_{C},N_{R}$ as the number of positive samples for coarse and refined
predictions individually.

Training Once having both the coarse and refined prediction of temporal
boundary and class label, we can optimize the model with the following
objective function:

$\mathcal{L}=\ell^{C}_{cls}+\lambda\ell^{C}_{loc}+\ell^{R}_{cls}+\lambda\ell^{R}_{loc}+\gamma\ell_{q},$
(11)

where $\lambda,\gamma$ are hyper-parameters, $\ell^{C}_{cls},\ell^{R}_{cls}$
are softmax focal loss $\ell_{focal}$ [21] between both classification
prediction $\left\\{\hat{y}^{C},\hat{y}^{R}\right\\}$ and ground truth labels
$y$:

$\ell_{cls}(\\{\hat{y}_{i}\\})=\frac{1}{N}\sum_{i}\ell_{focal}(\hat{y}_{i},y_{i}),$
(12)

where $N\in\\{N_{C},N_{R}\\}$. $\ell^{C}_{loc}$ is a tIoU loss between coarse
boundaries $\hat{\phi}_{i}=(\hat{\psi}_{i},\hat{\xi}_{i})$ and the
corresponding ground truth $\phi_{i}=(\psi_{i},\xi_{i})$. $\ell^{R}_{loc}$ is
a L1 loss between the predicted offset
$\hat{\Delta}_{i}=(\Delta\hat{\psi}_{i},\Delta\hat{\xi}_{i})$ and the
corresponding offset label $\Delta_{i}=(\Delta\psi_{i},\Delta\xi_{i})$:

$\displaystyle\ell^{C}_{loc}(\\{\hat{\phi}_{i}\\})$
$\displaystyle=\frac{1}{N_{C}}\sum_{i}\mathbb{I}(y_{i}\geq
1)(1-\frac{|\hat{\phi}_{i}\cap\phi_{i}|}{|\hat{\phi}_{i}\cup\phi_{i}|}),$ (13)
$\displaystyle\ell^{R}_{loc}(\\{\hat{\Delta}_{i}\\})$
$\displaystyle=\frac{1}{N_{R}}\sum_{i}\mathbb{I}(y_{i}\geq
1)(|\hat{\Delta}_{i}-\Delta_{i}|).$

$\ell_{q}$ is a quality loss used to suppress the proposals with low quality.
As a counterpart in the object detection, FCOS [35] proposes the centerness of
each spatial location as the quality target. However, such definition of
centerness for actions is vague since it is hard to decide the exact frame
being a start or end signal of an action, and thus it is inappropriate to
directly use centerness in TAL. To better predict quality of proposals, we
utilize tIoU between boundary prediction $\tilde{\phi}$ and the location
labels as the learning target of quality confidence $\eta$ generated from
refined feature $\hat{f}$:

$\ell_{q}(\\{\eta_{i}\\})=\frac{1}{N_{R}}\sum_{i}\mathbb{I}(y_{i}\geq
1)\text{BCE}(\eta_{i},\frac{|\tilde{\phi}_{i}\cap\phi_{i}|}{|\tilde{\phi}_{i}\cup\phi_{i}|}).$
(14)

For each batch in training, we first optimize the model with $\mathcal{L}$.
Then we seek the data available for BCL in Sec. 3.3 and train the model with
$\ell_{con}$ in Eq. 10.

Inference For the $i$-th temporal location in $l$-th FPN layer, the final
predictions are formalized through all outputs from our model, including
coarse predictions $\hat{\psi}_{l,i},\hat{\xi}_{l,i},\hat{y}^{C}_{l,i}$ and
refined ones
$\Delta\hat{\psi}_{l,i},\Delta\hat{\xi}_{l,i},\hat{y}^{R}_{l,i},\eta_{l,i}$,
in the following form:

$\displaystyle\hat{w}_{l,i}$ $\displaystyle=\hat{\xi}_{l,i}-\hat{\psi}_{l,i},$
(15) $\displaystyle\tilde{\psi}_{l,i}$
$\displaystyle=\hat{\psi}_{l,i}+\frac{1}{2}\hat{w}_{l,i}\Delta\hat{\psi}_{l,i},$
$\displaystyle\tilde{\xi}_{l,i}$
$\displaystyle=\hat{\xi}_{l,i}+\frac{1}{2}\hat{w}_{l,i}\Delta\hat{\xi}_{l,i},$
$\displaystyle\hat{y}_{l,i}$
$\displaystyle=\frac{1}{2}(\hat{y}^{C}_{l,i}+\hat{y}^{R}_{l,i})\eta_{l,i}.$

We then assemble all predictions and process them with Soft-NMS [3] to
suppress redundant proposals.

##  4 Experiments

Type | Model | Backbone | THUMOS14 | ActivityNet1.3
---|---|---|---|---
0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg. | 0.5 | 0.75 | 0.95 | Avg.
Anchor-based |  SSAD [19] | TS | 43.0 | 35.0 | 24.6 | — | — | — | — | — | — | —
TURN [10] | C3D | 44.1 | 34.9 | 25.6 | — | — | — | — | — | — | —
R-C3D [38] | C3D | 44.8 | 35.6 | 28.9 | — | — | — | 26.8 | — | — | —
CBR [11] | TS | 50.1 | 41.3 | 31.0 | 19.1 | 9.9 | 30.3 | — | — | — | —
TAL [7] | I3D | 53.2 | 48.5 | 42.8 | 33.8 | 20.8 | 39.8 | 38.2 | 18.3 | 1.3 | 20.2
GTAN [23] | P3D | 57.8 | 47.2 | 38.8 | — | — | — | 52.6 | 34.1 | 8.9 | 34.3
Actionness |  CDC [32] | — | 40.1 | 29.4 | 23.3 | 13.1 | 7.9 | 22.8 | 45.3 | 26.0 | 0.2 | 23.8
SSN [44] | TS | 51.0 | 41.0 | 29.8 | — | — | — | 43.2 | 28.7 | 5.6 | 28.3
BSN [20] | TS | 53.5 | 45.0 | 36.9 | 28.4 | 20.0 | 36.8 | 46.5 | 30.0 | 8.0 | 30.0
BMN [18] | TS | 56.0 | 47.4 | 38.8 | 29.7 | 20.5 | 38.5 | 50.1 | 34.8 | 8.3 | 33.9
DBG [17] | TS | 57.8 | 49.4 | 42.8 | 33.8 | 21.7 | 41.1 | — | — | — | —
G-TAD [39] | TS | 54.5 | 47.6 | 40.2 | 30.8 | 23.4 | 39.3 | 50.4 | 34.6 | 9.0 | 34.1
BU-TAL [43] | I3D | 53.9 | 50.7 | 45.4 | 38.0 | 28.5 | 43.3 | 43.5 | 33.9 | 9.2 | 30.1
|  BC-GNN [2] | TS | 57.1 | 49.1 | 40.4 | 31.2 | 23.1 | 40.2 | 50.6 | 34.8 | 9.4 | 34.3
Other |  A2Net [40] | I3D | 58.6 | 54.1 | 45.5 | 32.5 | 17.2 | 41.6 | 43.6 | 28.7 | 3.7 | 27.8
G-TAD+PGCN [41] | I3D | 66.4 | 60.4 | 51.6 | 37.6 | 22.9 | 47.8 | — | — | — | —
Anchor-free | Ours | I3D | 67.3 | 62.4 | 55.5 | 43.7 | 31.1 | 52.0 | 52.4 | 35.3 | 6.5 | 34.4
Table 1:  Performance comparison with state-of-the-art methods on THUMOS14 and
ActivityNet1.3, measured by mAP at different IoU thresholds, and average mAP
in $[0.3:0.1:0.7]$ on THUMOS14 and $[0.5:0.05:0.95]$ on ActivityNet1.3.

###  4.1 Datasets and Settings

Datasets To validate the efficacy of our model, we conduct extensive
experiments on commonly-used benchmark THUMOS14 [14] and ActivityNet1.3 [5].
THUMOS14 is composed of 200 validation videos and 212 testing videos from 20
categories labeled for temporal localization. ActivityNet1.3 has 19,994 videos
with 200 action classes. We follow the former setting [20] to split this
dataset into training, validation and testing subset by 2:1:1.

Implementation Details On THUMOS14, we sample both RGB and optical flow frames
at 10 frames per second (fps) and split video into clips. The length of each
clip $T$ is set as 256 frames. Adjacent clips have a temporal overlap of $m$
frames and $m$ is set to 30 in training and 128 in testing. On ActivityNet1.3,
we sample frames using different fps and ensure the number of video frames is
768 for each video. Thus, each video has only one clip with 768 frames. On
both datasets, the frame spatial size is set to $96\times 96$. Random crop,
horizontal flipping are used as data augmentation in training. To extract
features of clips, we finetune a I3D [6] model pre-trained on Kinetics.

Our model is trained for 16 epoches using Adam [15] with learning rate of
$10^{-5}$, weight decay of $10^{-3}$. Batch size is set to 1. We set
$\delta_{a}$ to 4 and $\delta_{b}$ to 100 for $\ell_{con}$ and $\delta_{b}$ to
10 for other loss terms. The weight of loss $\lambda$ is set to 10 on THUMOS14
and 1 on ActivityNet1.3 and $\gamma$ is set to 1 empirically. In the testing
phase, the results of RGB and optical flow frames are averaged to obtain final
locations and class scores. The tIoU threshold in Soft-NMS is set to 0.5 for
THUMOS14 and 0.85 for ActivityNet1.3.

Metrics We report mean Average Precision (mAP) in all experiments. The
thresholds are $[0.3:0.1:0.7]$ for THUMOS14 and $[0.5:0.05:0.95]$ for
ActivityNet1.3.

###  4.2 Main Results

We compare our model with state-of-the-art methods in Tab. 1 and report the
backbone used by each method, including TS [34], C3D [36], P3D [29] and I3D
[6]. On THUMOS14, our AFSD outperforms the strongest competitor A2Net and
G-TAD on all thresholds by large margin, especially 7.7% on<EMAIL_ADDRESS>The
remarkable improvement comes along with high efficiency, thus making our model
more practical for real TAL scenarios. Note that while A2Net also has an
anchor-free module, the performance of their merged model is far worse than
ours, not to mention the sole anchor-free branch, which proves the superiority
of our architecture for anchor-free methods.

On ActivityNet1.3, the results are still comparable. Our model receives the
best<EMAIL_ADDRESS>and average mAP compared to the strongest competitor GTAN. It is
noteworthy that while all of the actionness-guided methods have less average
mAP than ours, most of them enjoy a higher<EMAIL_ADDRESS>One possible reason is
that they can enumerate all potential proposals, thus the ground truth
proposals are already contained in the alternative prediction set. With such
an enumeration strategy the actionness-based methods would be better when
dealing with harder datasets like ActivityNet1.3. Compared with actionness-
guided methods, our model is more efficacious in the sense of producing less
proposals and attaining better overall performance when considering multiple
thresholds. Additionally, compared with THUMOS14, ActivityNet is less well
annotated as explained by official report [1] showing ‘it is hard to agree
about the temporal boundaries’, which partially attributes to the slight
improvement.

###  4.3 Ablation Study

To further verify the efficacy of our contributions, we conduct several
ablation studies on THUMOS14 for the RGB model, including each part of our
model and the choice of hyper-parameters.

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
baseline | 43.1 | 31.0 | 19.0 | 40.4
+centerness | 43.3 | 31.6 | 17.7 | 40.2
+quality | 44.0 | 32.0 | 19.8 | 41.4

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
0-1 clip | 45.3 | 34.6 | 22.4 | 42.6
0-1 norm | 45.4 | 34.9 | 21.6 | 42.9
tanh | 45.9 | 35.0 | 23.4 | 43.5

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
$\delta_{a}=\delta_{b}$ | 45.0 | 33.4 | 21.2 | 42.3
$\delta_{a}>\delta_{b}$ | 45.3 | 34.7 | 21.8 | 42.6
$\delta_{a}<\delta_{b}$ | 45.9 | 35.0 | 23.4 | 43.5

(a) Training strategy: we compare our quality confidence with centerness loss
proposed in FCOS.

(b) Feature normalization: we compare different choice of feature
normalization in constraint.

(c) Boundary feature extraction: we compare different choice of boundary
regions.

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
naive | 44.9 | 32.5 | 19.9 | 41.6
self | 44.6 | 33.9 | 22.3 | 42.4
frame | 42.9 | 31.9 | 20.2 | 40.5
all | 45.9 | 35.0 | 23.4 | 43.5

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
mean | 45.1 | 34.6 | 22.1 | 42.7
conv | 44.8 | 34.4 | 22.3 | 42.6
stack | 44.9 | 33.6 | 22.3 | 42.9
max | 45.9 | 35.0 | 23.4 | 43.5

Model | 0.5 | 0.6 | 0.7 | Avg.
---|---|---|---|---
w/o BCL | 44.3 | 34.1 | 21.2 | 42.0
$\ell_{act}$ | 45.6 | 34.4 | 22.3 | 42.7
$\ell_{trip}$ | 45.5 | 34.8 | 22.4 | 42.7
$\ell_{act}+\ell_{trip}$ | 45.9 | 35.0 | 23.4 | 43.5

(d) Refinement: we compare different sources of information for refinement.

(e) Instantiations: we compare different forms of boundary feature extraction.

(f) Consistency learning: we compare models varied with Boundary Consistency
Learning.

Table 2:  Ablation studies of RGB model on THUMOS14, measured by mAP at 0.5, 0.6 and 0.7, and average mAP in $[0.3:0.1:0.7]$. Model | GPU | FPS
---|---|---
S-CNN [33] | — | 60
DAP [9] | — | 134
CDC [32] | TITAN Xm | 500
SS-TAD [4] | TITAN Xm | 701
R-C3D [38] | TITAN Xm | 569
R-C3D [38] | TITAN Xp | 1030
Ours | 1080 Ti | 3259
Ours | V100 | 4057
Table 3:  Comparison of inference speed between our method and other methods
on THUMOS14.

Effectiveness of Quality Confidence We first compare the basic prediction
module introduced in Sec. 3.1 trained and inferred with and without the
proposed quality confidence in Tab. 2(a). Besides, we add an extra model using
the centerness proposed in FCOS. The results show that centerness leads to
1.3% drop on<EMAIL_ADDRESS>doing no help for training TAL model. In contrast, model
with quality loss $\ell_{q}$ (Eq. 14) can have 1.0% average mAP improvement,
which supports our claim that the definition of centerness is somehow
inappropriate in TAL, thus cannot be directly applied. Compared with that, our
quality loss $\ell_{q}$ is a more suitable objective function for suppressing
low-quality action proposals.

Choice of Signal Normalization In Action Guided Learning introduced in Sec.
3.3, we adopt a tanh function to normalize the feature vector. We compare this
one with another two instantiations. One is a hard clipping between
$\left[0,1\right]$. The other is to use simple standardization in the
following form, which is denoted as 0-1 norm in Tab. 2(b):

$f(i)=\frac{f(i)-\min_{i=1}^{T}f(i)}{\max_{i=1}^{T}f(i)-\min_{i=1}^{T}f(i)}.$
(16)

The results show that using tanh can have 0.9% and 0.6% average mAP advantage
against two alternatives.

Choice of Boundary Region In Tab. 2(c) we assess choices of the boundary
region used for pooling boundary features, which is composed of (1) a
symmetric area with the coarse boundary as center point, denoted as
$\delta_{a}=\delta_{b}$. (2, 3) two asymmetric regions either focusing on
background or action. Through the results we can safely say that it is better
to keep a larger proportion of region inside the coarse action than the
background. The possible reason is that our naive predictor can already
produce a relatively accurate prediction. As shown in Tab. 2(a), the
performance of baseline can beat most of the competitors in Tab. 1 even if the
competitors fuse RGB and optical flow models for final prediction and our
baseline only utilizes RGB frames as input.

Effectiveness of Boundary Refinement In Tab. 2(d) we compare four forms of
boundary refinement, including: (1) Naive: Only several temporal convolutions
are directly applied to $f_{loc},f_{cls}$ to get another prediction. In this
way, the information disposed for refinement is the stacked neighbor
introduced by the convolutions. (2) Self: The saliency-based refinement module
is utilized without $f_{frame}$, and thus the consistency learning only
calculates losses corresponding to the FPN feature. (3) Frame-level: only
$f_{frame}$ is adopted in temporal refinement and FPN features are abandoned.
(4) All: All usable features are included, which is our final model. The
results demonstrate that: (1) Boundary information is more valuable for
refinement than that from neighborhood of each temporal location. (2) Frame-
level feature can only be taken as a complement of FPN features. Using only
frame-level feature would result in 1.1% average mAP drop against the naive
refinement model.

Instantiation of Boundary Pooling We compare our proposed boundary pooling
with the following three instantiations: (1) Mean: The max operation is
replaced with mean. (2) Conv: We sample three temporal positions for each
region, and a 1-layer temporal convolution is applied to aggregate them. (3)
Stack: Similar to (2), while instead of using convolution, we directly
concatenate these three features into one boundary feature. Note that all
models are trained with our boundary consistency learning. The results are
presented in Tab. 2(e). Among all instantiations, pooling with max receives
the best performance, showing 0.8%, 0.9% and 0.6% advantage of average mAP
against mean, conv and stack respectively. It is noteworthy that the
improvement of our model over others is larger than 1.0% on<EMAIL_ADDRESS>which
indicates that moment-level boundary feature helps the model generate more
accurate proposals.

Effectiveness of Boundary Consistency Learning We verify the proposed BCL by
comparing our full model with that trained only with $\mathcal{L}$ (Eq. 11)
and without $\ell_{con}$ (Eq. 10). Results in Tab. 2(f) suggest that when
trained without any consistency guarantee, the model cannot learn good
representations for boundary pooling. Therefore lack of useful information in
refinement leads to worse performance. Moreover, each loss term brings 0.7%
improvement on average mAP and by assembling $\ell_{act}$ (Eq. 8) and
$\ell_{trip}$ (Eq. 9) together, our model finally achieves the best
performance. To further validate the efficacy of BCL, we obtain the learned
frame-level feature $f_{frame}$ of an action instance and its neighboring
background, each half part of which could represent the start and end signals
in hypothesis.

Comparison on Inference Time As discussed, the proposed AFSD is highly
efficient. To verify this claim we report the inference speed on THUMOS14 in
terms if fps among different models in Tab. 3. Thus results show that our
model is much more faster than the existing methods. The main reasons is that
after features are extracted we can employ both a lighter predictor and a
lighter refinement module composed of 1D convolution due the the help of
anchor-free mechanism. Moreover, the number of proposals produced by our model
is less than that of these methods, which also helps speed our model.

##  5 Conclusion

In this paper we explore the possibility of a novel form of temporal action
localization model — anchor-free method. We discuss the merits over anchor-
based methods, and actionness-guided methods and design a dedicated anchor-
free model. Our model includes an end-to-end trainable basic predictor and a
temporal refinement module. For the refinement module, we analyze the
drawbacks of existing means to extract boundary features, and propose a novel
boundary pooling together with a Boundary Consistency Learning strategy. We
achieve remarkable results on THUMOS14 and comparable ones on ActivityNet1.3.
The results indicate the strength of anchor-free model as a promising choice
for solving temporal action localization.

## References

  * [1] Humam Alwassel, Fabian Caba Heilbron, Victor Escorcia, and Bernard Ghanem.  Diagnosing error in temporal action detectors.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 256–272, 2018.
  * [2] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu.  Boundary content graph neural network for temporal action proposal generation.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [3] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis.  Soft-nms–improving object detection with one line of code.  In Proceedings of the IEEE international conference on computer vision, pages 5561–5569, 2017.
  * [4] Shyamal Buch, Victor Escorcia, Bernard Ghanem, Li Fei-Fei, and Juan Carlos Niebles.  End-to-end, single-stream temporal action detection in untrimmed videos.  2019\.
  * [5] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [6] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [7] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1130–1139, 2018.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [9] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  Daps: Deep action proposals for action understanding.  In European Conference on Computer Vision, pages 768–784. Springer, 2016.
  * [10] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia.  Turn tap: Temporal unit regression network for temporal action proposals.  In Proceedings of the IEEE international conference on computer vision, pages 3628–3636, 2017.
  * [11] Jiyang Gao, Zhenheng Yang, and Ram Nevatia.  Cascaded boundary regression for temporal action detection.  arXiv preprint arXiv:1705.01180, 2017.
  * [12] Guoqiang Gong, Xinghan Wang, Yadong Mu, and Qi Tian.  Learning temporal co-attention models for unsupervised video action localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9819–9828, 2020.
  * [13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [14] Yu-Gang Jiang, Jingen Liu, A Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar.  Thumos challenge: Action recognition with a large number of classes, 2014\.
  * [15] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In ECCV, pages 734–750, 2018.
  * [17] Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, and Rongrong Ji.  Fast learning of temporal action proposal via dense boundary generator.  In AAAI, pages 11499–11506, 2020.
  * [18] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  Bmn: Boundary-matching network for temporal action proposal generation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3889–3898, 2019.
  * [19] Tianwei Lin, Xu Zhao, and Zheng Shou.  Single shot temporal action detection.  In Proceedings of the 25th ACM international conference on Multimedia, pages 988–996, 2017.
  * [20] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In ECCV, pages 3–19, 2018.
  * [21] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [22] Qinying Liu and Zilei Wang.  Progressive boundary refinement network for temporal action detection.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11612–11619, 2020.
  * [23] Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei.  Gaussian temporal awareness networks for action localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 344–353, 2019.
  * [24] Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei.  Learning to localize actions from moments.  arXiv preprint arXiv:2008.13705, 2020.
  * [25] Guillaume Lorre, Jaonary Rabarisoa, Astrid Orcesi, Samia Ainouz, and Stephane Canu.  Temporal contrastive pretraining for video action recognition.  In The IEEE Winter Conference on Applications of Computer Vision, pages 662–670, 2020.
  * [26] Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, and Huijuan Xu.  Weakly-supervised action localization with expectation-maximization multi-instance learning.  In European Conference on Computer Vision, pages 729–745. Springer, 2020.
  * [27] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou.  Sf-net: Single-frame supervision for temporal action localization.  In European Conference on Computer Vision, pages 420–437. Springer, 2020.
  * [28] Han Qiu, Yuchen Ma, Zeming Li, Songtao Liu, and Jian Sun.  Borderdet: Border feature for dense object detection.  arXiv preprint arXiv:2007.11056, 2020.
  * [29] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017.
  * [30] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [32] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang.  Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5734–5743, 2017.
  * [33] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1049–1058, 2016.
  * [34] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  arXiv preprint arXiv:1406.2199, 2014.
  * [35] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE international conference on computer vision, pages 9627–9636, 2019.
  * [36] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015.
  * [37] Yuxin Wu and Kaiming He.  Group normalization.  In ECCV, pages 3–19, 2018.
  * [38] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In Proceedings of the IEEE international conference on computer vision, pages 5783–5792, 2017.
  * [39] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-tad: Sub-graph localization for temporal action detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10156–10165, 2020.
  * [40] Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, and Junwei Han.  Revisiting anchor mechanisms for temporal action localization.  IEEE Transactions on Image Processing, 29:8535–8548, 2020.
  * [41] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7094–7103, 2019.
  * [42] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9759–9768, 2020.
  * [43] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian.  Bottom-up temporal action localization with mutual regularization.  In European Conference on Computer Vision, pages 539–555. Springer, 2020.
  * [44] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2914–2923, 2017.


# Integrability of a globally coupled complex Riccati array:
quadratic integrate-and-fire neurons, phase oscillators and all in between

Rok Cestnik<EMAIL_ADDRESS>Centre for Mathematical Science, Lund
University, Sölvegatan 18, 22100, Lund, Sweden  Erik A. Martens  Centre for
Mathematical Science, Lund University, Sölvegatan 18, 22100, Lund, Sweden

###### Abstract

We present an exact dimensionality reduction for dynamics of an arbitrary
array of globally coupled complex-valued Riccati equations. It generalizes the
Watanabe-Strogatz theory [Phys. Rev. Lett. 70, 2391 (1993)] for sinusoidally
coupled phase oscillators and seamlessly includes quadratic integrate-and-fire
neurons as the real-valued special case. This simple formulation reshapes our
understanding of a broad class of coupled systems – including a particular
class of phase-amplitude oscillators – which newly fall under the category of
integrable systems. Precise and rigorous analysis of complex Riccati arrays is
now within reach, paving a way to a deeper understanding of emergent behavior
of collective dynamics in coupled systems.

The study of complex systems often involves describing them using a few
relevant variables known as order parameters. Such dimensionality reduction
techniques are highly useful but challenging to discover, and they may not
exist for every system. In this regard, Watanabe and Strogatz [1] (WS) made
significant progress by demonstrating how globally coupled phase oscillators
can be effectively described using only three order parameters. This reduction
arises from a specific subclass of complex Riccati equations [2, 3] that
govern the dynamics of such oscillatory arrays. The macroscopic variables are
parameters of a Möbius transform that relates the dynamical state variables to
constants determined by the initial states of the oscillator array [4].
Remarkably, Ott and Antonsen [5] showed that for large ensembles of
nonidentical oscillators, the dynamics even collapses to an effectively 2D
manifold. These descriptions have proven immensely useful in studying the
collective dynamics of coupled oscillatory systems [6] and found many
applications, particularly in neuroscience [7, 8].

In this Letter, we present a framework that generalizes these dimensionality
reductions to a larger class of systems: an arbitrary array $j=1,...,N$ of
globally forced complex Riccati equations,

$\dot{x}_{j}=ax_{j}^{2}+bx_{j}+c\;,\qquad a,b,c\in\mathbb{C}\,,$ (1)

where all $a,b,c\in\mathbb{C}$ can be arbitrary complex functions of time and
$x_{j}\in\mathbb{C}$ can start with arbitrary complex values. The choice for
$a,b,c$ allows selecting from a range of coupled oscillator systems. Certain
choices reproduce known models, such as the quadratic integrate-and-fire
neurons (QIF) and phase oscillators; but the possibilities go far beyond that
and cover a broad class of 2D systems.

The flow of array variables $x_{j}$ in (1) is given by the following Möbius
transformation:

$x_{j}=Q+\frac{y\,\xi_{j}}{1+s\,\xi_{j}}\,,$ (2)

where the three global complex-valued variables $Q,y,s$ evolve according to:

$\displaystyle\dot{Q}$ $\displaystyle=aQ^{2}+bQ+c\,,$ (3a)
$\displaystyle\dot{y}$ $\displaystyle=(b+2aQ)y\,,$ (3b) $\displaystyle\dot{s}$
$\displaystyle=-ay\,,$ (3c)

and $\xi_{j}\in\mathbb{C}$ are constants determined by the initial values of
the oscillator array variables 111 There is some freedom in the way
transformation (2) relates original state variables $x_{j}$ with the dynamical
variables $Q,y,s$. One can always consider variables rotated in the complex
plane: $\Phi=e^{-\mathrm{i}\alpha}Q$, $\lambda=e^{-\mathrm{i}\beta}y$, in
which case the transformation is expressed as:
$x_{j}=e^{\mathrm{i}\alpha}\Phi+e^{\mathrm{i}\beta}\lambda\frac{\xi_{j}}{1+s\xi_{j}}$,
and the dynamical equations as:
$\dot{\Phi}=e^{\mathrm{i}\alpha}a\Phi^{2}+b\Phi+e^{-\mathrm{i}\alpha}c$,
$\dot{\lambda}=(b+e^{\mathrm{i}\alpha}2a\Phi)\lambda$,
$\dot{s}=-e^{\mathrm{i}\beta}a\lambda$, cf. Eqs. (31) in [18] where
$\alpha=\frac{\pi}{2}$ and $\beta=-\frac{\pi}{2}$.  (cf. Eqs. (4) in [10] and
Eqs. (15-17) in [11]). Since these equations completely generate the flow of
the original system (1), this implies that the dynamics of Eqs. (1) is
effectively six dimensional. See Supplemental Material [12] for a proof of
this result.

Choosing initial conditions.  Since we started with an array of $N$ variables
$\\{x_{j}\\}$ and now we describe the system with the three macroscopic
variables $Q,y,s$ and $N$ constants $\\{\xi_{j}\\}$, we have some freedom in
choosing the relationship between these variables. An additional constraint of
choice is set on the initial values, that defines the exact relationship
between $x_{j}$ and $Q,y,s,\xi_{j}$. We present two convenient options here.
(I) Identity conversion: a straightforward way to determine initial conditions
is requiring that variables $x_{j}$ initially coincide with constants
$\xi_{j}$ (cf. Eq. (3.7) in [2])

$Q(0)=0\,,\quad y(0)=1\,,\quad s(0)=0\,,\quad\xi_{j}=x_{j}(0)\,.$ (4)

(II) Möbius conversion: in general, the relation between $x_{j}$ and $\xi_{j}$
is a Möbius transform. We here use

$Q(0)=\mathrm{i}\,,\ y(0)=-2\mathrm{i}\,,\ s(0)=1\,,\
\xi_{j}=\frac{\mathrm{i}-x_{j}(0)}{\mathrm{i}+x_{j}(0)}\,.$ (5)

In different situations different initial condition choices are more
appropriate - we will see examples of both options in the later examples.
Other options for initial constraints are possible as well, e.g.,
$\sum_{j}\xi_{j}=0$ (cf. Eq. (4.12) in [2]), see Supplemental Material [12]
for more details.

Special case of real-valued arrays.  We can consider the special case of real
coefficients and real initial conditions: $a,b,c,x_{j}(0)\in\mathbb{R}$.
Consequently, the flow of variables is real-valued, $x_{j}(t)\in\mathbb{R}$
for all $t\geq 0$. The dynamics is then three dimensional, which is easiest to
see with constraint (4) which make $Q,y,s$ initially real, and since they
follow Eqs. (3) with real coefficients, they remain real with evolution. An
example of this special case is a globally coupled array of identical QIF
neurons [13, 14]. Individual voltages $x_{j}(t)$ obey

$\dot{x}_{j}=x_{j}^{2}+I\;,\qquad\text{if }\ x_{j}>x_{\text{thr}}\ \text{ then
}\ x_{j}\mapsto x_{\text{reset}}\,,$ (6)

where the voltage threshold and reset values are: $x_{\text{thr}}=\infty$ and
$x_{\text{reset}}=-\infty$. The current $I$ can have a constant component
$I_{0}$ associated with intrinsic neuronal dynamics, but it can also represent
external forcing (even noisy) or global coupling, e.g., the input current
generated by $N$ globally coupled QIF neurons with pulses $P(u)$ is expressed
as:

$I=I_{0}+\frac{\epsilon}{N}\ \sum_{j=1}^{N}P\left(1/x_{j}\right)\,.$ (7)

Within our formalism, the voltage spikes naturally occur when the denominator
$1+s\xi_{j}$ in (2) crosses zero. As a result, the voltage in that instance
reaches $+\infty$ upon which it is reset to $-\infty$. What has to be
considered, however, is that depending on the chosen initial constraint, $Q$
might also diverge. Indeed, this is the case if one chooses initial conditions
according to (4): $Q(0)=s(0)=y(0)-1=0$ in which case additional resetting of
variables would be needed 222 When $Q$ is diverging $Q\to\infty$, one can
reset the variables according to the initial condition (4) again: $Q\mapsto
0,\ y\mapsto 1,\ s\mapsto 0$ and $\xi_{j}\mapsto x_{j}$. . However, diverging
variables and additional resetting can be avoided by simply choosing the
appropriate initial conditions (5): $Q(0)=\mathrm{i},\ y(0)=-2\mathrm{i},\
s(0)=1$. The constants $\xi_{j}$ then relate to $x_{j}$ via a Möbius
transform: $\xi_{j}=\frac{\mathrm{i}-x_{j}(0)}{\mathrm{i}+x_{j}(0)}$. Since
$x_{j}(0)$ take real values, the constants $\xi_{j}$ are unitary
$|\xi_{j}|=1$, and can thus be described by their argument $\psi_{j}$:
$\xi_{j}=e^{\mathrm{i}\psi_{j}}$.

For this specific case of $a,b,c,x_{j}\in\mathbb{R}$ and initial constraint
(5) the following simplification is true:

$y=-(Q-\bar{Q})s\,,$ (8)

which reduces the dynamical equations (3) to

$\displaystyle\dot{Q}$ $\displaystyle=aQ^{2}+bQ+c\,,$ (9a)
$\displaystyle\dot{\zeta}$
$\displaystyle=-\mathrm{i}a(Q-\bar{Q})=2a\text{Im}[Q]\,,$ (9b)

where $\zeta\in\mathbb{R}$ is the argument of $s=e^{\mathrm{i}\zeta}$. Note
that these dynamics are three dimensional since $Q$ now takes on complex
values (even though $x_{j}\in\mathbb{R}$). The transformation (2) is reduced
to:

$x_{j}\ =\
Q-(Q-\bar{Q})\frac{e^{\mathrm{i}(\psi_{j}+\zeta)}}{1+e^{\mathrm{i}(\psi_{j}+\zeta)}}\
=\ \bar{Q}+\frac{(Q-\bar{Q})}{1+e^{\mathrm{i}(\psi_{j}+\zeta)}}\,.$ (10)

The variable $Q$ remains bounded for all times, no additional resetting is
needed and the spikes occur when the denominator
$1+e^{\mathrm{i}(\psi_{j}+\zeta)}$ crosses 0, i.e., when $\zeta=\pi-\psi_{j}$.
A numerical example of this special case is shown later in Fig. 1. One can
arrive at the same dynamics by transforming the QIF neurons into theta neurons
via the transformation $x_{j}=\tan(\theta_{j}/2)$ and then employing the
Watanabe-Strogatz theory for phase oscillators [1, 2, 4]. For more on how our
formalism relates to the Watanabe-Strogatz theory for phase oscillators see
Supplemental Material [12], which includes Refs. [16, 17].

Such a description of QIF neurons has already been considered in the continuum
limit $N\to\infty$, cf. Eqs. (42) in [18]. If additionally the distribution of
$x_{j}$ variables is Cauchy-Lorenztian, the dynamics further simplifies and
variable $Q$ completely determines the macroscopic state, its real and
imaginary component correlating to the firing rate and mean voltage [19].

Examples.  We now present some specific systems where the dimensionality
reduction can be applied and with numerical simulations validate its
exactness. First (I), a known and relatable example of pulse-coupled real-
valued QIF neurons. It is known that this system possesses low-dimensional
dynamics since the QIF model can be transformed into a theta neuron to which
the WS theory applies. Our formalism provides not only a new perspective of
this fact, but also justifies the voltage resetting at infinity by viewing the
model as a limiting case of the extended complex model. Next, we show two
examples which generalize known models to complex values. Example (II), the
complex generalization of the QIF model, where we simply allow the “voltage”
variables to attain complex values. And example (III) concerns a complex-
valued generalization of phase oscillators, specifically we choose overdamped
Josephson junctions. An additional example is found in Supplemental Material
[12] of how our description applies to infinite ensembles in the thermodynamic
limit, and how particular integrals can simplify with set initial conditions.

Example (I): real QIF model. We consider $N=8$ excitable QIF neurons (6),(7)
with $I_{0}=-0.001$, interacting via Gaussian pulses
$P(u)=\sqrt{\sigma/\pi}\exp(-\sigma u^{2})$ where $\sigma=5$ and coupling
strength $\epsilon=2.3$. Initial conditions are
$\\{x_{j}(0)\\}=\\{-(N-1)/2+j\\}$, $j=1,...,N$. These parameters yield chaotic
dynamics, see Fig. 1. We integrate the reduced three dimensional system using
Eqs. (9) and compare the resulting trajectories with the ones obtained from
$N=8$ coupled voltage equations (6), we see an exact overlap, as expected.

Figure 1: $N=8$ quadratic integrate-and-fire neurons with global coupling
interacting via Gaussian pulses. The dynamics is exactly described by the low
dimensional Eqs. (9). Left panel: time series of the input current $I(t)$,
alongside the individual neurons’ firing events (top). Right panel: trajectory
of the macroscopic variable $Q(t)$ determining voltages via (10).

For ensembles of pure phase oscillators the Kuramoto order parameter is
defined as $Z_{1}=1/N\sum_{j=1}^{N}e^{\mathrm{i}\varphi_{j}}$ where its
amplitude $|Z_{1}|$ quantifies the order in the system. This is easily
generalized to the full complex plane by simply invoking the mean 333Higher
order parameters generalize to higher moments:
$Z_{n}=\frac{1}{N}\sum_{j=1}^{N}x_{j}^{n}$.:

$Z_{1}=\frac{1}{N}\sum_{j=1}^{N}x_{j}\,,$ (11)

which is neatly expressed with dynamical variables $Q,y,s$ and constants
$\xi_{j}$,

$Z_{1}=Q+y\frac{1}{N}\sum_{j=1}^{N}\frac{\xi_{j}}{1+s\xi_{j}}\,.$ (12)

The sum $1/N\sum_{j}\xi_{j}/(1+s\xi_{j})$ can be seen as a function constant
in time, that holds the information of initial conditions and only depends on
the variable $s$.

Example (II): complex QIF model. Let us consider a simple generalization of
the QIF neurons to the complex plane. If the voltages $x_{j}$ start off the
real axis, then they never diverge and there is no need for resetting
conditions in (6). Let us consider such generalized QIF neurons, globally
coupled via the first moment $Z_{1}$ (11),

$\dot{x}_{j}=x_{j}^{2}+I_{0}\ \ +\epsilon\left(Z_{1}-x_{0}^{+}\right)\,,$ (13)

where $I_{0}$ is the intrinsic input current and $x_{0}^{+}$ the positive-
imaginary fixed point of individual neurons (which can be absorbed in the
current $I_{0}\mapsto I_{0}-\epsilon x_{0}^{+}$). In the form of the initial
Riccati equation (1) this model translates to the parameters: $a=1$, $b=0$,
$c=I_{0}+\epsilon(Z_{1}-x_{0}^{+})$. On the real line the behavior of an
individual unit $x_{j}$ tends towards infinity in finite time and hence one
needs a reset rule (6) as well as an implementation of a pulse during that
event. However, if the dynamics occurs instead in the complex plane, the
trajectory of this unit naturally oscillates around a fixed point. The
uncoupled system is time-reversible, invariant under the involution:
$\text{Re}[x_{j}]\mapsto-\text{Re}[x_{j}]$, $t\mapsto-t$, which allows for a
continuum of closed orbits symmetric with respect to the imaginary axis if
$I_{0}>0$, see Supplemental Material for details [12]. Indeed, we find two
center fixed points of the single unit dynamics:
$x_{0}^{\pm}=\pm\mathrm{i}\sqrt{I_{0}}$. When we couple several units
together, they remain oscillatory. In our example we use $N=8$ oscillators
with $I_{0}=1$, $\epsilon=-5$ and initial conditions:
$\\{x_{j}(0)\\}=\\{x_{0}^{+}+\frac{j^{2}}{20}\exp(\mathrm{i}\frac{\pi}{2N}(j-1))\\}$,
$j=1,...,N$. The dynamics settles into periodic motion with a nontrivial limit
cycle, as shown in Fig. 2.

Figure 2: Complex generalization of coupled quadratic integrate-and-fire
neurons (13). Unlike the real QIF model, the voltage resetting in this complex
generalization is redundant since the dynamics everywhere (except the special
case of real line with real coupling) loops back and stays finite; see also
Example (I) in Fig. 1 where voltages diverge, but the resetting naturally
results from the transformation (2). $N=8$ units settle into periodic motion
with a nontrivial limit cycle. Trajectories of oscillators on the limit cycle
in the complex plane are depicted with colored lines. Initial conditions are
marked with (color-coded) points. The fixed point $x_{0}^{+}$ is emphasized
with a black dot.

Example (III): complex generalization of phase oscillators. Now let us
generalize phase oscillators to include a free amplitude. Consider the phase
dynamics equation in complex exponential form:

$\dot{x}_{j}=-\bar{h}x_{j}^{2}+\mathrm{i}\omega x_{j}+h\,,$ (14)

but allow that oscillators have amplitude $r_{j}$ different from 1:
$x_{j}=r_{j}e^{i\varphi_{j}}$. This choice results in a special family of
phase-amplitude oscillators:

$\displaystyle\dot{\varphi}_{j}$
$\displaystyle=\omega-\left(\frac{1}{r_{j}}+r_{j}\right)\text{Im}[he^{-i\varphi_{j}}]\,,$
(15a) $\displaystyle\dot{r}_{j}$
$\displaystyle=\left(1-r_{j}^{2}\right)\text{Re}[he^{-i\varphi_{j}}]\,.$ (15b)

Note that $r_{j}=1$ defines an invariant subspace that oscillators cannot
cross: oscillators that start on the inside of the unit disk stay inside the
disk forever. For this example we consider a complex generalization of coupled
Josephson junctions (cf. Eq. (3.16) in [2]),

$a=-c=0.75\;,\quad b=\mathrm{i}-0.7\mathrm{i}\ \text{Im}[Z_{1}]\,,$ (16)

where $Z_{1}$ is the generalized Kuramoto order parameter (11). We use $N=8$
units with initial conditions:
$\\{x_{j}(0)\\}=\\{-\mathrm{i}\sin(\frac{\pi}{N}j)\exp(\mathrm{i}\frac{2\pi}{N}j)\\}$,
$j=1,...,N$. Just like in the case of pure phase oscillators, for this
parameter choice we observe chaos, see Fig. 3, cf. Fig. 4(c) in [2] and Fig. 2
in [21].

Figure 3: Array of $N=8$ Josephson junctions (16) with complex initial
conditions inside the unit disk (colored points) generalize pure phase
oscillators bound on the unit circle, see Fig. 4(c) in [2] and Fig. 2 in [21].
Trajectory of one oscillator in the complex plane is shown in orange. Small
blue scatter points depict the value of $Z_{1}$ on the Poincaré section where
the oscillator on the unit circle passes phase $\pi/2$.

Discussion. Our study presents a novel low-dimensional description that
generalizes the well-known WS [1, 2] theory to arbitrary arrays of complex
Riccati equations, and so includes the real QIF model as a limiting case. This
exact formalism enables the consideration of a whole new class of complex
oscillatory models, including a special case of phase-amplitude oscillators.
To showcase its correctness and applicability, we provide numerical
simulations for several interesting examples.

The new formalism is effectively six dimensional, as compared to the three
dimensional WS theory – this is not surprising as the generalization from real
$\mathbb{R}$ to complex $\mathbb{C}$ variables implies twice the
dimensionality. However, it should be noted that the generalization does not
simply involve allowing the dynamical variables to take complex values;
rather, the equations we obtain are fundamentally different from those
described by WS theory. What is common with the WS theory is that the
overarching motif is the Möbius transform between initial values of the
dynamical variables $x_{j}$ and the constants $\xi_{j}$. As was explored later
[4], cross-ratios of dynamical variables
$C_{j}=\frac{(x_{j}-x_{j+2})}{(x_{j}-x_{j+3})}\frac{(x_{j+1}-x_{j+3})}{(x_{j+1}-x_{j+2})}$
are invariant under the Möbius transform and thus constants of motion. Where
in the WS context these cross-ratios are real (even though
$x_{j}=e^{\mathrm{i}\varphi_{j}}\in\mathbb{C}$), here they can be complex:
$C_{j}\in\mathbb{C}$. Just as was shown in [4], there are $N-3$ independent
ratios $C_{j}$, $j=1,...,N-3$. Since the initial problem contains $N$ complex
variables (1), and there are $N-3$ complex constants of motion, this leaves
three complex variables to describe dynamics (3), thus confirming that the
description is six dimensional.

For real-phase oscillators in the thermodynamic limit the inclusion of
heterogeneity [5, 22] or noise [23] leads to terms of complex-valued effective
frequencies. Our description (3) is clearly applicable to describe those
scenarios as well, as we have shown in previous work [24, 18]. This remarkable
equivalence between noise and heterogeneity, and complex valued frequencies
can be studied further with our approach.

The generalization to complex numbers is substantial and provides room for
qualitatively different dynamics. A complex extension of the Kuramoto model
has recently been explored in [25]; but our description applies to a much
broader class of coupled systems described by complex Riccati arrays defined
in Eqs. (1). This unlocks a whole spectrum of 2D dynamical systems, including
a special case of phase-amplitude oscillators (15) we showcased here. The
examples considered here are simply complex generalizations of known models:
Example (II) a generalization of QIF and Example (III) a generalization of
phase oscillators. One can explore systems that go beyond just generalizing
known models to complex initial conditions, and really consider complex units
with complex coupling, thus tapping into the rich 2D dynamics of intrinsic
units defined by Riccati equations (1).

Moreover, the new description provides a fresh perspective and insights on the
relationship between phase oscillators and QIF neurons. In fact, the voltage
resetting in QIF neurons arises naturally in our framework, providing
additional motivation for its use in studying neuronal dynamics and
development of new models.

Our findings have significant implications for both theoretical and practical
research. The new description opens up many avenues for investigating the
dynamics of complex systems. Several ideas for future work directly follow
from our framework, and more are expected to arise from the research
community. We briefly outline three ideas here. (A) It is well known that the
general Riccati equation can be transformed into a linear second order ODE
[26], which means that our formalism applies there as well. A more detailed
study will be performed in a forthcoming work. (B) Most likely similar
descriptions exist for higher dimensional systems as well; both for systems
with larger spatial dimensionality [27], $x_{j}\in\ \mathbb{R}^{n}$, $n>1$,
such as the generalizations to vectors [28] and matrices [11], as well as
allowing for states $x_{j}$ that belong to higher number systems, such as
quaternions or octonions. (C) For the special case of phase oscillators and
QIF neurons, in the thermodynamic limit $N\to\infty$ a particular initial
state leads to an additional dynamical reduction as described by Ott and
Antonsen [5]. One would expect that similar reductions can be found for the
more general case of $a,b,c,x_{j}\in\mathbb{C}$. It would be interesting to
know which states allow for additional reductions and what are the reduced
dynamics. (D) Throughout this work we strictly considered identical
oscillators, i.e., at all times every oscillator felt the same global forcings
exerted by $a,b,c$. Thus, the oscillators only differed by their states
$x_{j}$ which are determined by the initial conditions. However, one may
consider adding heterogeneity in the forces by assuming that $a,b,c$ in some
way differ between oscillators. Just like Ott and Antonsen incorporated
Cauchy-Lorentzian inhomogeneities in frequencies [5], one can add them to our
formalism in the thermodynamic limit as well. It is even likely that
particular heterogeneity can be incorporated into finite arrays.

Acknowledgments.  We thank Arkady Pikovsky and Steven Strogatz for useful
discussion and comments. We gratefully acknowledge financial support from the
Royal Swedish Physiographic Society of Lund and the DFG (Grant No. PI
220/21-1).

## References

  * Watanabe and Strogatz [1993] S. Watanabe and S. H. Strogatz, Integrability of a globally coupled oscillator array, Phys. Rev. Lett. 70, 2391 (1993).
  * Watanabe and Strogatz [1994] S. Watanabe and S. H. Strogatz, Constants of motion for superconducting Josephson arrays, Physica D 74, 197 (1994).
  * Pikovsky and Rosenblum [2008] A. Pikovsky and M. Rosenblum, Partially integrable dynamics of hierarchical populations of coupled oscillators, Phys. Rev. Lett 101, 264103 (2008).
  * Marvel  _et al._ [2009] S. A. Marvel, R. E. Mirollo, and S. H. Strogatz, Identical phase oscillators with global sinusoidal coupling evolve by Möbius group action, Chaos 19, 043104 (2009).
  * Ott and Antonsen [2008] E. Ott and T. M. Antonsen, Low dimensional behavior of large systems of globally coupled oscillators, Chaos 18, 037113 (2008).
  * Pikovsky and Rosenblum [2015] A. Pikovsky and M. Rosenblum, Dynamics of globally coupled oscillators: Progress and perspectives, Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 1 (2015), 1504.06747 .
  * Bick  _et al._ [2020] C. Bick, M. Goodfellow, C. R. Laing, and E. A. Martens, Understanding the dynamics of biological and neural oscillator networks through exact mean-field reductions: a review, The Journal of Mathematical Neuroscience 10, 9 (2020).
  * Coombes and Byrne [2018] S. Coombes and A. Byrne, Next generation neural mass models, in  _Nonlinear dynamics in computational neuroscience_ (Springer, 2018) pp. 1–16.
  * Note [1] There is some freedom in the way transformation (2) relates original state variables $x_{j}$ with the dynamical variables $Q,y,s$. One can always consider variables rotated in the complex plane: $\Phi=e^{-\mathrm{i}\alpha}Q$, $\lambda=e^{-\mathrm{i}\beta}y$, in which case the transformation is expressed as: $x_{j}=e^{\mathrm{i}\alpha}\Phi+e^{\mathrm{i}\beta}\lambda\frac{\xi_{j}}{1+s\xi_{j}}$, and the dynamical equations as: $\dot{\Phi}=e^{\mathrm{i}\alpha}a\Phi^{2}+b\Phi+e^{-\mathrm{i}\alpha}c$, $\dot{\lambda}=(b+e^{\mathrm{i}\alpha}2a\Phi)\lambda$, $\dot{s}=-e^{\mathrm{i}\beta}a\lambda$, cf. Eqs. (31) in [18] where $\alpha=\frac{\pi}{2}$ and $\beta=-\frac{\pi}{2}$.
  * Goebel [1995] C. J. Goebel, Comment on “Constants of motion for superconductor arrays”, Physica D: Nonlinear Phenomena 80, 18 (1995).
  * Lohe [2019] M. A. Lohe, Systems of matrix Riccati equations, linear fractional transformations, partial integrability and synchronization, Journal of Mathematical Physics 60, 10.1063/1.5085248 (2019).
  * [12] See Supplemental Material at https://arxiv.org/src/2305.17683v4/anc/supplemental.pdf for a proof showing equivalence of Eqs. (1) and Eqs. (3), the relation of our formalism with the Watanabe-Strogatz theory, and an additional example of how our description applies to infinite ensembles in the thermodynamic limit.
  * Ermentrout and Kopell [1986] G. B. Ermentrout and N. Kopell, Parabolic bursting in an excitable system coupled with a slow oscillation, SIAM Journal on Applied Mathematics 46, 233 (1986).
  * Izhikevich [2007] E. M. Izhikevich, Dynamical systems in neuroscience 10.7551/mitpress/2526.001.0001 (2007).
  * Note [2] When $Q$ is diverging $Q\to\infty$, one can reset the variables according to the initial condition (4) again: $Q\mapstochar\rightarrow 0,\ y\mapstochar\rightarrow 1,\ s\mapstochar\rightarrow 0$ and $\xi_{j}\mapstochar\rightarrow x_{j}$.
  * Pikovsky and Rosenblum [2011] A. Pikovsky and M. Rosenblum, Dynamics of heterogeneous oscillator ensembles in terms of collective variables, Physica D 240, 872 (2011).
  * Strogatz [2018] S. H. Strogatz,  _Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering_ (CRC press, 2018).
  * Pietras  _et al._ [2023] B. Pietras, R. Cestnik, and A. Pikovsky, Exact finite-dimensional description for networks of globally coupled spiking neurons, Phys. Rev. E 107, 024315 (2023).
  * Montbrió  _et al._ [2015] E. Montbrió, D. Pazó, and A. Roxin, Macroscopic Description for Networks of Spiking Neurons, Physical Review X 5, 021028 (2015), 1506.06581 .
  * Note [3] Higher order parameters generalize to higher moments: $Z_{n}=\frac{1}{N}\sum@\slimits@_{j=1}^{N}x_{j}^{n}$.
  * Cestnik and Pikovsky [2022a] R. Cestnik and A. Pikovsky, Hierarchy of exact low-dimensional reductions for populations of coupled oscillators, Phys. Rev. Lett. 128, 054101 (2022a).
  * Skardal [2018] P. S. Skardal, Low-dimensional dynamics of the Kuramoto model with rational frequency distributions, Phys. Rev. E 98, 022207 (2018).
  * Tönjes and Pikovsky [2020] R. Tönjes and A. Pikovsky, Low-dimensional description for ensembles of identical phase oscillators subject to cauchy noise, Phys. Rev. E 102, 052315 (2020).
  * Cestnik and Pikovsky [2022b] R. Cestnik and A. Pikovsky, Exact finite-dimensional reduction for a population of noisy oscillators and its link to Ott-Antonsen and Watanabe-Strogatz theories, Chaos 32, 113126 (2022b).
  * Thümler  _et al._ [2023] M. Thümler, S. G. M. Srinivas, M. Schröder, and M. Timme, Synchrony for weak coupling in the complexified kuramoto model, Phys. Rev. Lett. 130, 187201 (2023).
  * Ince [1956] E. Ince,  _Ordinary differential equations_ (Courier Corporation, 1956).
  * Chandra  _et al._ [2019] S. Chandra, M. Girvan, and E. Ott, Continuous versus discontinuous transitions in the D-dimensional generalized Kuramoto model: Odd $D$ is different, Physical Review X 9, 011002 (2019).
  * Lohe [2018] M. A. Lohe, Higher-dimensional generalizations of the Watanabe–Strogatz transform for vector models of synchronization, Journal of Physics A: Mathematical and Theoretical 51, 225101 (2018).