# Multi-Task Consistency for Active Learning

Aral Hekimoglu
Technical University of Munich
<EMAIL_ADDRESS>Philipp Friedrich
BMW Group
<EMAIL_ADDRESS>Walter Zimmer
Technical University of Munich
<EMAIL_ADDRESS>Michael Schmidt
BMW Group
<EMAIL_ADDRESS>Alvaro Marcos-Ramiro
BMW Group
<EMAIL_ADDRESS>Alois Knoll
Technical University of Munich
<EMAIL_ADDRESS>
###### Abstract

Learning-based solutions for vision tasks require a large amount of labeled
training data to ensure their performance and reliability. In single-task
vision-based settings, inconsistency-based active learning has proven to be
effective in selecting informative samples for annotation. However, there is a
lack of research exploiting the inconsistency between multiple tasks in multi-
task networks. To address this gap, we propose a novel multi-task active
learning strategy for two coupled vision tasks: object detection and semantic
segmentation. Our approach leverages the inconsistency between them to
identify informative samples across both tasks. We propose three constraints
that specify how the tasks are coupled and introduce a method for determining
the pixels belonging to the object detected by a bounding box, to later
quantify the constraints as inconsistency scores. To evaluate the
effectiveness of our approach, we establish multiple baselines for multi-task
active learning and introduce a new metric, mean Detection Segmentation
Quality (mDSQ), tailored for the multi-task active learning comparison that
addresses the performance of both tasks. We conduct extensive experiments on
the nuImages and A9 datasets, demonstrating that our approach outperforms
existing state-of-the-art methods by up to 3.4% mDSQ on nuImages. Our approach
achieves 95% of the fully-trained performance using only 67% of the available
data, corresponding to 20% fewer labels compared to random selection and 5%
fewer labels compared to state-of-the-art selection strategy. Our code will be
made publicly available after the review process.

##  I Introduction

Object localization and classification are critical for planning and executing
safe and comfortable autonomous driving. Recent deep learning methods have
demonstrated state-of-the-art (SOTA) performance on 2D object detection [1, 2,
3] and semantic segmentation [4, 5] tasks. However, achieving high accuracy in
these tasks comes at a high computational cost when handled separately, making
them unsuitable to be used together for real-time autonomous driving. To
address this challenge, multi-task learning has emerged as a promising
solution. By sharing computations between related tasks, multi-task learning
can achieve high accuracy while meeting real-time requirements. Recent
publications showed that networks that predict both 2D object detection, and
pixel-wise semantic segmentation perform better on both tasks compared to the
single-task trained networks [6, 7]. In this paper, we focus on the problem of
multi-task active learning in autonomous driving, aiming to maximize
performance across multiple tasks while minimizing the need for large amounts
of labeled training data.

Figure 1: Consistency constraints between object detection and semantic
segmentation. (a) Mask covers all pixels of a detected object. (b) A detected
object and the segmentation mask that covers it share the same predicted class
distribution. (c) No pixels outside of the detected boxes are segmented with
an object class. Figure 2: The proposed inconsistency-based selection
strategy. Detection boxes and the segmentation mask are obtained from a multi-
task network architecture consisting of a backbone and two task heads. The
BoxMask is generated by cropping the region around the detected box, passing
it through the backbone and segmentation head, and applying a threshold $\tau$
to the class probability corresponding to the class of the detected box. Our
proposed strategy focuses on three inconsistency scores by using the BoxMask
and segmentation mask.

Active learning (AL) [8, 9] is a technique for selecting the most informative
samples for training a machine learning model under labeling budget
constraints. In a typical single-task active learning loop, the model’s
predictions on the remaining unlabeled data are used to identify the samples
that would be most beneficial for further training. In vision-based single-
task settings, AL has been used to improve performance in tasks such as image
classification [10, 11], object detection [12, 13, 14, 15, 16, 17], and
semantic segmentation [18, 19]. One effective approach is inconsistency-based
selection, which identifies samples for annotation by measuring the
inconsistency of the model’s predictions across different augmentations of the
input data. For instance, CALD [20] explores various augmentations for object
detection, and EquAL [19] measures inconsistency between an image and its
flipped version for semantic segmentation. However, to our knowledge, no
existing work has explored the use of inconsistency between multiple tasks in
the context of multi-task active learning.

Our novel strategy incorporates the concept of inconsistency-based selection
from active learning and applies it to multi-task learning by leveraging the
inconsistency between two coupled vision tasks, namely 2D object detection and
semantic segmentation. Our approach quantifies the inconsistency between these
two tasks to identify informative samples across both of them to maximize the
performance while minimizing the amount of labeled data needed for training.

To measure the inconsistency between 2D object detection and semantic
segmentation, we define three constraints that specify how the two tasks are
coupled together. The first constraint requires that the segmentation mask
covers all pixels of the detected objects (Fig. 1a). The second constraint
requires that a detected object and the segmentation mask that covers it share
the same predicted class distribution (Fig. 1b). The third constraint requires
that no pixels outside the detected boxes are segmented with an object-class
(Fig. 1c). To map the object detection predictions to a similar pixel-wise
output as the semantic segmentation predictions, we define BoxMask to identify
the pixels belonging to the object detected by a bounding box. BoxMask enables
us to quantify the three constraints by measuring the overlap between the
object detection and semantic segmentation predictions. Based on these
constraints, we propose three scores utilizing the BoxMask that quantify the
inconsistency between the two coupled tasks.

Our main contributions are the following:

  * •

A novel multi-task active learning strategy that effectively leverages the
inconsistency between 2D object detection and semantic segmentation to improve
performance on both tasks and reduce the amount of labeled data needed for
training.

  * •

A novel method for identifying the pixels belonging to a detected object
(BoxMask) and using it to quantify the constraints between two tasks into
selection scores.

  * •

A comprehensive qualitative and quantitative comparison of the proposed
approach and multi-task active learning baselines against SOTA baselines that
are outperformed by up to 3.4% mDSQ and 5% in data savings rate.

##  II Related work

###  II-A Active learning

AL methods for object detection measure uncertainty of a detected box through
either classification or localization uncertainty [21, 22, 8, 23, 24, 16].
Recent methods [8, 20] leverage inconsistency between the predictions of the
network when given different augmented versions of the sample to define the
robustness of a sample and select samples that are less robust. For example,
Elezi et al. [8] use horizontal flipping, while Yu et al. [20] explore various
augmentations to obtain multiple outputs and measure the inconsistency to
define a selection score. Once the uncertainty of an object is estimated, the
scores of all detections are aggregated using either the sum, the average, or
the maximum of the scores, and the resulting image score is used to rank the
images for annotation.

AL methods for semantic segmentation also utilize inconsistency methods to
define uncertainty [19, 25, 26, 18]. For instance, Golestaneh et al. [19]
apply horizontal flipping to the image and measure the inconsistency through
the KL-divergence of the predictions from the original and the flipped image.
The unit of data queried in AL methods for semantic segmentation varies, with
methods querying either whole images [19, 27] or regions [25, 28, 26, 29, 30].
In our work, we choose to query whole images for labeling since we are also
interested in obtaining object detection labels. Notably, our approach is the
first to utilize inconsistency between predictions of different tasks to
define multi-task uncertainty and use it for AL selection.

Learning Loss is a task-agnostic strategy proposed by Yoo et al.[11], using a
loss prediction module. The network learns to predict the target loss for
unlabeled inputs, and samples with the highest predicted loss are selected for
labeling. As this approach is task-agnostic, it can be adapted for multi-task
networks and serves as a relevant comparison in our work.

Diversity-based methods aim to ensure a diverse training set that covers the
input space. One such method is the use of a core-set, as proposed by Sener et
al. [31], where diversity is defined as the Euclidean distance between
intermediate network features for each image. Another method, CDAL, proposed
by Agarwal et al. [32], exploits contextual diversity with respect to the
predicted classes, and has been applied to object detection and semantic
segmentation. These methods are applicable to the multi-task active learning
scenario and provide additional baselines for comparison.

###  II-B Multi-task learning

Multi-task learning has been studied extensively, and readers are referred to
a survey by Crawshaw et al.[33]. In multi-task architectures, the hidden
layers of a backbone model are shared among different tasks, and have separate
heads that predict each task [34, 35, 36]. Multi-task optimization deals with
the joint objective function in multi-task settings, such as how to weigh
losses of individual loss functions for different tasks [37, 38, 35, 39].
Kendall et al.[38] explored weighting each loss by its corresponding single-
task uncertainty, using homoscedastic uncertainty for weighing the multi-task
loss.

The idea of joint semantic segmentation and object detection was first
investigated for shallow networks in [40, 41, 42]. These studies demonstrated
that learning both tasks simultaneously can be better than learning them
independently. Salscheider et al.[43] employed a shared backbone and heads in
their work, and we adopted this approach as our multi-task network.

###  II-C Multi-task active learning

To our knowledge, no prior research has investigated multi-task active
learning (MTAL) for object detection and semantic segmentation. However,
active learning has been successfully applied to multi-task settings in other
domains, such as Natural Language Processing (NLP). Reichart and Rappoport
[44] proposed alternating two single-task focused data selection strategies in
each cycle, while Ikhwantri et al. [45] randomly selected a task for each
cycle. Instead of alternating between two single-task scores, we propose a
novel approach that generates a single score for selecting interesting samples
relevant to both tasks.

##  III Methodology

###  III-A MTAL problem overview

The goal of this work is to tackle the AL problem of iteratively selecting
samples from a large pool of unlabeled data $X^{U}$ to be labeled by an
oracle, to improve the performance of a multi-task object detection and
semantic segmentation network. Specifically, we consider each sample
$(x,y_{det},y_{seg})$ as a triplet, where $x$ is an image, $y_{det}$ is the
set of objects, and $y_{seg}$ is the pixel-wise segmentation label belonging
to one of the semantic classes $C_{seg}$. Detection labels $y_{det}$ consists
of bounding box coordinates $(y_{box})$ and corresponding categories
$(y_{cls})$ belonging to one of the object classes $C_{det}$, where
$C_{det}\subseteq C_{seg}$.

Our multi-task network consists of a shared backbone and two single-task
heads, as shown in Fig. 2. The network predicts object boxes $p_{det}$
consisting of $(p_{box},p_{cls})$ and a segmentation mask $p_{seg}$ for each
input image $x$. In each AL cycle, the network is trained on the labeled data
$(X^{L},Y^{L})$, and a subset $S$ of unlabeled samples is selected for
labeling. In the next cycle, the selected samples are added to the labeled
dataset, and the network is trained again on the updated labeled dataset.

###  III-B Method overview

Object detection and semantic segmentation are two interconnected tasks shown
to benefit from each other when combined [6]. The predictions from both tasks
are inherently coupled, as objects detected in the former should align with
the labeled regions in the latter. We propose an AL selection strategy that
identifies samples where either task fails. To achieve this, we measure the
inconsistency between the predictions of object detection and semantic
segmentation. These inconsistent areas indicate potential points of failure
for both tasks and, as such, are deemed interesting for further labeling.

To this end, we define three constraints between the tasks to formulate a
selection score as illustrated in Fig. 2.

  1. 1.

The segmentation mask should cover all pixels belonging to the detected
object, ensuring that the entire object is accurately segmented for the given
class. (Sec. III-D)

  2. 2.

The segmentation mask and the detected object should have consistent class
distributions. (Sec. III-E)

  3. 3.

There should be no segmented pixel belonging to a class from the object
detection outside the predicted bounding boxes. (Sec. III-F)

###  III-C BoxMask generation strategy

To quantify the constraints, we define a binary segmentation mask, BoxMask,
that covers all pixels within a detected box belonging to the class of the
detected object. A perfect BoxMask covers each pixel of the entire object of
interest. Fig. 2 illustrates our BoxMask generation strategy. We begin by
cropping a region of the image around the detected box, and then pass it
through the network. Using the segmentation head of our multi-task network, we
generate a segmentation mask for the cropped region $p_{seg}(x_{crop})$.
BoxMask is then defined as a binary mask where the predicted class probability
in the new segmentation label is above a threshold $\tau$ for the class of the
detected object.

###  III-D Localization consistency

Our localization-focused inconsistency score measures the alignment between a
detected object and its corresponding segmentation mask. To ensure consistent
detection and segmentation, the segmentation mask within the detected box
should cover the entire detected object without any missing regions.

We define the localization consistency score as the number of pixels in the
predicted segmentation mask that match the corresponding pixels in the BoxMask
for that detection. To account for varying object sizes and scales, we
normalize the score by $|BM|$, the number of pixels in the BoxMask, resulting
in a scale-invariant score. Mathematically, the localization consistency score
is given by the following equation:

$S_{loc}=\frac{1}{|BM|}\sum_{i,j\in BM}{\mathbb{I}{(p_{seg}(i,j)==c)}}$ (1)

where, $\mathbb{I}$ represents the indicator function, $c$ denotes the
predicted class of the detected object, and $i,j$ represent the pixel
coordinates in the BoxMask ($BM$).

###  III-E Classification consistency

In Sec. III-D, we addressed the localization inconsistency between the BoxMask
and the segmentation mask. However, this approach treats classes that are very
different, such as Truck and Pedestrian, the same as classes that are more
similar, such as Truck and Bus. To account for this, we propose a
classification inconsistency score that considers the predicted class
probabilities during the sample score calculation.

To achieve this, we transform the predicted object class distribution
$p_{cls}$ into the same probability domain as the class probabilities
predicted by semantic segmentation. Specifically, we set the probability of
the classes not trained in object detection, i.e., $C_{seg}-C_{det}$, such as
Road and Sky, to a negligible value to ensure that both tasks have the same
number of classes, without impacting the score calculation. Since pixels
within BoxMask should not contain any classes from $C_{seg}-C_{det}$, we
consider this a valid assumption. The transformed probability vector, denoted
as $\tilde{p}_{cls}$, contains the same classes as the segmentation task
($C_{seg}$).

The classification inconsistency score, $S_{cls}$, for a sample is calculated
using the transformed object class probability distribution $\tilde{p}_{cls}$
and the class probability distribution of the segmentation $p_{seg}$ for each
pixel $i,j$ in BoxMask as follows:

$S_{cls}=\frac{1}{2|BM|}\sum_{i,j\in
BM}KL\big{(}p_{seg}(i,j),\tilde{p}_{cls}\big{)}+\\\
KL\big{(}\tilde{p}_{cls},p_{seg}(i,j)\big{)}$ (2)

where $KL$ represents the Kullback-Leibler divergence between the predicted
probability distributions, and the resulting value is averaged to obtain a
similarity measure.

###  III-F Segmentation consistency

In Sec. III-D and III-E, we addressed inconsistencies within the boundaries of
detected boxes. However, another constraint that must be met between the two
tasks is that there should be no segmentation mask outside the boundaries of
the detected box for a class that also belongs to the set of classes predicted
in the object detection $C_{det}$.

To account for the constraint that outside of the detected boxes, there should
be no segmentation mask for classes predicted in object detection, we combine
all BoxMasks by taking the pixel-wise maximum and taking the inverse of the
resulting binary mask to define the region falling outside of the detected
objects, which we denote as $BM^{\prime}$. The inconsistency score for the
remaining segmented areas $S_{seg}$ is calculated using Eq. 3:

$S_{seg}=\frac{1}{|BM^{\prime}|}\sum_{i,j\in
BM^{\prime}}{\mathbb{I}{(p_{seg}(i,j)\in C_{det})}}$ (3)

where $\mathbb{I}$ is the indicator function. For each pixel falling within
the inverted BoxMask region, we penalize the class probabilities for the
classes predicted by object detection. The resulting score is normalized by
the number of pixels, ensuring it is scale-invariant and shares the same range
as the other inconsistency scores.

###  III-G Combination of all constraints

The pseudo-code of combining the individual constraint scores into a single
inconsistency score between the two tasks is given in Algorithm 1. We first
calculate each detected box’s BoxMask as described in Sec. III-C. Then, we
calculate the localization and classification consistency scores using Eq. 1
and Eq. 2, respectively. We then add the localization and classification
scores to obtain a per-box consistency score $S^{box}$. Next, as explained in
Sec. III-F, we combine all BoxMasks and traverse the inverse region to search
for segmentation pixels belonging to classes from the object detection and
calculate the segmentation inconsistency using Eq. 3. Finally, we add this
score with the maximum per-box score to estimate the inconsistency between two
tasks as a single selection score.

Algorithm 1 The pseudo-code of combining the constraints

1:$p_{det}$, $p_{seg}$

2:score S

3:$BM^{comb}=\\{\\}$

4:for $box\in p_{det}$ do

5: Obtain $BM^{box}$ explained in Sec. III-C

6: Compute $S_{loc}$ using Eq. 1

7: Compute $S_{cls}$ using Eq. 2

8: $S^{box}=S_{loc}+S_{cls}$

9: $BM^{comb}=pixelwise\\_max(BM^{comb},BM^{box})$

10:end for

11:$BoxMask^{\prime}=inverse(BM^{comb})$

12:Compute $S_{seg}$ using Eq. 3

13:$S=S_{seg}+\underset{box\in p_{det}}{\max}(S^{box})$

##  IV Experiments

((a)) Multi-task performance (mDSQ)

((b)) Detection performance (mAP)

((c)) Segmentation performance (mIoU)

Figure 3: Comparison of our proposed method with SOTA AL methods on the
nuImages dataset. Lines indicate the average results over three trials. Note
that all the methods start with the same network trained with 40% of labeled
samples.

((a)) Multi-task performance (mDSQ)

((b)) Detection performance (mAP)

((c)) Segmentation performance (mIoU)

Figure 4: Comparison of our proposed method with SOTA AL methods on the A9
dataset. Lines indicate the average results over three trials. Note that all
the methods start with the same network trained with 40% of labeled samples.

###  IV-A Experimental setup

Datasets. We evaluate the performance of our approach on two publicly
available datasets: nuImages [46] and the A9-Dataset [47]. The nuImages
dataset provides 3D and 2D sensor data collected from autonomous vehicles
operating in urban settings. We use images taken from the front camera,
resulting in a training set size of 13,187 images and 3,249 images for the
validation set with a total of 138,569 objects. The first release of the A9
dataset offers camera and LiDAR frames from two overhead gantry bridges on the
A9 autobahn near Munich, Germany. It provides annotations for object detection
and semantic segmentation with 33,378 labeled image frames and a total of
672,049 3D and 2D object labels. The A9 dataset was created using the proAnno
labeling toolbox which is based on [48].

Implementation details. We employ the multi-task network architecture proposed
by Salscheider et al.[43], augmented with the loss prediction module and the
active learning framework. The hyperparameters proposed in the original work
are used. The experiments are performed using a batch size of 4 and a learning
rate of 0.001.

To perform active learning, we randomly divide the training set into a labeled
pool of 40% and an unlabeled pool of 60%. The initial labeled pool is used to
pre-train the network, and at each iteration, the top 10% of the samples with
the highest scores are selected from the unlabeled pool to add to the labeled
pool, based on the available annotations. We perform six active learning
iterations of 30,000 steps per iteration for each dataset. We employ a
continuous training strategy, where each active learning iteration is
initialized with the best-performing checkpoint from the previous iteration.
All experiments are conducted using two Tesla V100 GPUs and evaluated on the
respective validation sets.

Evaluation metrics. The evaluation metrics for object detection and semantic
segmentation are typically measured using mean Average Precision (mAP) [49]
and mean Intersection-over-Union (mIoU) [50], respectively. However, a new
metric that can capture the performance of both tasks is necessary to evaluate
the performance of multi-task active learning methods. Therefore, we propose
the mean Detection Segmentation Quality (mDSQ) metric, which normalizes mAP
and mIoU by the performance of the fully-trained network and combines them, as
shown in Eq. 4.

$mDSQ=(\frac{mAP}{mAP_{fully}}+\frac{mIoU}{mIoU_{fully}})/2$ (4)

where $mAP_{fully}$ and $mIoU_{fully}$ represent the performance of the
network trained with 100% of data for 300,000 steps. This metric is more
suitable for comparing multi-task active learning methods than the individual
metrics used in each task, as it combines both metrics into a single score
normalized by the fully-trained performance.

We evaluate the performance of our method using the mDSQ metric and report the
mean of the metric by running three experiments with three random initial data
pools. We present each experiment’s numerical values and variance in the
supplementary.

###  IV-B Baselines

To compare the effectiveness of our multi-task active learning method, we
compare it against several baselines from the literature. We choose two
inconsistency-based AL methods from the literature: we use CALD [20] as the
SOTA method for object detection, and EquAL [19] for the semantic
segmentation. We also compare against the alternating selection strategy,
Alternation, proposed by Reichart and Rappoport [44], and alternate between
two SOTA detection and segmentation selection scores CALD and EquAL. Due to
its task-agnostic nature, we also compare our method against the loss
prediction strategy, LL4AL, proposed by Yoo et al.[11]. We extend the network
architecture by two loss prediction modules that learn to predict the loss of
each task. The loss of both tasks is then summed together to form a combined
loss score. We use CDAL [32] as our diversity-based baseline, and to mimic
passive learning, we use Random selection, where each sample is assigned a
score following a uniform distribution.

###  IV-C Qualitative results

We present qualitative results to compare the final performances obtained by
following different selection strategies in Fig. 5. Compared to the CALD
baseline, our method provides more fitting segmentation masks within each
detection. For example, for the bottom vehicle in the last row, our method
correctly segments the regions as Car instead of Truck, demonstrating the
effectiveness of using class inconsistency. Additionally, our method produces
more accurate localization for the left front wheel, which is another
constraint we set in our method. Finally, the pixels outside of the boxes are
worse than our method, demonstrating the effectiveness of the segmentation
constraint. Overall, our qualitative comparison shows that our method
outperforms the baselines in producing accurate and consistent object
detection and segmentation results.

Figure 5: Qualitative comparison of the Random (second), CALD (third), and Our
(fourth) sampling strategies on the A9 dataset. Light blue and dark blue
correspond to the Truck and Car classes.

Score | 50% | 70% | 90%
---|---|---|---
$S_{loc}$ | 86.9 | 94.0 | 96.4
$S_{cls}$ | 86.5 | 93.5 | 96.6
$S_{seg}$ | 87.1 | 94.4 | 96.9
$S_{loc}+S_{cls}$ | 87.4 | 93.7 | 96.6
$S_{cls}+S_{seg}$ | 87.0 | 95.0 | 97.0
$S_{loc}+S_{seg}$ | 87.7 | 94.6 | 97.5
$S_{loc}+S_{cls}+S_{seg}$ | 88.0 | 95.6 | 97.3

TABLE I: Ablation study of the contribution of each scoring constraint for
each amount of used data on nuImages.

###  IV-D Quantitative results

nuImages. Our results on the nuImages dataset are presented in Fig. 3, which
shows the mDSQ, mAP, and mIoU metrics. In the initial AL cycle, our method
outperforms all the baselines by at least 0.71%. As the number of actively
selected labels increases, for example, using 80% of all available data, with
50% actively labeled, our method outperforms Random by 3.39% and the second-
best method, LL4AL, by 1.07%.

Our approach reaches 95% of the fully-trained performance using only 67% of
the data, compared to 74% of LL4AL and 87% of random selection, corresponding
to 20% of more data savings. We observe that both of the multi-task selection
scores (Ours, LL4AL) outperform the single-task scores, their alternation and
the diversity-based method. This demonstrates that a score that considers both
tasks is more suitable for multi-task networks, compared to alternating
between single-task scores as previously done.

Regarding single-task performance, as shown in Fig. 3(b) and Fig. 3(c), our
method is on par with the SOTA detection algorithm CALD and even outperforms
it as the number of actively selected samples increases. For semantic
segmentation, our method outperforms the SOTA segmentation algorithm EquAL.
These results demonstrate that both tasks benefit from inconsistency
information from the other task.

A9. We present the mDSQ, mAP, and mIoU metrics for the A9 dataset in Fig. 4.
Our method outperforms all the baselines for all data percentages,
demonstrating the effectiveness of our data selection strategy in a larger
dataset. Our selection strategy has the highest performance in the first
cycle, leading to the best performance throughout the remaining cycles. We
achieve 95% mDSQ using only 66% of the data, which means a 34% savings in
labeling budget compared to full training.

For single-task detection performance (Fig. 4(b)), our method is on par with
the Alternation baseline. For semantic segmentation (Fig. 4(c)), our method is
on par with the SOTA segmentation algorithm EquAL. These results demonstrate
that our multi-task approach can achieve comparable or better performance than
state-of-the-art single-task active learning methods for both object detection
and semantic segmentation.

###  IV-E Ablation studies

Ablation on each component. We perform an ablation study to evaluate the
contribution of each scoring constraint function to the overall performance of
our method. The results are shown in Tab. I. Among the single-score versions,
$S_{seg}$ has the highest performance, indicating the importance of avoiding
segmented pixels outside the bounding boxes. $S_{loc}$ and $S_{cls}$ have
comparable results, as they are both focused on different types of
inconsistency, namely classification, and localization. We observe that the
two combinations lead to better performance compared to their single-score
counterparts. The best performance is achieved when we combine all three
scores, as shown in the last row of the table. These results demonstrate the
effectiveness of each constraint and the importance of combining them to
achieve optimal performance.

Figure 6: Correlation of each constraint and losses

Analysis of BoxMask accuracy and threshold. The accuracy of the BoxMask is a
crucial factor in obtaining accurate constraints in our method. We conduct an
experiment to evaluate the accuracy of the BoxMask and compared it with using
a separate network trained solely for semantic segmentation. We calculate the
mIoU between the ground-truth binary segmentation label and the predict
segmentation mask. We are only interested in the areas in the ground-truth
segmentation that belong to the class from the detected box and only for the
area bounded by the detected boxes. Based on our results, we select a
threshold value $\tau$ of 0.3 for BoxMask generation. Even though our method
does not have any additional parameters, it still performs comparably to using
a separate network. Therefore, we use the same segmentation head from our
multi-task network for generating the BoxMask predictions.

Correlation of each consistency score with the actual error. We analyze the
correlation of each consistency score with respect to the losses and each
other, presented in Fig. 6. We observe that $S_{seg}$ is better at measuring
segmentation error, while $S_{cls}$ and $S_{loc}$ are the most effective at
measuring classification and localization error in the detected boxes,
respectively. Since these three losses are the main components in a joint
detection and segmentation loss function, all three constraints effectively
capture areas where the individual losses are high. We also observe that
$S_{loc}$ and $S_{seg}$ have the lowest correlation across the selection
scores, which explains the highest performance when combined in Tab. I.

Mask Gen. | $\tau$ | Add Mem. | Accuracy
---|---|---|---
BoxMask | 0.1 | - | 74.7
0.3 | - | 78.4
0.5 | - | 77.9
0.7 | - | 73.2
HR-Net | - | 6.31 GB | 80.2

TABLE II: Performance of BoxMask across different thresholds.

Single-task LL4AL and alternation. To support our hypothesis that multi-task
active learning is more effective in dealing with multiple tasks
simultaneously, we compare the performance of the LL4AL strategy when using
two single-task scores and one multi-task score. As shown in Fig. 7, using a
single multi-task score outperforms the single-task scores and their
alternation. This suggests that a score considering both tasks is more
suitable for multi-task networks than alternating between single-task scores.

Figure 7: Comparison between multi-task LL4AL to single-task losses and their
alternation. Seg and Det refer to LL4AL using only the segmentation and
detection losses, respectively. All methods start with the same network
trained with 40% of labeled data.

##  V Conclusion

This study addressed the gap in research on active learning for multi-task
networks in the vision domain. Our proposed selection strategy combines
knowledge from the two task domains, object detection and semantic
segmentation, into a single multi-task selection score. This strategy relies
on three constraints between the two tasks and measures them by identifying
the pixels belonging to a detected object through the BoxMask. Our experiments
on two multi-task datasets demonstrate the effectiveness of our approach, as
it outperforms all the baselines by 3.4% and achieves 5% fewer annotations.
Future work will focus on adapting our task inconsistency-based selection
strategy to other multi-task networks.

## References

  * [1] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong _et al._ , “Swin transformer v2: Scaling up capacity and resolution,” in _CVPR_ , 2022.
  * [2] M. Xu, Z. Zhang, H. Hu, J. Wang, L. Wang, F. Wei, X. Bai, and Z. Liu, “End-to-end semi-supervised object detection with soft teacher,” in _ICCV_ , 2021.
  * [3] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _ICCV_ , 2019.
  * [4] Y. Yuan, X. Chen, and J. Wang, “Object-contextual representations for semantic segmentation,” in _ECCV_ , 2020.
  * [5] B. Cheng, M. D. Collins, Y. Zhu, T. Liu, T. S. Huang, H. Adam, and L.-C. Chen, “Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” in _CVPR_ , 2020.
  * [6] N. Dvornik, K. Shmelkov, J. Mairal, and C. Schmid, “Blitznet: A real-time deep network for scene understanding,” in _ICCV_ , 2017.
  * [7] N. Ebert, P. Mangat, and O. Wasenmuller, “Multitask network for joint object detection, semantic segmentation and human pose estimation in vehicle occupancy monitoring,” in _IV_ , 2022.
  * [8] I. Elezi, Z. Yu, A. Anandkumar, L. Leal-Taixe, and J. M. Alvarez, “Not all labels are equal: Rationalizing the labeling costs for training object detection,” in _CVPR_ , 2022.
  * [9] P. Colling, L. Roese-Koerner, H. Gottschalk, and M. Rottmann, “Metabox+: A new region based active learning method for semantic segmentation using priority maps,” in _ICPRAM_ , 2020.
  * [10] W. H. Beluch, T. Genewein, A. Nürnberger, and J. M. Köhler, “The power of ensembles for active learning in image classification,” in _CVPR_ , 2018\.
  * [11] D. Yoo and I. S. Kweon, “Learning loss for active learning,” in _CVPR_ , 2019\.
  * [12] S. Roy, A. Unmesh, and V. P. Namboodiri, “Deep active learning for object detection.” in _BMVC_ , 2018.
  * [13] S. V. Desai, A. L. Chandra, W. Guo, S. Ninomiya, and V. N. Balasubramanian, “An adaptive supervision framework for active learning in object detection,” in _BMVC_ , 2019.
  * [14] F. Tang, D. Wei, C. Jiang, H. Xu, A. Zhang, W. Zhang, H. Lu, and C. Xu, “Towards dynamic and scalable active learning with neural architecture adaption for object detection,” _BMVC_ , 2021.
  * [15] Y. Li, B. Fan, W. Zhang, W. Ding, and J. Yin, “Deep active learning for object detection,” _Information Sciences_ , 2021.
  * [16] J. Choi, I. Elezi, H.-J. Lee, C. Farabet, and J. M. Alvarez, “Active learning for deep object detection via probabilistic modeling,” in _ICCV_ , 2021.
  * [17] A. Hekimoglu, M. Schmidt, A. Marcos-Ramiro, and G. Rigoll, “Efficient active learning strategies for monocular 3d object detection,” in _IV_ , 2022.
  * [18] S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou, “Semi-supervised active learning with temporal output discrepancy,” in _ICCV_ , 2021.
  * [19] S. A. Golestaneh and K. M. Kitani, “Importance of self-consistency in active learning for semantic segmentation,” in _BMVC_ , 2020.
  * [20] W. Yu, S. Zhu, T. Yang, and C. Chen, “Consistency-based active learning for object detection,” in _CVPR_ , 2022.
  * [21] C.-A. Brust, C. Käding, and J. Denzler, “Active learning for deep object detection,” in _VISAPP_ , 2019.
  * [22] H. H. Aghdam, A. Gonzalez-Garcia, J. v. d. Weijer, and A. M. López, “Active learning for deep detection neural networks,” in _ICCV_ , 2019.
  * [23] T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye, “Multiple instance active learning for object detection,” in _CVPR_ , 2021.
  * [24] C.-C. Kao, T.-Y. Lee, P. Sen, and M.-Y. Liu, “Localization-aware active learning for object detection,” in _ACCV_ , 2018.
  * [25] Y. Siddiqui, J. Valentin, and M. Nießner, “Viewal: Active learning with viewpoint entropy for semantic segmentation,” in _CVPR_ , 2020.
  * [26] B. Li and T. Alstrøm, “On uncertainty estimation in active learning for image segmentation,” in _ICMLW_ , 2020.
  * [27] S. Xie, Z. Feng, Y. Chen, S. Sun, C. Ma, and M. Song, “Deal: Difficulty-aware active learning for semantic segmentation,” in _ACCV_ , 2020.
  * [28] A. Casanova, P. O. Pinheiro, N. Rostamzadeh, and C. J. Pal, “Reinforced active learning for image segmentation,” in _ICLR_ , 2020.
  * [29] T. Kasarla, G. Nagendar, G. M. Hegde, V. Balasubramanian, and C. Jawahar, “Region-based active learning for efficient labeling in semantic segmentation,” in _WACV_ , 2019.
  * [30] B. Xie, L. Yuan, S. Li, C. H. Liu, and X. Cheng, “Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation,” in _CVPR_ , 2022.
  * [31] O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” in _ICLR_ , 2018.
  * [32] S. Agarwal, H. Arora, S. Anand, and C. Arora, “Contextual diversity for active learning,” in _ECCV_ , 2020.
  * [33] M. Crawshaw, “Multi-task learning with deep neural networks: A survey,” _arXiv preprint arXiv:2009.09796_ , 2020.
  * [34] X. Zhao, H. Li, X. Shen, X. Liang, and Y. Wu, “A modulation module for multi-task learning with applications in image retrieval,” in _ECCV_ , 2018\.
  * [35] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in _CVPR_ , 2019.
  * [36] G. Ghiasi, B. Zoph, E. D. Cubuk, Q. V. Le, and T.-Y. Lin, “Multi-task self-training for learning general representations,” in _ICCV_ , 2021.
  * [37] T. Gong, T. Lee, C. Stephenson, V. Renduchintala, S. Padhy, A. Ndirango, G. Keskin, and O. H. Elibol, “A comparison of loss weighting strategies for multi task learning in deep neural networks,” _IEEE Access_ , 2019.
  * [38] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in _CVPR_ , 2018.
  * [39] X. Liu, P. He, W. Chen, and J. Gao, “Multi-task deep neural networks for natural language understanding,” in _ACL_ , 2019.
  * [40] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, “Simultaneous detection and segmentation,” in _ECCV_ , 2014.
  * [41] D. P. Papadopoulos, J. R. Uijlings, F. Keller, and V. Ferrari, “Extreme clicking for efficient object annotation,” in _ICCV_ , 2017.
  * [42] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _NeurIPS_ , 2015.
  * [43] N. O. Salscheider, “Simultaneous object detection and semantic segmentation,” in _ICPRAM_ , 2020.
  * [44] R. Reichart, K. Tomanek, U. Hahn, and A. Rappoport, “Multi-task active learning for linguistic annotations,” in _ACL_ , 2008.
  * [45] F. Ikhwantri, S. Louvan, K. Kurniawan, B. Abisena, V. Rachman, A. F. Wicaksono, and R. Mahendra, “Multi-task active learning for neural semantic role labeling on low resource conversational corpus,” in _ACLW_ , 2018.
  * [46] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _CVPR_ , 2020.
  * [47] C. Creß, W. Zimmer, L. Strand, M. Fortkord, S. Dai, V. Lakshminarasimhan, and A. Knoll, “A9-dataset: Multi-sensor infrastructure-based dataset for mobility research,” in _IV_ , 2022.
  * [48] W. Zimmer, A. Rangesh, and M. Trivedi, “3d bat: A semi-automatic, web-based 3d annotation toolbox for full-surround, multi-modal data streams,” in _IV_ , 2019.
  * [49] D. Feng, A. Harakeh, S. L. Waslander, and K. Dietmayer, “A review and comparative study on probabilistic object detection in autonomous driving,” in _ITS_ , 2021.
  * [50] M. Everingham, S. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _IJCV_ , 2015.


# On the iterates of the Laguerre operator

Smiljana Jakšić  Faculty of Forestry, University of Belgrade, Kneza Višeslava
1, Belgrade, Serbia<EMAIL_ADDRESS>,  Stevan Pilipović
Department of Mathematics, Faculty of Sciences, University of Novi Sad, Trg
Dositeja Obradovića 4, Novi Sad, Serbia<EMAIL_ADDRESS>,
Nenad Teofanov  Department of Mathematics, Faculty of Sciences, University of
Novi Sad, Trg Dositeja Obradovića 4, Novi Sad, Serbia
tnenad<EMAIL_ADDRESS>and  Đorđe Vučković  Đorđe Vučković,
Technical Faculty “Mihajlo Pupin”, Đure Đakovića bb, 23000 Zrenjanin, Serbia
<EMAIL_ADDRESS>
###### Abstract.

We use the iterates of the Laguerre operator to introduce Pilipović spaces on
positive orthants. It is shown that such spaces coincide with $G-$type spaces
$g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$ and
$G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$, when $\alpha>1$, and $\alpha\geq
1$, respectively. However, in contrast to $G$-type spaces, Pilipović spaces
are nontrivial below the critical index $\alpha=1$. We also remark that there
is a natural isomorphism between subspaces of Pilipović spaces on
$\mathbb{R}^{d}$ consisting of even functions, and Pilipović spaces on
positive orthants.

###### Key words and phrases:

Laguerre expansions; Laguerre operator; ultradifferentiable functions;
Pilipović spaces.

###### 2010 Mathematics Subject Classification:

46F05; 33C45

##  1\. Introduction

Counterparts of Schwartz functions and of Gelfand-Shilov type functions on
$(0,\infty)$ were studied in [1, 2]. These results were thereafter generalized
in [6], were the test spaces of ultradifferentiable functions of Roumieu and
Beurling type, denoted by $G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$, and
$g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$, $\alpha\geq 1$, respectively, are
introduced and characterized via decay properties of their coefficients in
Laguerre expansions.

Toft in [10] studied Pilipović spaces $\mathbfcal
S_{\alpha}^{\alpha}(\mathbb{R}^{d})$, $\alpha>0$, by considering the iterates
of the linear Harmonic oscillator. These spaces are non trivial even if
$0<\alpha<\frac{1}{2}$, whereas when $\alpha\geq\frac{1}{2}$ they coincide
with the Gelfand-Shilov spaces usually denoted by
$\mathcal{S}_{\alpha}^{\alpha}(\mathbb{R}^{d})$, see e.g. [4]. Recently, Toft,
Bhimani and Manna demonstrated that Pilipović spaces on $\mathbb{R}^{d}$, and
their distribution spaces naturally appear in the study of well-posednedness
of certain Schrödinger equations, which are ill-posed in the class of Gelfand-
Shilov type distributions, see [11].

In this report we use the iterates of the Laguerre operator to introduce
Pilipović spaces on positive orthants, and relate them with with $G-$type
spaces $g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$ and
$G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$ of ultradifferentiable functions
when $\alpha>1$, and $\alpha\geq 1$, respectively.

The proofs will be given in our forthcoming contribution [8], where we also
study natural isomorphisms between Pilipović spaces on $\mathbb{R}^{d}$
consisting of even functions and Pilipović spaces on $\mathbb{R}^{d}_{+}$.
Related questions in the context of $G$-type spaces were considered in [7].

###  1.1. Notation

By $\mathbb{N}$, $\mathbb{Z}$, $\mathbb{R}$ and $\mathbb{C}$ the sets of
positive integers, integers, real and complex numbers, respectively;
$\mathbb{N}_{0}=\mathbb{N}\cup\\{0\\}$,
$\mathbb{N}^{d}_{0}=(\mathbb{N}\cup\\{0\\})^{d}$, $\mathbb{R}_{+}=(0,\infty)$,
$\mathbb{R}^{d}_{+}=(0,\infty)^{d}$ and
$\overline{\mathbb{R}^{d}_{+}}=[0,\infty)^{d}$. The standard multi-index
notation will be used: for $\alpha\in\mathbb{N}_{0}^{d}$ and
$x\in\mathbb{R}^{d}$ (or $x\in\overline{\mathbb{R}^{d}_{+}}$),
$|\alpha|:=\sum_{i=1}^{d}\alpha_{j}$,
$x^{\alpha}=x_{1}^{\alpha_{1}}...x_{d}^{\alpha_{d}}$, and
$\partial^{\alpha}=\partial^{\alpha_{1}}/{\partial
x_{1}^{\alpha_{1}}}...\partial^{\alpha_{d}}/\partial x_{d}^{\alpha_{d}}$. If
$x,\gamma\in\overline{\mathbb{R}^{d}_{+}}$, then
$x^{\gamma}=x_{1}^{\gamma_{1}}...x_{d}^{\gamma_{d}}$, and if $x_{j}=0$ and
$\gamma_{j}=0$, $j=1,\dots,d$, we use the convention $0^{0}=1$.

We use the standard notation for common spaces of functions, distributions,
and sequences. For example, $\ell_{p}$, $p\geq 1$, denotes the Banach space of
Lebesgue sequences, endowed with the usual norm:

$\\{a_{n}\\}_{n\in\mathbb{N}^{d}_{0}}\in\ell_{p}\qquad\Longleftrightarrow\qquad\|a_{n}\|_{\ell^{p}}=(\sum_{n}|a_{n}|^{p})^{1/p}<\infty,$

and with the usual modification when $p=\infty.$

A measurable function belongs to the Hilbert space $L^{2}(\mathbb{R}^{d}_{+})$
if

$\|\cdot\|_{L^{2}(\mathbb{R}^{d}_{+})}:=(\int_{\mathbb{R}^{d}_{+}}|f(x)|^{2}dx)^{1/2}<\infty,$

and the scalar product is given in the usual way. By
$\mathcal{S}(\mathbb{R}_{+}^{d})$ we denote the space of all smooth functions
on positive orthant, $f\in\mathcal{C}^{\infty}(\mathbb{R}^{d}_{+})$ such that
all of its derivatives $D^{p}f$, $p\in\mathbb{N}^{d}_{0}$, extend to
continuous functions on $\overline{\mathbb{R}^{d}_{+}}$, and

$\sup_{x\in\mathbb{R}^{d}_{+}}x^{k}|D^{p}f(x)|<\infty\;,\qquad
k,p\in\mathbb{N}_{0}^{d}.$

With this system of seminorms $\mathcal{S}(\mathbb{R}_{+}^{d})$ becomes an
$(F)$-space, see e.g. [5].

##  2\. Preliminaries

###  2.1. Sequence spaces

We use the notation from [10].

Let $\alpha>0$, $h>1$, and put
$\vartheta_{h,\alpha}(n)=e^{h|n|^{1/(2\alpha)}},$ $n\in\mathbb{N}^{d}_{0}$. By
$\ell^{p}_{[\vartheta_{h,\alpha}]}(\mathbb{N}_{0}^{d})$, $p\geq 1$, we denote
the space of all complex sequences $\\{a_{n}\\}_{n\in\mathbb{N}^{d}_{0}}$ for
which

$\|\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}}\|_{\ell^{p}_{[\vartheta_{h,\alpha}]}(\mathbb{N}_{0}^{d})}:=\|\\{|a_{n}|\cdot\vartheta_{h,\alpha}(n)\\}_{n\in\mathbb{N}^{d}_{0}}\|_{l^{p}}<\infty.$
(1)

Equipped with this norm
$\ell^{p}_{[\vartheta_{h,\alpha}(n)}(\mathbb{N}_{0}^{d})$ becomes a Banach
space, with the usual modification when $p=\infty$.

We also introduce

$\ell_{\alpha}(\mathbb{N}^{d}_{0})=\bigcup_{h>0}\ell^{\infty}_{[\vartheta_{h,\alpha}]}(\mathbb{N}_{0}^{d}),\qquad\ell_{0,\alpha}(\mathbb{N}_{0}^{d})=\bigcap_{h>0}\ell^{\infty}_{[\vartheta_{h,\alpha}]}(\mathbb{N}_{0}^{d}),$

and endow these locally convex spaces with inductive and projective limit
topology, respectively.

For consistency, if $\alpha=0$, then $\ell_{0}(\mathbb{N}^{d}_{0})$ denotes
the space of all sequences $\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}}$
such that $a_{n}\not=0$ for only finitely many $n\in\mathbb{N}_{0}^{d}$, and
$\ell_{0,0}(\mathbb{N}^{d}_{0})=\\{0\\}.$

Observe that different choices of $p\in[1,\infty]$ give rise to equivalent
norms in (1), cf [10]. In particular, if
$\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}}\in\ell^{\infty}_{[\vartheta_{h_{1},\alpha}]}(\mathbb{N}_{0}^{d})$
then there exists $h_{1}>0$ and $C_{1}>0$ such that

$\|\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}\\}}\|_{\ell^{2}_{[\vartheta_{h_{1},\alpha}]}(\mathbb{N}_{0}^{d})}=\Big{(}\sum_{n\in\mathbb{N}^{d}_{0}}|a_{n}|^{2}\vartheta_{h_{1},\alpha}(n)\Big{)}^{1/2}\leq
C_{1}\|\\{a_{n}\cdot\vartheta_{h,\alpha}(n)\\}_{n\in\mathbb{N}^{d}_{0}}\|_{l^{\infty}},$

and conversely, if

$\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}}\in\ell^{2}_{[\vartheta_{h,\alpha}]}(\mathbb{N}_{0}^{d})$

then there exist $C_{2},h_{2}>0$ such that

$\|\\{a_{n}\\}_{n\in\mathbb{\mathbb{N}}^{d}_{0}\\}}\|_{\ell^{\infty}_{[\vartheta_{h_{2},\alpha}]}(\mathbb{N}_{0}^{d})}\leq
C_{2}\|\\{a_{n}\cdot\vartheta_{h,\alpha}(n)\\}_{n\in\mathbb{N}^{d}_{0}}\|_{l^{2}}.$

###  2.2. Laguerre functions and Laguerre operator

For $j\in\mathbb{N}_{0}$ and $\gamma\geq 0$, the $j$-th Laguerre polynomial of
order $\gamma$ is defined by

$L_{j}^{\gamma}(x)=\frac{x^{-\gamma}e^{x}}{j!}\frac{d^{j}}{dt^{j}}(e^{-x}x^{\gamma+j}),\qquad
x\geq 0.$

The $j$-th Laguerre function is then given by $l_{j}(x)=L_{j}(x)e^{-x/2}$,
$x\geq 0$. The $d$-dimensional Laguerre polynomials and Laguerre functions,
are given by tensor products $L_{n}(x)=\prod_{l=1}^{d}L_{n_{l}}(x_{l})$, and
$l_{n}(x)=\prod_{l=1}^{d}l_{n_{l}}(x_{l})$, $n\in\mathbb{N}^{d}_{0}$,
respectively. We are interested in the case $\gamma=0$, and we write $L_{n}$
and $l_{n}$ instead of $L_{n}^{0}$ and $\l_{n}^{0}$, respectively.

Laguerre functions $\\{l_{n}\\}_{n\in\mathbb{N}^{d}_{0}}$ form an orthonormal
basis in $L^{2}(\mathbb{R}^{d}_{+})$, cf. e.g. [9, 12]. Thus every $f\in
L^{2}(\mathbb{R}^{d}_{+})$ can be expressed via its Laguerre series expansion,
and by $a_{n}(f)$ we denote the corresponding Laguerre coefficients:

$a_{n}(f)=\int_{\mathbb{R}^{d}_{+}}f(x)l_{n}(x)dx,\qquad
n\in\mathbb{N}^{d}_{0}.$

Similarly as it is done in [10], we may consider the spaces of formal Laguerre
series expansions $f=\sum_{n\in\mathbb{N}_{0}^{d}}a_{n}l_{n}$ that correspond
to the sequence $\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}$ from
$\ell_{\alpha}(\mathbb{N}_{0}^{d})$ or $\ell_{0,\alpha}(\mathbb{N}_{0}^{d})$,
$\alpha\geq 0$. These spaces will be denoted by
$\mathcal{G}_{\alpha}(\mathbb{R}_{+}^{d})$ and
$\mathcal{G}_{0,\alpha}(\mathbb{R}_{+}^{d})$, respectively, and the natural
mapping

$T:\ell_{\alpha}(\mathbb{N}_{0}^{d})\rightarrow\mathcal{G}_{\alpha}(\mathbb{R}_{+}^{d})\quad(T:\ell_{0,\alpha}(\mathbb{N}_{0}^{d})\rightarrow\mathcal{G}_{0,\alpha}(\mathbb{R}_{+}^{d})),\\\
T(\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}})=\sum_{n\in\mathbb{N}_{0}^{d}}a_{n}l_{n}$
(2)

induces topology on the target space.

Next we introduce the Laguerre operator:

$E=-\sum_{j=1}^{d}\Big{(}x_{j}\frac{\partial^{2}}{\partial
x_{j}^{2}}+\frac{\partial}{\partial x_{j}}-\frac{x}{4}+\frac{1}{2}\Big{)}.$

Then the Laguerre functions are the eigenfunctions of $E$:

$E(l_{n}(x))=\Big{(}\sum_{i=1}^{d}n_{j}\Big{)}\cdot l_{n}(x)=|n|\cdot
l_{n}(x),\qquad x>0,$

see [3, (11), p.188], and for $N\in\mathbb{N}$, and $n\in\mathbb{N}_{0}^{d}$
we have

$E^{N}(l_{n}(x))=|n|^{N}l_{n}(x).$ (3)

##  3\. Pilipović spaces on positive orthants

Before we define spaces of ultradifferentiable functions in terms of the
iterates of the Laguerre operator we give a simple but useful lemma.

######  Lemma 3.1.

Let $u$ be a smooth and square integrable function on $\mathbb{R}_{+}^{d}$,
i.e. $u\in L^{2}(\mathbb{R}_{+}^{d})\cap C^{\infty}(\mathbb{R}^{d}_{+})$. If
$E^{N}u\in L^{2}(\mathbb{R}_{+}^{d})$ for every $N\in\mathbb{N}$, then
$u\in\mathcal{S}(\mathbb{R}_{+}^{d}).$

The proof follows from (3) and the characterization of
$\mathcal{S}(\mathbb{R}_{+}^{d})$ via coefficients in Laguerre expansions of
its elements.

######  Definition 3.2.

Let $h>0$, and $\alpha>0$. Then a smooth functions $f$ belongs to the space
$\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})$, if

$\eta_{h}^{\alpha}(f):=\sup_{N\in\mathbb{N}}\frac{\|E^{N}f\|_{L^{2}(\mathbb{R}^{d}_{+})}}{h^{|N|}N!^{\alpha}}<\infty.$
(4)

From Lemma 3.1 it immediately follows that
$\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})\subset\mathcal{S}(\mathbb{R}^{d}_{+})$.
$\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})$ is a Banach space with
the norm $\eta_{h}^{\alpha}$, and clearly, if $0<\alpha<\beta$, then
$\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})$ is continuously
injected into $\mathbf{G}^{\beta,h}_{\beta,h}(\mathbb{R}^{d}_{+})$.

The Laguerre functions $l_{p}$, $p\in\mathbb{N}^{d}_{0}$, satisfy (4) for
every $h>0$ and $\alpha>0$, since

$\sup_{N\in\mathbb{N}}\frac{\|E^{N}l_{p}\|_{L^{2}(\mathbb{R}^{d}_{+})}}{h^{N}N!^{\alpha}}=\sup_{N\in\mathbb{N}}\frac{(\frac{|p|}{h})^{N}}{N!^{\alpha}}<\infty.$

It follows that $\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})$ are
non-trivial spaces since they contain at least finite linear combinations of
the Laguerre functions.

We notice that even if $\alpha=0$, and if $h>1$, then
$l_{p}\in\mathbf{G}^{0,h}_{0,h}(\mathbb{R}^{d}_{+})$ for all
$p\in\mathbb{N}^{d}$ such that $|p|\leq h$.

Now we are ready to introduce Pilipović spaces on positive orthants.

######  Definition 3.3.

Let $\alpha>0$. Then the Pilipović spaces on positive orthant
$\mathbb{R}^{d}_{+}$, $\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ are defined to be the
union, respectively the intersection of
$\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+})$, with respect to $h>0$:

$\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\bigcup_{h>0}\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+}),\qquad\text{and}\qquad\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\bigcap_{h>0}\mathbf{G}^{\alpha,h}_{\alpha,h}(\mathbb{R}^{d}_{+}).$

The spaces $\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ are endowed with projective
and inductive limit topologies, respectively. Thus,
$\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ are $(F)$-spaces
continuously injected into $\mathcal{S}(\mathbb{R}^{d}_{+})$, and if
$0<\alpha<\beta$ we have

$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})\hookrightarrow\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})\hookrightarrow\mathbf{g}^{\beta}_{\beta}\hookrightarrow\mathbf{G}^{\beta}_{\beta}(\mathbb{R}^{d}_{+}),$
(5)

where $\hookrightarrow$ denotes dense and continuous inclusion.

Next we state our main result.

######  Theorem 3.4.

Let $\alpha>0$, $f\in L^{2}(\mathbb{R}_{+}^{d})\cap
C^{\infty}(\mathbb{R}_{+}^{d})$, and let
$\\{a_{n}(f)\\}_{n\in\mathbb{N}^{d}_{0}}$ be the sequence of Laguerre
coefficients of $f$. Then the following is true:

  * (i)

$f\in\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ if and only if
$\\{a_{n}(f)\\}_{n\in\mathbb{N}^{d}_{0}}\in\ell_{\alpha/2}(\mathbb{N}^{d}_{0})$

  * (ii)

$f\in\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ if and only if
$\\{a_{n}(f)\\}_{n\in\mathbb{N}^{d}_{0}}\in\ell_{0,\alpha/2}(\mathbb{N}^{d}_{0})$.

The proof will be given in a separate contribution, [8]

By Theorem 3.4 it follows that
$\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\mathcal{G}_{\alpha}(\mathbb{R}^{d}_{+})$
and
$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\mathcal{G}_{0,\alpha}(\mathbb{R}^{d}_{+})$
where $\mathcal{G}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$\mathcal{G}_{0,\alpha}(\mathbb{R}^{d}_{+})$ are intriduced in subsection 2.2.
Moreover, the mapping
$T:{\ell_{\alpha}(\mathbb{N}_{0}^{d})}\rightarrow\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$
(resp.
$T:{\ell_{0,\alpha}(\mathbb{N}_{0}^{d})}\rightarrow\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$
) is a well defined topological isomorphism, see (2) for the definition of
$T$.

###  3.1. $G$-type spaces as Pilipović spaces

Following [6], we define $G$-type spaces of ultradifferentiable functions
$G^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$g^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ as follows:

Let $A>0$. By $g^{\alpha,A}_{\alpha,A}(\mathbb{R}^{d}_{+})$, we denote the
space of all $f\in\mathcal{S}(\mathbb{R}^{d}_{+})$ such that

$\displaystyle\sup_{p,k\in\mathbb{N}^{d}_{0}}\frac{\|x^{(p+k)/2}\partial^{p}f(x)\|_{L^{2}(\mathbb{R}^{d}_{+})}}{A^{|p+k|}k^{(\alpha/2)k}p^{(\alpha/2)p}}<\infty.$

With the following seminorms

$\displaystyle\sigma_{A,j}^{\alpha,\alpha}(f)=\sup_{p,k\in\mathbb{N}^{d}_{0}}\frac{\|x^{(p+k)/2}\partial^{p}f(x)\|_{L^{2}(\mathbb{R}^{d}_{+})}}{A^{|p+k|}k^{(\alpha/2)k}p^{(\beta/2)p}}+\sup_{\begin{subarray}{c}|p|\leq
j\\\ |k|\leq
j\end{subarray}}\sup_{x\in\mathbb{R}^{d}_{+}}|x^{k}D^{p}f(x)|,\,\,j\in\mathbb{N}_{0},$

it becomes an $(F)$-space.

The spaces $G^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$g^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ are then defined as union,
respectively intersection of $g^{\alpha,A}_{\alpha,A}(\mathbb{R}^{d}_{+})$
with respect to $A$:

$G^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\bigcup_{A>0}g^{\alpha,A}_{\alpha,A}(\mathbb{R}^{d}_{+}),\qquad
g^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=\bigcap_{A>0}g^{\alpha,A}_{\alpha,A}(\mathbb{R}^{d}_{+}).$
(6)

Then $G^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ and
$g^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ are endowed with inductive,
respectively projective limit topology.

We note that $G^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ is non-trivial when
$\alpha\geq 1$, and $g^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})$ is non-trivial
when $\alpha>1$.

Next, we rephrase and summarize results from [2, Theorem 3.6] and [6, Theorem
5.7] as follows:

######  Theorem 3.5.

Let $f\in\mathcal{S}(\mathbb{R}^{d}_{+})$. Then the following is true.

  1. $(i)$

Let $\alpha\geq 1$. If $f\in G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$, and
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}$ is the sequence of its Laguerre
coefficients, then
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}\in\ell_{\alpha}(\mathbb{N}_{0}^{d})$.

Conversely if
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}\in\ell_{\alpha}(\mathbb{N}_{0}^{d})$,
then there exists $f\in G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$ such that
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}$ is the sequence of its Laguerre
coefficients.

  2. $(ii)$

Let $\alpha>1$. If $f\in g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$, and
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}$ is the sequence of its Laguerre
coefficients, then
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}\in\ell_{0,\alpha}(\mathbb{N}_{0}^{d})$.

Conversely if
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}\in\ell_{0,\alpha}(\mathbb{N}_{0}^{d})$,
then there exists $f\in g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$ such that
$\\{a_{n}\\}_{n\in\mathbb{N}_{0}^{d}}$ is the sequence of its Laguerre
coefficients.

By Theorems 3.5 and 3.4 we have the following:

######  Corollary 3.6.

$G$-type spaces coincide with Pilipović spaces whenever the former are non-
trivial. More precisely,
$\mathbf{G}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=G_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$,
when $\alpha\geq 1$ and
$\mathbf{g}^{\alpha}_{\alpha}(\mathbb{R}^{d}_{+})=g_{\alpha}^{\alpha}(\mathbb{R}^{d}_{+})$,
when $\alpha>1$.

Therefore, $G$-type spaces can be also defined by the iterates of the Laguerre
operator, which gives a new insight into their structure.

It should be noticed that $\mathbf{g}_{1}^{1}(\mathbb{R}^{d}_{+})$ is of a
particular interest, since it is the largest Pilipović space on orthant which
is not a $G$-type space.

## Acknowledgement

This research was supported by the Science Fund of the Republic of Serbia,
$\\#$GRANT No. 2727, Global and local analysis of operators and distributions
\- GOALS. The first author is supported by the Ministry of Science,
Technological Development and Innovation of the Republic of Serbia Grant No.
451-03-65/2024-03/ 200169. The second author is supported by the project F10
financed by the Serbian Academy of Sciences and Arts. The third author is
supported by the Ministry of Science, Technological Development and Innovation
of the Republic of Serbia (Grants No. 451–03–66/2024–03/200125 $\&$
451–03–65/2024–03/200125).

## References

  * [1] Duran A. J.: Laguerre expansions of Tempered distributions and Generalized functions, J. Math. Anal. Appl. 150 166–180 (1990)
  * [2] Duran, A. J.: Laguerre expansions of Gel’fand-Shilov spaces, J. Approx. Theory 74, 280–300 (1993)
  * [3] Erdelyi, A.: Higher Transcedentals Function, Vol. 2, McGraw-Hill, New York (1953)
  * [4] I. M. Gel’fand, G. E. Shilov, Generalized Functions Volume 2 - Spaces of Fundamental and Generalized functions - Academic Press, (1968)
  * [5] Jakšić, S., Prangoski, B.: Extension theorem of Whitney type for $S(\mathbb{R}_{+}^{d})$ by the use of the Kernel Theorem, Publ. Inst. Math. Beograd, 99(113), 59–65 (2016)
  * [6] Jakšić, S., Pilipović, S., Prangoski, B.: G-type spaces of ultradistributions over $\mathbb{R}_{+}^{d}$ and the Weyl pseudo-differential operators with radial symbols, RACSAM 111, 613–640 (2017)
  * [7] Jakšić, S., Maksimović, S., Pilipović, S., Prangoski, B.: Relations between Hermite and Laguerre expansions of ultradistributions over $\mathbb{R}^{d}$ and $\mathbb{R}^{d}_{+}$, J. Pseudo-Differ. Oper. Appl., 8, 275–296 (2017)
  * [8] Jakšić, S., Pilipović, S., Teofanov, N., Vučković, Dj: Pilipović type spaces on $\mathbb{R}^{d}_{+}$, in preparation (2024)
  * [9] Thangavelu, S.: Lectures on Hermite and Laguerre Expansions, Princeton University Press, Princeton New Jersey (1993)
  * [10] Toft, J.: Images of function and distribution spaces under the Bargmann transform, J. Pseudo-Differ. Oper. Appl. 8 83–139 (2017)
  * [11] Toft, J., Bhimani, D. G. , Manna, R.: Fractional Fourier transforms, harmonic oscillator propagators and Strichartz estimates on Pilipović and modulation spaces, Appl. Comput. Harmon. Anal. 67, 101580 (2023)
  * [12] Wong, M. W.: Weyl Transform, Springer-Verlag, New York (1998)