# Dynamic Unicast-Multicast Scheduling for Age-Optimal Information
Dissemination in Vehicular Networks

Ahmed Al-Habob$\dagger$, Hina Tabassum$\dagger$, Omer Waqar$\ddagger$
†Department of Electrical Engineering and Computer Science, York University,
Toronto, ON, Canada
‡ Department of Engineering, Thompson Rivers University, Kamloops, BC, Canada
Email: {alhabob<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper investigates the problem of minimizing the age-of-information (AoI)
and transmit power consumption in a vehicular network, where a roadside unit
(RSU) provides timely updates about a set of physical processes to vehicles.
Each vehicle is interested in maintaining the freshness of its information
status about one or more physical processes. A framework is proposed to
optimize the decisions to unicast, multicast, broadcast, or not transmit
updates to vehicles as well as power allocations to minimize the AoI and the
RSU’s power consumption over a time horizon. The formulated problem is a
mixed-integer nonlinear programming problem (MINLP), thus a global optimal
solution is difficult to achieve. In this context, we first develop an ant
colony optimization (ACO) solution which provides near-optimal performance and
thus serves as an efficient benchmark. Then, for real-time implementation, we
develop a deep reinforcement learning (DRL) framework that captures the
vehicles’ demands and channel conditions in the state space and assigns
processes to vehicles through dynamic unicast-multicast scheduling actions.
Complexity analysis of the proposed algorithms is presented. Simulation
results depict interesting trade-offs between AoI and power consumption as a
function of the network parameters.

###### Index Terms:

Age-of-information (AoI), ant colony optimization (ACO), deep reinforcement
learning (DRL), multicast and unicast transmission, vehicular networks.

##  I Introduction

With the increasing diversity of vehicular applications that require real-time
information updates, such as blind spot/lane change and forward collision
warnings, vehicular communications over the upcoming 6G mobile networks become
time-critical, and thus, fresh updates are of high importance [1]. While the
conventional communication latency and throughput are effective metrics to
evaluate the performance of the vehicular communication networks, these
metrics do not capture the information freshness which is critical to obtain
the real-time knowledge about the location, orientation, and speed of the
vehicles. To this end, the age-of-information (AoI) is emerging as a useful
metric to quantify the freshness of the information while taking into account
the transmission latency, update generation time, and inter-update time
interval. Specifically, AoI is defined as the elapsed time between the
received information at the destination and the time when it was generated at
the source [2]. It should be noted that the inter-update time —which is a
scheduling parameter— is a crucial parameter in the AoI [2], and thus
optimizing AoI is totally different from optimizing other metrics such as the
throughput and latency.

Moreover, the dramatic upsurge in the number of vehicles requires the roadside
infrastructures—such as roadside units (RSUs)—to serve more vehicles
simultaneously and support their time-critical update requirements. In this
context, unicast and multicast transmissions are typically considered to
transmit independent data messages of interest for a user and a group of
users, respectively. For instance, in [3], the authors considered maximizing
the spectral efficiency by optimizing the downlink training and transmit power
allocations. However, the unicast-multicast transmissions are predefined such
that a user is either a unicast or belongs to a group of the multicast groups.
Another predefined unicast-multicast transmission scenario was considered in
[4], in which each user receives a private message and a common message is
broadcasted to all users. The authors maximized the desired effective channel
gain by designing the unicast power allocation and multicast beamformers. In
[5], the energy efficiency was maximized while considering a predefined
unicast-multicast scenario with simultaneous wireless information and power
transfer.

None of the aforementioned research works considered optimizing a unicast,
multicast, broadcast transmission scheduling, while minimizing AoI. Note that
the consideration of minimizing a time-dependent metric, such as AoI
necessitates a dynamic transmission scheduling over the time horizon. This
paper develops a dynamic transmission scheduling and power allocation
framework, in which at each time slot a vehicle receives either unicast,
multicast, or broadcast message from the RSU with optimized power allocations.
The main contributions of this paper are summarized as follows:

  * •

We consider minimizing both the AoI at the vehicles and the RSU’s power
expenditure, while optimizing the unicast-multicast scheduling decisions and
their corresponding power allocations. The two objectives are coupled in a
conflicting manner, due to the transmit power allocations. Therefore, we
formulate a multi-objective optimization problem for the two conflicting
objectives.

  * •

We develop a metaheuristic solution based on the ant colony optimization (ACO)
to solve the optimization problem, which provides a near-optimal solution.

  * •

A computationally efficient solution for the real-time implementation purposes
is also developed using deep reinforcement learning (DRL) model, which
captures the vehicles’ demands and the channel conditions in the state space
and assigns processes to vehicles through dynamic unicast-multicast scheduling
actions.

  * •

Complexity analysis of the proposed algorithms is presented. Simulation
results demonstrate interesting trade-offs between AoI and power consumption
as a function of system parameters.

The remainder of this paper is organized as follows. Section II presents the
system model. The performance metrics and problem formulation are discussed in
Section III. Section IV presents the ACO and DRL solutions. Section V
illustrates simulation results and Section VI concludes the paper.

##  II System Model and Assumptions

The considered system consists of a set $\mathcal{V}=\\{v_{i}\\}_{i=1}^{V}$ of
$V$ vehicles supported by an RSU that disseminates timely status updates to
the vehicles. The RSU is equipped with a uniform linear array of $N$ antennas.
A multi-modal data dissemination scenario is considered, in which the RSU is
capable of providing timely status updates about a set
$\mathcal{F}=\\{f_{l}\\}_{l=1}^{F}$ of $F$ physical processes. The payload
size of an update is $L$ bits. Each vehicle is interested in maintaining
freshness of its information status about a subset of processes
$\mathcal{R}_{i}\subseteq\mathcal{F}$. To represent the information demands of
the vehicles, we define $\bm{R}=[r_{il}]_{\footnotesize{V\times F}}$ such that

$r_{il}=\begin{cases}1,&\mbox{if vehicle }i~{}\mbox{is interested in
process}~{}l,\\\ 0,&\mbox{otherwise}.\end{cases}$ (1)

The time is divided into $T$ time slots each of duration $\delta$. Let
$\psi_{0}=\\{x_{0},y_{0}\\}$ be the coordinates of the RSU and
$\psi^{(t)}_{i}=\\{x^{(t)}_{i},y^{(t)}_{i}\\}$ be the coordinates of vehicle
$i$ at time slot $t$. The angle of vehicle $i$ relative to the RSU at time
slot $t$ can be expressed as
$\phi_{i}^{(t)}=\arccos\frac{x^{(t)}_{i}-x_{0}}{\ell_{i}^{(t)}}$, where
$\ell_{i}^{(t)}=\lVert\psi^{(t)}_{i}-\psi_{0}\rVert$ is the distance between
the vehicle $i$ and the RSU. Let us define the process-vehicle assignment
decision variable $\bm{\eta}^{(t)}=[\eta_{il}^{(t)}]_{\footnotesize{V\times
F}}$, such that

$\eta_{il}^{(t)}\\!\\!=\\!\\!\begin{cases}1,&\\!\\!\\!\mbox{if the update of
}f_{l}~{}\mbox{is assigned to}~{}v_{i}~{}\mbox{at time slot}~{}t,\\\
0,&\\!\\!\\!\mbox{otherwise}.\end{cases}$

It is worth noting that $\sum_{i=1}^{V}\eta_{il}^{(t)}=1$ implies that the
update of $f_{l}$ is unicasted to a single vehicle with $\eta_{il}^{(t)}=1$,
$\sum_{i=1}^{V}\eta_{il}^{(t)}=V^{\prime}<V$ implies that the update of
$f_{l}$ is multicasted to a group of vehicles with $\eta_{il}^{(t)}=1$.
Finally, $\sum_{i=1}^{V}\eta_{il}^{(t)}=V$ implies that the update message of
$f_{l}$ is broadcasted to all vehicles and $\sum_{i=1}^{V}\eta_{il}^{(t)}=0$
implies that the information of process $f_{l}$ is not transmitted to any
vehicle at time slot $t$. The communication channel between the RSU and
vehicle $i$ at time slot $t$ is modeled as follows:

$\mbox{{h}}_{i}^{(t)}=\sqrt{\frac{c_{0}}{4\pi
f_{c}\ell_{i}^{(t)^{2}}}}\mbox{{a}}^{H}(\phi_{i}^{(t)})e^{j2\pi\varrho_{i}^{(t)}},$
(2)

where $f_{c}$ is the carrier frequency, $c_{0}$ is the speed of light, $H$
denotes the Hermitian transpose of a, and $\varrho_{i}^{(t)}$ is the Doppler
shift due to the movement of vehicle $i$ expressed as
$\varrho_{i}^{(t)}=\frac{c_{i}f_{c}\cos\phi_{i}^{(t)}}{c_{0}},$ where $c_{i}$
is the speed of vehicle $i$ [6]. Assuming a uniform linear antenna array at
the RSU, the transmit array steering vector
$\mbox{{a}}(\phi_{i}^{(t)})\in\mathbb{C}^{N\times 1}$ (with $\phi_{i}^{(t)}$
as the azimuth angle between the RSU and vehicle $i$ at time slot $t$) can be
expressed as follows:

$\mbox{{a}}(\phi_{i}^{(t)})\\!=\\![1,e^{j\pi\sin\phi_{i}^{(t)}}\\!,e^{j2\pi\sin\phi_{i}^{(t)}}\\!,\cdots\\!,e^{j(N-1)\pi\sin\phi_{i}^{(t)}}],$
(3)

where $j=\sqrt{-1}$ and the antenna spacing is $\lambda/2$ with $\lambda$ as
the carrier wavelength.

##  III Performance Metrics and Problem Statement

###  III-A Decoding Error Probability

To guarantee the vehicles’ quality-of-service (QoS) requirements, the decoding
error probability of each message should be less than a tolerable decoding
error. The decoding error probability can be expressed as [7] follows:

$\varepsilon_{i}(\gamma_{i}^{(t)})=\Phi\left(\sqrt{\frac{\delta_{2}\omega}{\Gamma^{(t)}_{i}}}\left[\ln\left(1+\gamma_{i}^{(t)}\right)-\frac{L\ln
2}{\delta_{2}\omega}\right]\right),$ (4)

where
$\Phi\left(q\right)\triangleq\frac{1}{\sqrt{2\pi}}\int_{q}^{\infty}\exp(-\frac{u^{2}}{2})du$,
$\Gamma^{(t)}_{i}\triangleq 1-\frac{1}{(1+\gamma_{i}^{(t)})^{2}}$ is the
channel dispersion, $\gamma_{i}^{(t)}$ is the signal-to-interference plus-
noise ratio (SINR) at vehicle $i$ at time slot $t$, $\omega$ is the bandwidth
of the channel, and $\delta_{2}\triangleq\delta-\delta_{1}$ is the information
transmission time, with $\delta_{1}$ as the dedicated time to acquire the
vehicles’ angular parameters (i.e., location and speed).

###  III-B SINR Model with MRT Beamforming

The maximum ratio transmission (MRT) beamforming scheme is considered, in
which the asymptotically optimal beamformer vector for the vehicles that
assigned the same update message is a linear combination of channels of these
vehicles [8],[3]. Consequently, for a given $\bm{\eta}^{(t)}$, the linear
combination of the channel vectors of vehicles that receive a message about
$f_{l}$ is expressed as $\sum_{i=1}^{V}\eta_{il}\mbox{{h}}_{i}^{(t)}$. Let
$\mbox{{p}}^{(t)}=$ $[p^{(t)}_{1},\cdots,p^{(t)}_{F}]$ be the power allocation
decision with $p^{(t)}_{l}$ as the allocated power to transmit message update
of $f_{l}$. For a given process-vehicle assignment $\bm{\eta}^{(t)}$ and power
allocation decision $\mbox{{p}}^{(t)}$, the MRT beamforming vector of message
$f_{l}$ (the beamforming vector of the group of vehicles that receive an
update about $f_{l}$) is expressed as follows:

${\mbox{{{w}}}}_{l}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})=\sum_{i=1}^{V}\eta_{il}\frac{\sqrt{{p}_{l}^{(t)}}\mbox{{h}}_{i}^{(t)}}{\sqrt{N\chi^{(t)}_{i}\tilde{\xi}_{l}}},$
(5)

where $\chi^{(t)}_{i}={\frac{c_{0}}{4\pi
f_{c}\ell_{i}^{(t)^{2}}}}e^{-j2\pi\varrho_{i}^{(t)}}$ is the large-scale
channel attenuation of vehicle $i$ and $\tilde{\xi}_{l}$ is a normalization
factor [3, 9]. The SINR at vehicle $i$ can thus be expressed as follows:

$\gamma_{i}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})=\frac{|\mbox{{h}}_{i}^{(t)^{H}}\mbox{{w}}_{l}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})|^{2}}{\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq
l\end{subarray}}^{F}|\mbox{{h}}_{i}^{(t)^{H}}\mbox{{w}}_{m}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})|^{2}+\sigma^{2}}.$
(6)

###  III-C Age of Information

The instantaneous AoI of the physical process $f_{l}$ at vehicle $i$ evolves
according to

$\Delta_{il}^{(t)}\\!(\bm{\eta}^{(t)}\\!,\\!\mbox{{p}}^{(t)}\\!)\\!=\\!\\!\begin{cases}\delta,&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\mbox{if}~{}\eta_{il}^{(t)}\\!\\!\\!=\\!1~{}\mbox{and}~{}\varepsilon_{i}(\gamma_{i}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\\!)\\!\leq\varepsilon^{\mbox{\scriptsize
max}}_{i}\\!,\\\
\Delta_{il}^{(t-1)}\\!+\\!\delta,&\mbox{otherwise},\end{cases}$ (7)

where $\varepsilon^{\mbox{\scriptsize max}}_{i}$ is the maximum allowed error
probability at vehicle $i$. The time-average AoI of $f_{l}$ at vehicle $i$
over $T$ time slots is
$\bar{\Delta}_{{il}}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\triangleq\mathbb{E}_{T}\\!\\!\left[\Delta_{il}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\right]=$
$\frac{1}{T}\sum_{t=1}^{T}\Delta_{il}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})$.
Consequently, the total time-average AoI can be expressed as follows:

$\begin{split}\bar{\Delta}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})&=\sum_{i=1}^{V}\sum_{l=1}^{F}r_{il}\bar{\Delta}_{{il}}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\\\
&=\frac{1}{T}\sum_{i=1}^{V}\sum_{t=1}^{T}\sum_{l=1}^{F}r_{il}\Delta_{il}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)}).\end{split}$
(8)

Note that the maximum value of $\bar{\Delta}_{{il}}$ is $\delta(T+1)/2$, which
corresponds the case of no update about $f_{l}$ is received at vehicle $i$
over the $T$ time slots. Thus, the maximum value (upper bound) of the total
time-average AoI $\bar{\Delta}^{\mbox{\scriptsize max}}$ can be expressed as:

$\bar{\Delta}^{\mbox{\scriptsize
max}}=\frac{\delta(T+1)}{2}\sum_{i=1}^{V}\sum_{l=1}^{F}r_{il},$ (9)

which corresponds the case of no update is received by any vehicle during the
$T$ time slots. The minimum value (lower bound) of the total time-average AoI
$\bar{\Delta}^{\mbox{\scriptsize min}}$ corresponds the case of updating each
vehicle in each time slot. Keeping in mind that a vehicle $i$ is interested in
$|\mathcal{R}_{i}|=\sum_{l=1}^{F}r_{il}$ processes and the best option is to
alternate the updates between the $|\mathcal{R}_{i}|$ processes,
$\bar{\Delta}^{\mbox{\scriptsize min}}$ can be expressed as111The expression
in (10) is valid for $T>|\mathcal{R}_{i}|+1$; for the special case
$T<|\mathcal{R}_{i}|-1$, we have $\bar{\Delta}^{\mbox{\scriptsize
min}}=\sum_{i=1}^{V}\frac{\delta}{T}\left[|\mathcal{R}_{i}|\frac{T(T+1)}{2}-\sum_{r=1}^{T}\frac{r(r+1)}{2}\right]$;
both expressions are equivalent for $|\mathcal{R}_{i}|-1\leq
T\leq|\mathcal{R}_{i}|+1$.:

$\bar{\Delta}^{\mbox{\scriptsize
min}}=\sum_{i=1}^{V}\frac{\delta}{T}\left[T\frac{|\mathcal{R}_{i}|(|\mathcal{R}_{i}|+1)}{2}-\sum_{r=1}^{|\mathcal{R}_{i}|-1}\frac{r(r+1)}{2}\right].$
(10)

Figure 1: Illustrative example with $V=5$ vehicles and $F=4$ processes.

###  III-D Significance of Process-Vehicle Assignment and Power Allocation - A
Toy Example

Let us consider $V=5$ vehicles, where each vehicle is interested in a subset
of $\mathcal{F}=\\{f_{1},f_{2},f_{3},f_{4}\\}$ processes with demands as
illustrated in Fig. 1. The associated information demand matrix $\mathbf{R}$
can be represented as follows:

$\mathbf{R}=[r_{il}]_{\footnotesize{5\times
4}}=\blockarray{ccccc}f_{1}&f_{2}f_{3}f_{4}\\\ \block{[cccc]c}1001v_{1}\\\
1010v_{2}\\\ 1001v_{3}\\\ 1110v_{4}\\\ 1100v_{5}\\\ .$ (11)

Let us assume that the observation interval is $T=7$ time slots and other
parameters are as listed in Table I. According to (9) and (10), the maximum
and minimum time-average AoI at the vehicles can be computed
$\bar{\Delta}^{\mbox{\scriptsize max}}=308/7=44$s and
$\bar{\Delta}^{\mbox{\scriptsize min}}=108/7\approx 15.4$s, respectively. Let
us assume that the RSU assigns the processes to vehicles and allocates the
power at random, the time-average AoI will vary around $210/7\approx 30$s and
the power consumption will be around $0.5$W. By examining all possible
process-vehicle assignment and power allocation decisions with objective of
minimizing both AoI and power consumption, the time-average AoI can be reduced
to $124/7\approx 17.7$s while the time-average consumed power is $0.18$W if
the following decisions are performed. At $t=1$ no message is transmitted. At
$t=2$, an update about $f_{1}$ is broadcasted to all vehicles. At $t=3$, an
update about $f_{3}$ is unicasted to $v_{2}$ and updates about $f_{4}$ and
$f_{2}$ are multicasted to $\\{v_{1},v_{3}\\}$ and $\\{v_{4},v_{5}\\}$,
respectively, and so on. The question is how to select optimal scheduling and
power allocation decisions at each time slot for arbitrary number of physical
processes and vehicles with different demands. This motivates the problem
formulation in the following.

###  III-E Problem Formulation

We consider minimizing the time-average AoI of each process at the vehicles as
well as the time-average power consumption at RSU. Keeping in mind the trade-
off between these two objectives and the fact that they have different units,
ranges, and orders of magnitude, they should be normalized such that they have
similar ranges [10]. Consequently, we define a multi-objective weighted sum
utility function as:

$\begin{split}{O}(\bm{\eta}^{(t)}\\!,\\!\mbox{{p}}^{(t)})\\!=&\zeta\frac{\bar{\Delta}(\bm{\eta}^{(t)}\\!,\mbox{{p}}^{(t)}\\!)\\!-\\!\bar{\Delta}^{\mbox{\scriptsize
min}}}{\bar{\Delta}^{\mbox{\scriptsize max}}-\bar{\Delta}^{\mbox{\scriptsize
min}}}\\!+\\!(1\\!-\\!\zeta)\frac{\bar{{p}}^{(t)}\\!-\\!P^{\mbox{\scriptsize
min}}}{P^{\mbox{\scriptsize max}}\\!-\\!P^{\mbox{\scriptsize
min}}},\end{split}$ (12)

where the time-average power consumption can be given as
$\bar{{p}}^{(t)}=\frac{1}{T}\sum_{t=1}^{T}\sum_{l=1}^{F}p^{(t)}_{l}$,
$0<\zeta\leq 1$ is a relative weight to give preference to minimize the AoI or
the power, $\bar{\Delta}^{\mbox{\scriptsize max}}$ and
$\bar{\Delta}^{\mbox{\scriptsize min}}$ are the maximum and minimum total
time-average AoI as expressed in (9) and (10), respectively,
$P^{\mbox{\scriptsize max}}$ is the maximum transmission power of the RSU, and
$P^{\mbox{\scriptsize min}}=0$. The optimization problem is thus formulated as
follows:

P1 $\displaystyle\\!\min_{\bm{\eta}^{(t)},\mbox{{p}}^{(t)}}$
$\displaystyle{O}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})$ (13a) s.t.
$\displaystyle{\sum_{l=1}^{F}p_{l}^{(t)}\leq P^{\mbox{\scriptsize max}}},$
(13b) $\displaystyle\sum_{l=1}^{F}\eta_{il}^{(t)}\leq
1,~{}\forall~{}v_{i}\in\mathcal{V},$ (13c)
$\displaystyle\eta_{il}^{(t)}r_{il}=\eta_{il}^{(t)},~{}\forall~{}1\leq i\leq
V,1\leq l\leq F,$ (13d) $\displaystyle{p_{l}^{(t)}\geq
0,~{}\forall~{}f_{l}\in\mathcal{F}},$ (13e)
$\displaystyle\eta_{il}^{(t)}\in\\{0,1\\},~{}\forall
v_{i}\in\mathcal{V},f_{l}\in\mathcal{F}.$ (13f)

Constraint (13b) guarantees that the allocated power is less than the maximum
transmission power of the RSU. Constraint (13c) guarantees that at most one
process is assigned to each vehicle at each time slot. Keeping in mind that
$\bm{R}$ and $\bm{\eta}^{(t)}$ are binary variables, (13d) guarantees that if
a vehicle $i$ is not interested in process $l$ (i.e., $r_{il}=0$) then process
$l$ will not be assigned to vehicle $i$ (i.e., $\eta_{il}^{(t)}$ should be
$0$).

The optimization problem in (13) is a mixed-integer non-linear programming
(MINLP) problem, where the discontinuity in the objective function comes from
(8), the non-linearity comes from (6), and the integer constraint arises from
(14f).

##  IV Proposed Age-Optimal Solutions

First, it is important to note that from (7), the AoI of process $f_{l}$ at
vehicle $i$ can be minimized if both $\eta_{il}^{(t)}=1$ and
$\varepsilon_{i}(\gamma_{i}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\\!)\leq\varepsilon^{\mbox{\scriptsize
max}}_{i}$. To ensure that
$\varepsilon_{i}(\gamma_{i}^{(t)}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\\!)\leq\varepsilon^{\mbox{\scriptsize
max}}_{i}$, we denote $\hat{\gamma}_{i}$ be the SINR at vehicle $v_{i}$ that
ensures $\varepsilon_{i}(\hat{\gamma}_{i})=\varepsilon^{\mbox{\scriptsize
max}}_{i}$ which can be found by solving (4) numerically. Then, from (4) and
(6), the AoI of the process $f_{l}$ at vehicle $v_{i}$ can be minimized if
both $\eta_{il}^{(t)}=1$ and the following inequality are satisfied, i.e.,

$\frac{{p}_{l}^{(t)}|\mbox{{h}}_{i}^{(t)^{H}}\mbox{{w}}_{l}^{(t)}(\bm{\eta}^{(t)})|^{2}}{\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq
l\end{subarray}}^{F}{p}_{m}^{(t)}|\mbox{{h}}_{i}^{(t)^{H}}\mbox{{w}}_{m}^{(t)}(\bm{\eta}^{(t)})|^{2}+\sigma^{2}}\geq\hat{\gamma}_{i}.$
(14)

Consequently, a solution for the optimization problem in (13) can be obtained
by solving the following optimization problem:

$\displaystyle\textbf{P2}\min_{\bm{\eta}^{(t)}{,\mbox{{p}}^{(t)}}}$
$\displaystyle~{}{O}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)}_{\bm{\eta}})$ s.t.
$\displaystyle\eqref{con3},\eqref{con4},~{}\mbox{and}~{}\eqref{con5},\ $

where
$\mbox{{p}}^{(t)}_{\bm{\eta}}\\!=[{p}^{(t)}_{\bm{\eta}_{1}},\cdots,{p}^{(t)}_{\bm{\eta}_{F}}]$
is the solution of the following linear programming problem:

$\displaystyle\mbox{{p}}^{(t)}_{\bm{\eta}}\\!=\\!\min_{\mbox{{p}}^{(t)}}\sum_{l=1}^{F}p_{l}^{(t)}$
$\displaystyle\mbox{s.t.}\frac{\eta_{il}^{(t)}{p}_{l}^{(t)}|\mbox{{h}}_{i}^{(t)^{H}}\\!\\!\mbox{{w}}_{l}^{(t)}(\bm{\eta}^{(t)})|^{2}}{\sum\limits_{\begin{subarray}{c}m=1\\\
m\neq
l\end{subarray}}^{F}\\!{p}_{m}^{(t)}|\mbox{{h}}_{i}^{(t)^{H}}\\!\\!\mbox{{w}}_{m}^{(t)}(\bm{\eta}^{(t)})|^{2}\\!+\\!\sigma^{2}\\!}\\!\\!\geq\\!\eta_{il}^{(t)}\hat{\gamma}_{i},~{}~{}\forall
f_{l},v_{i},$ (16a)
$\displaystyle~{}~{}\eqref{con1}~{}\mbox{and}~{}\eqref{con2}$

It is worth noting that for a given $\bm{\eta}^{(t)}$, constraint (16a)
guarantees that the allocated power minimizes the AoI. If (16) is infeasible
for a given $\bm{\eta}^{(t)}$, then $\bm{\eta}^{(t)}$ does not minimize the
objective function and will be discarded. The strategy of the following ACO
and DRL solution approaches can be summarized as follows. Search for an
optimized $\bm{\eta}^{(t)}$ by exploring its search space and for each
candidate $\bm{\eta}^{(t)}$ find the corresponding best power allocation by
solving (16). The fitness of a candidate $\bm{\eta}^{(t)}$ is reflected by its
ability to minimize the objective function with a feasible power allocation.

###  IV-A Ant Colony Optimization (ACO)

The optimization problem in (15) can be solved by enumerating all the feasible
decisions. Such exhaustive search approach is computationally inefficient,
which motivates designing a metaheuristic solution based on the ACO for rapid
discovery of good solutions and guarantee convergence [11, Ch.4.3]. In the
proposed ACO algorithm, a colony of $A$ ants collaborate to solve  P1. Each
ant $a\in A$ travels a tour of $T$ steps. In each step, it assigns a process
for each vehicle. The probability that ant $a$ assigns process $f_{l}$ to
vehicle $v_{i}$ and the probability it does not assign a process to vehicle
$v_{i}$ at time $t$ are obtained as:

$\begin{split}\bar{\pi}_{il}^{\left(t\right)}=\frac{(\bar{\tau}^{(t)}_{il})^{\iota_{1}}(\bar{\varrho}^{(t)}_{il})^{\iota_{2}}}{1+\sum_{l=1}^{F}(\bar{\tau}^{(t)}_{il})^{\iota_{1}}(\bar{\varrho}^{(t)}_{il})^{\iota_{2}}},~{}\bar{\pi}_{i0}^{\left(t\right)}=1-\sum_{l=1}^{F}\bar{\pi}_{il}^{\left(t\right)},\end{split}$
(17)

respectively, where $\bar{\tau}^{(t)}_{ir}$ is the trail pheromone and
$\bar{\varrho}^{(t)}_{il}$ is the attractiveness of assigning process $f_{l}$
to vehicle $v_{i}$. The latter is set to be

$\bar{\varrho}^{(t)}_{il}=\frac{r_{il}\Delta_{il}^{(t)}}{t},$ (18)

to give higher attractiveness to assigning the process with high AoI and of
interest to $v_{i}$. The parameters $\iota_{1}$ and $\iota_{2}$ control the
influence of the pheromone and attractiveness, respectively. At each step, ant
$a$ obtains $\bm{\eta}^{(t,a)}$ based on (17) and obtain the power allocation
$\mbox{{p}}^{(t,a)}$ by solving (16). Only ants that obtain the highest and
second-highest $O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$ $\forall a\in A$
deposit their pheromone [12]. The pheromone is updated as follows:

$\bar{\tau}_{il}^{(t)}\leftarrow\left(1-\kappa\right)\bar{\tau}_{il}^{(t)}+{\eta}_{il}^{(t,a)}\nabla{\tau}^{(a)},$
(19)

where $\kappa$ is the pheromone evaporation coefficient and
$\Delta{\tau}^{(a)}$ is the deposit pheromone, which is expressed as

$\nabla{\tau}^{(a)}\\!\\!=\\!\begin{cases}\exp[-O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})],&\\!\\!\\!\\!\mbox{if}~{}\sum\limits_{l=1}^{F}p_{l}^{(t,a)}\\!\leq\\!P^{\mbox{\scriptsize
max}}\\!,\\\ 0,&\mbox{otherwise}.\end{cases}$ (20)

The ACO algorithm is illustrated in Algorithm 1 which iterates until the
improvement in the best solution of the whole colony is less than a threshold
$\epsilon_{0}$ or a maximum number of colonies $I$ has been generated.

Algorithm 1 ACO algorithm for age-optimum dynamic transmission.

1:Input $\bm{R}$, $\mbox{{h}}_{i}^{(t)}$, $\kappa$, $I$, $A$, $\epsilon_{0}$;

2:$O^{*}\leftarrow\infty$; $O_{p}\leftarrow 0$;

3:while $I\geq 1$ and $|O^{*}-O_{p}|\geq\epsilon_{0}$ do

4: $O_{1}\leftarrow\infty$; $O_{2}\leftarrow\infty$; $I=I-1$; $O_{p}=O^{*}$;

5: for $a=1$ to $A$ do

6: for $t=1$ to $T$ do

7: Obtain $\bm{\eta}^{(t,a)}$ using (17); Obtain $\mbox{{p}}^{(t,a)}$ by
solving (16);

8: end for

9: Evaluate $O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$ using (12);

10: if $O^{*}>O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$,
$O^{*}=O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$; end if;

11: if $O_{1}>O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$, $a_{1}=a$;

12: else if $O_{2}>O(\bm{\eta}^{(t,a)},\mbox{{p}}^{(t,a)})$, $a_{2}=a$; end if

13: end for

14: Deposit pheromone of $a_{1}$ and $a_{2}$ using (19);

15:end while

16:Return $O^{*}$.

###  IV-B DRL-Based Solution Approach

In this section, we aim to design a real-time solution approach for the
optimization problem in (13). In this context, we develop a DRL model that
involves the definition of the environment state space
$\bm{s}^{t}\in\bm{\mathcal{S}}$, the action space
$\bm{\mu}^{(t)}\in\bm{\mathcal{A}}$, and the immediate reward function
$\rho^{(t)}$.

  * •

The state of the environment at time slot $t$ is given by

$\bm{s}^{t}\\!=\\!\\!\left\\{\\{\chi^{(t)}_{i}\\},\\{r_{i1}\Delta_{i1}^{(t)},r_{i2}\Delta_{i2}^{(t)},\cdots\\!,r_{iF}\Delta_{iF}^{(t)}\\}\right\\}_{i=1}^{V}\\!,$
(21)

which captures the channel state and the AoI of the processes of interest for
the $V$ vehicles.

  * •

The action at time slot $t$ is defined as
$\bm{\mu}^{(t)}=[\mu^{(t)}_{i}]_{\footnotesize{V\times 1}}$, which is a vector
of integers $\mu^{(t)}_{i}\in\\{0\cup\mathcal{R}_{i}\\}$, such that

$\mu^{(t)}_{i}=\begin{cases}l,&\mbox{implies }f_{l}~{}\mbox{is assigned
to}~{}v_{i},\\\ 0,&\mbox{implies}~{}v_{i}~{}\mbox{is not updated
at}~{}t.\end{cases}$ (22)

It is worth noticing that $\bm{\mu}^{(t)}$ is an equivalent representation to
$\bm{\eta}^{(t)}$ with a dimension of $V\times 1$ instead of $V\times F$.

  * •

The immediate reward at time slot $t$ is expressed as

$\rho^{(t)}\\!\\!=\\!\zeta\frac{\bar{\Delta}_{t}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})\\!-\\!\bar{\Delta}_{t}^{\mbox{\scriptsize
min}}}{\bar{\Delta}_{t}^{\mbox{\scriptsize
max}}-\bar{\Delta}_{t}^{\mbox{\scriptsize
min}}}\\!+\\!(1\\!-\zeta)\frac{\hat{\mbox{{p}}}^{(t)}-P^{\mbox{\scriptsize
min}}}{P^{\mbox{\scriptsize max}}-P^{\mbox{\scriptsize min}}},$ (23)

where
$\hat{\mbox{{p}}}^{(t)}=\frac{1}{t}\sum_{t^{\prime}=1}^{t}{\mbox{{p}}}^{(t^{\prime})}$,
$\bar{\Delta}_{t}(\bm{\eta}^{(t)},\mbox{{p}}^{(t)})$,
$\bar{\Delta}_{t}^{\mbox{\scriptsize max}}$, and
$\bar{\Delta}_{t}^{\mbox{\scriptsize min}}$ are obtained by replacing $T$ by
$t$ in (8), (9), and (10), respectively. The reward is set to
$-\log(\rho^{(t)}+\nu)$, where $\nu$ is a very small number that introduced to
avoid infinite reward.

The DRL training algorithm is illustrated in Algorithm 2. It is worth
mentioning that the trained DRL agent estimates an action $\bm{\mu}^{(t)}$ for
a given state, the corresponding $\bm{\eta}^{(t)}$ is utilized to find the
power allocation in (16) and evaluate the objective function in (13a).

Algorithm 2 DRL training algorithm for age-optimum dynamic transmission.

1:Initialize the weights of the deep network $\theta$ and the replay buffer B;

2:For $episode=1$ to Max no. of episodes do

3:Initialize the environment and receive the initial state $\bm{s}^{(1)}$;

4: Repeat:

5: With probability $\epsilon$, select a random action
$\bm{\mu}^{(t)}\in\mathcal{A}$; Otherwise, select
$\bm{\mu}^{(t)}=\arg\max\limits_{\forall\bm{\mu}^{(t)}\in\bm{\mathcal{A}}}Q(\bm{s}^{(t)},\bm{\mu}^{(t)}\mid\bm{\theta})$;

6: Obtain $\bm{\eta}^{(t)}$; Obtain $\mbox{{p}}^{(t)^{*}}$ by solving (16);

7: Observe the reward $\rho^{(t)}$ and next state $\bm{s}^{({t+1})}$;

8: Store the transition
$\\{\bm{s}^{({t})},\bm{\mu}^{(t)},\rho^{(t)},\bm{s}^{({t+1})}\\}$ in B;
$t=t+1$;

9: Until terminal state $t=T$.

10: Sample a random mini-batch of $M$ transitions from B;

11: For each transitions in $M$ obtain $y_{m}$ such that
$y_{m}\\!=\\!\\!\begin{cases}\rho^{(m)},&\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\mbox{if
}\bm{s}^{(m+1)}~{}\mbox{is a terminal state},\\\
\rho^{(m)}\\!+\\!\gamma\max\limits_{\forall\bm{\mu}^{(m)}\in\mathcal{A}}Q(\bm{s}^{{t}},\bm{\mu}^{(m)}\mid\bm{\theta}),&\\!\\!\\!\\!\mbox{otherwise}.\end{cases}$

12: Update the weight of the DNN network by minimizing the loss
$L(\bm{\theta})=\frac{1}{M}\sum_{m=1}^{M}(y_{m}-Q(\bm{s}^{(m)},\bm{\mu}^{(m)}\mid\bm{\theta}))^{2}$;

13:End for

###  IV-C Complexity Analysis

Keeping in mind that a process-vehicle assignment decision $\bm{\eta}^{(t)}$
can be represented by a vector
$\bm{\mu}^{(t)}=[\mu_{i}^{(t)}]_{\footnotesize{1\times V}}$, the

TABLE I: Simulation Parameters.

Parameter Value Parameter Value Parameter Value $P^{\mbox{{\footnotesize
max}}}$ $0$ dB $c_{i}$ $\sim U(10,15)$ m/s [6] $\omega$ $10$ MHz [6] $f_{c}$
$3$ GHz [6] $N$ $64$ [6] $L$ $128$ byte $\varepsilon^{\mbox{{\footnotesize
max}}}$ $10^{-}6$ $c_{0}$ $2.99\times 10^{8}$ m/s $\sigma^{2}$ $0.1$ $I$ $400$
iteration $A$ $100$ individuals $\delta$ $1$ s $\kappa$ $0.1$
$\iota_{1}/\iota_{2}$ $1$ ${\epsilon_{0}}$ $0.01$ B $10^{4}$ mini-batch $64$
learning rate $0.001$

search space of (15) over the $T$ time slots is $\mathcal{O}((F+1)^{V}T)$. The
worst-case computational complexity of evaluating the objective function in
(12) is $\mathcal{O}(VFT^{2})$ operations and solving (16) requires
$\mathcal{O}((V+F)^{2.5})$ operations. Consequently, the computational
complexity of the exhaustive search is $\mathcal{O}(V^{3.5}T^{3}F^{V})$. An
ant agent performs $\mathcal{O}(VF)$ operations to find a process-vehicle
assignment decision, $\mathcal{O}((V+F)^{2.5})$ operations to allocate the
power, and $\mathcal{O}(VFT^{2})$ operations to evaluate (12). Consequently,
the worst-case computational complexity of the ACO is
$\mathcal{O}((VF)^{4.5}T^{3}AI)$, where $A$ is the number of ants in the
colony and $I$ is the maximum number of colonies. Considering a scenario of
$V=5$ vehicles and $F=4$ processes, the average execution time of the
exhaustive search, ACO, and DRL solution approaches is $1.8$ s, $0.2$ s, and
$0.01$ s, respectively. Average execution time using MATLAB on an Intel(R)
Core(TM) i7-3770 CPU machine working at a clock frequency of 3.4 GHz and 16 GB
of RAM.

##  V Simulation Results and Discussions

This section introduces simulation results to evaluate the proposed framework
and compare its performance with the random solution approach, in which the
process-vehicle assignment and power allocations decisions are selected at
random. Unless otherwise stated, the considered numerical values of the system
parameters and solution approaches are listed in Table I.

Figure 2: Objective function in (12) versus the relative weight $\zeta$ with
total number of $F=4$ process and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$.

Figure 3: Average normalized AoI and power expenditure versus the relative
weight $\zeta$ with total number of processes $F=4$ process and
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}=2$.

Figure 4: Average normalized AoI and power expenditure versus number of
process of interest per vehicle
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}$ with total number of
processes $F=10$ process and $\zeta=0.5$.

Figure 4 illustrates the objective function versus the relative weight $\zeta$
for the proposed framework obtained using the exhaustive search, ACO, and DRL
solution approaches as well as the random approach. It is seen that the ACO
and DRL approaches achieve performance close to that of the exhaustive search
approach and the random approach provides the worst performance in comparison
with the proposed framework with the three solution approaches.

To get more insight into this result, Fig. 4 shows the average normalized AoI
and power consumption versus the relative weight $\zeta$ for the random
solution and the proposed framework using the ACO and DRL solutions. The
curves of the exhaustive search follow a similar trend to that of the ACO and
are omitted to make Fig. 4 less crowded. It is noticed that the proposed
framework provides a good trade-off between AoI and power expenditure as for
low values of $\zeta$ it minimizes the power expenditure and as $\zeta$
increases it minimizes the AoI. That is not the case for the random solution,
in which both the AoI and power expenditure are not function of the relative
weight and the power consumption is higher than that of the proposed
framework.

Figure 4 depicts the average normalized AoI and power expenditure of the
random solution and the proposed framework using the ACO solution versus the
number of process of interest per vehicle
$\mathinner{\\!\left\lvert\mathcal{R}_{i}\right\rvert}$. It is clear that the
AoI increases as the vehicles’ demand increases in both random solution and
proposed framework, with less AoI in the proposed framework. On the other
hand, the power expenditure in the proposed framework is decreased as the
vehicles’ demand increases. This is attributed to the fact that for a fixed
set of processes, as the number of process of interest per vehicle increases
the demand of the vehicles becomes more similar which enables the proposed
framework to transmit the same update to more vehicles, which reduces the
interference, and thus reduces the power expenditure.

##  VI Conclusion

This paper has proposed a dynamically unicast, multicast, and broadcast
transmission framework to minimize both AoI and power consumption in vehicular
networks. To solve the formulated mixed integer optimization problem, two
solution approaches have been developed, namely a metaheuristic solution based
on ACO and less computational complex in real-time evaluation solution based
on DRL approach. Simulation results have illustrated that the proposed
framework minimizes both the AoI and power consumption and provides a good
trade-off between them. Results also have showed that ACO and DRL solution
approaches provide close to the optimal solution, which is obtained through
exhaustive search.

## Acknowledgment

This research was supported by two Discovery Grants funded by the Natural
Sciences and Engineering Research Council of Canada.

## References

  * [1] Noor-A-Rahim _et al._ , “6G for vehicle-to-everything (v2x) communications: Enabling technologies, challenges, and opportunities,” _Proc. IEEE_ , vol. 110, no. 6, pp. 712–734, 2022.
  * [2] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update,” in _Proc. IEEE INFOCOM_ , 2012, pp. 2731–2735.
  * [3] M. Sadeghi _et al._ , “Joint unicast and multi-group multicast transmission in massive MIMO systems,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 10, pp. 6375–6388, 2018.
  * [4] J. Wang, H. Xu, B. Zhu, L. Fan, and A. Zhou, “Hybrid beamforming design for mmWave joint unicast and multicast transmission,” _IEEE Commun. Lett._ , vol. 22, no. 10, pp. 2012–2015, 2018.
  * [5] W. Hao, G. Sun, F. Zhou, D. Mi, J. Shi, P. Xiao, and V. C. M. Leung, “Energy-efficient hybrid precoding design for integrated multicast-unicast millimeter wave communications with SWIPT,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 11, pp. 10 956–10 968, 2019.
  * [6] W. Yuan, Z. Wei, S. Li, J. Yuan, and D. W. K. Ng, “Integrated sensing and communication-assisted orthogonal time frequency space transmission for vehicular networks,” _IEEE J. Sel. Top. Signal Process._ , vol. 15, no. 6, pp. 1515–1528, 2021.
  * [7] C. She, C. Sun, Z. Gu, Y. Li, C. Yang, H. V. Poor, and B. Vucetic, “A tutorial on ultrareliable and low-latency communications in 6G: Integrating domain knowledge into deep learning,” _Proc. IEEE_ , vol. 109, no. 3, pp. 204–246, 2021.
  * [8] M. Sadeghi and C. Yuen, “Multi-cell multi-group massive MIMO multicasting: An asymptotic analysis,” in _Proc. IEEE Global Communications Conference (GLOBECOM)_ , 2015, pp. 1–6.
  * [9] M. Sadeghi, E. Björnson, E. G. Larsson, C. Yuen, and T. L. Marzetta, “Max–min fair transmit precoding for multi-group multicasting in massive MIMO,” _IEEE Trans. Wirel. Commun._ , vol. 17, no. 2, pp. 1358–1373, 2018\.
  * [10] R. T. Marler and J. S. Arora, “Survey of multi-objective optimization methods for engineering,” _Structural and multidisciplinary optimization_ , vol. 26, no. 6, pp. 369–395, 2004.
  * [11] M. Dorigo and T. Stützle, _Ant Colony Optimization_ , 1st ed. A Bradford Book, Cambridge, MA, U.S.A., 2004.
  * [12] K. Doerner _et al._ , “Pareto ant colony optimization: A metaheuristic approach to multiobjective portfolio selection,” _Annals of Operations Research_ , vol. 131, no. 1-4, pp. 79–99, Oct. 2004.