# URLLC-Awared Resource Allocation for Heterogeneous Vehicular Edge Computing

Qiong Wu, , Wenhua Wang, Pingyi Fan, ,
Qiang Fan, Jiangzhou Wang, , Khaled B. Letaief  Qiong Wu and Wenhua Wang are
with the School of Internet of Things Engineering, Jiangnan University, Wuxi
214122, China, and also with the State Key Laboratory of Integrated Services
Networks (Xidian University), Xi’an 710071, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Pingyi Fan is with
the Department of Electronic Engineering, Beijing National Research Center for
Information Science and Technology, Tsinghua University, Beijing 100084, China
(Email<EMAIL_ADDRESS>Qiang Fan is with Qualcomm, San Jose, CA 95110,
USA (e-mail<EMAIL_ADDRESS>Jiangzhou Wang is with the School of
Engineering, University of Kent, CT2 7NT Canterbury, U.K. (Email:
<EMAIL_ADDRESS>K. B. Letaief is with the Department of Electrical and
Computer Engineering, the Hong Kong University of Science and Technology
(HKUST), Hong Kong, and also with the Pengcheng Laboratory, Shenzhen 518055,
China<EMAIL_ADDRESS>
###### Abstract

Vehicular edge computing (VEC) is a promising technology to support real-time
vehicular applications, where vehicles offload intensive computation tasks to
the nearby VEC server for processing. However, the traditional VEC that relies
on single communication technology cannot well meet the communication
requirement for task offloading, thus the heterogeneous VEC integrating the
advantages of dedicated short-range communications (DSRC), millimeter-wave
(mmWave) and cellular-based vehicle to infrastructure (C-V2I) is introduced to
enhance the communication capacity. The communication resource allocation and
computation resource allocation may significantly impact on the ultra-reliable
low-latency communication (URLLC) performance and the VEC system utility, in
this case, how to do the resource allocations is becoming necessary. In this
paper, we consider a heterogeneous VEC with multiple communication
technologies and various types of tasks, and propose an effective resource
allocation policy to minimize the system utility while satisfying the URLLC
requirement. We first formulate an optimization problem to minimize the system
utility under the URLLC constraint which modeled by the moment generating
function (MGF)-based stochastic network calculus (SNC), then we present a
Lyapunov-guided deep reinforcement learning (DRL) method to convert and solve
the optimization problem. Extensive simulation experiments illustrate that the
proposed resource allocation approach is effective.

###### Index Terms:

Heterogeneous, Vehicular edge computing, URLLC, Resource allocation

##  I Introduction

With the coming of the fifth generation (5G) era, the demands of the real-time
vehicular applications such as the online three-dimensional (3D) game,
augmented/virtual reality (AR/VR) as well as high definition video are
increasing[1]. These applications need the supports of large amount of data
collected by the high definition (HD) resolution cameras, light detection and
ranging (LiDAR) and HD maps with high rate [2]. Such large amount of data may
result in intensive computation tasks to be processed in time. However, the
computation and storage capability of vehicles are usually insufficient, how
to deal with the computing-thirsty tasks is becoming a great challenge.
Vehicular edge computing (VEC) is a promising approach for real-time vehicular
applications, where a VEC server deployed near the roadside can process tasks
offloaded by vehicles and then return the processed result[3, 4, 5].

Up to now, there are three major communication technologies in VEC to support
task offloading, i.e., dedicated short range communications (DSRC), cellular-
based vehicle to infrastructure (C-V2I) and millimeter-wave (mmWave). DSRC is
a short-range communication technology, which operates from 5.85GHz to 5.92GHz
band based on the 802.11p standard[6, 7], but it exhibits poor performance in
the case of high vehicle density [8]. C-V2I works with the cellular licensed
spectrum to provide wide geographical coverage [9, 10], but it is not capable
to support the real-time information exchange with very high data rate [10,
11, 12]. mmWave is the wireless communication technology which works on the
underutilized spectrum (i.e., 3-300GHz) to achieve multi-gigabit data rate for
autonomous driving. However, the propagation loss of mmWave channel is high
and obstructions will lead to high attenuation of mmWave channel [13, 14, 15,
16]. Traditional VECs that rely solely on one communication technology often
face challenges in meeting the communication and computing requirements for
automatic driving, especially when it comes to the support needed for task
offloading. By integrating the collective strengths of C-V2I, DSRC, and
mmWave, a heterogeneous VEC can substantially enhance the communication and
computing capability, making it facilitate the high-demand services for
automatic driving and others[2].

Ultra-reliable low-latency communication (URLLC) at VEC should satisfy the
basic performance requirements, i.e., low-latency between 10-100ms and store
1000 times higher data volumes compared with 4G systems[17, 18]. The system
utility, which includes communication fees and CPU energy consumption of the
VEC server, also plays a crucial role in the system planning and design. The
heterogeneous VEC usually needs to offload and process the tasks of various
types such as 3D game, VR and AR tasks. For the limited communication and
computation resources in the heterogeneous VEC, the resource allocation for
each type of tasks will significantly impact on the URLLC and system utility.
In heterogeneous VEC, it becomes crucial to explore how to create a resource
allocation strategy to meet the requirement of URLLC and minimize the system
utility in a VEC, where DSRC, C-V2I, and mmWave are integrated.

In this paper, we consider a heterogeneous VEC with multiple communication
technologies and various types of tasks and propose a resource allocation
policy to minimize the system utility while guaranteeing the URLLC requirement
in the heterogeneous VEC111The source code has been released at:
https://github.com/qiongwu86/URLLC-Awared-Resource-Allocation-for-
Heterogeneous-Vehicular-Edge-Computing. The major contributions of this paper
are summarized as follows:

  * •

An optimization problem is formulated. Since the stochastic network calculus
(SNC) offers a comprehensive model to describe the communication process, we
first adopt a moment generating function (MGF)-based SNC to construct the
network service of mmWave, DSRC and C-V2I. Then, the system utility consisting
of the communication utility and computation utility is characterized with an
explicit form. Later on, we formulate an optimization problem to minimize the
system utility while meeting URLLC requirement.

  * •

The optimization problem is converted. The SNC-based optimization problem
introduces the long-term constraints and complex interactions, which makes it
extremely difficult to address the optimization problem using the traditional
optimization methods. To overcome the challenges, we propose a Lyapunov-guided
DRL method to convert and solve the optimization problem. We adopt the
Lyapunov optimization to convert the long-term ultra-reliability constraint
into a short-term constraint. Based on the results, we reformulate the
optimization problem based on the short-term constraint and make it to be a
tractable optimization problem.

  * •

The optimization problem is solved. Deep reinforcement learning (DRL) emerges
as a potent and efficient solution for resource allocation in VEC due to its
rapid decision-making capabilities and adaptability to dynamic conditions.
Hence, we present a DRL-based solution after the problem conversion.
Specifically, we first design a DRL framework including the state, action and
reward, then adopt the soft actor-critic (SAC) algorithm to learn the optimal
resource allocation policy.

The remainder of this paper is organized as follows. Section II reviews the
related work. Section III introduces the system model which includes the data
arrival model, network service model and computing service model, then we
formulate the URLLC constraint and the system utility, and then formulate the
optimization problem. In Section IV we convert the ultra-reliability
constraint to be the short-term constraint and reformulate the optimization
problem. Then we present the DRL based solution. Various simulation results
are shown to demonstrate the performance of our proposed VEC system in Section
V. The conclusions are drawn in Section VI.

##  II Related Work

In this section, we first review the related works on the URLLC performance in
VEC, then survey the existing works on heterogeneous vehicular networks
(HetVNETs).

###  II-A URLLC in VEC

In recent years, there are many works studied the URLLC in VEC. In [19], Pan
_et al._ established the URLLC constraint based on the extreme value theory,
where the Lyapunov optimization was employed to decompose the task offloading
and computation optimization, while considering the long-term URLLC
constraints to present an asynchronous federated DQN-based algorithm to
maximize the throughput. In [20], Liao _et al._ developed a novel task
offloading framework for the air-ground integrated VEC. They modeled the long-
term URLLC constraints by putting a probabilistic requirement on the extreme
queue length and a high-order statistical requirement on the excess backlog,
and proposed an intent-aware upper confidence bound algorithm to maximize the
constraints including the quality of service (QoS) and URLLC. In [21],
Batewela _et al._ studied an URLLC communication problem for VEC, where the
risk-sensitive notion was leveraged to define a reliability metric, and
proposed the joint utility and policy estimation-based learning algorithm to
minimize the end-to-end task offloading delay of each vehicle. In [22], Cui
_et al._ combined communication and computation resource allocation to reduce
the total system cost consisting of latency and reliability, where a multi-
objective reinforcement learning policy was adopted to approach the optimal
solution in VEC. In [23], Zhu _et al._ explores reliability and queue length
violation in VEC with finite blocklength codes. They proposed the optimal and
learning-based solutions to minimize error probabilities to guarantee the
URLLC requirement. However, these works have not considered the multiple
wireless technologies to support the heterogeneous VEC.

###  II-B Heterogeneous Vehicular Networks

Some works have recently investigated the performance of vehicular networks by
integrating various communication technologies. In [2], Xiong _et al._
proposed a task offloading framework in HetVNETs by integrating DSRC, C-V2I
and mmWave communication technologies. In [6], Zheng _et al._ provided a
comprehensive survey on the advanced techniques applied to the vehicular
networks, and proposed a HetVNET where cellular communication technology was
integrated with DSRC to provide a potential solution to meet the communication
requirements of the intelligent transportation system. In [24], Posner _et
al._ investigated a federated vehicular network to support computation
intensive applications such as distributed machine learning and federated
learning by utilizing both DSRC and mmWave. In [25], Zhang _et al._ proposed a
distributed message dissemination scheme for vehicle-to-vehicle (V2V)
communications, where each vehicle was equipped with the two communication
interfaces including the DSRC interface and mmWave interface. In [26], Sheng
_et al._ proposed an intelligent 5G heterogeneous wireless network
architecture including a Q-learning based DSRC and mmWave to support V2V and
vehicle to infrastructure (V2I) communications, respectively. In [27], Ming
_et al._ proposed a hybrid V2V communication selection policy based on the
evolutionary game, where vehicles could select the communications mode (i.e.,
DSRC or long term evolution-based vehicle to everything (LTE-V2X) mode 4) to
improve the flexibility of packet transmissions. Despite the emerging trend of
integrating DSRC, C-V2I, and mmWave for enhanced vehicular communication, so
far works have not studied the crucial URLLC requirements within the
heterogeneous VEC that integrates DSRC, C-V2I, and mmWave. This is the issue
our work aims to address.

As mentioned above, no work investigated the resource allocation problem in
the heterogeneous VEC with multiple communications technologies and various
types of tasks to guarantee the URLLC requirement.

Figure 1: Heterogeneous VEC scenario

##  III System Model

Similar to[2], we consider the heterogeneous VEC network where vehicles are
equipped with three typical communication technologies, namely C-V2I, DSRC,
and mmWave, represented by the set
$\mathcal{G}=\left\\{mmw,dsrc,cv2i\right\\}$. This configuration allows for
offloading the tasks of $N$ types, denoted by the set $\mathcal{N}=\
\left\\{1,2,\cdots,N\right\\}$. The vehicles within the coverage of BS and RSU
offload tasks to the BS or RSU, which will forward the tasks to the server for
processing. Similar to [2], for the sake of research we assume that all
vehicles are always within the transmission range of these three communication
technologies, which is practice in urban scenarios where RSUs, BSs and other
infrastructures are densely deployed. The server has $N$ queues. It receives
the tasks from the vehicles and stores them in dedicated queues before tasks
are processed. In addition, it is equipped with $N_{E}$ CPU cores and the
total CPU frequency of all CPU cores is $f_{E}$ cycles per second.

The offloading process is divided into $T$ discrete time slots with the equal
duration $\varDelta t$. In the beginning of the each time slot $t$, each
vehicle randomly generates one type tasks. The primary responsibility of the
VEC server installed to some RSUs or BSs along the road is making the
communication resource allocation policy and the computation resource
allocation policy through training. Note that the VEC server’s computation
resource is substantial, and it is well-equipped to handle the computing
demands of training. VEC server determines how to allocate communication
resource and computation resource. The communication resource allocation
policy among three communication technologies, which are responsible for
offloading tasks of each type $i$. The communication resource allocation
proportions for these technologies are denoted as $\varphi^{mmw}_{i}(t)$ for
mmWave, $\varphi^{dsrc}_{i}(t)$ for DSRC, and $\varphi^{cv2i}_{i}(t)$ for
CV2I. VEC server also has a pivotal role in allocating computation resource.
Specifically, it is tasked with deciding the CPU frequency allocation for
processing tasks of each type $i$, denoted as $\alpha_{i}(t)$. Once the
communication and computation resource allocation policies are made, vehicles
offload tasks of each type $i$ in accordance with the guidelines set by the
VEC server. Following this, the BS or the RSU receive the offloaded tasks and
subsequently forwards them to the queue within the VEC server. Then the VEC
server processes these tasks methodically and efficiently according to the
established computation resource allocation policy. After the processing is
finished, the VEC server provides feedback on the results to the respective
vehicles. The heterogeneous VEC scenario is shown in Fig. 1.

Note that due to the fact that the processing results are normally much
smaller than the tasks, the latency caused by the downlink transmission is
neglected in this paper. In addition, the transmission rate of the wired link
is relatively large[28], the delay at wired link is also not considered. Hence
we consider the uplink transmission to construct the network service model and
the computing process to construct the computing service model, then we derive
the low-latency constraint based on the network service model and computing
service model.

###  III-A Data Arrival Model

The cumulative amount of the tasks that arrive at queue $i$ within time
interval $[s_{1},s_{2})$ ($0\leq s_{1}<s_{2}$) is denoted as
$A_{i}\left(s_{1},s_{2}\right)=\sum^{s_{2}-1}_{t=s_{1}}a_{i}(t)$, where
$a_{i}(t)=n_{i}d_{i}$ is the amount of the $i$th type tasks arriving to queue
$i$ in time slot $t$, $n_{i}$ is the number of the $i$th type tasks which
follows Poisson distribution with arrival rate $\lambda_{i}$, and $d_{i}$ is
the constant size of the $i$th type tasks. $A_{i}\left(s_{1},s_{2}\right)$ has
the statistical envelope $\left(\rho_{i},\sigma_{i}\right)$ which is referred
to as the exponentially bounded burstiness (EBB) and defined to provide a
guarantee of the form with a violation probability $\varepsilon^{a}_{i}$[29],

$P[A_{i}\left(s_{1},s_{2}\right)>\rho_{i}(s_{2}-s_{1})+\sigma_{i}]\leq\varepsilon^{a}_{i},$
(1)

where $\rho_{i}$ and $\sigma_{i}$ are the long-term average arrival rate and
burstiness of the amount of the $i$th type tasks, respectively.

###  III-B Ultra-reliability Constraint

The required frequencies to process per bit of data are different for various
types of tasks. Let $w_{i}$ be the required frequency to process per bit of
the $i$th type tasks, thus the amount of the $i$th types of tasks processed by
the VEC server in time slot $t$ is calculated as
$\frac{f_{E}\alpha_{i}(t)}{\omega_{i}}$. Thus the backlog of queue $i$ in time
slot $t+1$ is calculated as

$q_{i}(t+1)=\left[q_{i}(t)+a_{i}(t)-\frac{f_{E}\alpha_{i}(t)}{\omega_{i}}\right]^{+},$
(2)

where $q_{i}(t)$ is the backlog of queue $i$ in time slot $t$.

The reliability of communication is influenced by the stability of the queue.
If a queue becomes unstable, the tasks arriving at the queue may be dropped.
The ultra-reliable communications are achieved by maintaining strong stability
of each queue $i$ [30], i.e.,

$\mathop{\lim}\limits_{t\to\infty}\sup\frac{1}{t}\sum^{t-1}_{m=0}\sum\limits_{i=1}^{N}\mathbb{E}\left[q_{i}\left(m\right)\right]<\infty,\quad\forall
i\in\mathcal{N},\forall t\in\mathcal{T}.$ (3)

Our work focuses on the long-term stability of the queue in the networking
layer of a dynamic vehicular network to achieve ultra-reliable performance.
While conserving resources, not all CPU frequencies are used to process tasks
instantly, resulting in tasks being stored in a queue. The key is ensuring
that this queue length remains stable over time rather than consistently
growing. While factors like dynamic vehicular network, changing V2R channel
status, and vehicle mobility affect latency, they are primarily physical layer
concerns and don’t directly affect the long-term constraints of the networking
layer.

###  III-C Network Service Model

Next, we will introduce the network service model of mmWave, DSRC and C-V2I.
The network service of a communication technology $g$ is the transmission
capacity of $g$, i.e., the largest amount of all arrival tasks that $g$ can
provide after excluding interference.

####  III-C1 Network Service Model of mmWave

The small-scale fading effects are shown to be negligible in mmWave bands due
to the short wavelength and the channel fading is dominated by the shadowing
effect[31], thus its transmission rate in time slot $t$ can be calculated
according to the Shannon theory, i.e.,
$C(t)=Blog_{2}\left(1+\zeta(t)\gamma_{sinr}(t)l^{-\delta}\right)$, where $l$
is the average transmission distance for the vehicles, $\delta$ is the path
loss exponent, $\gamma_{sinr}(t)$ is the signal to interference plus noise
ratio (SINR) in time slot $t$, $B$ is the aggregated system bandwidth,
$\zeta(t)$ is the amplitude of the mmWave channel gain coefficient in time
slot $t$. Similar with [31] and [32], we consider $\zeta(t)$ follows
Nakagami-$m$ distribution, i.e., $\zeta(t)\sim\Gamma\left(M,M^{-1}\right)$,
$M$ is the Nakagami index. Thus the network service of mmWave within a time
interval $[s_{1},s_{2})$ is given by

$\begin{aligned} &\beta^{mmw}\left(s_{1},s_{2}\right)\\\
&\qquad=\sum^{s_{2}-1}_{t=s_{1}}C(t)=\eta\sum^{s_{2}-1}_{t=s_{1}}In\left(1+\zeta(t)\gamma_{sinr}(t)l^{-\delta}\right)\end{aligned},$
(4)

where $\eta=Blog_{2}e$ and $t\in[s_{1},s_{2})$. Assuming that
$\gamma_{sinr}(t)$ and $\zeta(t)$ are independent and identically distributed,
we can further get $\beta^{mmw}\left(s_{1},s_{2}\right)=\eta
In\left(1+\zeta\gamma_{sinr}l^{-\delta}\right)\left(s_{2}-s_{1}\right)$.

According to Leftover service theorem, the network service of mmWave for the
$i$th type of tasks is calculated as

$\begin{aligned} &\beta_{i,comm}^{mmw}\left({s_{1},s_{2}}\right)=\\\
&{\left[{{\beta^{mmw}}(s_{1},s_{2})-\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{mmw}(t)}a_{j}(t)}\right]^{+}}\end{aligned},$
(5)

where $\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{mmw}(t)}a_{i}(t)$ is the
network service of mmWave for the tasks of the other types, which reflects the
interference from transmitting the tasks of the other types with the same
spectrum resources, and $\left[x\right]^{+}=max\left(x,0\right)$. Similar with
[2], we consider the network service of mmWave is much larger than the network
service of mmWave for the tasks of the other types, the symbol "$+$" in Eq.
(6) can be ignored and we can further get

$\begin{aligned}
\beta_{i,comm}^{mmw}\left({s_{1},s_{2}}\right)={{{\beta^{mmw}}(s_{1},s_{2})-\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{mmw}(t)}a_{j}(t)}}\end{aligned},$
(6)

####  III-C2 Network Service Model of DSRC

According to the IEEE 802.11p standard, the access delay is the predominant
delay of DSRC [2]. Therefore, according to the classical latency-rate
service[33], the network service of DSRC is calculated as

${\beta^{dsrc}}\left({s_{1},s_{2}}\right)={R^{dsrc}}{\left[{(s_{2}-s_{1})-{{\hat{t}}_{serv}}}\right]}{\rm{}},$
(7)

where $R^{dsrc}$ is the largest transmission rate of DSRC, $\hat{t}_{serv}$ is
the average access delay of DSRC which has a Pareto-type tail with an exponent
of $\partial$ [34], i.e., $\hat{t}_{serv}\sim
u\left(R^{dsrc}\right)^{-\partial}$, here $u$ is a constant[35]. Similar to
Eq. (6), the network service of DSRC for the $i$th type of tasks is modeled as

$\begin{aligned}
\beta_{i,comm}^{dsrc}\left({s_{1},s_{2}}\right)={{{\beta^{dsrc}}(s_{1},s_{2})-\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{dsrc}(t)}a_{j}(t)}}\end{aligned},$
(8)

####  III-C3 Network Service Model of C-V2I

C-V2I is a communication technology pre-reserved bandwidth resources for the
tasks of different types in advance, thus the tasks of different types would
not compete the network service. Hence, the network service of C-V2I for the
$i$th type tasks is

${\beta_{i,comm}^{cv2i}}\left({s_{1},s_{2}}\right)=R_{i}^{cv2i}(s_{2}-s_{1}),$
(9)

where $R_{i}^{cv2i}$ is the largest transmission rate of C-V2I.

###  III-D Computing Service Model

The computing service for the $i$th type tasks is the amount of the $i$th type
tasks processed by the VEC server within time interval $[s_{1},s_{2})$, thus
the computing service provided for the $i$th type tasks is calculated as

$\beta_{i,{comp}}(s_{1},s_{2})=\sum^{s_{2}-1}_{t=s_{1}}\frac{{{f_{E}}{\alpha_{i}(t)}}}{{{\omega_{i}}}}.$
(10)

According to the Leftover service theorem, the $i$th type tasks offloaded by a
communication technology $g$ would compete the computing service with those
offloaded by the other communication technologies. The computing service of
the other communication technologies for the $i$th type tasks is

$\begin{aligned}
\sum_{\mathcal{G}/{\rm{g}}}\sum^{s_{2}-1}_{t=s_{1}}{a_{i,comp}^{\rm{g}}\left(t\right)}=\sum_{\mathcal{G}/{\rm{g}}}\sum^{s_{2}-1}_{t=s_{1}}\varphi_{i}^{g}(t)a_{i}(t)\end{aligned},$
(11)

where ${\mathcal{G}/{\rm{g}}}$ indicates the communication technologies
excluding communication technology $g$, thus the computing service of
communication technology $g$ for the $i$th type tasks is calculated as

$\begin{aligned}
\beta_{i,comp}^{g}\left({s_{1},s_{2}}\right)={\beta_{i,{comp}}(s_{1},s_{2})-\sum_{\mathcal{G}/{\rm{g}}}\sum^{s_{2}-1}_{t=s_{1}}{a_{i,comp}^{\rm{g}}\left(t\right)}}\end{aligned}.$
(12)

###  III-E Low-latency Constraint

Since the task arrival and network service are stationary random process, the
probabilistic delay upper bound is adopted to define the delay upper bound of
communication technology $g$ for the $i$th type tasks, which is denoted by
$\omega_{i}^{g}(t)$, i.e.,

$P\left({W_{i}^{g}\left(t\right)\geq\omega_{i}^{g}(t)}\right)\leq{\varepsilon_{i}},$
(13)

where ${\varepsilon_{i}}$ is the violation probability for the $i$th type
tasks, $W_{i}^{g}\left(t\right)$ is the delay of communication technology $g$
for the $i$th type tasks, which is calculated as [36]

$W_{i}^{g}\left(t\right)=inf\left\\{{w_{i}\geq 0:\left({A_{i}\oslash
S_{i}^{g}}\right)\left({t+\omega_{i}^{g}(t),t}\right)\leq 0}\right\\},$ (14)

where $\oslash$ is the mini-plus deconvolution operator, here
$S_{i}^{g}(t+\omega_{i}^{g}(t),t)$ is the system service of communication
technology $g$ for the $i$th type of tasks, which consists of network service
and computing service.

According to Eq. (14) and delay bound theorem [36], Eq. (13) can be converted
to

$P\left({W_{i}^{g}\left(t\right)\geq\omega_{i}^{g}(t)}\right)=P\left({A_{i}\oslash
S_{i}^{g}\left({t+\omega_{i}^{g}(t),t}\right)\geq 0}\right).$ (15)

The upper bound of
$P\left({W_{i}^{g}\left(t\right)\geq\omega_{i}^{g}(t)}\right)$ can be obtained
as

$\begin{aligned}
&P\left({W_{i}^{g}\left(t\right)\geq\omega_{i}^{g}(t)}\right)\leq\frac{{{e^{\theta\left({{\sigma_{i}}+\eta_{i,comm}^{g}(t)+\eta_{i,comp}^{g}(t)}\right)}}}}{{{e^{-\theta\xi_{i,{comp}}^{g}(t)}}-{e^{-\theta\xi_{i,{comm}}^{g}(t)}}}}\\\
&\left\\{{\frac{{{e^{-\theta\xi_{i,{comp}}^{g}(t)\omega_{i}^{g}(t)}}}}{{{e^{\theta\xi_{i,{comp}}^{g}(t)}}-{e^{\theta\varphi^{g}_{i}(t){\rho_{i}}}}}}-\frac{{{e^{-\theta\xi_{i,{comm}}^{g}(t)\omega_{i}^{g}(t)}}}}{{{e^{\theta\xi_{i,{comm}}^{g}(t)}}-{e^{\theta\varphi^{g}_{i}(t){\rho_{i}}}}}}}\right\\}\end{aligned}.$
(16)

proof: See Appendix A.

After setting the upper bound equal to ${\varepsilon_{i}}$, a closed-form
solution of $\omega_{i}^{g}(t)$ can be obtained, i.e.,

$\omega_{i}^{g}(t)=\begin{cases}&\frac{{-In({\varepsilon_{i}})}}{{\theta\xi_{i,{comm}}^{g}(t)}}+\frac{\Delta}{{\xi_{i,{comm}}^{g}(t)}}-{\chi^{g}},\\\
&\qquad\qquad\qquad{\xi_{i,{comp}}^{g}(t)>\xi_{i,{comm}}^{g}(t)}\\\
&\frac{{-In({\varepsilon_{i}})}}{{\theta\xi_{i,{comp}}^{g}(t)}}+\frac{\Delta}{{\xi_{i,{comp}}^{g}(t)}}-{\chi^{comp}},\\\
&\qquad\qquad\qquad{\xi_{i,{comm}}^{g}(t)>\xi_{i,{comp}}^{g}(t)}\end{cases},$
(17)

here
${\chi^{g}}={\frac{{In[({e^{-\theta\xi_{i,{comm}}^{g}(t)}}-{e^{-\theta\xi_{i,{comp}}^{g}(t)}})({e^{\theta\xi_{i,{comm}}^{g}(t)}}-{e^{\theta{\varphi^{g}_{i}(t)}{\rho_{i}}}})]}}{{\theta\xi_{i,{comm}}^{g}(t)}}},{\chi^{comp}}={\frac{{In[({e^{-\theta\xi_{i,{comp}}^{g}(t)}}-{e^{-\theta\xi_{i,{comm}}^{g}(t)}})({e^{\theta\xi_{i,{comp}}^{g}(t)}}-{e^{\theta{\varphi^{g}_{i}(t)}{\rho_{i}}}})]}}{{\theta\xi_{i,{comp}}^{g}(t)}}}$,
$\Delta={\sigma_{i}}+\eta_{i,comm}^{g}(t)+\eta_{i,comp}^{g}(t)$. From Eq.
(17), we can get an important conclusion, i.e., the upper bound
$\omega_{i}^{g}(t)$ is determined by the less service between the computing
service $\xi_{i,{comp}}^{g}(t)$ and the network service
$\xi_{i,{comm}}^{g}(t)$.

The communications in the VEC are low-latency if the delay upper bound
satisfies the maximum latency requirement, i.e.,

$\max\\{\omega_{i}^{mmw}(t),\omega_{i}^{dsrc}(t),\omega_{i}^{cv2i}(t)\\}\leq
T_{i}^{\max},\forall i\in\mathcal{N},\forall t\in\mathcal{T},$ (18)

where $\omega_{i}^{mmw}(t)$, $\omega_{i}^{dsrc}(t)$ and $\omega_{i}^{cv2i}(t)$
are the delay upper bounds of the $i$th type tasks transmitted by mmWave, DSRC
and C-V2I in time slot $t$, respectively, and $T_{i}^{max}$ is the maximum
latency requirement of the $i$th type tasks.

###  III-F System Utility

The system utility consists of the communication utility and computation
utility.

####  III-F1 Communication Utility

The communication utility is defined as the charges to offload tasks. As C-V2I
works on the licensed band[17], while DSRC and mmWave operate on the license-
free band [37, 38], thus only C-V2I will generate charges for data
transmission. In this case, the communication utility to offload the $i$th
type tasks in time slot $t$ is

${F_{comm,i}}(t)={c^{comm}}\sum\limits_{t=s_{1}}^{s_{2}-1}\varphi_{i}^{cv2i}(t)a_{i}(t),$
(19)

where $c^{comm}$ is the unit price to offload one Megabyte tasks through
C-V2I.

####  III-F2 Computation Utility

The computation utility is defined as the charges of the power consumption at
the VEC server due to task processing. The unit price of power consumption is
$c^{comp}$ and the power consumption of each CPU $n_{e}$
(${n_{e}}=1,2,...,N_{E}$) is $P_{n_{e}}^{comp}$. Thus, the computation utility
of each CPU $n_{e}$ in time slot $t$ can be expressed as
${F_{comp,n_{e}}}(t)={c^{comp}}P_{n_{e}}^{comp}$. According to the dynamic
voltage frequency scaling (DVFS) approach which has been widely applied to
construct the realistic CPU power consumption [39], $P_{n_{e}}^{comp}$ is
calculated as

$P_{n_{e}}^{comp}=\kappa{\left({{f_{E}}\sum\limits_{j=1}^{N}{{\alpha_{j}(t)}}/{N_{E}}}\right)^{3}},$
(20)

where $\kappa$ is the hardware-related effective switching capacitance
parameter.

Based on the above analysis, the system utility is

$F(t)={\varpi_{1}}\sum\limits_{{n_{e}}=1}^{{N_{E}}}{{F_{comp,{n_{e}}}}(t)}+{\varpi_{2}}\sum\limits_{i=1}^{N}{{F_{comm,i}}(t)},$
(21)

where ${\varpi_{1}}$ and ${\varpi_{2}}$ are the normalized weighting factors
to ensure that the magnitudes of the communication utility and computation
utility are uniformed.

###  III-G Optimization Problem

Our optimization problem aims to minimize the system utility under the
resource constraint and the URLLC constraint. Thus the optimal problem is
formulated as

$\displaystyle
P1:\qquad\qquad\mathop{\min}\limits_{{\boldsymbol{\alpha}}\left(t\right),{\boldsymbol{\varphi}}\left(t\right)}F(t)$
(22a) $\displaystyle s.t.,\;\;$
$\displaystyle\mathop{\lim}\limits_{t\to\infty}sup\frac{1}{t}\sum\limits_{m=0}^{t-1}\sum\limits_{i=1}^{N}\mathbb{E}\left[{{q_{i}}\left(m\right)}\right]<\infty$
(22b) $\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad,\forall
i\in\mathcal{N},\forall t\in\mathcal{T}$
$\displaystyle\max\\{\omega_{i}^{mmw}(t),\omega_{i}^{dsrc}(t),\omega_{i}^{cv2i}(t)\\}\leq
T_{i}^{\max}$ (22c) $\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad,\forall
i\in\mathcal{N},\forall t\in\mathcal{T}$
$\displaystyle\sum\limits_{i=1}^{N}{{\alpha_{i}(t)}}\leq 1,\quad\forall
i\in\mathcal{N},\forall t\in\mathcal{T}$ (22d) $\displaystyle\sum\limits_{g\in
G}{\varphi_{i}^{g}(t)}=1,\quad\forall i\in\mathcal{N},\forall
t\in\mathcal{T},$ (22e)

where
${\boldsymbol{\alpha}}(t)=\left[{{\alpha_{1}}(t),{\alpha_{2}}(t),...,{\alpha_{N}}(t)}\right]$
is the computation resource allocation policy in time slot $t$,
${\boldsymbol{\varphi}}(t)=\left[{\boldsymbol{\varphi_{1}^{g}}(t),\boldsymbol{\varphi_{2}^{g}}(t),...,\boldsymbol{\varphi_{N}^{g}}(t)}\right]$
is the communication resource allocation policy in time slot $t$, here
${\boldsymbol{\varphi_{i}^{g}}}(t)=[\varphi_{i}^{mmw}(t),\varphi_{i}^{dsrc}(t),\varphi_{i}^{cv2i}(t)]$.
Constraint (22c) is the low-latency constraint and constraint (22b) is the
ultra-reliability constraint; constraint (22d) guarantees that the CPU
frequency allocated to process the tasks of all types cannot exceed the total
available CPU frequency; constraint (22e) imposes the tasks of all types
should be offloaded.

It is noteworthy that the decision variables in constraints (22b) and (22c)
are intertwined. Moreover, addressing the long-term constraint (22b) needs the
future insights into the queue backlog, which are difficult to estimate.
Furthermore, the delay upper bound for offloading each type tasks through each
communication technology is not isolated. The complexity of the above
interactions makes it extremely difficult to address the optimization problem
$P1$ using the traditional optimization methods. To overcome these challenges,
we propose a Lyapunov-guided DRL method to convert and solve the optimization
problem.

##  IV Lyapunov-guided DRL Based Conversion and Solution

In this section, we propose a Lyapunov-guided DRL method to convert and solve
the optimization problem. Specifically, since the Lyapunov optimization
technique can be employed to deal with these long-term constraints
effectively, we first adopt the Lyapunov optimization to convert the long-term
constraints to a tractable form that does not require the knowledge of future
events. After that, due to that DRL can optimize policies over time through
learning and dynamically adapt the policies according to the real-time network
conditions, we then employ DRL for efficient decision-making in face of
complex interactions among the tasks of different types and different
communication technologies in the heterogeneous VEC.

###  IV-A Lyapunov-based Conversion

In this section, we apply the Lyapunov optimization to convert $P1$ into
short-term decision problem. Let $\Delta\left(\boldsymbol{Q}(t)\right)$ be the
conditional Lyapunov drift in time slot $t$ which is calculated as [30]

$\Delta\left(\boldsymbol{Q}(t)\right)\buildrel\Delta\over{=}\mathbb{E}\left\\{{L\left(\boldsymbol{Q}(t+1)\right)-L\left(\boldsymbol{Q}(t)\right)|\boldsymbol{Q}(t)}\right\\},$
(23)

where $\boldsymbol{Q}(t)=\left[q_{1}(t),q_{2}(t),...q_{N}(t)\right]$ is the
queue backlog in time slot $t$, $L\left(\boldsymbol{Q}(t)\right)$ is the
Lyapunov function which measures the average queue backlog in time slot $t$,
and is calculated as
$L\left(\boldsymbol{Q}(t)\right)=\frac{1}{2}\sum\limits_{i=1}^{N}{{q_{i}}}{\left(t\right)^{2}}$.
Squaring both sides of Eq. (2) and substituting it into Eq. (23), then using
the fact that $\left\\{[x]^{+}\right\\}^{2}={(\max[x,0])^{2}}\leq{x^{2}}$
($x\in\mathbb{R}$), $\Delta\left(\boldsymbol{Q}(t)\right)$ is upper bounded by

$\begin{aligned}
&\Delta\left(\boldsymbol{Q}(t)\right)\leq\frac{1}{2}\mathbb{E}\left\\{\sum\limits_{i=1}^{N}{{{\left({{a_{i}}(t)-\left[\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}\right]}\right)^{2}}}\left|{\boldsymbol{Q}\left(t\right)}\right.}\right\\}+\\\
&\quad\mathbb{E}\left\\{\sum\limits_{i=1}^{N}{{q_{i}}(t)\left({a_{i}}(t)-\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}\right)\left|{\boldsymbol{Q}\left(t\right)}\right.}\right\\}\end{aligned},$
(24)

Let ${a^{max}_{i}}$ be the upper bound of ${a_{i}}(t)$, thus we have
${a_{i}}(t)\leq{a^{max}_{i}}$. Moreover, since ${\alpha_{i}}(t)\leq 1$,
according to drift-plus-penalty bound[40], the first term in the right hand
side of Eq. (24) can be upper bounded by

$\begin{aligned}
&\frac{1}{2}\mathbb{E}\left\\{\sum\limits_{i=1}^{N}{{{\left({{a_{i}}(t)-\left[\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}\right]}\right)^{2}}}\left|{\boldsymbol{Q}\left(t\right)}\right.}\right\\}\leq\\\
&\qquad\frac{1}{2}\mathbb{E}\left\\{\sum\limits_{i=1}^{N}{{{\left({[{a^{max}_{i}}]^{2}-\left[\frac{{{f_{E}}}}{{{\omega_{i}}}}\right]^{2}}\right)}}\left|{\boldsymbol{Q}\left(t\right)}\right.}\right\\}=B\end{aligned}.$
(25)

Substituting Eq. (25) into Eq. (24), we have

$\begin{aligned} \Delta\left(\boldsymbol{Q}(t)\right)\leq
B+\mathbb{E}\left\\{\sum\limits_{i=1}^{N}{{q_{i}}(t)\left({a_{i}}(t)-\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}\right)\left|{\boldsymbol{Q}\left(t\right)}\right.}\right\\}\end{aligned}.$
(26)

We then will apply the opportunistic expectation minimization technique to
process the last term in the right hand side of Eq. (26), thus
$\Delta\left(\boldsymbol{Q}(t)\right)$ can be upper bound by

$\begin{aligned} \Delta\left(\boldsymbol{Q}(t)\right)&\leq
B-\sum\limits_{i=1}^{N}{q_{i}}(t)\left(\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}-a_{i}(t)\right)\end{aligned}.$
(27)

Letting $\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}^{g}(t)}}}-a_{i}(t)$ be
$\epsilon$, and thus Eq. (27) can be rewritten as
$\Delta\left(\boldsymbol{Q}(t)\right)\leq
B-\epsilon\sum\limits_{i=1}^{N}q_{i}(t)$. According to the conditional
Lyapunov drift theorem[30], if conditional Lyapunov drift
$\Delta\left(\boldsymbol{Q}(t)\right)$ is upper bounded by
$B-\epsilon\sum\limits_{i=1}^{N}q_{i}(t)$ for each time slot
$t\in\mathcal{T}$, we have

$\begin{aligned}
\mathop{\lim}\limits_{t\to\infty}sup\frac{1}{t}\sum\limits_{m=0}^{t-1}\sum\limits_{i=1}^{N}\mathbb{E}\left[{{q_{i}}\left(m\right)}\right]\leq\frac{B}{\epsilon}\end{aligned}.$
(28)

In this case, the ultra-reliability constraint (22b) can be achieved. Hence,
in order to satisfy the ultra-reliability constraint (22b), we just need to
ensure that the conditional Lyapunov drift
$\Delta\left(\boldsymbol{Q}(t)\right)$ has a minimum tight upper bound, i.e.,
an optimal ${\boldsymbol{\alpha}}(t)$ is existed to minimize the right hand
side of Eq. (27), thus we have

$\begin{aligned}
\mathop{\min}\limits_{{\boldsymbol{\alpha}}\left(t\right)}:\sum\limits_{i=1}^{N}{q_{i}}(t)a_{i}(t)-\sum\limits_{i=1}^{N}{{{q_{i}}(t)\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}}}\end{aligned}.$
(29)

According to Eq. (29), ${\boldsymbol{\alpha}}(t)$ is obtained based on the
current queue backlog $\boldsymbol{Q}(t)$, but not the long-term queue
backlog, hence the long-term constraint (22b) can be transformed to a
tractable short-term decision problem.

Then we combine the objective of $P$1 with the short-term optimization
objective, i.e., Eq. (29), thus the optimization problem $P$1 can be converted
into the optimization problem $P$2 as follows

$\displaystyle\mathop{\min}\limits_{{\boldsymbol{\alpha}}\left(t\right),{\boldsymbol{\varphi}}\left(t\right)}$
$\displaystyle:V\cdot\left(\sum\limits_{i=1}^{N}{q_{i}}(t)a_{i}(t)-\sum\limits_{i=1}^{N}{{{q_{i}}(t)\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}}}\right)+F(t)$
(30a) $\displaystyle s.t.,$ $\displaystyle\ \ \eqref{eq32_b}-\eqref{eq32_e},$
(30b)

where $V$ is a non-negative coefficient that enables a tradeoff between the
ultra-reliability constraint and system utility.

However, problem $P$2 is non-convex and the dimensionality is high for the
complex and dynamic heterogeneous VEC environment, thus solving problem $P$2
based on the traditional optimization method such as the convex optimization
method would cause the curse of dimensionality. Moreover, the traditional
optimization method struggle to adapt to dynamic environments and can not
provide the real-time solutions, which is critical requirement for VEC.
Consequently, finding a solution for problem $P$2 is far from trivial. In
recent years, there has been a growing trend in the research community to
employ DRL to solve the non-convex optimization problems. This is due to its
robustness and adaptability in solving complex, non-convex problems in dynamic
and high-dimensional environments. Therefore, DRL emerges as an ideal choice
for addressing problem $P$2 in our study.

###  IV-B DRL-based Solution

####  IV-B1 DRL Framework Construction

To address problem $P$2, we model the offloading process as a DRL process,
where the VEC server acts as a agent. In each time slot $t$, the VEC server
first observes the current state $s_{t}$, then decides the current action
$a_{t}$ based on $s_{t}$ according to policy and executes it. After that the
VEC server receives a reward $r_{t}$ from the system, then the current state
$s_{t}$ is transited to the next state $s_{t+1}$ and the process enters the
next time slot. Next, we construct the DRL framework, namely state, action,
and reward.

##### State

Considering that the task arrival is random in VEC and can significantly
influence the system performance, we set the amount of the arrival tasks in
each time slot $t$, denoted as ${{\cal
A}_{t}}=[{a_{1}(t)},{a_{2}(t)},...{a_{N}(t)}]$, as the first element of the
state. Furthermore, the queue backlog is another critical factor that
influences the ultra-reliability of the system, particularly in the scenarios
with high vehicular density[41]. Therefore, we consider ${\cal
Q}_{t}=\left[{{q_{1}}\left(t\right),{q_{2}}\left(t\right),...{q_{N}}\left(t\right)}\right]$
as the second element of the state. Moreover, according to Eq. (17), the upper
bound delay $\omega_{i}^{g}(t)$ is determined by ${\bf{\xi}}_{i,comm}^{g}(t)$
and ${\bf{\xi}}_{i,comp}^{g}(t)$. Since mmWave has sufficient radio resources,
we do not consider the competition for the network service of mmWave among
tasks of different types. Moreover, as explained in sub-section III-C3, there
is no competition for the network service of C-V2I among different types of
tasks, thus we do not consider ${\bf{\xi}}_{i,comm}^{mmw}(t)$ and
${\bf{\xi}}_{i,comm}^{cv2i}(t)$ as the state. Hence the third element of the
state is designed as
${{\boldsymbol{\xi}}_{t}}=\left[{{{\boldsymbol{\xi}}_{1}}(t),{{\boldsymbol{\xi}}_{2}}(t),...{{\boldsymbol{\xi}}_{N}}(t)}\right]$,
where
${{\boldsymbol{\xi}}_{i}}(t)=\left[{{\bf{\xi}}_{i,comm}^{dsrc}(t),{\bf{\xi}}_{i,comp}^{cv2i}(t),{\bf{\xi}}_{i,comp}^{mmw}(t),{\bf{\xi}}_{i,comp}^{dsrc}(t)}\right]$.
Therefore, the state in time slot $t$ is formulated as

$\boldsymbol{s}_{t}=\left[{{{\cal A}_{t}},{{\cal
Q}_{t}},{{\boldsymbol{\xi}}_{t}}}\right].$ (31)

##### Action

The VEC server takes actions to adjust ${\boldsymbol{\alpha}}(t)$ and
${\boldsymbol{\varphi}}(t)$ in each time slot $t$, thus the action taken by
the VEC server in time slot $t$ is given by

$\boldsymbol{a}_{t}=\left[{{\boldsymbol{\alpha}}(t),{\boldsymbol{\varphi}}(t)}\right].$
(32)

##### Reward Function

The target of the DRL is to maximize the long-term discount reward, while we
aim to minimize the objective of $P2$, thus the reward of the DRL framework is
formulated as the negative of the objective of problem $P2$. In addition, we
put a large penalty term into the reward for the penalty that violates the
constraint (22c). Hence, the reward function is defined as

$\displaystyle{r_{t}}$
$\displaystyle=V\cdot\left(-\sum\limits_{i=1}^{N}{q_{i}}(t)a_{i}(t)+\sum\limits_{i=1}^{N}{{{q_{i}}(t)\frac{{{f_{E}}{\alpha_{i}}(t)}}{{{\omega_{i}}}}}}\right)-F(t)$
(33) $\displaystyle-
P_{e}\cdot\\{\max\\{\omega_{i}^{mmw}(t),\omega_{i}^{dsrc}(t),\omega_{i}^{cv2i}(t)\\}-T_{i}^{\max}\\},$

where $P_{e}$ is a penalty weight.

####  IV-B2 Solution

The SAC algorithm is known for its stability and robustness, and SAC
encourages exploration and maintains a more diverse range of actions, thus SAC
is suitable for optimizing the policy in the complex VEC with multiple types
of tasks and communication technologies.

According to SAC, we formulate the expected long-term discount reward when the
policy under $\boldsymbol{a}_{t}$ and $\boldsymbol{s}_{t}$, denoted by
${\boldsymbol{\pi}\left({\boldsymbol{a}_{t}|\boldsymbol{s}_{t}}\right)}$, is
adopted, i.e.,

$J({\boldsymbol{\pi}\left({\boldsymbol{a}_{t}|\boldsymbol{s}_{t}}\right)})={E}\left[{\sum\limits_{t=0}^{T}{{\gamma^{t-1}}}r_{t}+\beta{\cal
H}\left({{\boldsymbol{\pi}\left({\cdot|\boldsymbol{s}_{t}}\right)}}\right)}\right],$
(34)

where $\boldsymbol{\pi}\left({\cdot|\boldsymbol{s}_{t}}\right)$ is the policy
when all available actions are taken under $\boldsymbol{s}_{t}$,
$\gamma\in\left[{0,1}\right]$ is the discounting factor, ${\cal
H}\left({{\pi\left({\cdot|\boldsymbol{s}_{t}}\right)}}\right)={E}\left[\log{\pi\left({\cdot|\boldsymbol{s}_{t}}\right)}\right]$
is the policy entropy, $\beta$ is the tradeoff weight of the policy entropy
between exploring feasible policy and maximizing reward, which can be adjusted
dynamically and formulated as follows,

$\beta^{*}=\arg{\min_{{\beta}}}{E}\left[{-{\alpha_{t}}log\pi^{*}\left({{\boldsymbol{a}_{t}}|{\boldsymbol{s}_{t}}}\right)-{\beta}\overline{\cal
H}}\right],$ (35)

where $\bar{\cal H}=dim({\boldsymbol{a}_{t}})$,
${\pi^{*}\left({{\boldsymbol{a}_{t}}|{\boldsymbol{s}_{t}}}\right)}$ is the
optimal policy under $\boldsymbol{a}_{t}$ and $\boldsymbol{s}_{t}$ to maximize
$J({\boldsymbol{\pi}\left({{\boldsymbol{a}_{t}}|{\boldsymbol{s}_{t}}}\right)})$.

##### Training stage

The architecture of the SAC algorithm includes an actor network, two critic
networks and two target critic networks, where all these networks are the deep
neural network (DNN). Let $\phi$ be the parameters of the actor network,
${\psi_{1}}$ and ${\psi_{2}}$ be the parameters of the two critic networks,
$\bar{\psi}_{1}$ and $\bar{\psi}_{2}$ be the parameters of the two target
critic networks. The pseudocode of the training stage for the SAC algorithm is
described in Algorithm 1.

Input: $V$,$\gamma$, $\phi$, ${\psi_{1}}$, ${\psi_{2}}$, $\bar{\psi}_{1}$,
$\bar{\psi}_{2}$, $\beta$

Output: optimized $\phi^{*}$

1 Randomly initialize the $\phi$, ${\psi_{1}}$, ${\psi_{2}}$, $\beta$;

2

3Initialize target networks by $\bar{\psi}_{1}\leftarrow\psi_{1}$,
$\bar{\psi}_{2}\leftarrow\psi_{2}$;

4 Initialize replay experience buffer $\mathcal{R}$;

5 for  _episode from $1$ to $K^{train}_{max}$ _ do

6 Initialize millimeter-wave channel coefficients $\zeta$;

7 Reset simulation parameters for the system model;

8

9 for  _time slot $t$ from $0$ to $T^{train}_{max}$ _ do

10 Receive observation state $\boldsymbol{s}_{t}$;

11 Generate the CPU allocation policy and communication offloading policy
$\boldsymbol{a}_{t}$ ;

12 Execute action $\boldsymbol{a}_{t}$, observe reward $r_{t}$ and new state
$\boldsymbol{s}_{t+1}$ from the system model;

13 Store tuple
$(\boldsymbol{s}_{t},\boldsymbol{a}_{t},r_{t},\boldsymbol{s}_{t+1})$ in
$\mathcal{R}$;

14 if  _episode $k$ can be exactly divided by $K_{u}$ _ then

15

16 for  _iteration $r$ from $1$ to $R_{u}$ _ do

17 Randomly sample a mini-batch of $\mathcal{M}$ transitions tuples from
$\mathcal{R}$;

18 Update $\beta$ according to Eq. (36);

19 Update $\phi$ based on Eq. (37);

20 Update ${\psi_{1}}$ and ${\psi_{2}}$ based on Eq. (39);

21

22 if  _iteration $r$ can be exactly divided by $R_{t}$ _ then

23 Update $\bar{\psi}_{1}$ and $\bar{\psi}_{2}$ based on Eq. (40).;

24

25

26

27

28

Algorithm 1 Training stage for the SAC based framework

Firstly, $\phi$, ${\psi_{1}}$, ${\psi_{2}}$ and $\beta$ are initialized
randomly, $\bar{\psi}_{1}$ and $\bar{\psi}_{2}$ are set to be the same as
${\psi_{1}}$ and ${\psi_{2}}$. A replay buffer $\mathcal{R}$ with sufficient
space is constructed. The algorithm runs for ${K_{\max}}$ episodes. In the
first episode, the mmWave channel coefficient $\zeta$ is initialized based on
the Nakagami-$m$ distribution.

For each episode, the algorithm will iteratively be carried out for
$T^{train}_{max}$ time slots. In the first time slot $t=0$, for each task type
$i$, $n_{i}$ is generated according to Poisson distribution with arrival rate
$\lambda_{i}$, then ${a_{i}(0)}$ is obtained, thus ${\cal A}_{0}$ is obtained.
$q_{i}(0)$ is set to zero, thus ${{\cal Q}_{0}}$ is obtained.
${\xi}_{i,comm}^{dsrc}(0)$ is initialized to $R^{dsrc}$. Given that tasks of
$N$ types evenly share the CPU frequency $f_{E}$, ${\xi}_{i,comp}^{mmw}(0)$,
${\xi}_{i,comp}^{dsrc}(0)$, and ${\xi}_{i,comp}^{cv2i}(0)$ are all initialized
as $\frac{f_{E}}{N\omega_{i}}$, thus ${{\boldsymbol{\xi}}_{0}}$ is obtained.
Hence, state $\boldsymbol{s}_{0}$ is obtained according to Eq. (31). Then, we
can input state $\boldsymbol{s}_{0}$ into the actor network and output the
policy $\boldsymbol{\pi}_{\phi}(\boldsymbol{a}_{0}|\boldsymbol{s}_{0})$, which
follows the multivariate Gaussian distribution with the mean
${\boldsymbol{\mu}_{\phi}}(0)$ and variance
$\boldsymbol{\sum\nolimits}_{\phi}{(0)}$. Then action
$\boldsymbol{a}_{0}^{\prime}$ comprising of
${\boldsymbol{\alpha}}^{\prime}(0)$ and ${\boldsymbol{\varphi}}^{\prime}(0)$
is generated based on
$\boldsymbol{\pi}_{\phi}(\boldsymbol{a}_{0}|\boldsymbol{s}_{0})$. Note that
the dimension of the generated $\boldsymbol{a}_{0}^{\prime}$ is $N+1$. After
that a $softmax$ function is applied to ${\boldsymbol{\alpha}}^{\prime}(0)$
and ${\boldsymbol{\varphi}}^{\prime}(0)$ to ensure
$\sum_{i=1}^{N+1}\alpha_{i}(0)=1$ and $\sum_{g\in G}\varphi_{i}^{g}(0)=1$,
thus constraint (22e) is satisfied. Then ${\boldsymbol{\alpha}}(0)$ is set as
$[{a_{1}(0)},{a_{2}(0)},...{a_{N}(0)}]$. As $\sum_{i=1}^{N}\alpha_{i}(0)\leq
1$, thus constraint (22d) is satisfied. Finally, we can obtain action
$\boldsymbol{a}_{0}$ based on ${\boldsymbol{\alpha}}(0)$ and
${\boldsymbol{\varphi}}(0)$. Next, the VEC server takes action
$\boldsymbol{a}_{0}$, and based on this action, the delay upper bound
$\omega_{i}^{g}(0)$ and reward $r_{0}$ are calculated according to Eq. (17)
and Eq. (33), respectively. Then ${a_{i}(1)}$ is updated based on $n_{i}$
which is generated following Poisson distribution, $q_{i}(1)$ is updated
according to Eq. (2), then ${\bf{\xi}}_{i,comm}^{dsrc}(1)$ and
${\xi}_{i,comp}^{g}(1)$ are calculated according to Eqs. (47) and (49), thus
state $\boldsymbol{s}_{0}$ is transited to $\boldsymbol{s}_{1}$. After that
the tuple
$\left({{\boldsymbol{s}_{0}},{\boldsymbol{a}_{0}},{r_{0}},{\boldsymbol{s}_{1}}}\right)$
is stored in the replay buffer, and the algorithm moves to the next time slot.
The above process iterates until time slot $t$ reaches $T^{train}_{max}$. If
the episode number $k$ is not divisible by $K_{u}$, the algorithm moves to the
next episode; otherwise, the algorithm updates $\beta$ and the network
parameters $\phi$, ${\psi_{1}}$, ${\psi_{2}}$, $\bar{\psi}_{1}$, and
$\bar{\psi}_{2}$ through $R_{u}$ iterations. The update process is described
as follows.

For each iteration, $\mathcal{M}$ tuples are randomly selected from the replay
buffer to constitute a mini-batch of training data. Let
$\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m}},{r_{m}},{\boldsymbol{s}_{m}^{\prime}}}\right)(m=1,2,\cdots,\mathcal{M})$
be the $m$th tuple in the minibatch. After that, for each tuple $m$, input
$\boldsymbol{s}_{m}$ into the actor network and then obtain action
$\boldsymbol{a}_{m,new}$ and
${\boldsymbol{\pi}_{\phi}}\left({{\boldsymbol{a}_{m,new}}|{\boldsymbol{s}_{m}}}\right)$
according to the process described previously. Then the gradient of the loss
function of $\beta$ is calculated as

${\nabla_{\beta}}J\left(\beta\right)={\nabla_{\beta}}\left[\frac{1}{\mathcal{M}}\sum^{\mathcal{M}}_{m=1}\left[{-\beta
log{\boldsymbol{\pi}_{\phi}}\left({{\boldsymbol{a}_{m,new}}|{\boldsymbol{s}_{m}}}\right)-\beta\overline{\cal
H}}\right]^{2}\right].$ (36)

Then after inputting $\boldsymbol{s}_{m}$ and $\boldsymbol{a}_{m,new}$ into
the two critic networks, respectively, the two critic networks will output the
action-value functions
${Q_{{\psi_{1}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m,new}}}\right)$
and
${Q_{{\psi_{2}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m,new}}}\right)$,
respectively. The gradient of the loss function of ${\phi}$ can be calculated
based on
${Q_{{\psi_{1}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m,new}}}\right)$
and
${Q_{{\psi_{2}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m,new}}}\right)$,
i.e.,

$\begin{aligned}
&{\nabla_{\phi}}{J}\left(\phi\right)={\nabla_{\phi}}\left[\frac{1}{\mathcal{M}}\sum^{\mathcal{M}}_{m=1}[\beta
log\left({{\boldsymbol{\pi}_{\phi}}\left({{\boldsymbol{a}_{m,new}}|{\boldsymbol{s}_{m}}}\right)}\right)]^{2}\right]+\\\
&{\nabla_{\phi}}[\frac{1}{\mathcal{M}}\sum^{\mathcal{M}}_{m=1}[f\left({{\varepsilon};{\boldsymbol{s}_{m}}}\right)\cdot{\nabla_{{\boldsymbol{a}_{m,new}}}}(\beta
log\left({{\boldsymbol{\pi}_{\phi}}\left({{\boldsymbol{a}_{m,new}}|{\boldsymbol{s}_{m}}}\right)}\right)\\\
&\qquad\qquad\qquad\qquad\qquad\qquad-Q\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m,new}}}\right))]^{2}]\end{aligned},$
(37)

where $Q(\boldsymbol{s}_{m},\boldsymbol{a}_{m,new})$ is calculated as the
minimum of $Q_{{\psi_{1}}}(\boldsymbol{s}_{m},\boldsymbol{a}_{m,new})$ and
$Q_{{\psi_{2}}}(\boldsymbol{s}_{m},\boldsymbol{a}_{m,new})$, ${\varepsilon}$
is a noise sampled from multivariate normal distribution and
${f_{\phi}}\left({{\varepsilon};{\boldsymbol{s}_{m}}}\right)$ is a function to
reparameterize action $\boldsymbol{a}_{m,new}$ [42].

After that the gradients of the loss functions for ${\psi_{1}}$ and
${\psi_{2}}$ are calculated as follows. $\boldsymbol{s}_{m}$ and
$\boldsymbol{a}_{m}$ are input into two critic networks to produce action-
value functions ${Q_{{\psi 1}}}(\boldsymbol{s}_{m},\boldsymbol{a}_{m})$ and
${Q_{{\psi 2}}}(\boldsymbol{s}_{m},\boldsymbol{a}_{m})$. Also, it feeds
$\boldsymbol{s}_{m}^{\prime}$ into the actor network to obtain
$\boldsymbol{a}_{m}^{\prime}$ and
$\boldsymbol{\pi}_{\phi}^{\prime}(\boldsymbol{a}_{m}^{\prime}|\boldsymbol{s}_{m}^{\prime})$.
Next, $\boldsymbol{s}_{m}^{\prime}$ and $\boldsymbol{a}_{m}^{\prime}$ are
input into two target critic networks, yielding
${Q_{{{\bar{\psi}}1}}}(\boldsymbol{s}_{m}^{\prime},\boldsymbol{a}_{m}^{\prime})$
and
${Q_{{{\bar{\psi}}2}}}(\boldsymbol{s}_{m}^{\prime},\boldsymbol{a}_{m}^{\prime})$.
The target value is then calculated as

$\begin{aligned}
&\hat{Q}\left({{\boldsymbol{s}_{m}^{\prime}},{\boldsymbol{a}_{m}^{\prime}}}\right)=-\beta
log\boldsymbol{\pi}_{\phi}^{\prime}\left({{\boldsymbol{a}_{m}^{\prime}}|{\boldsymbol{s}_{m}^{\prime}}}\right)+\\\
&\qquad\qquad
min\left\\{{{Q_{{{\bar{\psi}}_{1}}}}\left({{\boldsymbol{s}_{m}^{\prime}},{\boldsymbol{a}_{m}^{\prime}}}\right),{Q_{{{\bar{\psi}}_{2}}}}({\boldsymbol{s}_{m}^{\prime}},{\boldsymbol{a}_{m}^{\prime}})}\right\\}\end{aligned}.$
(38)

Then the gradients of the loss functions of ${\psi_{1}}$ and ${\psi_{2}}$ are
calculated as

$\begin{aligned}
&{\nabla_{{\psi_{b}}}}{J}\left({{\psi_{b}}}\right)={\nabla_{{\psi_{b}}}}[\frac{1}{\mathcal{M}}\sum^{\mathcal{M}}_{m=1}[{Q_{{\psi_{b}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m}}}\right)\cdot\\\
&\quad\left({{Q_{{\psi_{b}}}}\left({{\boldsymbol{s}_{m}},{\boldsymbol{a}_{m}}}\right)-{r_{m}}+\gamma\hat{Q}\left({{\boldsymbol{s}_{m}^{\prime}},{\boldsymbol{a}_{m}^{\prime}}}\right)}\right)]^{2}],b\in\left\\{1,2\right\\}\end{aligned}.$
(39)

Then Adam optimizer is adopted to update $\beta$, $\phi$, ${\psi_{1}}$ and
${\psi_{2}}$ through gradient ascending based on
${\nabla_{\beta}}J\left(\beta_{t}\right)$,
${\nabla_{\phi}}{J}\left(\phi\right)$,
${\nabla_{{\psi_{1}}}}{J}\left({{\psi_{1}}}\right)$ and
${\nabla_{{\psi_{2}}}}{J}\left({{\psi_{2}}}\right)$. After every $R_{t}$
iterations, the parameters of the two target critic network are updated as

${\bar{\psi}_{b}}:=\tau_{b}{\psi_{b}}+\left({1-\tau_{b}}\right){\bar{\psi}_{b}},b\in\left\\{1,2\right\\}.$
(40)

where $\tau_{1}$ and $\tau_{2}$ are constants satisfying $\tau_{1}\ll 1$ and
$\tau_{2}\ll 1$.

After $R_{u}$ iterations, the update process is finished. Then the algorithm
enters the next episode. The algorithm of the training stage will be finished
when $k=K^{train}_{max}$ which yields the optimal parameters $\phi$, denoted
as $\phi^{*}$.

##### Testing stage

Compared with the training process, the testing stage omits the two critic
networks, two target networks and updating processes of parameters $\beta$,
$\phi$, ${\psi_{1}}$, ${\psi_{2}}$, $\bar{\psi}_{1}$ and $\bar{\psi}_{2}$. The
test stage is executed by using the parameters $\phi^{*}$ to obtain the
optimal policy
$\pi_{\phi}^{*}\left({{\boldsymbol{a}_{t}}|{\boldsymbol{s}_{t}}}\right)$.
Based on
$\pi_{\phi}^{*}\left({{\boldsymbol{a}_{t}}|{\boldsymbol{s}_{t}}}\right)$, the
optimal ${\boldsymbol{\alpha}^{*}}(t)$ and ${\boldsymbol{\varphi}^{*}}(t)$ can
be obtained.

###  IV-C Computational Complexity and Processing Running Time Analysis

In this section, we will analyze the computational complexity and processing
running time of our approach. Our analysis focuses on the training stage due
to the significant computation resource and time consumption in the training
stage. Our methodology for analyzing the computational complexity is inspired
by [5].

####  IV-C1 Computational Complexity Analysis

We first analyze the computational complexity during the training stage.
Because the training processing requires a significant computational resources
to compute gradients and update parameters, the computational complexity
mainly consists of the complexity of computing gradients and the complexity of
updating parameters. Let $G_{E}$, $G_{A}$, and $G_{C}$ be the computational
complexity of computing gradients for the tradeoff weight of the policy
entropy $\beta$, the actor network $\phi$ and two critic networks ${\psi_{1}}$
and ${\psi_{2}}$, respectively, and $U_{E}$, $U_{A}$, and $U_{C}$ be the
computational complexity of updating parameters for $\beta$, $\phi$,
${\psi_{1}}$ and ${\psi_{2}}$, respectively. Since the structures of two
target critic networks are the same as that of the critic networks and they
only need update their parameters, the target networks have the same
complexity of parameter updating as critic networks.

The computational complexity of our approach is affected by the number of
episodes. Throughout the training stage, $\beta$, $\phi$, ${\psi_{1}}$ and
${\psi_{2}}$ require total $\frac{K^{train}_{max}}{K_{u}}R_{u}$ times to
calculate the gradients and update parameters. Therefore, the computational
complexity for calculating the gradients and updating parameters of $\beta$,
$\phi$, ${\psi_{1}}$ and ${\psi_{2}}$ is
$O\left(\frac{K^{train}_{max}}{K_{u}}R_{u}(G_{E}+G_{A}+2G_{C}+U_{E}+U_{A}+2U_{C})\right)$.
For two target critic networks, they only require
$\frac{K^{train}_{max}}{K_{u}}\frac{R_{u}}{r}$ times to update parameters.
Thus their total computational complexity for updating parameters is is
$O\left(\frac{K^{train}_{max}}{K_{u}}R_{u}\frac{2}{r}U_{C}\right)$. Thus, the
total computational complexity of our approach in the training stage is
$O\left(\frac{K^{train}_{max}}{K_{u}}R_{u}(G_{E}+G_{A}+2G_{C}+U_{E}+U_{A}+2(1+\frac{1}{r})U_{C})\right)$.

####  IV-C2 Processing Time Analysis

Then we analyze the processing time of our approach in the training stage. We
assume that the time of initializing the neural networks and experience replay
buffer is denoted by $t_{0}$, and the time of resetting environment is
represented by $t_{1}$. Moreover, the time of decisions making is expressed by
$t_{2}$ .The total time to update all networks for $R_{u}$ iterations is
$t_{4}$. Consequently, the processing running time for the SAC algorithm in
the training stage can be written as

$\begin{aligned} T_{SAC}&=t_{0}+(t_{1}+t_{2}\times
T^{train}_{max}+t_{3}\times\frac{1}{K_{u}})\times K^{train}_{max}\\\
&=t_{0}+t_{1}\times K^{train}_{max}\\\ &\qquad+t_{2}\times
T^{train}_{max}\times
K^{train}_{max}+t_{3}\times\frac{K^{train}_{max}}{K_{u}}\\\ &\approx
t_{2}\times T^{train}_{max}\times
K^{train}_{max}+t_{3}\times\frac{K^{train}_{max}}{K_{u}}\end{aligned}.$ (41)

As the values of $K^{train}_{max}$ and $T^{train}_{max}$ have a great
influence on the processing running time and value of $t_{3}$ is large enough,
we can ignore the influence of $t_{0}$ and $t_{1}\times K^{train}_{max}$ when
$K^{train}_{max}$ and $T^{train}_{max}$ are very large.

TABLE I: Environment parameters in the simulation. Parameter | Value | Parameter | Value
---|---|---|---
$f_{E}$ |  ${6\cdot 10^{4}}$ GHz | ${\varepsilon_{i}}$ | 0.01
$R_{i}^{cv2i}$ | 27 Mbps | $R_{i}^{dsrc}$ | 27 Mbps
$B$ | 20 MHz | $N_{E}$ | 10 cores
$\kappa$ | $\frac{1}{{{{(400GHz)}^{3}}}}$ | $\lambda_{i}$ | 10
$c^{comp}$ | 1000 dollars/W | $c^{comm}$ | 500 dollars/Mbps
${\varpi_{1}}$ | 0.5 | ${\varpi_{2}}$ | 0.5
$\theta$ | $10^{-8}$ | $T^{max}$ | 30 ms
$l$ | 1m | $M$ | 5
$\rho$ | 0.62Mbps | $\sigma$ | 18.6Mbps
$M$ | 5 | $\delta$ | 2.45

##  V Simulation Results

TABLE II: Hyperparameters in the SAC algorithm. SAC Hyperparameters
---
Parameter | Value | Parameter | Value
optimizer | Adam | $\gamma$ | 0.99
$\alpha^{A}$ | $3\cdot{10^{-4}}$ | $\alpha^{C}$ | $3\cdot{10^{-3}}$
nonlinearity | ReLU | $\mathcal{M}$ | 256
$R_{t}$ | 1 | $K^{train}_{max}$ | 16000
$K_{u}$ | 2 | $T^{train}_{max}$ | 2000
$\tau_{1}$ | 0.005 | $\tau_{2}$ | 0.005
$R_{u}$ | 80 | $P_{e}$ | $10^{6}$
$T^{test}_{max}$ | 5000 |  |

The simulations are realized via Python 3.7 and the scenario is described in
the system model. Simulation parameters are listed in Table I. Three task
types are set as 3D game, VR and AR[41], and their $w_{i}$ are 54633 GHz/bit,
40305 GHz/bit and 34532 GHz/bit, respectively. According to [43], AR has the
strictest requirements among the three task types for latency between 20-40ms,
hence for the sake of simulation we set the average latency requirement for
AR, i.e., 30ms, as the maximum latency requirement for any task type, which is
denoted as $T^{max}$. The actor network and two critic networks have four-
layer fully connected DNN with two hidden layers, where each layer is equipped
with 256 neurons. In addition, the other hyperparameters we used are same as
[44], which are listed in Table II.

Figure 2: Learning curve

###  V-A Training Stage

Fig. 2 shows the learning curve of the training process under different $V$.
The transparent point line is the average reward curve where the average
reward in a episode is calculated by averaging the rewords of all time steps
in the episode, while the dark line is the curve after smoothing the average
reward curve. We can see that the learning curves under different $V$ first
increase, then decrease, and finally increase slowly to a stable value. This
is because that the VEC server will first learn the policy to maximize the
reward and guarantee the low-latency constraint (22c), thus the learning curve
first increases. In the learning process, the constraint (22c) may be violated
and incur a large penalty according to Eq. (33), hence the learning curve will
degrade. Afterwards, the VEC will eventually learns an optimal policy to
maximize the reward while satisfying constraints (22c), therefore the learning
curve will finally become stable. We can also see that the average rewards
have some jitters after reaching stable, because the VEC server stays in a
stochastic environment due to the random task arrival, which will affect the
learning of the VEC server.

###  V-B Testing Stage

All the results in testing stage are obtained by averaging the simulation
results for 50 times. We will validate the performance of our policy, which is
referred to as LySAC, by comparing the performance of five baseline policies,
i.e.,

  * •

Particle Swarm Optimization (PSO)[45]: PSO is an optimization algorithm
inspired by the foraging behavior of birds, which has been widely applied in
the research on resource allocation. Its widespread application in resource
allocation studies offers a traditional benchmark for comparison.

  * •

Heuristic Gibbs Random Algorithm (HGRA)[46]: HGRA iteratively determines
resource allocation policy based on the Gibbs distribution, which is composed
of a uniform distribution $\rho_{g}$ and an empirical performance-related
distribution $\omega_{g}$. It iteratively updates policy based on the value of
optimization objective $P2$. The dual composition of a uniform distribution
and an empirical performance-related distribution makes it a notable
comparison policy.

  * •

Heuristic Gibbs Greedy Algorithm (HGGA): HGGA is a variation of HGRA Algorithm
that greedily selects the communication technology with the highest
probability as the policy. This policy can offer an alternative perspective on
resource allocation strategies and provide insights into the trade-offs
between iterative and greedy approaches.

  * •

Equal Allocation and Equal Offloading (EAEO): The total CPU frequency $f_{E}$
is equally allocated to $N$ task types, and each communication technology
offloads equivalent percentage of tasks, i.e.,
$\varphi^{mmw}_{i}(t)=\varphi^{dsrc}_{i}(t)=\varphi^{cv2i}_{i}(t)=\frac{1}{3}$
and $\alpha_{i}(t)=\frac{1}{3}f_{E}$, hence EAEO is independent with $V$. It
provides a reference policy on the effect of a uniform distribution approach,
which is decoupled from varying parameters like $V$.

  * •

Greedy Algorithm (Greedy): The total CPU frequency $f_{E}$ is allocated to the
queue with the largest length, and each communication technology also still
offloads the same proportion of tasks. Hence Greedy is also independent with
$V$. It offers insights into the results of a simplistic greedy approach in
resource allocation without considering dynamic factors such as $V$.

Figure 3: Average computation utility Figure 4: Average communication utility
Figure 5: Average system utility

((a)) V = 0

((b)) V = 50

((c)) V = 100

((d)) V = 200

Figure 6: Queue backlog of different policies under different $V$

((a)) V = 0

((b)) V = 50

((c)) V = 100

((d)) V = 200

Figure 7: Violation probability vs delay upper bound of different policies
under different $V$ Figure 8: Delay upper bound vs task arrival rate under
different policies

Figs. 3 \- 5 present the average computation utility, communication utility,
and system utility under different policies and $V$. These results are
obtained by taking the average results across all time steps. From Figs. 3 \-
5, we can see that the LySAC utilizes the lower computation utility, this is
because that LySAC is capable of intelligently and dynamically adjusting the
usage of resources to meet the requirements of URLLC and minimize the utility.
From Fig. 3, it can be observed that EAEO and Greedy utilize the highest
computation utility. This is because EAEO and Greedy employ all CPU
frequencies to process tasks. Moreover, we can see that HGGA, HGRA, PSO and
Greedy also have a higher computation utility than LySAC. This is due to the
fact that they cannot intelligently and dynamically allocate resources, thus
using more CPU frequencies than LySAC to ensure URLLC performance. From Fig.4,
it can be seen that EAEO, HGGA, HGRA, PSO and Greedy use roughly the same
communication utility. This is because these five policies do not consider the
differences of task types when selecting communication technology for task
offloading.

Fig. 5 depicts the system utility, which is the sum of computation utility and
communication utility. On the other hand, in Fig. 3, the average computation
utility used by LySAC increases as $V$ increases. This is because as $V$
increases, LySAC places more emphasis on the ultra-reliable requirement and
thus allocates more CPU frequency. However, when $V=0$ the average computation
utility is at its maximum, this is because LySAC pays more attention to meet
the low-latency requirements of the tasks at this point, which typically
requires more CPU frequencies. In Fig. 4, the trend of the communication
utility of LySAC with respect to the change of $V$ is exactly opposite to that
in Fig. 3. This is because as LySAC allocates more computation resources to
maintain queue stability, it reduces the dependence on communication
resources. In Fig. 5, the system utility decreases as $V$ increases. This is
because the system utility is the sum of the computation and communication
utility, and the communication utility constitutes a large proportion of the
system utility. Moreover, in Figs. 3 \- 5, we can see that HGGA, HGRA, PSO and
Greedy do not exhibit significant changes in terms of average computation
utility, communication utility, and system utility as $V$ changes. This is
because these four policies are designed with a focus on overall optimization
of resource allocation, without fully considering the impact of $V$
variations. Meanwhile, we also see that the three utilities of EAEO and Greedy
do not change with $V$, this is because that the EAEO and the Greedy are
independent of $V$. This demonstrates that EAEO, HGGA, HGRA, PSO and Greedy
fail to adapt to actual requirements.

Figs. 6(a) \- 6(d) illustrate the queue backlog of different policies under
different $V$, where the queue backlog is obtained by averaging the queue
backlogs for the three task types. We can see that under different $V$, the
queue backlog of LySAC increase at the beginning and then almost keep
constant, which means the queue backlogs of LySAC reaches a stable state under
different $V$ and thus our proposed LySAC policy can achieve ultra-reliable
efficiency. This is because LySAC can allocate computation resources according
to real-time demands, thus achieve a stable state of queue backlog. We also
can see as $V$ increases, the stable queue backlog of LySAC decreases and the
queue backlog can reach a stable state faster. This because as $V$ increases,
LySAC emphasizes more ultra-reliable requirement, thus allocates higher CPU
frequencies to process tasks. In addition, it is seen that there are almost no
queue backlogs for the five other policies under different $V$, this is
because that according to Fig. 3 they utilize the higher CPU frequencies to
process tasks.

Figs. 7(a)-7(d) depict the relationship between the violation probability and
the delay upper bound of various policies and different $V$, where the delay
upper bound is obtained by averaging the delay upper bounds of the three task
types, i.e., 3D game, VR and AR, and the violation probability is calculated
as the percentage of the events that the delay exceeds the delay upper bound.
From the figures, we can observe that for LySAC, the violation probability
that the delay upper bound is lower than the maximum latency requirement for
any task type, i.e., $T^{max}=30$ms, is below 0.01, which is relatively small,
hence our proposed LySAC policy can ensure the low-latency requirement. We can
also see that EAEO, HGRA, and HGGA are also capable of ensuring low-latency
requirements, but according to Figs. 3 \- 5 they consume more resources
compared with our policy. In addition, it is seen that PSO policy fails to
ensure the low-latency requirement. This because PSO can not adequately
allocate resource according to the relationships between task types and their
respective latency requirements. The Greedy algorithm exhibits the poorest
latency performance. This is because that Greedy algorithm allocates the total
CPU frequencies to one single type of task. Consequently, tasks of other types
have no additional CPU frequency for processing, resulting in the highest
offloading latency.

Fig. 8 depicts the relationship between the delay upper bound and task arrival
rate under different policies when $V=200$, where the delay upper bound is
obtained by averaging the delay upper bounds of the three task types, the task
arrival rate is the arrival rate of each type tasks. We choose $V=200$ because
according to Fig. 3 our LySAC policy allocates a higher CPU frequency when
$V=200$, thus we can better observe the performance under higher task arrival
rates. We can see that LySAC can almost meet the low-latency performance,
i.e., the delay upper bound is smaller than 30ms, under the task arrival rate
ranging from 5 tasks to 50 tasks per time slot, as it effectively leverages
the advantages of three communication technologies. We also see that the delay
upper bound for HGGA increases rapidly as the task arrival rate increases,
this because HGGA uses only one communication technology to offload task. It
is also seen that PSO, EAEO and HGRA have inferior latency performance, this
is due to they allocate reasonable communication resource among the three
communication technologies, which deteriorates the latency performance. The
Greedy algorithm exhibits the poorest latency performance and remains
unaffected by the variations of task arrival rate. One reason for this is that
the Greedy algorithm allocates all CPU frequencies to a single type of task.
Additionally, the Greedy algorithm’s principle for CPU allocation is based on
queue length, making it insensitive to the changes of the task arrival rate.

##  VI Conclusions

In this paper, we considered a heterogeneous VEC with multiple communication
technologies and various types of tasks technologies and proposed a Lyapunov-
guided DRL resource allocation policy based on SAC to minimize the system
utility while guaranteeing the URLLC requirement. We first derived the delay
upper bound of offloading tasks based on the SNC theory, then adopted the
Lyapunov optimization to transform the ultra-reliability constraint into a
short-term constraint and reformulated the optimization problem. Finally, we
employed SAC algorithm to solve the optimization problem, thus the optimal
allocation policy can be obtained. Extensive simulation results demonstrated
that the proposed policy can minimize the system utility while satisfying
URLLC requirement. The conclusions are summarized as follows:

  * •

Our approach supports real-time decision-making, and adapts to the environment
where vehicles run at higher speeds. Since decisions are based solely on the
current environmental state, it can provide decisions in time.

  * •

Our approach emphasizes efficient resource allocation, which makes a better
balance between computation and communication requirements while meeting URLLC
performance.

  * •

The approach is able to dynamically adjust decisions based on real-time
demands. This inherent adaptability ensures an optimal balance among latency,
reliability, and resource utilization.

Although our approach presents considerable strengths, there is one notable
limitation, i.e., model retraining is required when the types of tasks keep
increasing. In the future work, we would address this concern to ensure the
wider applicability and robustness of our proposed approach.

##  Appendix A Derivation of the Tractable Upper Bound

For Eq. (15), we have

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\\\ &=P\left\\{{\mathop{\sup}\limits_{0\leq s\leq
u^{\prime}}\left[{A_{i}\left({s,t}\right)-S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)}\right]\geq
0}\right\\}\\\
&=P\left\\{{\bigcup\limits_{s=0}^{u^{\prime}}\left\\{A_{i}\left({s,t}\right)-S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)\right\\}\geq
0}\right\\}\\\
&\leq\sum\limits_{s=0}^{u^{\prime}}{P\left\\{\left\\{A_{i}\left({s,t}\right)-S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)\right\\}\geq
0\right\\}}\\\
&\leq\sum\limits_{s=0}^{{u^{\prime}}}{E[{e^{\theta\\{A_{i}\left({s,t}\right)-S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)\\}}}]}\\\
&=\sum\limits_{s=0}^{{u^{\prime}}}E[e^{\theta
A_{i}\left({s,t}\right)}]E[e^{-\theta
S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)}]\end{aligned}.$ (42)

where the second line of Eq. (42) holds by using the notion of $\oslash$, the
third line holds by using union bound, the fourth line holds by using Boole’s
inequality, the fifth line holds by using Chernoff’s bound [36], and the last
line holds by assuming $A_{i}\left({s,t}\right)$ and
$S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)$ are independent.

For the result of Eq. (42), $E[e^{-\theta
S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)}]=M_{S_{i}^{g}}(-\theta,s,t+\omega_{i}^{g}(t))=\overline{M}_{S_{i}^{g}}(\theta,s,t+\omega_{i}^{g}(t))$,
where $\overline{M}_{S_{i}^{g}}(\theta,s,t+\omega_{i}^{g}(t))$ is the moment
generation function (MGF) of $-S_{i}^{g}\left({s,t+\omega_{i}^{g}(t)}\right)$.
According to concatenation theorem in SNC, $S_{i}^{g}(s,t+\omega_{i}^{g}(t))$
is calculated as
$\left({\beta_{i,comm}^{g}\otimes\beta_{i,comp}^{g}}\right)({s,t+\omega_{i}^{g}(t)})$[33],
where $\otimes$ is the mini-plus convolution operator[47]. According to MGF of
$\otimes$ lemma[36], we have

$\begin{aligned} &\overline{M}_{S_{i}^{g}}(\theta,s,t+\omega_{i}^{g}(t))\\\
&\leq\sum\limits_{z=s}^{t+\omega_{i}^{g}(t)}\overline{M}_{\beta_{i,comm}^{g}}(\theta,s,z)\overline{M}_{\beta_{i,comp}^{g}}(\theta,z,t+\omega_{i}^{g}(t))\end{aligned},$
(43)

where
$\overline{M}_{\beta_{i,comm}^{g}}(\theta,s,z)=E[e^{-\theta\beta_{i,comm}^{g}(s,z)}]$,
$\overline{M}_{\beta_{i,comp}^{g}}(\theta,z,t+\omega_{i}^{g}(t))=E[e^{-\theta\beta_{i,comp}^{g}(z,t+\omega_{i}^{g}(t))}]$.
Substituting Eq. (43) into Eq. (42), we have

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\leq\sum\limits_{s=0}^{{u^{\prime}}}E[e^{\theta
A_{i}\left({s,t}\right)}]\cdot\\\
&\qquad\sum\limits_{z=s}^{t+\omega_{i}^{g}(t)}E[e^{-\theta\beta_{i,comm}^{g}(s,z)}]E[e^{-\theta\beta_{i,comp}^{g}(z,t+\omega_{i}^{g}(t))}]\end{aligned}.$
(44)

According to the MGF of affine envelope model, when $t\geq s\geq 0$, we have
$M_{A_{i}}(\theta,s,t)=E[{e^{\theta{A_{i}}\left({s,t}\right)}}]\leq{{e^{\theta{\rho_{i}}(t-s)+\theta{\sigma_{i}}}}}$
[48]. According to Eqs. (6), (8) and (12), $\beta_{i,comm}^{g}(s,z)$ and
$\beta_{i,comp}^{g}(z,t+\omega_{i}^{g}(t))$ are both calculated by the linear
functions related with $\rho_{i}$ and $\sigma_{i}$, thus similar with
${E[{e^{\theta{A_{i}}\left({s,t}\right)}}]}$, we have
${E[{e^{-\theta\beta_{i,comm}^{g}(s,z)}}]}\leq
e^{\theta\beta_{i,comp}^{g}(s,z)}$ and
${E[{e^{-\theta\beta_{i}^{g}(z,t+\omega_{i,comp}^{g}(t))}}]}\leq
e^{\theta\beta_{i,comp}^{g}(z,t+\omega_{i}^{g}(t))}$. Thus Eq. (44) can
further be upper bounded as

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\leq\sum\limits_{s=0}^{{u^{\prime}}}{{e^{\theta{A_{i}}\left({s,t}\right)}}}\cdot\\\
&\qquad\qquad\sum\limits_{z=s}^{t+\omega_{i}^{g}(t)}{{e^{-\theta\beta_{i,comm}^{g}(s,z)}}}{e^{-\theta\beta_{i,comp}^{g}(z,t+\omega_{i}^{g}(t))}}\end{aligned}.$
(45)

Since the interval time $[s_{1},s_{2})$ is relatively small,
${\varphi_{j}^{g}(t)}$ in time interval $[s_{1},s_{2})$ can be deemed as a
constant, thus $\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{g}(t)}a_{i}(t)$ in Eqs. (6)
and (8) can be approximated as $\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{g}(t)}\sum\limits_{t=s_{1}}^{s_{2}-1}a_{i}(t)=\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{g}(t)}A_{j}(s_{1},s_{2})$. According to Leftover service
theorem, Eqs. (6) and (8) can be written as

$\begin{aligned}
&\beta_{i,comm}^{mmw}\left({s_{1},s_{2}}\right)=-\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{mmw}(t)}\sigma_{j}+\\\
&\qquad\left[{\beta^{mmw}}-\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{mmw}(t)}\rho_{j}\right](s_{2}-s_{1})\end{aligned}.$ (46)

and

$\begin{aligned}
&\beta_{i,comm}^{dsrc}\left({s_{1},s_{2}}\right)=-{R^{dsrc}}{\hat{t}_{serv}}-\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{dsrc}}(t){\sigma_{j}}+\\\
&\qquad\left[{\beta^{dsrc}}-\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{dsrc}(t)}\rho_{j}\right](s_{2}-s_{1})\end{aligned}.$ (47)

Let $\eta_{i,comm}^{mm\omega}(t)=\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{mmw}(t)}\sigma_{j}$ and
$\xi_{i,comm}^{mm\omega}(t)={\beta^{mmw}}-\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{mmw}(t)}\rho_{j}$,
$\eta_{i,comm}^{dsrc}(t)={R^{dsrc}}{\hat{t}_{serv}}+\sum\limits_{j\neq
i}^{N}{\varphi_{j}^{dsrc}(t)}{\sigma_{j}}$, and
$\xi_{i,comm}^{dsrc}(t)={\beta^{dsrc}}-\sum\limits_{j\neq
i}^{N}\sum\limits_{t=s_{1}}^{s_{2}-1}{\varphi_{j}^{dsrc}(t)}\rho_{j}$ We also
let $\xi_{i,comm}^{cv2i}(t)={R_{i}^{cv2i}}$ and $\eta_{i,comm}^{cv2i}(t)=0$ in
Eq. (9) to unify the form of Eqs. (9), (46) and (47) as

$\beta_{i,{comm}}^{g}(s_{1},s_{2})=\xi_{i,{comm}}^{g}(t)(s_{2}-s_{1})-\eta_{i,{comm}}^{g}(t).$
(48)

Similarly $\alpha_{i}(t)$ also can be deemed as a constant, thus Eq. (12) is
approximated as

$\begin{aligned}
&\beta_{i,comp}^{g}\left({s_{1},s_{2}}\right)=[\frac{{{f_{E}}{\alpha_{i}(t)}}}{{{\omega_{i}}}}-\sum\nolimits_{\mathcal{G}/{\rm{g}}}{\varphi_{i}^{g}(t)}{\rho_{i}}](s_{2}-s_{1})-\\\
&\qquad\qquad\qquad\sum\nolimits_{\mathcal{G}/{\rm{g}}}{\varphi_{i}^{g}(t)}{\sigma_{i}}\end{aligned}.$
(49)

Let
$\eta_{i,comp}^{g}(t)=\sum\nolimits_{\mathcal{G}/{\rm{g}}}{\varphi_{i}^{g}(t)}{\sigma_{i}}$
and
$\xi_{i,comp}^{g}(t)=\frac{{{f_{E}}{\alpha_{i}(t)}}}{{{\omega_{i}}}}-\sum\nolimits_{\mathcal{G}/{\rm{g}}}{\varphi_{i}^{g}(t)}{\rho_{i}}$
Hence the network service and computing service can be expressed as a same
form, i.e.,
$\beta_{i,{serve}}^{g}(s_{1},s_{2})=\xi_{i,{serve}}^{g}(t)(s_{2}-s_{1})-\eta_{i,{serve}}^{g}(t)$,
where $serve\in\\{comm,comp\\}$ and
$g\in\mathcal{G}=\left\\{mmw,dsrc,cv2i\right\\}$.Performing a geometric series
summation on the second term of the right hand side of Eq. (45), we have

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\leq\\\
&{e^{\theta{\Delta_{2}}}}\sum\limits_{s=0}^{{u^{\prime}}}{{e^{\theta(t+\omega_{i}^{g}(t)-s)({\rho_{i}}-\xi_{i,{comp}}^{g}(t))}}}\cdot\\\
&\qquad\qquad\sum\limits_{{z^{\prime}}=0}^{t+\omega_{i}^{g}(t)-s}{{e^{-\theta{z^{\prime}}(\xi_{i,{comm}}^{g}(t)-\xi_{i,{comp}}^{g}(t))}}}\end{aligned}.$
(50)

where
$\Delta_{2}=-{\rho_{i}}\omega_{i}^{g}(t)+{\sigma_{i}}+\eta_{i,comm}^{g}(t)+\eta_{i,comp}^{g}(t)$.

Let ${s^{\prime}}=t+\omega_{i}^{g}(t)-s$ and
$\tau_{0}=\max\\{0,\omega_{i}^{g}(t)\\}$. Since $0\leq s\leq{u^{\prime}}$ and
$u^{\prime}=\min\\{t+\omega_{i}^{g}(t),t\\}$, we have
${s^{\prime}}\in[\tau_{0},t+\omega_{i}^{g}(t)]$, Eq. (50) can be written as

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\leq{e^{\theta\cdot{\Delta_{2}}}}\cdot\\\
&\sum\limits_{{s^{\prime}}=\tau_{0}}^{t+\omega_{i}^{g}(t)}{{e^{\theta({s^{\prime}})({\rho_{i}}-\xi_{i,{comp}}^{g}(t))}}}\sum\limits_{{z^{\prime}}=0}^{{s^{\prime}}}{{e^{-\theta{z^{\prime}}(\xi_{i,{comm}}^{g}(t)-\xi_{i,{comp}}^{g}(t))}}}\end{aligned}.$
(51)

After performing a geometric scaling on third term of the right hand side of
Eq. (51), then let $t+\omega_{i}^{g}(t)\to\infty$, and as the network service
is much larger than the network service for the tasks of all types, i.e.,
${\rho_{i}}<<\xi_{i,{comp}}^{g}(t)$ and ${\rho_{i}}<<\xi_{i,{comm}}^{g}(t)$,
we have ${{e^{\theta({\rho_{i}}-\xi_{i,{comp}}^{g}(t))}}}<1$ and
${{e^{\theta({\rho_{i}}-\xi_{i,{comp}}^{g}(t))}}}<1$, finally we have

$\begin{aligned} &P(A_{i}\oslash S_{i}^{g}({t+{\omega_{i}^{g}(t)},t})\geq
0)\leq\\\
&\qquad\qquad\frac{{{e^{\theta\cdot{\Delta}}}}}{{{e^{-\theta\xi_{i,{comp}}^{g}(t)}}-{e^{-\theta\xi_{i,{comm}}^{g}(t)}}}}\cdot\\\
&\qquad\qquad\\{\frac{{{e^{-\theta\xi_{i,{comp}}^{g}(t)\omega_{i}^{g}(t)}}}}{{{e^{\theta\xi_{i,{comp}}^{g}(t)}}-{e^{\theta{\rho_{i}}}}}}-\frac{{{e^{-\theta\xi_{i,{comm}}^{g}(t)\omega_{i}^{g}(t)}}}}{{{e^{\theta\xi_{i,{comm}}^{g}(t)}}-{e^{\theta{\rho_{i}}}}}}\\}\end{aligned}.$
(52)

Notice that $\omega_{i}^{g}(t)\geq 0$, thus
$\tau_{0}=\max\\{0,\omega_{i}^{g}(t)\\}=\omega_{i}^{g}(t)$, and
${\Delta}={\sigma_{i}}+\eta_{i,comm}^{g}(t)+\eta_{i,comp}^{g}(t)$.

## References

  * [1] Z. Chen and X. Wang, “Decentralized computation offloading for multi-user mobile edge computing: A deep reinforcement learning approach,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2020, no. 1, pp. 1–21, 2020.
  * [2] K. Xiong, S. Leng, C. Huang, C. Yuen, and Y. L. Guan, “Intelligent task offloading for heterogeneous v2x communications,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 4, pp. 2226–2238, 2021.
  * [3] T. Taleb, K. Samdanis, B. Mada, H. Flinck, S. Dutta, and D. Sabella, “On multi-access edge computing: A survey of the emerging 5g network edge cloud architecture and orchestration,” _IEEE Communications Surveys Tutorials_ , vol. 19, no. 3, pp. 1657–1681, 2017.
  * [4] Q. Wu, Y. Zhao, Q. Fan, P. Fan, J. Wang, and C. Zhang, “Mobility-aware cooperative caching in vehicular edge computing based on asynchronous federated and deep reinforcement learning,” _IEEE Journal of Selected Topics in Signal Processing_ , pp. 1–16, 2022.
  * [5] H. Zhu, Q. Wu, X.-J. Wu, Q. Fan, P. Fan, and J. Wang, “Decentralized power allocation for mimo-noma vehicular edge computing based on deep reinforcement learning,” _IEEE Internet of Things Journal_ , vol. 9, no. 14, pp. 12 770–12 782, 2022.
  * [6] K. Zheng, Q. Zheng, P. Chatzimisios, W. Xiang, and Y. Zhou, “Heterogeneous vehicular networking: A survey on architecture, challenges, and solutions,” _IEEE Communications Surveys Tutorials_ , vol. 17, no. 4, pp. 2377–2396, 2015\.
  * [7] J. B. Kenney, “Dedicated short-range communications (dsrc) standards in the united states,” _Proceedings of the IEEE_ , vol. 99, no. 7, pp. 1162–1182, 2011.
  * [8] C. Han, M. Dianati, R. Tafazolli, R. Kernchen, and X. Shen, “Analytical study of the ieee 802.11p mac sublayer in vehicular networks,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 13, no. 2, pp. 873–886, 2012.
  * [9] X. Wang, S. Mao, and M. X. Gong, “An overview of 3gpp cellular vehicle-to-everything standards,” _GetMobile: Mobile Computing and Communications_ , vol. 21, no. 3, pp. 19–25, 2017.
  * [10] G. Naik, B. Choudhury, and J.-M. Park, “Ieee 802.11bd $\&$ 5g nr v2x: Evolution of radio access technologies for v2x communications,” _IEEE Access_ , vol. 7, pp. 70 169–70 184, 2019.
  * [11] K. Abboud, H. A. Omar, and W. Zhuang, “Interworking of dsrc and cellular network technologies for v2x communications: A survey,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 12, pp. 9457–9470, 2016.
  * [12] R. Molina-Masegosa and J. Gozalvez, “Lte-v for sidelink 5g v2x vehicular communications: A new 5g technology for short-range vehicle-to-everything communications,” _IEEE Vehicular Technology Magazine_ , vol. 12, no. 4, pp. 30–39, 2017.
  * [13] L. Kong, M. K. Khan, F. Wu, G. Chen, and P. Zeng, “Millimeter-wave wireless communications for iot-cloud supported autonomous vehicles: Overview, design, and challenges,” _IEEE Communications Magazine_ , vol. 55, no. 1, pp. 62–68, 2017.
  * [14] J. Choi, V. Va, N. Gonzalez-Prelcic, R. Daniels, C. R. Bhat, and R. W. Heath, “Millimeter-wave vehicular communication to support massive automotive sensing,” _IEEE Communications Magazine_ , vol. 54, no. 12, pp. 160–167, 2016.
  * [15] H. Wymeersch, G. Seco-Granados, G. Destino, D. Dardari, and F. Tufvesson, “5g mmwave positioning for vehicular networks,” _IEEE Wireless Communications_ , vol. 24, no. 6, pp. 80–86, 2017.
  * [16] W. Roh, J.-Y. Seol, J. Park, B. Lee, J. Lee, Y. Kim, J. Cho, K. Cheun, and F. Aryanfar, “Millimeter-wave beamforming as an enabling technology for 5g cellular communications: theoretical feasibility and prototype results,” _IEEE Communications Magazine_ , vol. 52, no. 2, pp. 106–113, 2014.
  * [17] S. Chen, J. Hu, Y. Shi, L. Zhao, and W. Li, “A vision of c-v2x: Technologies, field testing, and challenges with chinese development,” _IEEE Internet of Things Journal_ , vol. 7, no. 5, pp. 3872–3881, 2020.
  * [18] Q. Wang, D. O. Wu, and P. Fan, “Delay-constrained optimal link scheduling in wireless sensor networks,” _IEEE Transactions on Vehicular Technology_ , vol. 59, no. 9, pp. 4564–4577, 2010.
  * [19] C. Pan, Z. Wang, H. Liao, Z. Zhou, X. Wang, M. Tariq, and S. Al-Otaibi, “Asynchronous federated deep reinforcement learning-based urllc-aware computation offloading in space-assisted vehicular networks,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–13, 2022.
  * [20] H. Liao, Z. Zhou, W. Kong, Y. Chen, X. Wang, Z. Wang, and S. Al Otaibi, “Learning-based intent-aware task offloading for air-ground integrated vehicular edge computing,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 8, pp. 5127–5139, 2021.
  * [21] S. Batewela, C.-F. Liu, M. Bennis, H. A. Suraweera, and C. S. Hong, “Risk-sensitive task fetching and offloading for vehicular edge computing,” _IEEE Communications Letters_ , vol. 24, no. 3, pp. 617–621, 2020.
  * [22] Y. Cui, L. Du, H. Wang, D. Wu, and R. Wang, “Reinforcement learning for joint optimization of communication and computation in vehicular networks,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 12, pp. 13 062–13 072, 2021.
  * [23] Y. Zhu, Y. Hu, T. Yang, T. Yang, J. Vogt, and A. Schmeink, “Reliability-optimal offloading in low-latency edge computing networks: Analytical and reinforcement learning based designs,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 6, pp. 6058–6072, 2021.
  * [24] J. Posner, L. Tseng, M. Aloqaily, and Y. Jararweh, “Federated learning in vehicular networks: Opportunities and solutions,” _IEEE Network_ , vol. 35, no. 2, pp. 152–159, 2021.
  * [25] X. Zhang, S. Pan, and Q. Miao, “Adaptive beamforming-based gigabit message dissemination for highway vanets,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 7666–7679, 2022.
  * [26] Z. Sheng, A. Pressas, V. Ocheri, F. Ali, R. Rudd, and M. Nekovee, “Intelligent 5g vehicular networks: An integration of dsrc and mmwave communications,” in _2018 International Conference on Information and Communication Technology Convergence (ICTC)_ , 2018, pp. 571–576.
  * [27] Y. Ming, J. Chen, Y. Dong, and Z. Wang, “Evolutionary game based strategy selection for hybrid v2v communications,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 2, pp. 2128–2133, 2022.
  * [28] S. Samarakoon, M. Bennis, W. Saad, and M. Debbah, “Distributed federated learning for ultra-reliable low-latency vehicular communications,” _IEEE Transactions on Communications_ , vol. 68, no. 2, pp. 1146–1159, 2020\.
  * [29] C. Guo, L. Liang, and G. Y. Li, “Resource allocation for low-latency vehicular communications: An effective capacity perspective,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 4, pp. 905–917, 2019.
  * [30] M. J. Neely, “Stochastic network optimization with application to communication and queueing systems,” _Synthesis Lectures on Communication Networks_ , vol. 3, no. 1, pp. 1–211, 2010.
  * [31] G. Yang, M. Xiao, and H. V. Poor, “Low-latency millimeter-wave communications: Traffic dispersion or network densification?” _IEEE Transactions on Communications_ , vol. 66, no. 8, pp. 3526–3539, 2018.
  * [32] G. Yang, M. Xiao, H. Al-Zubaidy, Y. Huang, and J. Gross, “Analysis of millimeter-wave multi-hop networks with full-duplex buffered relays,” _IEEE/ACM Transactions on Networking_ , vol. 26, no. 1, pp. 576–590, 2018\.
  * [33] Y. Jiang, Y. Liu _et al._ , _Stochastic network calculus_. Springer, 2008, vol. 1.
  * [34] K. Katsaros, M. Dianati, R. Tafazolli, and X. Guo, “End-to-end delay bound analysis for location-based routing in hybrid vehicular networks,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 9, pp. 7462–7475, 2016.
  * [35] J.-W. Cho and Y. Jiang, “Fundamentals of the backoff process in 802.11: Dichotomy of the aggregation,” _IEEE Transactions on Information Theory_ , vol. 61, no. 4, pp. 1687–1701, 2015.
  * [36] M. Fidler, “An end-to-end probabilistic network calculus with moment generating functions,” in _200614th IEEE International Workshop on Quality of Service_ , 2006, pp. 261–270.
  * [37] Y. Ghasempour, C. R. C. M. da Silva, C. Cordeiro, and E. W. Knightly, “Ieee 802.11ay: Next-generation 60 ghz communication for 100 gb/s wi-fi,” _IEEE Communications Magazine_ , vol. 55, no. 12, pp. 186–192, 2017.
  * [38] I. Mavromatis, A. Tassi, and R. J. Piechocki, “Operating its-g5 dsrc over unlicensed bands: A city-scale performance evaluation,” in _2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2019, pp. 1–7.
  * [39] S. Mittal, “Power management techniques for data centers: A survey,” _arXiv preprint arXiv:1404.6681_ , 2014.
  * [40] Y. Li, S. Xia, M. Zheng, B. Cao, and Q. Liu, “Lyapunov optimization-based trade-off policy for mobile cloud offloading in heterogeneous wireless networks,” _IEEE Transactions on Cloud Computing_ , vol. 10, no. 1, pp. 491–505, 2022.
  * [41] S. Bae, S. Han, and Y. Sung, “A reinforcement learning formulation of the lyapunov optimization: Application to edge computing systems with queue stability,” _CoRR_ , 2020.
  * [42] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” 2018. [Online]. Available: https://arxiv.org/abs/1801.01290
  * [43] S. Mangiante, G. Klas, A. Navon, G. Zhuang, J. Ran, and M. Silva, “Vr is on the edge: How to deliver 360° videos in mobile networks,” 08 2017, pp. 30–35.
  * [44] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [45] N. Keshari, T. S. Gupta, and D. Singh, “Particle swarm optimization based task offloading in vehicular edge computing,” in _2021 IEEE 18th India Council International Conference (INDICON)_ , 2021, pp. 1–8.
  * [46] Z. Zhou, Z. Wang, H. Yu, H. Liao, S. Mumtaz, L. Oliveira, and V. Frascolla, “Learning-based urllc-aware task offloading for internet of health things,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 2, pp. 396–410, 2021.
  * [47] J.-Y. Le Boudec and P. Thiran, Eds., _Network Calculus_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp. 3–81. [Online]. Available: https://doi.org/10.1007/3-540-45318-0_1
  * [48] C.-S. Chang, _Performance guarantees in communication networks_. Springer Science & Business Media, 2000.

## References

  * [1] Z. Chen and X. Wang, “Decentralized computation offloading for multi-user mobile edge computing: A deep reinforcement learning approach,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2020, no. 1, pp. 1–21, 2020.
  * [2] K. Xiong, S. Leng, C. Huang, C. Yuen, and Y. L. Guan, “Intelligent task offloading for heterogeneous v2x communications,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 4, pp. 2226–2238, 2021.
  * [3] T. Taleb, K. Samdanis, B. Mada, H. Flinck, S. Dutta, and D. Sabella, “On multi-access edge computing: A survey of the emerging 5g network edge cloud architecture and orchestration,” _IEEE Communications Surveys Tutorials_ , vol. 19, no. 3, pp. 1657–1681, 2017.
  * [4] Q. Wu, Y. Zhao, Q. Fan, P. Fan, J. Wang, and C. Zhang, “Mobility-aware cooperative caching in vehicular edge computing based on asynchronous federated and deep reinforcement learning,” _IEEE Journal of Selected Topics in Signal Processing_ , pp. 1–16, 2022.
  * [5] H. Zhu, Q. Wu, X.-J. Wu, Q. Fan, P. Fan, and J. Wang, “Decentralized power allocation for mimo-noma vehicular edge computing based on deep reinforcement learning,” _IEEE Internet of Things Journal_ , vol. 9, no. 14, pp. 12 770–12 782, 2022.
  * [6] K. Zheng, Q. Zheng, P. Chatzimisios, W. Xiang, and Y. Zhou, “Heterogeneous vehicular networking: A survey on architecture, challenges, and solutions,” _IEEE Communications Surveys Tutorials_ , vol. 17, no. 4, pp. 2377–2396, 2015\.
  * [7] J. B. Kenney, “Dedicated short-range communications (dsrc) standards in the united states,” _Proceedings of the IEEE_ , vol. 99, no. 7, pp. 1162–1182, 2011.
  * [8] C. Han, M. Dianati, R. Tafazolli, R. Kernchen, and X. Shen, “Analytical study of the ieee 802.11p mac sublayer in vehicular networks,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 13, no. 2, pp. 873–886, 2012.
  * [9] X. Wang, S. Mao, and M. X. Gong, “An overview of 3gpp cellular vehicle-to-everything standards,” _GetMobile: Mobile Computing and Communications_ , vol. 21, no. 3, pp. 19–25, 2017.
  * [10] G. Naik, B. Choudhury, and J.-M. Park, “Ieee 802.11bd $\&$ 5g nr v2x: Evolution of radio access technologies for v2x communications,” _IEEE Access_ , vol. 7, pp. 70 169–70 184, 2019.
  * [11] K. Abboud, H. A. Omar, and W. Zhuang, “Interworking of dsrc and cellular network technologies for v2x communications: A survey,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 12, pp. 9457–9470, 2016.
  * [12] R. Molina-Masegosa and J. Gozalvez, “Lte-v for sidelink 5g v2x vehicular communications: A new 5g technology for short-range vehicle-to-everything communications,” _IEEE Vehicular Technology Magazine_ , vol. 12, no. 4, pp. 30–39, 2017.
  * [13] L. Kong, M. K. Khan, F. Wu, G. Chen, and P. Zeng, “Millimeter-wave wireless communications for iot-cloud supported autonomous vehicles: Overview, design, and challenges,” _IEEE Communications Magazine_ , vol. 55, no. 1, pp. 62–68, 2017.
  * [14] J. Choi, V. Va, N. Gonzalez-Prelcic, R. Daniels, C. R. Bhat, and R. W. Heath, “Millimeter-wave vehicular communication to support massive automotive sensing,” _IEEE Communications Magazine_ , vol. 54, no. 12, pp. 160–167, 2016.
  * [15] H. Wymeersch, G. Seco-Granados, G. Destino, D. Dardari, and F. Tufvesson, “5g mmwave positioning for vehicular networks,” _IEEE Wireless Communications_ , vol. 24, no. 6, pp. 80–86, 2017.
  * [16] W. Roh, J.-Y. Seol, J. Park, B. Lee, J. Lee, Y. Kim, J. Cho, K. Cheun, and F. Aryanfar, “Millimeter-wave beamforming as an enabling technology for 5g cellular communications: theoretical feasibility and prototype results,” _IEEE Communications Magazine_ , vol. 52, no. 2, pp. 106–113, 2014.
  * [17] S. Chen, J. Hu, Y. Shi, L. Zhao, and W. Li, “A vision of c-v2x: Technologies, field testing, and challenges with chinese development,” _IEEE Internet of Things Journal_ , vol. 7, no. 5, pp. 3872–3881, 2020.
  * [18] Q. Wang, D. O. Wu, and P. Fan, “Delay-constrained optimal link scheduling in wireless sensor networks,” _IEEE Transactions on Vehicular Technology_ , vol. 59, no. 9, pp. 4564–4577, 2010.
  * [19] C. Pan, Z. Wang, H. Liao, Z. Zhou, X. Wang, M. Tariq, and S. Al-Otaibi, “Asynchronous federated deep reinforcement learning-based urllc-aware computation offloading in space-assisted vehicular networks,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–13, 2022.
  * [20] H. Liao, Z. Zhou, W. Kong, Y. Chen, X. Wang, Z. Wang, and S. Al Otaibi, “Learning-based intent-aware task offloading for air-ground integrated vehicular edge computing,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 8, pp. 5127–5139, 2021.
  * [21] S. Batewela, C.-F. Liu, M. Bennis, H. A. Suraweera, and C. S. Hong, “Risk-sensitive task fetching and offloading for vehicular edge computing,” _IEEE Communications Letters_ , vol. 24, no. 3, pp. 617–621, 2020.
  * [22] Y. Cui, L. Du, H. Wang, D. Wu, and R. Wang, “Reinforcement learning for joint optimization of communication and computation in vehicular networks,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 12, pp. 13 062–13 072, 2021.
  * [23] Y. Zhu, Y. Hu, T. Yang, T. Yang, J. Vogt, and A. Schmeink, “Reliability-optimal offloading in low-latency edge computing networks: Analytical and reinforcement learning based designs,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 6, pp. 6058–6072, 2021.
  * [24] J. Posner, L. Tseng, M. Aloqaily, and Y. Jararweh, “Federated learning in vehicular networks: Opportunities and solutions,” _IEEE Network_ , vol. 35, no. 2, pp. 152–159, 2021.
  * [25] X. Zhang, S. Pan, and Q. Miao, “Adaptive beamforming-based gigabit message dissemination for highway vanets,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 7666–7679, 2022.
  * [26] Z. Sheng, A. Pressas, V. Ocheri, F. Ali, R. Rudd, and M. Nekovee, “Intelligent 5g vehicular networks: An integration of dsrc and mmwave communications,” in _2018 International Conference on Information and Communication Technology Convergence (ICTC)_ , 2018, pp. 571–576.
  * [27] Y. Ming, J. Chen, Y. Dong, and Z. Wang, “Evolutionary game based strategy selection for hybrid v2v communications,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 2, pp. 2128–2133, 2022.
  * [28] S. Samarakoon, M. Bennis, W. Saad, and M. Debbah, “Distributed federated learning for ultra-reliable low-latency vehicular communications,” _IEEE Transactions on Communications_ , vol. 68, no. 2, pp. 1146–1159, 2020\.
  * [29] C. Guo, L. Liang, and G. Y. Li, “Resource allocation for low-latency vehicular communications: An effective capacity perspective,” _IEEE Journal on Selected Areas in Communications_ , vol. 37, no. 4, pp. 905–917, 2019.
  * [30] M. J. Neely, “Stochastic network optimization with application to communication and queueing systems,” _Synthesis Lectures on Communication Networks_ , vol. 3, no. 1, pp. 1–211, 2010.
  * [31] G. Yang, M. Xiao, and H. V. Poor, “Low-latency millimeter-wave communications: Traffic dispersion or network densification?” _IEEE Transactions on Communications_ , vol. 66, no. 8, pp. 3526–3539, 2018.
  * [32] G. Yang, M. Xiao, H. Al-Zubaidy, Y. Huang, and J. Gross, “Analysis of millimeter-wave multi-hop networks with full-duplex buffered relays,” _IEEE/ACM Transactions on Networking_ , vol. 26, no. 1, pp. 576–590, 2018\.
  * [33] Y. Jiang, Y. Liu _et al._ , _Stochastic network calculus_. Springer, 2008, vol. 1.
  * [34] K. Katsaros, M. Dianati, R. Tafazolli, and X. Guo, “End-to-end delay bound analysis for location-based routing in hybrid vehicular networks,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 9, pp. 7462–7475, 2016.
  * [35] J.-W. Cho and Y. Jiang, “Fundamentals of the backoff process in 802.11: Dichotomy of the aggregation,” _IEEE Transactions on Information Theory_ , vol. 61, no. 4, pp. 1687–1701, 2015.
  * [36] M. Fidler, “An end-to-end probabilistic network calculus with moment generating functions,” in _200614th IEEE International Workshop on Quality of Service_ , 2006, pp. 261–270.
  * [37] Y. Ghasempour, C. R. C. M. da Silva, C. Cordeiro, and E. W. Knightly, “Ieee 802.11ay: Next-generation 60 ghz communication for 100 gb/s wi-fi,” _IEEE Communications Magazine_ , vol. 55, no. 12, pp. 186–192, 2017.
  * [38] I. Mavromatis, A. Tassi, and R. J. Piechocki, “Operating its-g5 dsrc over unlicensed bands: A city-scale performance evaluation,” in _2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2019, pp. 1–7.
  * [39] S. Mittal, “Power management techniques for data centers: A survey,” _arXiv preprint arXiv:1404.6681_ , 2014.
  * [40] Y. Li, S. Xia, M. Zheng, B. Cao, and Q. Liu, “Lyapunov optimization-based trade-off policy for mobile cloud offloading in heterogeneous wireless networks,” _IEEE Transactions on Cloud Computing_ , vol. 10, no. 1, pp. 491–505, 2022.
  * [41] S. Bae, S. Han, and Y. Sung, “A reinforcement learning formulation of the lyapunov optimization: Application to edge computing systems with queue stability,” _CoRR_ , 2020.
  * [42] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” 2018. [Online]. Available: https://arxiv.org/abs/1801.01290
  * [43] S. Mangiante, G. Klas, A. Navon, G. Zhuang, J. Ran, and M. Silva, “Vr is on the edge: How to deliver 360° videos in mobile networks,” 08 2017, pp. 30–35.
  * [44] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [45] N. Keshari, T. S. Gupta, and D. Singh, “Particle swarm optimization based task offloading in vehicular edge computing,” in _2021 IEEE 18th India Council International Conference (INDICON)_ , 2021, pp. 1–8.
  * [46] Z. Zhou, Z. Wang, H. Yu, H. Liao, S. Mumtaz, L. Oliveira, and V. Frascolla, “Learning-based urllc-aware task offloading for internet of health things,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 2, pp. 396–410, 2021.
  * [47] J.-Y. Le Boudec and P. Thiran, Eds., _Network Calculus_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp. 3–81. [Online]. Available: https://doi.org/10.1007/3-540-45318-0_1
  * [48] C.-S. Chang, _Performance guarantees in communication networks_. Springer Science & Business Media, 2000.