# Elastic execution of checkpointed MPI applications

Sumeet Gajjar  Khoury College of Computer Sciences
Northeastern University
Boston, MA
<EMAIL_ADDRESS>Saurabh Vaidya  Khoury College of Computer Sciences
Northeastern University
Boston, MA
<EMAIL_ADDRESS>
###### Abstract

MPI applications begin with a fixed number of rank and, by default, the rank
remains constant throughout the application’s lifetime. The developer can
choose to increase the rank by dynamically spawning MPI processes. However
doing this manually adds complexity to the MPI application. Making the MPI
applications malleable [20] would allow HPC applications to have the same
elasticity as that of cloud applications. We propose multiple approaches to
change the rank of an MPI program agnostic to the modification of the user
code. We use checkpointing as a tool to achieve mutability of rank by halting
the execution and resuming the MPI program with a new state. In this paper, we
focus on the scenario of increasing the rank of an MPI program using ExaMPI as
the implementation for MPI.

###### Index Terms:

ExaMPI, Elastic MPI, MPI_Fork

##  I Introduction

Message Passing remains a dominant programming model for distributed memory
systems. Message Passing Interface (MPI) [1] is the de facto method for
developing applications in scientific computing. The applications rely on MPI
to achieve horizontal scaling with multiple nodes in HPC clusters. This kind
of scaling is necessary to support memory and performance requirements.

The MPI applications running on HPC clusters have mostly been designed to use
a fixed number of resources. Thus, the application cannot exploit elasticity
when additional resources are available. MPI versions 1.x did not have any
support for changing the number of processes during the execution. MPI 2.0
introduced a feature to increase the number of processes running under MPI
using MPI_Comm_spawn [2] and MPI_Comm_spawn_multiple [3]. However, this
feature is not supported by many of the available MPI implementations. The
developer must put in extra effort to manage the newly spawned processes and
redistribute the data efficiently amongst those new processes.

Elastic execution of MPI applications is the process of modifying the
allocated resources to the MPI application thereby increasing or decreasing
the number of MPI processes running under the hood. There are a couple of
cases that the elastic execution might help. One, for a long-running
application, a user may want to increase the allocated resources to try and
reduce the completion time of the application. Secondly, when an application
is not scaling linearly, the number of instances can be reduced, resulting in
lower nodes×hours, and thus a lower cost.

In this paper, we propose three approaches to allow elastic execution of MPI
applications. All of these approaches are implemented in userspace. Approaches
A and C require modification to the underlying MPI library, while Approach B
leverages the existing MPI APIs to achieve elasticity, with a caveat of
modifying the MPI library to implement the corresponding APIs.

##  II Related work

There are several research efforts related to elastic execution models. The
literature for elastic execution models mostly relates to grid and cloud
computing. The requirements for synchronization are low when it comes to grid
and cloud computing as compared to scientific applications.

Raveendran et. al. [4] propose an approach to elastically execute existing MPI
applications in the cloud. Nodes can be added and removed on the fly without
hindering the current execution of the MPI application. To demonstrate this
idea, the authors had to manually modify existing MPI applications to persist
data at every iteration. However, this approach requires the MPI application
to persist its state. The authors argue that this can be easily automated
using a source-to-source translation tool.

Comprés et. al. [5] demonstrate the idea of elastic execution by proposing an
MPI extension. The authors introduce four new MPI operations to achieve this.
As proof of concept, the authors modified the MPI library — MPICH [6] — and
the resource manager — SLURM. [7]

##  III Background

###  III-A DMTCP

Modern supercomputers comprise thousands of nodes, and this number is
constantly increasing. With the increasing number of hardware components in
the supercomputers, a hardware fault is likely to occur sooner or later on
these man-made marvels. The applications are however not developed to
accommodate these hardware faults. Fault tolerance mechanisms can be added to
the application; however, this is not the primary objective of the developer
building a scientific application.

Checkpointing [8] is a technique that provides fault tolerance for computing
systems. It consists of saving a snapshot of the application’s state, so that
the application can restart from that point in case of failure. This is
particularly important for long-running applications that are executed in
failure-prone computing systems. DMTCP — Distributed Multi Threaded
Checkpointing [9] — is a user space library to transparently checkpoint
applications. In this paper, we use DMTCP as our primary tool to checkpoint
and restore an MPI process or the entire MPI application as a whole.

###  III-B DMTCP plugins

DMTCP plugins [10] provide the flexibility to extend DMTCP generic library and
write an add-on library. The user can use the DMTCP plugins to modify the
state or any internal data structures of the program during, before or after
checkpointing. In this paper we use the event hooks functionality of DMTCP
plugins to capture various events generated by DMTCP and manipulate the data-
structures to suit our needs.

###  III-C ExaMPI

ExaMPI [11] is one of the implementations of the MPI standard developed with
the goal of enabling researchers to experiment rapidly and easily with its
internals. It supports a modern MPI-3.x subset with a robust MPI-4.x road-map.

For the purpose of this paper, we need to understand the MPI runtime viz.
mpiexec. mpiexec is a Python program which takes the parameters to start the
MPI application. mpiexec spawns multiple _fault_daemons_ , one for each rank,
with fairly similar but different environment. _fault_daemons_ are the
wrappers which execute the actual MPI process. Each _fault_daemon_ with local-
id 0 launches a _head_daemon_. Each node has one _head_daemon_ which is
responsible for managing all the _fault_daemons_ on that node. After launching
the _head_daemon_ , if the rank of the _fault_daemon_ is 0 then a
_controller_daemon_ is launched. The _controller_daemon_ is responsible for
managing the entire MPI cluster. Once the _controller_daemon_ is launched, all
the _fault_daemons_ try to establish a communication link with the respective
_head_daemon_ s. On success, the _fault_daemon_ now commences the user defined
MPI program and the MPI cluster is initialized and ready for message passing.

##  IV Proposed Solution

###  IV-A Approach A

In the current design of ExaMPI, the only way to spawn a _fault_daemon_ is
using mpiexec. Since mpiexec exits after spawning the specified number of
_fault_daemons_ , there is no clean way to spawn additional _fault_daemons_
for the same MPI application further in time.

We propose to change the design of the mpiexec and modify the order in which
various daemons are launched. According to the proposed design, mpiexec
launches the _head_daemons_ first. The _head_daemon_ spawns the
_fault_daemons_ which in turn starts MPI processes. The _head_daemon_ with
rank 0 spawns the _controller_daemon_. Because of this change in the ordering,
we now have the ability to spawn _fault_daemons_ without using mpiexec. We can
signal _controller_daemon_ to spawn new MPI processes. The _controller_daemon_
based on the load balancing scheme will signal the corresponding
_head_daemons_. The _head_daemons_ spawns the respective number of
_fault_daemons_ which finally spawn the MPI processes.

The modified design of ExaMPI is needed to adapt to launching under DMTCP. We
launch the MPI application under DMTCP with –ckpt-open-files –allow-file-
overwrite parameters –with-plugin exampi_dmtcp_plugin.so. The –ckpt-open-files
flag allows user to checkpoint the open files. The ExaMPI library has config
files for each MPI process. The config file contains the information to
establish TCP/UDP connection with any of the MPI ranks. These config files are
created during the initialization of _fault_daemons_. As per the current
implementation these config files are deleted once the application exits
gracefully or if the application is interrupted in any manner for e.g. CTRL+C.
The –ckpt-open-files flag also enables the restore of these deleted temporary
files during the restart phase.

A further problem in the integration of ExaMPI with DMTCP is that ExaMPI also
creates files for logging purposes. DMTCP must checkpoint these files and try
to restore them during the restart phase. But unlike config files the log
files are available during the restart phase and when DMTCP tries restoring
them, it encounters an error as the files already exists and hence cannot be
overwritten. Specifying –allow-file-overwrite parameters flag resolves the
issue and allows successful restart of the checkpointed MPI application.

During the init phase of DMTCP, the control is transferred momentarily to the
exampi_dmtcp_plugin and a thread is created which waits on a
conditional_variable. After the thread is created, the control is returned
back to DMTCP, and it continues the execution of the MPI application. During
the restart phase, DMTCP restores the MPI application in memory and gives the
control to the exampi_dmtcp_plugin. The conditional_variable is signaled by
the plugin and the control is returned back to DMTCP which resumes the
execution of the MPI application.

The waiting thread now tries to establish a TCP connection with the
_controller_daemon_ in the ExaMPI runtime. Once the connection is established,
a message is sent to the _controller_daemon_ to increase the rank of MPI
application. The _controller_daemon_ now determines how many more MPI
processes are required and the corresponding nodes to spawn the MPI processes.
The controller now sends a message to the respective _head_daemons_ to spawn
_fault_daemons_. The _head_daemons_ now spawns the _fault_daemons_ which in
turn spawn the MPI processes. The newly spawned processes are now blocked on
the barrier [12] in MPI_Init [13].

During the restart phase, connection information of the newly spawned
_fault_daemons_ needs to be provided to the old MPI processes. This connection
information and metadata in stored in config files and environment variables.
The metadata information stored in the environment variables e.g.
”EXAMPI_WORLD_SIZE” also needs to be updated. ExaMPI also uses ”epoch” files
to facilitate checkpoint and rollback to a previous state. However this
feature is still in development stage and not included in the production
release. Hence we are ignoring it in our design.

The old MPI processes now receive a error code MPI_WORLD_RESIZED on executing
any of the MPI functions. The error code is a way to notify the already
running MPI processes that the size of the world has changed. The processes
can decide to re-partition data and distribute the work amongst the old as
well as the newly spawned processes and make a call to enter the barrier.
Thus, our new cluster will now have an increased rank.

Once the cluster upgrade is complete the new MPI processes are added to the
MPI_COMM_WORLD communicator. No new communicator is created, the
MPI_COMM_WORLD acts a unified communicator for old and new MPI processes.

In addition to the approach described previously, there can be another
alternative. The alternative approach is to convert the MPI_COMM_WORLD into a
versioned communicator. Initially the MPI_COMM_WORLD has the tag $v0$ And with
each cluster upgrade the communicator expands but has a version tag associated
with it. Let’s consider a scenario where the MPI cluster was upgraded twice.
For the upgrade from $x\xrightarrow{}y$ where $(y>x)$, the communicator gets
the tag $v1$ and for the upgrade from $y\xrightarrow{}z$ where $(z>y)$, the
communicator gets the tag $v2$. The communicator tag can be used to identify
the membership of a given MPI process in a communicator at any given point in
time. The MPI process can choose to use the tag along with the communicator to
communicate with another MPI process. If the tag is not specified then the tag
is defaulted to the latest one.

⬇

void repartition_data(int status_code){

if (status_code == MPI_WORLD_RESIZED){

/* code to repartition

* or redistribute the data

*/

MPI_barrier();

}

}

int main(){

//...

int i, status_code;

for (i = 0; i < TOTAL_ITERATIONS; i++)

{

status_code = MPI_Send();

repartition_data(status_code);

status_code = MPI_Recv();

repartition_data(status_code);

/* after any MPI operation,

* repartition_data should be

* called with the status code

* of the operation to check

* if data repartition is required

*/

}

//...

}

Listing 1: Code Snippet for Approach A

###  IV-B Approach B

In this approach we use MPI_Comm_spawn and MPI_Intercomm_merge to launch new
MPI processes and create a new communicator MPI_COMM_RESIZED_WORLD which
represents the resized world after restarting the checkpointed MPI
application.

We launch the MPI application under DMTCP in a similar manner as described in
Approach A. The plugin however functions differently. During the plugin init
phase we initialize the global variable MPI_COMM_RESIZED_WORLD to
MPI_COMM_NULL. MPI_COMM_NULL indicates the communicator is invalid and it is
semantically correct since there is no resized world when the MPI application
starts. A thread is also created which waits on a conditional_variable. After
the thread is created the control is returned back to the DMTCP and continues
the execution of MPI application.

During the restart phase the control comes back to our plugin and the
conditional_variable is signaled. The thread waiting on the
conditional_variable now calls MPI_Comm_spawn and spawns additional required
processes. MPI_COMM_WORLD is passed as the intracommunicator [15] to
MPI_Comm_spawn while creating the additional processes. The child_comm is used
as intercommunicator to communicate with the newly spawned MPI processes. We
now use MPI_Intercomm_merge to merge the child_comm into resized_world_comm.
The resized_world_comm is a intracommunicator which contains the old as well
as the newly spawned MPI processes. We now assign resized_world_comm to
MPI_COMM_RESIZED_WORLD and relay the control back to the MPI application.

⬇

/* intercommunicator to communicate

* with new spawned processes

*/

MPI_Comm child_comm;

MPI_Comm_spawn("./worker", MPI_ARGV_NULL,

2, MPI_INFO_NULL, 0,

MPI_COMM_WORLD,

&child_comm,

MPI_ERRCODES_IGNORE);

MPI_Comm resized_world_comm;

MPI_Intercomm_merge(child_comm, 0,

&resized_world_comm);

MPI_COMM_RESIZED_WORLD = resized_world_comm;

Listing 2: Code Snippet to create the resized world communicator

This approach however has a caveat. The user MPI program needs to check in
every iteration of the computation whether the MPI_COMM_RESIZED_WORLD is not
equal to MPI_COMM_NULL. As soon as this condition is satisfied the MPI old
processes can perform the data re-partition to distribute the work amongst the
newly spawned processes. This step is similar to receiving the
MPI_WORLD_RESIZED error code in approach A. Listing 3 provides a code snippet
to demonstrate the idea.

One can ask why not use the error code to indicate to the user about the
resized world as described in approach A. The reason for this is to remain MPI
implementation agnostic. This method does not require any changes in the MPI
implementation. The implementation to increase the number of processes is
completely encapsulated in the DMTCP plugin.

###  IV-C Approach C

Here, we formulate a new API to the MPI standard: MPI_Fork. MPI_Fork can have
a wide applicability and prove to be a very useful tool to developers. One can
argue there already exists MPI_Comm_spawn and MPI_Comm_spawn_multiple which
serves the purpose of spawning more processes. However the MPI processes
started using MPI_Comm_spawn and MPI_Comm_spawn_multiple have none of the
parents data. The newly spawned processes needs to fetch the data to process
from the other processes. This operation involves a use of intercommunicators
[17] and these transforms a fairly simple MPI application into a complex one.

MPI_Fork tries to reduce this complexity by creating a clone of the parent and
hence starting the child with already initialized parents data. The developers
of MPI applications are usually scientists but not computer scientists. and
introducing a simpler API of MPI_Fork leads to its easy adoption amongst the
community.

MPI_Fork draws inspiration from UNIX fork for creating new processes. We
propose using the same idea like UNIX to be able to fork new ranks from old
ranks such that each new rank(child) forks off with the same memory of its
parent. This way, we achieve new ranks that are initialized with some data i.e
their parent’s data. Hence we now the fully initialized data, and we can
”forget” the part of the data that we don’t need.

One fundamental difference between the UNIX fork and MPI_Fork is that the UNIX
forked process is spawned on the same machine. However the process created
using MPI_Fork may or may not be spawned on the same node as its parent rank.

The ability to fork a new child can allow sharing of responsibility. The newly
forked processes can derive the mutually exclusive work based on who the
parent is. Thus we bring the traditional parallelism model of forking can be
made possible in MPI universe. One such example can be relaxation models e.g
membrane of a soap bubble. The core idea here would be that at the beginning
of the analysis we will have a coarser grid. Then on availability of new
resources new children are forked. As the number of processes increases the
grid becomes finer and finer and the accuracy of the model increases. Thus our
multi-grid application would use limited resources at the beginning and would
demand additional nodes only later.

The ability to increase the resources as the computation demands results into
a more effective and optimal use of resources in computational clusters.
Hellenbrand et. al. [16] demonstrates and concludes that the CPU hours to
complete one simulation run is reduced almost half in the elastic MPI run.

⬇

void repartition_data_if_required(){

if (MPI_COMM_RESIZED_WORLD != MPI_COMM_NULL){

/* code to repartition or

* redistribute the data

*/

MPI_barrier();

}

}

int main(){

//...

int i;

for (i = 0; i < TOTAL_ITERATIONS; i++)

{

/*

* user code

*/

repartition_data_if_required();

}

//...

}

Listing 3: Code Snippet for Approach B

We now propose the design for implementing MPI_Fork in ExaMPI. We discussed
the order of launching the Controller, Head and Fault daemons in the ExaMPI
subsection of background. In order to implement MPI_Fork we propose a design
change in ExaMPI for the order in which the daemons are launched. We propose
the following ordering where mpiexec launches the controller_daemon. The
controller_daemon determines the number of head_daemons required and launches
the corresponding head_daemons. The head_daemons determined the number of
fault_daemons required and launches the corresponding fault_daemons.

The new design for MPI_Fork necessitates changing the role of the
fault_daemons in launching other processes. Recall that the fault_daemons in
the current design directly launches the MPI Application. In the proposed
design, each fault_daemon launches the MPI Application under DMTCP. This can
be achieved by using dmtcp_launch command provided by the DMTCP library. Each
MPI process has its own DMTCP coordinator and the fault_daemon tracks the
information to communicate with its corresponding DMTCP coordinator.

There is a reason why we propose that each MPI process should be launched with
separate instance of DMTCP. If we start the entire MPI cluster under a single
instance of DMTCP then the entire cluster )along with Controller, Head and
Fault daemons) will be checkpointed. The requirement of the implementation is
to clone a single MPI process fundamentally which can only be achieved if each
MPI process is started under a separate instance of DMTCP.

Next, we discuss the sequence of events that occur after a call to MPI_Fork. A
call to MPI_Fork signals the controller_daemon. The controller_daemon
determines the MPI processes that should be cloned. This is determined by
using the following deterministic algorithm. Since we have constraints on the
the value of m, the controller_daemon can select the first m ranks for the
cloning. Once decided the fault_daemons managing these MPI processes are
signaled to checkpoint the corresponding MPI processes. This is achieved by
using the dmtcp_command –coord-host [host] –checkpoint. For the purpose of
this paper we will consider the scenario of forking additional m processes. m
can be between 1 and n, where n is the total number of MPI processes in the
MPI application. Since MPI_Fork is designed to be a collective operation, each
MPI process should participate in it.

After MPI_Fork has signaled the controller_daemon, the controller_daemon must
organize the remaining work. The controller_daemon determines the number of
head_daemons required based on the number of additional processes required.
The controller_daemon now launches the additional head_daemons. The
head_daemons launches the corresponding number of fault_daemons. The
fault_daemons launches or in this case restarts the corresponding checkpointed
MPI processes. This is achieved using the dmtcp_restart [checkpoint_file].
During the restart process the environment variables and configs file of old
processes are modified and connection information of the newly started
processes is populated. This is to ensure the newly spawned processes are
recognised by the old MPI processes.

The proposed ordering of launching the daemons can also be applied to approach
A. Approach A requires head_daemon to launch a fault_daemon. Since the
ordering between head_daemon and fault_daemon is same for both the approaches
we can easily support both approach A and approach C using the proposed
ordering in approach C.

While the new processes are being started the old MPI processes are waiting on
a barrier which includes the parent as well the newly started processes. On
successful restart the child processes now enter the barrier. Since the MPI
processes are in the barrier, the control returns back to the MPI_Fork in both
old and the new MPI processes. In the old processes, MPI_Fork returns a
positive value which corresponds to number of new MPI processes spawned. In
the new processes, MPI_Fork returns zero. A negative return value indicates
the error encountered during the forking process.

For successful forking, an intracommunicator is required for peer-to-peer or
collective communication across the old and new MPI processes. There are
multiple options to solve this problem. A very straight forward solution is to
add all the newly forked to MPI_COMM_WORLD. Hence MPI_COMM_WORLD represents
the entire universe of old and new MPI processes. There can arise a need where
an operation needs to be performed on the processes which existed before
forking. We will call those processes as the parents. Similarly a need can
arise to address only the newly spawned processes. We will call those
processes as the children.

We introduce two new intracommunicators MPI_COMM_PARENTS and
MPI_COMM_CHILDREN. MPI_COMM_PARENTS as the name suggests encapsulates the old
processes or parent processes and MPI_COMM_CHILDREN encapsulates the newly
spawned processes or the children processes. The initial value or the value
before forking of MPI_COMM_PARENTS and MPI_COMM_CHILDREN would be
MPI_COMM_NULL. However this solution does not consider the scenario where
forking occurs repeatedly.

In order to solve the problem of multiple forking, we can use a versioned
communicator as discussed in approach A. Each time MPI_Fork is invoked, we add
a version tag to all the three intracommunicators, viz. MPI_COMM_WORLD,
MPI_COMM_PARENTS and MPI_COMM_CHILDREN. Now using the tag along with the
communicator the MPI processes can easily distinguish between the processes
which existed at various forking times.

##  V Discussion

The approach A, in our opinion, is transparent to the user since it requires
no modification to the user code, except for the part where redistribution of
data after the world has been resized needs to be handled. However the
disadvantage of approach A is it would require a rewrite of mpiexec and other
_fault_daemons_. It also changes the way in which jobs are spawned in slurm.
Now we would require to spawn a slurm job(_fault_daemon_) from within the
slurm job(_head_daemon_). This might increase the complexity. Another
disadvantage is that this approach is limited to ExaMPI and cannot be extended
to other implementations of MPI.

Approach B is cleaner and better than the Approach A since there is no need
for any change in the underlying MPI implementation. However we completely
rely on the APIs specified in MPI standard. However since ExaMPI
MPI_Comm_spawn and MPI_Intercomm_merge are not supported in ExaMPI. And hence
this approach cannot be used by application using ExaMPI.

Approach C introduces totally a new API and moreover a model to perform the
computation. It brings MPI closer to the traditional fork and spawn model of
programming and distributing the working amongst the child processes.

With minimal change that do not alter the user code structure or logic, we
showed that elastic MPI execution can be achieved. We showed that
transformation of a tightly coupled parallel application into a malleable one
is possible. This transformation could result into a better and resource
efficient application.

##  VI Future Work

Until now we discussed how to increase the MPI processes given the allocated
resources increases. However there is also a scenario of reducing the
allocated resources. In this case we need to reduce the number of MPI
processes or migrate the existing MPI from the node which is supposed to freed
up. Migration can be orchestrated using DMTCP and carefully selecting the
process to migrate and restore it on the new node.

Another approach can be to have a event-sourcing model, where each MPI message
is considered an event. Each event has sufficient information to identify the
sender and receiver. Given that the MPI application goes through a process
reduction phase, the event for the corresponding rank could be redirected to a
proxy rank. A proxy rank is a live MPI process which gets elected by to send
and receive events of the reduced process.

##  VII Acknowledgements

We thank Dr. Gene Cooperman [18] for their guidance throughout the course of
this project and for their comments on the work and presentation. We wish to
thank Dr. Anthony Skjellum [19] and the ExaMPI team for allowing us the access
to ExaMPI source code and helping us understand the internals and architecture
of ExaMPI.

## References

  * [1] Argonne National Labs, https://www.mcs.anl.gov/research/projects/mpi/
  * [2] MPI_Comm_spawn, https://www.mpich.org/static/docs/v3.1.x/www3/MPI_Comm_spawn.html
  * [3] MPI_Comm_spawn_multiple, https://www.mpich.org/static/docs/v3.1.x/www3/MPI_Comm_spawn_multiple.html
  * [4] Raveendran, A., Bicer, T. and Agrawal, G., 2011, May. A framework for elastic execution of existing mpi programs. In 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (pp. 940-947). IEEE.
  * [5] Comprés, I., Mo-Hellenbrand, A., Gerndt, M. and Bungartz, H.J., 2016, September. Infrastructure and api extensions for elastic execution of mpi applications. In Proceedings of the 23rd European MPI Users’ Group Meeting (pp. 82-97).
  * [6] MPICH, https://www.mpich.org/about/overview/
  * [7] SLURM, https://slurm.schedmd.com/overview.html
  * [8] Checkpointing, https://en.wikipedia.org/wiki/Application_checkpointing
  * [9] Ansel, J., Arya, K. and Cooperman, G., 2009, May. DMTCP: Transparent checkpointing for cluster computations and the desktop. In 2009 IEEE International Symposium on Parallel & Distributed Processing (pp. 1-12). IEEE.
  * [10] DMTCP Plugins, http://dmtcp.sourceforge.net/plugins.html
  * [11] Skjellum, A., Rüfenacht, M., Sultana, N., Schafer, D., Laguna, I. and Mohror, K., 2019, September. ExaMPI: A Modern Design and Implementation to Accelerate Message Passing Interface Innovation. In Latin American High Performance Computing Conference (pp. 153-169). Springer, Cham.
  * [12] Barrier, https://en.wikipedia.org/wiki/Barrier_(computer_science)
  * [13] MPI_Init, https://www.mpich.org/static/docs/v3.1.x/www3/MPI_Init.html
  * [14] Rieker, M., Ansel, J. and Cooperman, G., 2006, June. Transparent User-Level Checkpointing for the Native Posix Thread Library for Linux. In PDPTA (Vol. 6, pp. 492-498).
  * [15] MPI Intra-communicators, https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node94.htm
  * [16] Mo-Hellenbrand, A., Comprés, I., Meister, O., Bungartz, H.J., Gerndt, M. and Bader, M., 2017, May. A large-scale malleable tsunami simulation realized on an elastic MPI infrastructure. In Proceedings of the Computing Frontiers Conference (pp. 271-274).
  * [17] MPI Intercommunicators, https://www.mpi-forum.org/docs/mpi-2.0/mpi-20-html/node142.htm
  * [18] Dr Gene Cooperman, http://www.ccs.neu.edu/home/gene/
  * [19] Dr. Anthony Skjellum, https://www.utc.edu/college-engineering-computer-science/profiles/computer-science-engineering/faculty/zyz852.php
  * [20] Feitelson, D.G., Rudolph, L., Schwiegelshohn, U., Sevcik, K.C. and Wong, P., 1997, April. Theory and practice in parallel job scheduling. In Workshop on Job Scheduling Strategies for Parallel Processing (pp. 1-34). Springer, Berlin, Heidelberg.


# Towards a Heterogeneous Smart Electromagnetic Environment for Millimeter-
Wave Communications: An Industrial Viewpoint

Roberto Flamini  Danilo De Donno  Jonathan Gambini  Francesco Giuppi
Christian Mazzucco  Angelo Milani  and Laura Resteghini Manuscript received
October xx, 2021.The authors are with the Milan Research Center of Huawei
Technologies Italia S.r.l, Milan, Italy, (e-mail: name.surname@huawei.com).

###### Abstract

Fifth generation (5G) and beyond communication systems open the door to
millimeter Wave (mmWave) frequency bands to leverage the extremely large
operating bandwidths and deliver unprecedented network capacity. These
frequency bands are affected by high propagation losses that severely limit
the achievable coverage. The simplest way to address this problem would be to
increase the number of installed mmWave base stations (BSs), at the same time
augmenting the overall network cost, power consumption and ElectroMagnetic
Field (EMF) levels. As alternative direction, here we propose to complement
the macro-layer of mmWave BSs with a heterogeneous deployment of Smart
Electromagnetic (Smart EM) Entities – namely IAB nodes, Smart Repeaters,
Reconfigurable Intelligent Surfaces (RISs) and passive surfaces – that is
judiciously planned to minimize the total installation costs while at the same
time optimizing the network spectral efficiency. Initial network planning
results underline the effectiveness of the proposed approach. The available
technologies and the key research directions for achieving this view are
thoroughly discussed by accounting for issues ranging from system-level design
to the development of new materials.

###### Index Terms:

Millimeter wave systems, Reconfigurable Intelligent Surface, Smart Repeater,
Network-Controlled Repeater, Smart Skin, Reflecting Surface.

##  I Introduction

The 5G New Radio (NR) is the first mobile technology generation to make use of
the underutilized mmWave spectrum between 24.25 GHz and 52.6 GHz, which has
been defined by the 3GPP standard as Frequency Range 2 (FR2) [1, 2]. At the
same time, the large amount of bandwidth availability in the unlicensed mmWave
band at 60 GHz has led to the formulation of next generation WiFi standards
like IEEE 802.11ad and 802.11ay, also known as Wireless Gigabit (WiGig) [3].
The mmWave spectrum is extremely attractive for two main reasons: (i) it
allows to accommodate large antenna arrays in very compact systems, thus
obtaining high antenna gain and beamforming capability; and (ii) it provides
large channel bandwidths. These points make mmWave suitable to accommodate a
rapidly growing number of intelligent devices and services and to address
highly demanding applications, where very low latencies and very high data
rates are required [4, 5].

As a critical aspect, communications in the mmWave frequency range are
essentially coverage-limited if compared with sub-6GHz and this limitation
represents the most important issue that mobile operators, ICT manufacturers
and standardization bodies are currently facing in order to be able to
successfully plan and deploy 5G networks operating at FR2. Coverage limitation
at mmWave is much more severe than at sub-6GHz because of two main reasons:
(i) the higher free space path loss (FSPL) and (ii) the higher sensitivity to
obstacles in the propagation environment. In terms of FSPL, it is sufficient
to consider that a radio link at 28GHz suffers from a loss of around 29dB
higher than that at 1GHz and this dramatically affects the link budget.
Regarding the role of the environment, mmWave signals are subject to higher
attenuation in presence of blockage from obstacles of different kinds,
including the human body, foliage and building walls. Moreover, due to the
small wavelength, the roughness and imperfections of the surfaces tend to
create a scattering effect rather than a specular theoretical reflection,
leading to a weaker signal strength [6].

To mitigate these limitations, mmWave applications typically resort to high
gain and narrow beamforming antenna arrays combined with high transmit power,
in other words high EIRP systems. While this solution is sufficient for
adequately recovering the propagation losses in line-of-sight (LoS) or near-
LoS scenarios, it fails in delivering acceptable performance in urban areas
characterized by deep non-LoS (NLoS) conditions. For this cases, a natural
solution would be to resort to network densification. A denser deployment of
5G mmWave base stations (gNBs according to the 5G NR terminology) would be
preferred from the performance point of view since it would guarantee the
desired minimum signal strength in the served area, but it may not be always a
feasible or economically viable solution, e.g., due to the lack of
wired/wireless backhaul, the higher costs for the acquisition of new sites,
rental fees, maintenance and power supply. As a matter of fact, alternative
solutions to the approach ”more information and data through more power and
higher costs” are mandatory. Aligned to this view, standardization bodies,
mobile operators and ICT industries are now evaluating a number of trade-off
solutions in terms of network performance and complexity, total cost of
ownership (TCO) and electromagnetic field (EMF) levels. Such solutions
include, among the others, the deployment of new types of network nodes, much
cheaper and easy to be deployed than traditional Macro and Micro BSs, that
allow to achieve an homogeneous coverage in the radio access network without
necessarily resorting to the installation of additional macro-sites.

Contribution. In this paper, we focus on the concept of a Heterogeneous Smart
Electromagnetic (EM) Environment as a new paradigm where novel types of
network nodes – specifically, passive static surfaces, Reconfigurable
Surfaces, Smart Repeaters, and Integrated Access and Backahul (IAB) nodes –
coexist in the same network and are optimized in terms of deployment and
functionality to maximize network performance (e.g., coverage and capacity)
while reducing costs and end-to-end power consumption. In Fig. 1, we sketch
our vision of Smart EM Environment. In this framework, we propose a system-
level study to show the advantages of the outlined solution with respect to
more conservative network planning approaches. Moreover, we clearly identify
the main technological directions as well as the most promising design
approaches that can be successfully taken into account for an effective
development, manufacturing and deployment of the new Smart EM Entities. To
this end, we provide an inspiring review of the most interesting tunable
materials suitable for reconfigurable antennas and passive/active surfaces.

Figure 1: Smart EM Environment: an environment leveraging the electromagnetic
properties of new network nodes, whose deployment and operation is optimized
to maximize network performance in terms of coverage and capacity while
reducing costs and E2E power consumption.

The paper is organized as follows. In §II, a detailed description of the
network nodes forming a Smart EM Environment is provided. In §III, an overview
of the potential application scenarios for the Smart EM Environment at mmWave
frequencies is given, along with some updates on industrial progress and
standardization activities related to this topic. The concept of heterogeneous
deployment is analyzed in §IV where preliminary system-level evaluations are
presented to show the advantages of the proposed solution. Finally, the main
system-level and hardware-level challenges, as well as the most promising
design approaches to manufacture and deploy Smart EM nodes are outlined in §V
and §VI, respectively.

##  II Smart EM Environment: the actors

The use of higher frequencies, especially – but not only – in the mmWave
spectrum, is extremely attractive in terms of available bandwidth, but must
cope with higher propagation losses with respect to the sub-6GHz bands. For
this reason, new types of network nodes have been considered in order to
increase deployment flexibility, that at the end translates into new paradigms
for mobile operators to design and operate their networks. In this section, we
take Fig. 2 as a reference to introduce the four types of network nodes
envisioned for the Smart EM Environment in order of decreasing complexity,
cost, power consumption, and architectural impact. The main characteristics
are also summarized in Table I.

Figure 2: The actors forming a Smart EM Environment presented, from left to
right, in increasing order of complexity, architectural impact, cost, power
consumption and, from right to left, in terms of timeline for 3GPP
standardization.

###  II-A IAB node

As per 3GPP specifications, an IAB Radio Access Network (RAN) generally
consists of a network of micro base stations (IAB nodes) interconnected
through in-band wireless backhaul to a Macro BS (or Donor gNB) that, instead,
has direct, wired access to the core network. Consequently, IAB brings about
affordable dense networks without the need to densify backhaul wiring [7].
Moreover, IAB is particularly effective when applied to mmWave RAN: thanks to
the vast wireless capacity and highly directional transmissions, it is
possible to obtain dense access networks by installing only a few pricey Donor
gNBs.

The peculiar aspect of IAB is the self-backhauling approach, where both radio
access and wireless backhaul links share the same radio resources and
interfaces. Therefore, a proper management of the radio resource allocation is
fundamental to operate this network and it is carried out by the Donor gNB.
The key benefits of IAB are essentially three:

  1. 1.

it provides fast mechanisms to handle user mobility and handover without
directly involving the core network;

  2. 2.

it guarantees coverage extension for incremental network deployment;

  3. 3.

it enhances overall cell capacity and cell-edge user experience.

The Donor gNB consists of a central unit (CU) and one or more distributed
units (DUs). A CU is a logical node hosting radio resource control (RRC),
service data adaptation protocol (SDAP) and packet data convergence protocol
(PDCP) and controls the operation of one or more DUs. A DU is a logical node
hosting radio link control (RLC), medium access control (MAC) and physical
(PHY) layers. IAB nodes exploit the CU/DU functional split by hosting a DU and
a mobile termination (MT) function. The IAB-DU interfaces to the Donor gNB CU
via the F1 interface (with relevant extensions). The MT replicates the UE
functionality to establish connections to a gNB and the core network. In this
manner, IAB-MTs can connect to DUs on other IAB nodes or on the Donor gNB thus
realizing multi-hop backhaul. The CU on the Donor gNB becomes the central
control function for all IAB-DUs and IAB-MTs defining an interconnected IAB
topology.

Based on the above discussion, it is clear that IAB nodes are Layer-2 (L2)
regenerative relays, as every packet received (either from the Donor gNB or
from the UE or from another IAB Node) has to be properly decoded and re-
encoded before re-transmission. IAB has been standardized in 3GPP Rel-16 [8]
assuming half-duplex operation between access and backhaul transmission and
reception. Nevertheless, the plan is to evolve towards full-duplex operation
in 3GPP Rel-17 with significant expected advantages in terms of network sum
capacity.

From the hardware point of view, the IAB-MT could either use a separate
antenna or share the access antenna used by the IAB-DU (virtual IAB-MT). The
latter provides the ultimate level of integration and allows to use the high-
performance base station antennas for backhaul over longer distances. On the
access part, IAB nodes are a sort of access points with compact form factor,
approximately with 8 Kg weight, 50-60dBm Equivalent Isotropically Radiated
Power (EIRP), and 300-350W power consumption.

###  II-B Smart Repeater

The Smart Repeater will be introduced in 3GPP Rel-18 under the name of
Network-Controlled Repeater [9] as an evolution of the more classical RF
Repeater, which has been used in 2G, 3G and 4G deployments to supplement the
coverage provided by regular macro cells. Classical RF Repeaters are low-cost,
easy to deploy, and do not increase latency. On the other hand, as they
amplify both signal and noise, they can contribute to an increase of
interference in the system. RF repeaters are also non-regenerative type of
relay nodes, as they do not decode and re-encode the signal as done by IAB
nodes, but simply amplify-and-forward the received signal. Moreover, they are
full-duplex nodes, not able to differentiate between uplink and downlink, and
usually not designed to provide beamforming gain.

The concept of Smart Repeater has been proposed within the 3GPP as a new type
of network node that is able to make use of some side control information to
enable a more intelligent amplify-and-forward operation in a system with time-
division duplex (TDD) access and beamforming operation [10]. The Smart
Repeater is still non-regenerative and requires a low-capacity control channel
from the Donor gNB. From the hardware perspective a Smart Repeater can be
thought as consisting of two beamforming antennas (e.g., phased antenna
arrays), one oriented towards the serving Donor gNB while the other oriented
towards the service area to be covered. The two antennas allows to achieve the
highest flexibility in terms of installation and therefore in terms of
size/shape/location of the target area. A Smart Repeater with higher level of
integration (e.g., with the two antennas arranged back-to-back in a single
box) can be envisioned to further reduce costs, dimensions, and to have a
limited impact on urban architecture. Generally speaking, we imagine a Smart
Repeater with small form factor, light weight (roughly 4Kg), with less than
60dBm EIRP, and low power consumption ($\sim$20W).

###  II-C Reconfigurable Intelligent Surface (RIS)

Reconfigurable Intelligent Surface (RIS) is a new type of system node with
reconfigurable surface technology, where its response can be adapted to the
status of the propagation environment through control signaling, hence turning
the wireless environment from a passive to an intelligent actor as the channel
becomes somehow programmable [11, 12, 13, 14]. An Industry Specification Group
(ISG) has been recently created within the ETSI (European Telecommunications
Standards Institute) organization with the aim to study and specify use cases,
deployment scenarios, evaluation methods, system architecture and requirements
of RIS as well as its impact on the current specifications published by
Standards Development Organizations (SDOs) like the ETSI or 3GPP. A RIS can be
defined as a semi-passive or nearly-passive device since it does not include
any amplification stage and requires relatively low power to be configured and
controlled. For these reasons, RISs (i) do not amplify nor introduce noise
when reflecting/transmitting/reshaping the signals, (ii) do not increase the
latency providing an inherently full-duplex transmission, and (iii) do not
increase the EMF levels. From the hardware point of view, an RIS can be seen
as a planar surface composed by an arrangement of a massive amount of
controllable unit cells [15] – whose size is in the order of $\lambda/2$
(being $\lambda$ the wavelength) in case of reflectarray-like design or
smaller fractions of $\lambda$ for meta-material based approaches – that are
able to capture and re-radiate energy in a controllable manner [13, 14, 16].

RISs are designed to be installed on walls, building facades, ceilings or even
hung on lamp posts, poles or towers, generally in Line of Sight (LoS) with the
serving Donor gNB (or IAB node or Smart Repeater or another RIS). The size
(area) of a RIS must be chosen as the best compromise between coverage
improvement, cost, power consumption and control complexity, which in turn
depends on how fast the RIS configuration needs to be updated. Tipically,
surface sizes of 25x25cm2 or 50x50cm2 are suitable for mmWave applications,
while overall $<$2W power consumption can be considered as a target to make
RIS a valid, low-cost alternative to Smart Repeaters.

TABLE I: Overview of the main characteristics and parameters of Smart EM devices Network node |  Brief description |  Main parameters |  Configuration
---|---|---|---
IAB Node |  L2 regenerative relay which can decode and forward RF signals |  Relatively small form factor, around 8Kg weight, 60dBm EIRP, $\leq$350W power consumption |  5G control plane, real-time beamforming at TTI level
Smart Repeater |  Non-regenerative relay which can amplify and forward RF signals |  Small form factor, lightweight (around 4Kg), $<$60dBm EIRP, $\sim$20W power consumption |  Two possible modes of operation: (i) semi-dynamic with out-of-band control plane and small number of beams updated few times per day; (ii) dynamic with in-band control plane and real-time beam update at TTI level.
RIS |  Intelligent radio surface whose response can be reconfigured using control signalling |  Tipically 25x25cm2 or 50x50cm2 surface dimension for mmWave applications, $<$2W power consumption. |
Smart Skin |  Surface that can be appropriately engineered/designed to go beyond Snell’s law, providing, for instance, non-specular reflection and beam-shaping |  $>$25x25cm2 surface dimension, fully passive. |  Fixed configuration
Massive MIMO BS |  Antenna-integrated radio, typically comprising many transceiver chains |  Cell tower site, around 20Kg weight, up to 70dBm EIRP, $\leq$800W power consumption |  Real-time beamforming at TTI level

###  II-D Smart Skin

The fourth new type of network node is the Intelligent Surface (or Smart
Surface, or Smart Skin). From now on along the paper we simply refer to it as
Smart Skin. As shown in Fig. 2, a Smart Skin is the simplest node in a Smart
EM Environment, in the sense that it is a fully passive, non-reconfigurable
surface able to reflect (or focus, or absorb, or even transmit) an impinging
wave with a non-specular reflection angle. In addition to that, being non-
reconfigurable, a Smart Skin does not require any control signaling and
therefore it has no impact on the network operation, but only on network
planning. From the hardware perspective, a Smart Skin can be thought as a
surface, either planar or conformal, designed to be installed on walls,
building facades, ceilings or even hung on lamp posts, poles or towers, in
Line of Sight (LoS) with the serving Donor gNB (or IAB node or Smart Repeater
or RIS). Being fully-passive and non-reconfigurable, the Smart Skin does need
neither reconfigurable devices, nor power supply and therefore it can be
fabricated with extremely low manufacturing costs. This characteristic allows
the Smart Skin to be much larger in size than RISs or, equivalently, Smart
Skins can be deployed more massively compared to RISs.

##  III Application scenarios, industrial progress and standardization

This section is aimed at providing an industrial view on the potential
applications and scenarios for the Smart EM Environment at mmWave frequencies
along with a summary of industrial progress and standardization activities. A
pictorial overview of Smart EM application scenarios is given in Fig. 3.

###  III-A Fixed Wireless Access (FWA)

FWA brings high-speed broadband access to domestic and business sites. Users
connect to the network by a Customer Premises Equipment (CPE) that in turn is
served by a remote mmWave BS (placed at a distance of up to 2-3 km), typically
over a LoS connection. In order to increase market penetration, FWA should
guarantee the minimum Quality of Service (QoS) also in case of quasi-LOS (or
even Non-LOS) conditions. IAB nodes and Smart Repeaters with high end-to-end
gain (typically set in the order of 90-100dB) appear as the best candidates to
guarantee relevant spectral efficiencies in long distance regime.

###  III-B Enhanced Mobile Broadband (eMBB)

eMBB is one of the three primary 5G New Radio (NR) use cases defined by the
3GPP. eMBB is supposed to provide broadband access to radio terminals in
densely populated areas, both indoors and outdoors (like city centers, office
buildings or public venues such as stadiums or conference centers), as well as
mobile services to moving vehicles (including cars, buses, trains and planes).
Typical urban and dense urban cell radii range from 200m to 500 m, and the
target areas to be served are expected to include a high number of subscribers
with frequent coverage problems and shadowing issues. In this framework, a
heterogeneous deployment encompassing all the Smart EM actors (IAB Nodes,
Smart Repeaters, RISs and Smart Skins) can be judiciously planned in order to
strike an optimal balance among cost, capacity and coverage issues.

###  III-C Verticals

Differently from the previous generations of mobile systems, 5G (and beyond)
technologies also aim at defining communication scenarios that can
successfully lead to a global industrial transformation in many vertical
markets such as transportation, media, healthcare, manufacturing, automotive,
energy, public safety, smart cities and much more. One of the main challenges
in this extremely diversified landscape lies in guaranteeing a homogenous
coverage in industrial sites characterized by the presence of obstacles and
numerous metal objects. In this context, all the Smart EM actors are potential
candidates for enabling the use cases of Flexible Production Lines, Connected
Workers and Automated Guided Vehicles for Industry 4.0. Moreover, the
different network nodes can be deployed within the industrial sites with
reduced operating costs due to absence of rental fees. This can be an enabling
factor for the newest Smart EM solutions (Smart Repeater, RIS and Smart Skin)
that inherently trade lower cost of apparatuses and installations for a more
pervasive deployment with respect to, e.g., Macro BSs or IAB systems. Finally,
passive solutions such as Smart Skins can be installed with a higher degree of
flexibility than in public urban scenarios where architectural constraints and
site acquisition campaigns set stricter constraints.

Figure 3: Heterogeneous deployment for the Smart EM Environment.

###  III-D Industrial progress and standardization

The most recent 3GPP studies and contributions are now all focused on Release
18 (Rel-18) that will mark the mid-generation of 5G with commercialization
expected around 2025 [17]: the new term coined to indicate this new set of
specifications is ”5G-Advanced”. Smart EM Environment, among the others, has
been identified as a potential solution for enabling 5G-Advanced. In [9], an
evolution of the simple RF repeater, namely Smart Repeater or Network-
Controlled Repeater, is presented posing much emphasis on its integration with
IAB: the two technologies can be deployed independently, but also combined
together to complement each other. IAB nodes add cells to the RAN footprint,
while Smart Repeaters enhance the cell footprint of an IAB-DU or gNB-DU [10].
An important aspect is also that while IAB nodes are managed by the Donor gNB
CU, the operation of Smart Repeaters can be managed by the IAB-DU or Donor gNB
DU: this separation of control provides easier implementation and allows
independent evolution of IAB and Smart Repeater. Moreover, keeping the Smart
Repeater under local IAB-DU/gNB-DU control provides better scalability. Once
the Smart Repeater will be specified in the standard, the related control
mechanism may be the stepping stone towards RIS.

Multi-beam aware, adaptive repeaters for Rel-18 are proposed in [18] where it
is shown how repeater performance can be improved by introducing control
information such as slot transmission status, TDD slot configuration, TX/RX
beam information and adaptive repeater bandwidth.

In [19], Smart Repeaters and RISs with adaptive beamforming control
capabilities towards the UE in FR2 are proposed. Both solutions can serve not
only as countermeasures against dead spots or outdoor-to-indoor attenuation,
but also as boosters of MIMO layers.

Finally, in [20, 21], the introduction of RISs in 5G-Advanced is promoted by
stressing the need to design a proper channel and device model, identify
application scenarios, and analyze potential impacts on beam management,
control interface with gNB, random access, and interference coordination.

##  IV A promising solution: heterogeneous deployment

The issue of blockage due to opaque obstacles interrupting the line of sight
is the most detrimental effect of mmWave propagation. A denser deployment of
gNBs could provide the reliability level needed by 5G requirements, as failed
radio links can be easily recovered by connecting to alternative gNBs. While
such density increase might generate high deployment costs, the IAB paradigm
outlined in the previous sections promises to limit the increased complexity
and expenses, and it is thus considered as one among the most promising
enablers of mmWave deployment.

(a) Coverage for gNB-only deployment

(b) Coverage for gNB+RISs deployment

Figure 4: An example of heterogeneous deployment in a small area of Hong Kong
city. Ray-tracing prediction shows significant coverage improvement when RISs
are installed at strategic positions to fill 5G service holes.

However, as already discussed, IAB nodes are full-fledged devices (i.e., they
consists of DU and MT entities) compared to the other types of relay nodes for
coverage augmentation presented in §II, namely amplify-and-forward relays
(Smart Repeaters), passive static reflectors (Smart Skins), and reconfigurable
intelligent surfaces (RISs). In our view, these nodes all together are
essential for the design of a Smart EM Environment where the electromagnetic
field can be shaped through network control. More specifically, the
fundamental question we pose is: ”Do we always need a full-fledged IAB node to
relay signals, boost capacity, and enhance 5G mmWave coverage?”. For example,
in a typical urban scenario, different parts of the city may require different
types of Smart EM devices, depending on the performance objective of the
target use case (see §II). We refer to this concept as heterogeneous
deployment.

The key idea behind the heterogeneous deployment is that while IAB nodes
represent the most powerful tool to rapidly add 5G mmWave cells, they can be
replaced in many cases by much cheaper Smart Repeaters, RISs, or Smart Skins
without substantially affecting performance. To better investigate this
aspect, we simulated via ray-tracing prediction a small area (roughly
100x100m2) of Hong Kong city where we employed a 28GHz 5G gNB with 65dBm EIRP
and 33dBi antenna gain installed at 10m height in a road intersection (see
Fig. 4(a)). Several areas with very low received power can be observed,
especially along the cross streets of the main road. Overall, the achieved 5G
mmWave coverage is around 60% assuming omnidirectional user equipment. IAB
nodes in strategic positions could be installed to extend the coverage;
however, one may wonder whether resorting to full-fledged nodes like IAB is a
convenient choice. To answer this question, we supposed to install three
25x25cm2 RISs at the minor road intersections, in line-of-sight with the gNB,
as shown in Fig. 4(b). A significant improvement in received power can be
observed, which translates, approximately, into 20% coverage increase. The
impact of sequentially adding RIS 1, RIS 2, and RIS 3 as in Fig. 4(b) is
reported in Fig. 5, where the CDFs of UE received powers and achievable
capacities is showed. The beneficial effect of deploying RISs is evident,
especially for cell-edge users: received power is increased by more than 10dB
while capacity achieves almost 2x improvement.

(a) CDF of user received power

(b) CDF of user capacity

Figure 5: CDFs of user received power (a) and capacity (b) for the Hong Kong
simulation scenario: comparison between gNB-only and gNB+RISs deployments.

##  V System challenges

The heterogeneous deployment envisioned for the Smart EM Environment is able
to deliver the expected benefits when proper network planning strategies are
adopted and efficient coordination among the gNBs and the multiple IAB nodes,
RISs and Smart Repeaters is achieved. This poses several challenges under the
perspective of PHY layer and system design, as discussed in the following.

###  V-A Network planning

In the heterogeneous deployment envisioned in §IV, it is important to explore
network planning strategies that take into account performance, cost, and
energy-efficiency of each Smart EM node in order to improve the 5G mmWave
coverage in realistic scenarios. In particular, robustness against obstacle
blockages can be achieved by providing multiple connection alternatives to
users, via relaying or reflections, in order to always make available an entry
point for the mmWave access when random obstacles block some links. Smart EM
nodes are capable of focusing an impinging radio wave towards arbitrary
directions and this capability can be exploited at a network level to create a
reflected radio path between a transmitter and a receiver. The tuple
comprising a transmitter, a receiver and Smart EM device assisting the
communications is known as Smart Radio Connection (SRC) [22] and extends the
well-known concept of Smart Radio Environment [14]. SRCs can thus naturally
provide dual-connectivity, namely connecting the UE through multiple radio
links, which is effective in reducing outages due to obstacle obstructions in
mmWave access networks. However, a single, big nomadic obstacle could
interrupt both the primary and reflected path if these present low angular
separation, leading to an SRC outage and decreasing the solution’s efficacy. A
large angular separation between primary and reflected links can protect the
SRC against obstructions and mitigate the well-known effect of self-blockage,
caused by the human body of the device holder interrupting the line-of-sight.
Link length represents another factor influencing the network resilience that
can be controlled during the network planning phase. Indeed, longer radio
connections naturally experience higher probabilities of being blocked by one
or more obstacles. Independently on the link recovery procedure employed
during network operation, a reliable mmWave RAN design coincides with a
careful installation of gNB and Smart EM objects during the planning phase.

###  V-B Impact of hardware impairments

An intensively studied approach for implementing RISs consists in resorting to
the concept of metasurface, that is a digitally-tunable two-dimensional
metamaterial composed of a wide number of reflecting elements, also referred
to as meta-atoms, whose size is of the order of sub-wavelength of the
operating frequency [23]. Each meta-atom is equipped with reconfigurable
circuits that can be tuned to modify the phase of the impinging
electromagnetic wave. Therefore, the impact of the RIS onto the end-to-end
baseband channel model can be modeled by a diagonal matrix where the non-zero
entries represent the reflection coefficients. The phases of these
coefficients constitute the tunable part, while the amplitudes are fixed and
depend on the properties of the specific structure [24]. On the other hand,
Smart Repeaters can be seen as virtually omnidirectional amplify-and-forward
nodes that are composed of a number of tunable antennas (e.g., phased arrays)
able to synthesize transmit and receive beam patterns towards specific
directions. Their channel model is typically captured by a full complex matrix
where each entry depends on the choice of the transmit and receive beamforming
vectors and the gain introduced in the amplification process.

The possibility of tuning the phases (and when possible the amplitudes) over a
continuous domain, while desirable under a performance point of view, would
lead to a costly and not scalable manufacturing process. Practical
implementations of RISs and Smart Repeaters privilege the use of a quantized
set of phases and amplitudes to limit the number of controls for each
radiating element. Typically adopted quantization levels are in the order of
1-4 bits, depending on the applications. The beamforming design should in
principle explicitly account for this limitation, especially with
implementations based on 1 or 2 bits, even if this policy may lead to very
complex optimization routines. A more pragmatic approach consists in first
deriving beamforming vectors with continuous phases and amplitudes, and then
quantizing the obtained solutions to comply with the discrete sets.

The presence of phase quantization typically leads to the synthesis of
patterns with lower directivity and higher side lobes with respect to the case
with infinite resolution [25]. To emphasize this concept, Fig. 6 shows the
scan loss achieved when electronically pointing a phased array structure
towards different elevation angles by employing beam codewords appropriately
optimized with the respect to the admissible phases at each antenna element
(the phased array is characterized by 8 directive elements along the vertical
direction deployed with a mutual spacing of 1.5$\lambda$). Despite the
optimized approach, the figure suggests a remarkable directivity loss when a
1-bit quantization is adopted.

Figure 6: Radiation pattern envelope for an antenna array where beam codewords
have been optimized accounting for the available level of phase quantization
(1 bits for all elements, 2 bits for all elements or hybrid 1 or 2 bits for
the different elements).

###  V-C CSI acquisition and beamforming design

The transparent nature of the Smart EM Entities limits the possibility of
estimating the single contributions of the gNB-to-UE relayed channels. To
circumvent this issue, a practical solution relies on letting the RISs and
Smart Repeaters operate with pre-designed codebooks of phase configurations,
and consequently of beam patterns. During initial access, phase codewords
should be able to synthesize beams wide enough to keep the users search time
at an acceptable level. After the connection is established, convenient
procedures based on channel state information (CSI) feedback form the UEs
should be devised in order to increase the beam pattern directivity, combat
poor path loss conditions, and maximize the achievable spectral efficiencies.

The codebook-based approach requires that the beam synthesis at the gNBs and
the RISs or Smart Repeaters is tightly synchronized. Moreover, the gNB needs
to be aware of the beam pattern that maximizes the energy transfer towards
each Smart EM node deployed in the cell area. Since networks should be planned
to guarantee LoS communications between each gNB and all the Smart EM
entities, this issue can be handled by an initial network configuration phase.
We remark that the codebook design for RIS nodes is site-specific, as it
should depend jointly on both the source and the destination angles, and
cannot be defined a priori. This implies that the codebooks to be adopted at
the different RISs should be selected after the gNB-to-RIS relative azimuth
and elevation angles have been accurately estimated. Finally, motivations
related to cost and implementation issues will very likely lead to both RIS’s
and Smart Relays with phase-only tunable elements. This opens room to the
study and development of ad-hoc optimization routines for designing phase-only
codebooks able to generate beam patterns with variable widths and shapes [26,
27, 28].

###  V-D Energy efficiency

Beam switching of gNBs and Smart EM entities can in principle occur over
different time scales, thus entailing diversified capacity versus energy
efficiency trade-offs. In scenarios where high transmission gains are needed
to optimize capacity, the Smart EM entities should tailor the reflection
coefficients to maximize the directivity towards (or from) each intended user.
This requires that the RISs and Smart Repeaters synthesize highly directive
beams that need to be changed very frequently – typically in the order of one
or multiple Transmission Time Intervals (TTIs), according to the scheduling
needs. The vocation of low power consumption inherently characterizing the
RISs (each element consumes typically less than 1 mW) can be pushed forward by
letting them change their configuration at a much slower rate with respect to
the TTI support – e.g. in the order of one switch per units of second or even
hour. In this case, the RIS should be configured to synthesize wider beam
patterns to provide coverage to a target area that possibly encompasses
multiple users. In this scenario, the capacity performance is traded with
energy efficiency by setting the RIS circuits devoted to decode the control
and synchronization signals in idle or even deep sleep modes for a great
portion of the time.

When the configuration switching time becomes large, the RIS appears more as
an asynchronous network entity that can dynamically adjust the phase shifts of
each atomic reflecting element based on the propagation environment learned
through either appropriate control signals sent from the core network on a
slow timescale basis, or even through periodic sensing carried out via the
same passive array (when not reflecting) [29]. In the latter case, the RIS
should be equipped with very low-power receive RF chain(s) to enable the
sensing capability for CSI acquisition, while the RIS controller should be
able to select two different modes of operation: (i) receiving mode for
environment sensing; (ii) reflecting mode for scattering the incident signals
from the gNB.

###  V-E Users scheduling

The scheduling problem in the advocated Smart EM Environment acquires an
additional level of complexity with respect to traditional cellular systems,
due to the need of efficiently managing the multiple passive or active relay
nodes. After the beam management procedures have selected the candidate beam
patterns to reach each user, both directly from the gNB or through relayed
paths accounting for RISs and Smart Repeaters, the scheduler should assign the
available Radio Resource Blocks by accounting for the hierarchical structure
of the network architecture. Reports on the interference status by the UEs is
expected to have a crucial role in this framework, to be managed in a
centralized fashion. The resulting problem is non-convex in nature, and
suitable heuristic methods should be devised that are able to scale well with
the high problem dimensionality. The scheduler should be able to capitalize
scenarios when the signal attenuation of the gNB-to-UE link is comparable to
that of the RIS-relayed link by selecting the transmit beamforming of the gNB
to strike a balance between the UEs and RISs angular directions. In this case,
the phase shifts of the RIS need to be chosen in order to trigger a
constructive sum at the receiver between the reflected signal and the signal
coming directly from the gNB [30].

##  VI Hardware challenges

All the actors envisioned for the Smart EM Environment that have been
presented in §II are requested to realize the promises of enhanced coverage
and capacity while keeping the costs and the complexity of the network as low
as possible. In order to successfully achieve this combination of high-
performance, low-complexity and cost-effectiveness, the development of each
Smart EM node faces several challenges in terms of both hardware design and
technology, as discussed in the following.

###  VI-A IAB node

Being already standardized, the IAB node is probably the entity with the
lowest degree of freedom for alternative implementations. There are currently
two main research directions envisioned to improve the performance of the IAB
node while reducing its impact on the network TCO: (i) two-sector deployment
and (ii) Full-duplex (FD) operation.

The first direction focuses on extending the Field of View (FoV) of the IAB
antenna from a typical three-sectorial value of 120∘ to 180∘. In this way, the
IAB node would be composed of only two antennas (instead of three) to provide
omnidirectional coverage, hence reducing the overall hardware cost. The
realization of a 180∘-scanning antenna system is challenged by system-level
aspects, as the antenna has now to serve a larger FoV with same performance of
the three-sector case, as well as hardware-level issues, since the development
of an antenna with a 180∘ FoV requires innovative EM design techniques like
conformal arrays [31], deflecting lenses[32], metasurfaces [33],
reconfigurable pattern radiating elements [34] and wide-pattern radiating
elements suitable for phased arrays [35].

The second direction (FD operation) aims at improving drastically the node
performance thanks to the fact that uplink and downlink transmissions can
reuse the same radio resources [36]. This allows to enhance capacity, reduce
latency for both access and backhaul links and enable flexible and dynamic
uplink/downlink resource adaptation according to the traffic pattern.
Realizing efficient FD operation is challenging due to self-interference,
clutter echo, and cross-link interference. Key issues for FD IAB nodes are the
spatial isolation between antennas, the beam isolation between transmit and
receive antenna patterns, the digital/analog self-interference mitigation
techniques and suitable mechanisms to identify the optimal TX/RX beam pairing
for user scheduling.

TABLE II: Overview of main technologies for tunability and reconfigurability Technology |  FoM∗/performance |  Cost |  Speed |  Max. pwr/IP3 |  Comment
---|---|---|---|---|---
PIN diode |  Up to 2THz FoM [37], IL 0.6dB30GHz [38] |  $\sim$1$ |  $\sim$10ns |  1-10W Pmax |  Mature technology, few suppliers at microwave and mmWave frequencies
MEMS |  IL 0.5dB10GHz [39] |  $\sim$10$ |  $\sim$10$\mu$s |  10W Pmax |  Few suppliers in volume production
SOI CMOS |  2THz FoM [40], 1dB10GHz [41], 2dB30GHz [41] |  $<$1$ |  $\sim$50ns |  50dBm IP3 |  Suitable for RFIC integration, less as standalone device
PHEMT (GaN) |  0.5THz FoM [42] |  - |  $\sim$100ns |  $\sim$10W |  Suitable for MMIC integration, less as standalone device
Liquid crystals |  120∘/dB [43] |  ”low” |  $\sim$10ms |  $\sim$1W |  Suitable for system integration, less as standalone device
Phase change material |  7THz FoM [44] |  ”low” |  $\sim$1$\mu$s |  $\sim$1-10W |  Data refer to GeTe. V02 still in early research phase
Ferroelectric |  100∘/dB [45] |  $\sim$1$ |  $\sim$10ns |  $\sim$1-10W |  Either standalone device or integrated into systems
*FoM=$1/(2\pi C_{off}R_{on})$

###  VI-B Smart Repeater

Smart Repeaters are amplify-and-forward devices with a required end-to-end
(E2E) gain of approximately 90dB for mmWave applications. This figure
represents the actual bottleneck to Smart Repeater performance. In fact, in
order to avoid detrimental oscillations, the isolation between the two
antennas must be greater than the E2E gain. This has to be addressed providing
adequate isolation margin, e.g., by achieving a proper tradeoff between
antenna directivity, antenna spacing, and amplifier gain. However, being the
EM environment where the Smart Repeater is deployed not fully known and time
varying, an active stability check must be implemented, able to detect
oscillation risks and take proper countermeasures (e.g., gain reduction,
switch off, etc.). The use of echo/feedback cancelers can be envisioned here,
but complexity and cost increase has to be taken into account.

From an implementation point of view, the Smart Repeater is a simple apparatus
made of two beamforming antennas, one RF amplifier and a suitable control unit
that, connected with the Donor gNB, manages beamsteering/beamforming
algorithms, amplification control and uplink/downlink switching (see Fig. 7).
Considering a target EIRP of around 60dBm, the peak power required from the RF
amplifier is in the order of a few Watts and can be readily obtained with
current composite semiconductor technology. Uplink/downlink switches requires
a high level of isolation that, with a smart circuit implementation, is still
well within current technological capabilities. Beamforming antennas do not
require the ultimate RF performance level of those used for Macro BS or IAB
nodes, but rather have to be low cost and simple to produce. Conventional
solutions rely on the use of a plurality of radiating elements that, alone or
grouped into clusters (i.e., sub-arrays), are connected to a phase shifter and
an amplification stage. Clustering in a phased array already goes in the
direction of reducing the number of phase shifters in order to lower system
complexity as well as to simplify heat dissipation.

Figure 7: Block diagram of the Smart Repeater comprising beamforming antennas,
amplifiers, TDD switching and control unit.

Within this context, a significant effort in our research activities is
currently devoted to the development of low-cost and robust techniques to
achieve phase-control capabilities and reliable yet continuous beam-steering
with no need of conventional phase shifters. The first attempts to achieve
antenna reconfiguration relying on PIN diodes instead of phase shifters date
back to 1978, where preliminary studies for a PIN-based tunable dielectric
waveguide were presented [46]. In more recent studies, PIN diodes were used as
switches to control the period of the grating, thus demonstrating beam
steering in a number of discrete directions [47]. At the moment, the interest
of researchers is focused on the problem of reconfigurability applied to
metasurfaces and obtained through the use of a number of different techniques,
from PIN diodes to varactors, liquid crystals or other tunable materials [48].
In a recent research activity in collaboration with the University of Siena
and its spin off Wave-Up Srl, we have developed a solution based on a
reconfigurable metasurface leaky-wave antenna, in which a network of properly
biased PIN diodes is used to control the beam pointing angle in the elevation
plane [49, 50]. In this design, the metasurface has been designed as composed
of a plurality of sub-lambda radiating elements organized into different
channels to create a leaky-wave effect. Each channel contributes to a beam
whose pointing direction can be varied along the elevation plane; this
possibility is ensured by the presence of separately-driven PIN diodes
soldered across the radiating elements: by dynamically varying the modulation
period through the voltage control of each diode, the antenna beam angle is
steered accordingly, with a reconfiguration speed that falls in the order of
tens or hundreds of nanoseconds (compatible with the latest communications
technologies, such as 5G). One of the main advantages of such an approach is
that the leaky-wave antenna can be realized in simple PCB technology, leading
to a low-profile, cheap implementation suitable also for mass production. In
addition to that, compared to traditional clustered phased-array topologies,
this structure permits to extremely simplify the signal routing, as each
channel has a unique feeding point; the considerable number of PIN diodes
permits to achieve an arbitrary beam deflection (not constrained to a set of
discrete angles), but, on the other hand, the use of a great number of active
PIN diodes leads to relevant costs and some uncertainties in the actual yield
of the fabrication process. It appears to be imperative to develop, possibly
jointly with the manufacturers, a well-established and reliable
circuital/electromagnetic model for the active devices (e.g., PIN diodes). In
[51], an example of such an analysis for mm-Wave frequencies is presented. As
a matter of fact, the performance of such components can be very variable
according to the batch, the fabrication process and most importantly the kind
of packaging, which often introduces parasitic capacitance and inductance that
dominate the response. The unpredictability of the behavior of reconfigurable
devices strongly affects the propagation constant of the surface wave along
the metasurface. These considerations apply for RISs too (since are based on
reconfigurable devices as well) and are of particular interest when dealing
with mmWave systems.

###  VI-C RISs and Smart Skins

As described in §II, RISs and Smart Skins are surfaces able to reflect an
incident beam to an angle which is different with respect to the incident one
(non-specular reflection), or to provide additional focusing-defocusing and
beam-shaping effects. The main challenge for such devices is to make them
extremely inexpensive in order to create a real cost-effective solution that
can be massively deployed. As an example, PCB technology, despite the
extremely high optimization of the manufacturing processes achieved along
decades of wide use, is still probably too expensive, considering the wide
surfaces that are expected to be covered. The RIS poses additional challenges
in finding a suitable reconfigurable technology able to cope with the
performance and cost requirements. If we approach the design of a RIS as a
collection of controllable unit cells whose size is $\lambda/2$ (or much less
in case of a metasurface-based RIS), the cost of the controlling elements (PIN
diodes, varactors, switches) must be in the order of a few cents of USD to
keep overall price to an acceptable level. This is probably the most
interesting research direction for the actual deployment of such solution:
research on new, ultra-low-cost manufacturing technologies with the capability
to integrate seamlessly the tuning element, or the development of a new
concept of a virtually continuous (in space) tunable surface. On top of this,
being reconfigurable, a RIS requires a control circuitry and a low capacity
link with Donor gNB to synchronize its operation with the network; it would be
highly desirable that RIS could be disconnected from the public power network
and powered on its own, for example with batteries or by a suitable method of
energy harvesting, being photovoltaic panels the most obvious one. This sets a
limit of a few Watt for the RIS power consumption that is a severe additional
constraint for the tunable technology used. As mentioned in §V-D, switching
speed, e.g., for semi-static and programmable surfaces, can make a significant
difference in such an aspect.

###  VI-D Technologies for tunability and recofigurability

As discussed in the previous sections, Smart Repeaters and RISs need suitable
reconfigurable/tunable technologies with a different price/performance ratio
compared to the ones used in gNB or IAB apparatus. The remaining part of this
section provides a high-level overview of the main technologies that are
currently investigated for tunability and reconfigurability, alongside the
more classical semiconductor-based ones. For the sake of completeness, some of
them are listed in Table II. Generally speaking, while semiconductor-based
technologies (we consider MEMS along them, since their manufacturing process
stems from semiconductor industry) are well developed and can offer a good
level of performance, their price level is still not suitable for RIS targets.
Material-based technologies (LCD, Ferroelectric, PCM, etc.) have a huge
potential from this point of view, thanks to the simpler manufacturing process
(often involving just a single deposition step) or the possibility to be
seamlessly integrated into the RIS substrate, but they often lack from the
performance point of view. A description of the main material-based
technologies is provided below.

####  VI-D1 Ferroelectric ceramic materials

Ferroelectrics are special materials that show a non-null electric moment
generating a spontaneous polarization which can be reversibly switched by an
externally applied electric field. Ferroelectric materials (as Barium
Strontium Titanate, BST) change their dielectric constant as a function of an
external electric field and this makes them attractive for applications in
electrically tunable microwave devices, such as tunable oscillators, phase
shifters and varactors [52, 53, 54]. One critical aspect for practical
applications of ferroelectric materials is the reduction of the dielectric
losses, which tend to be high when high performance is required in microwave
devices. One strategy to reduce dielectric losses of BST ceramic thick or thin
films could be to exploit the addition of oxides such as magnesium oxide (MgO)
[52].

####  VI-D2 Phase change materials

Phase Change Materials (PCMs) are receiving considerable attention as reliable
candidates for next generation non-volatile memory devices, glass fibers, data
storage media, high density optical recording applications, microchips, and
reprogrammable photonic circuits [55, 56, 57]. These materials are
characterized by the capability to change from amorphous to crystalline phase,
in order to display different electrical features. The transformation between
states could be triggered by a thermal excitation generated by an optical or
electrical pulse. Germanium-Antimony-Tellurium (GST) alloys are the most
widespread PCMs due to the fast reversibility of the amorphous-to-crystalline
state transition and the resulting pronounced alternation of different optical
and electrical properties [55, 56]. GST-PCMs are fascinating materials to be
used in electrical and optical sectors, but some issues could arise in real
applications, such as power consumption, amorphous state instability after
repeated cycling, excessive resistivity contrast and materials degradation.
Among PCMs materials, it is worth mentioning also Vanadium Oxides, which show
semiconductor to metal/metal to insulator transitions [58]. Vanadium trioxide
(V2O3), in particular, shows a remarkable change in resistivity, but its
applicability is limited by the low transition temperature. For this reason,
vanadium dioxide (VO2), that has a transition temperature above room
temperature, is more frequently exploited.

####  VI-D3 Nematic Liquid Crystals

Ordered fluid mesophases, commonly called Liquid Crystals, simultaneously
possess liquid-like (fluidity) and solid-like (molecular order) properties
whose unique coexistence gives rise to many interesting properties. Nematic
Liquid Crystals (NLCs) are elongated molecules composed of a number of
aromatic rings and two end groups, at least one of which is polar [58] and
enables their easy reorientation by using a DC or low frequency electric
field. Low losses at high frequencies and low bias voltage have recently made
the use of NLCs interesting in the devices at microwave frequency range.

##  VII Conclusion

A Heterogeneous Smart Electromagnetic Environment for mmWave communications
has been presented, describing in detail the new network nodes that are part
of it and the related progresses in the activities of both the standardization
bodies and the industrial players. Within this framework, a system-level study
has been carried out showing some preliminary results of the heterogeneous
deployment in terms of both coverage and capacity and highlighting the major
system challenges that need to be faced. Lastly, some of the most recent
technological directions both for the hardware development and for the
research on tunable devices have been provided, giving useful insights from
the industrial point of view.

## References

  * [1] 3GPP, “TS 38.300: NR and NG-RAN Overall Description - Release 15,” 2018.
  * [2] ——, “TS 38.101-2: NR; User Equipment (UE) radio transmission and reception; Part 2: Range 2 Standalone - Release 17,” Jan. 2021.
  * [3] IEEE 802.11 working group, “IEEE Draft Standard for Information Technology-Telecommunications and Information Exchange Between Systems Local and Metropolitan Area Networks-Specific Requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications Amendment: Enhanced Throughput for Operation in License-Exempt Bands Above 45 GHz,” 2019.
  * [4] R. Flamini _et al._ , “Millimeter-wave phased arrays for 5g: An industry view on current issues and challenges,” in _2019 IEEE International Symposium on Phased Array System Technology (PAST)_ , 2019.
  * [5] T. S. Rappaport _et al._ , “Millimeter wave mobile communications for 5G cellular: It will work!” _IEEE Access_ , vol. 1, pp. 335–349, May 2013.
  * [6] T. Rappaport, R. Heath, R. Daniels, and J. Murdock, _Millimeter wave wireless communications_. Prentice Hall, 2015.
  * [7] 3GPP, “TR 38.874: NR; Study on integrated access and backhaul - Release 16,” 2019.
  * [8] ——, “TS 38.174: NR; Integrated Access and Backhaul (IAB) radio transmission and reception - Release 16,” 2021.
  * [9] ZTE Corporation, “RP-213700: Study on NR Network-controlled Repeaters,” 2021.
  * [10] Qualcomm, “RWS-210019: NR Smart Repeaters,” 2021.
  * [11] E. Björnson, Ö. Özdogan _et al._ , “Intelligent reflecting surface versus decode-and-forward: How large surfaces are needed to beat relaying?” _IEEE Wireless Communications Letters_ , vol. 9, no. 2, pp. 244–248, 2019.
  * [12] P. Wang, J. Fang, X. Yuan, Z. Chen, and H. Li, “Intelligent reflecting surface-assisted millimeter wave communications: Joint active and passive precoding design,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 14 960–14 973, 2020.
  * [13] M. Di Renzo _et al._ , “Reconfigurable intelligent surfaces vs. relaying: Differences, similarities, and performance comparison,” _IEEE Open Journal of the Communications Society_ , vol. 1, pp. 798–807, 2020.
  * [14] M. D. Renzo, A. Zappone, M. Debbah _et al._ , “Smart Radio Environments Empowered by Reconfigurable Intelligent Surfaces: How it Works, State of Research, and Road Ahead,” _IEEE Journal on Selected Areas in Communications_ , 2020.
  * [15] J. C. B. Garcia, A. Sibille, and M. Kamoun, “Reconfigurable intelligent surfaces: Bridging the gap between scattering and reflection,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 11, pp. 2538–2547, 2020.
  * [16] H. Gacanin and M. Di Renzo, “Wireless 2.0: Toward an intelligent radio environment empowered by reconfigurable meta-surfaces and artificial intelligence,” _IEEE Vehic. Tech. Mag._ , vol. 15, no. 4, pp. 74–82, 2020\.
  * [17] 3GPP, “RWS-210659: Summary of RAN Rel-18 Workshop,” https://www.3gpp.org/ftp/tsg_ran/TSG_RAN/TSGR_AHs/2021_06_RAN_Rel18_WS/Docs/RWS-210659.zip, 2021\.
  * [18] Rakuten Mobile, “RWS-210248: IAB and Smart Repeater Enhancements in Rel-18,” 2021.
  * [19] KDDI corporation, “RWS-210300: NR Repeaters and Reconfigurable Intelligent Surface,” 2021.
  * [20] Sony, “RWS-210306: Introducing Reconfigurable Intelligent Surfaces for 5G-Advanced,” 2021.
  * [21] China Unicom, “RWS-210390: Motivation paper for RIS requirement in 5G-Advanced,” 2021.
  * [22] E. Moro, I. Filippini, A. Capone, and D. De Donno, “Planning mm-wave access networks with reconfigurable intelligent surfaces,” in _2021 IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2021.
  * [23] C. Liaskos _et al._ , “A new wireless communication paradigm through software-controlled metasurfaces,” _IEEE Communications Magazine_ , vol. 56, no. 9, pp. 162–169, 2018.
  * [24] S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” _IEEE Transactions on Communications_ , vol. 68, no. 9, pp. 5849–5863, 2020\.
  * [25] J. Palacios, D. De Donno, and J. Widmer, “Lightweight and effective sector beam pattern synthesis with uniform linear antenna arrays,” _IEEE Antennas and Wireless Propagation Letters_ , vol. 16, pp. 605–608, 2017.
  * [26] B. Pompeo, L. Pralon, M. Pralon, and R. Mendes, “Phase-only pattern synthesis using a modified least squares method for phased arrays,” in _2013 European Radar Conference_ , 2013, pp. 443–446.
  * [27] M. Khalaj-Amirhosseini, “Phase-only power pattern synthesis of linear arrays using autocorrelation matching method,” _IEEE Antennas and Wireless Propagation Letters_ , vol. 18, no. 7, pp. 1487–1491, 2019.
  * [28] C.-N. Hu, C.-S. Chuang, and C.-I. Hung, “Phase-only pattern synthesis for the design of an active linear phased array,” in _Proceedings 2000 IEEE International Conference on Phased Array Systems and Technology_ , 2000, pp. 275–278.
  * [29] L. Subrt and P. Pechac, “Intelligent walls as autonomous parts of smart indoor environments,” _IET Communications_ , vol. 6, pp. 1004–1010, May 2012.
  * [30] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 11, pp. 5394–5409, 2019.
  * [31] N. H. Noordin, N. Haridas, A. O. El-Rayis, A. T. Erdogan, and T. Arslan, “Antenna array with wide angle scanning properties,” in _2012 6th European Conference on Antennas and Propagation (EUCAP)_ , 2012, pp. 1636–1640.
  * [32] E. Gandini _et al._ , “A dielectric dome antenna with reduced profile and wide scanning capability,” _IEEE Transactions on Antennas and Propagation_ , vol. 69, no. 2, pp. 747–759, 2021.
  * [33] G. Oliveri, E. T. Bekele, D. H. Werner, J. P. Turpin, and A. Massa, “Generalized qcto for metamaterial-lens-coated conformal arrays,” _IEEE Transactions on Antennas and Propagation_ , vol. 62, no. 8, pp. 4089–4095, 2014.
  * [34] X. Ding _et al._ , “A wide-angle scanning planar phased array with pattern reconfigurable magnetic current element,” _IEEE Transactions on Antennas and Propagation_ , vol. 65, no. 3, pp. 1434–1439, 2017.
  * [35] G. Yang and J. Li, “Study on wide-angle scanning linear phased array,” in _2017 IEEE International Symposium on Antennas and Propagation USNC/URSI National Radio Science Meeting_ , 2017, pp. 491–492.
  * [36] R. Askar _et al._ , “Interference handling challenges toward full duplex evolution in 5g and beyond cellular networks,” _IEEE Wireless Communications_ , vol. 28, no. 1, pp. 51–59, 2021.
  * [37] _MEST2GFC-010-25 AlGaAs PIN diode_ , Macom.
  * [38] _MASW-010646 bare die_ , Macom.
  * [39] _ADGM1304 SP4T MEMS switch (inlcuding package)_ , Analog Devices.
  * [40] D. Kelly, C. Brindle, C. Kemerling, and M. Stuber, “The state-of-the-art of silicon-on-sapphire cmos rf switches,” in _IEEE Compound Semiconductor Integrated Circuit Symposium, 2005. CSIC ’05._ , 2005.
  * [41] _PE42524 40GHz SPDT switch (flip chip die)_ , Psemi.
  * [42] S. Osmanoglu and E. Ozbay, “X-band high power gan spdt mmic rf switches,” in _2019 European Microwave Conference in Central Europe (EuMCE)_ , 2019, pp. 83–86.
  * [43] R. Jakoby, A. Gaebler, and C. Weickhmann, “Microwave liquid crystal enabling technology for electronically steerable antennas in satcom and 5g millimeter-wave systems,” _Crystals_ , vol. 10, no. 6, 2020.
  * [44] T. Singh and R. R. Mansour, “Chalcogenide phase change material gete based inline rf spst series and shunt switches,” in _2018 IEEE MTT-S International Microwave Workshop Series on Advanced Materials and Processes for RF and THz Applications (IMWS-AMP)_ , 2018, pp. 1–3.
  * [45] G. Velu _et al._ , “A 310/spl deg//3.6-db k-band phaseshifter using paraelectric bst thin films,” _IEEE Microwave and Wireless Components Letters_ , vol. 16, no. 2, pp. 87–89, 2006.
  * [46] T. Itoh and A. Hebert, “Simulation study of electronically scannable antennas and tunable filters integrated in a quasi-planar dielectric waveguide,” in _1978 IEEE-MTT-S International Microwave Symposium Digest_ , 1978, pp. 30–32.
  * [47] L. Huang, J.-C. Chiao, and M. De Lisio, “An electronically switchable leaky wave antenna,” _IEEE Transactions on Antennas and Propagation_ , vol. 48, no. 11, pp. 1769–1772, 2000.
  * [48] C. G. Christodoulou, Y. Tawk, S. A. Lane, and S. R. Erwin, “Reconfigurable antennas for wireless and space applications,” _Proceedings of the IEEE_ , vol. 100, no. 7, pp. 2250–2261, 2012.
  * [49] C. Della Giovampaola, F. Caminita, G. Labate, E. Martini, and S. Maci, “Reconfigurable metasurface antenna for 5g base stations,” in _2020 14th European Conference on Antennas and Propagation (EuCAP)_ , 2020.
  * [50] ——, “Experimental verification of a 5g active metasurface antenna with beam steering capabilities,” in _2021 15th European Conference on Antennas and Propagation (EuCAP)_ , 2021.
  * [51] T. Boles, J. Brogle, D. Hoag, and D. Curcio, “Algaas pin diode multi-octave, mmw switches,” in _2011 IEEE International Conference on Microwaves, Communications, Antennas and Electronic Systems (COMCAS 2011)_ , 2011, pp. 1–5.
  * [52] L. Kong _et al._ , “Electrically tunable dielectric materials and strategies to improve their performances,” _Progress in Materials Science_ , vol. 55, no. 8, pp. 840–893, 2010.
  * [53] A. Ahmed, I. A. Goldthorpe, and A. K. Khandani, “Electrically tunable materials for microwave applications,” _Applied Physics Reviews_ , vol. 2, no. 1, 2015.
  * [54] O. G. Vendik and S. P. Zubko, “Ferroelectric phase transition and maximum dielectric permittivity of displacement type ferroelectrics (baxsr1-xtio3),” _Journal of Applied Physics_ , vol. 88, pp. 5343–5350, 2000.
  * [55] P. Guo, A. M. Sarangan, and I. Agha, “A review of germanium-antimony-telluride phase change materials for non-volatile memories and optical modulators,” _Applied Sciences_ , vol. 9, no. 3, 2019.
  * [56] T. Cao, R. Wang, R. E. Simpson, and G. Li, “Photonic ge-sb-te phase change metamaterials and their applications,” _Progress in Quantum Electronics_ , vol. 74, p. 100299, 2020.
  * [57] B. G. Sangeetha, C. M. Joseph, and K. G. Suresh, “Preparation and characterization of ge1sb2te4 thin films for phase change memory applications,” _Microelectronic Engineering_ , vol. 127, pp. 77–80, 2014\.
  * [58] T. V. Slusar, J.-C. Cho, B. J. Kim, S. J. Yun, and H. tak Kim, “Epitaxial growth of higher transition-temperature vo2 films on aln/si,” _APL Materials_ , vol. 4, 2016.