# Generative AI and Semantic Communication: Evolution and Revolution

Eleonora Grassucci, , Jihong Park, , Sergio Barbarossa, , Seong-Lyun Kim, ,
Jinho Choi, , Danilo Comminiello E. Grassucci, S. Barbarossa, and D.
Comminiello are with the Department of Information Engineering, Electronics
and Telecommunication of Sapienza University of Rome, Italy. J. Park and J.
Choi are with the School of Information Technology, Deakin University,
Geelong, VIC 3220, Australia. S.-L. Kim is with the School of Electrical and
Electronic Engineering, Yonsei University, Seoul, South Korea.

###### Abstract

While deep generative models are showing exciting abilities in computer vision
and natural language processing, their adoption in communication frameworks is
still far underestimated. These methods are demonstrated to excel in solving
classic communication problems such as denoising, restoration, or compression.
Nevertheless, generative models can unveil their real potential in semantic
communication frameworks, in which the receiver is not asked to recover the
sequence of bits used to encode the transmitted (semantic) message, but only
to regenerate a sequence of symbols that is semantically equivalent to the
transmitted message. This change of perspective paves the way to a paradigm
shift with respect to conventional communication systems, which has great
potentials to reduce the amount of data traffic and offers greater versatility
to novel tasks and applications that were not even conceivable a few years
ago. In this paper, we present a unified perspective of deep generative models
in semantic communication and we unveil their transformative role in future
communication frameworks, enabling emerging applications and tasks. Finally,
we analyze the challenges and opportunities to face to develop generative
models specifically tailored for communication systems.

###### Index Terms:

Generative Semantic Communication, Generative Models, Semantic Communication,
Emerging Applications

##  I Introduction

In the last few months, deep generative models such as Stable Diffusion for
image and ChatGPT for conversational text generation forcefully entered our
daily lives, and it has immediately become clear that such models will
revolutionize many different fields of research and productivity. The
generative process behind these models starts by assuming that the initial set
of data obeys a probability distribution that has to be inferred from data.
During training, deep generative models learn to estimate such distribution to
be able to generate new samples from it at inference time. This generation
process can be guided or conditioned to obtain the desired generated output
(so that, for instance, the prompt “a dog running in the backyard” leads to
the generated of an image containing a dog running over a background).

In the past years, generative models have been employed in communication
frameworks providing improved results in solving inverse problems such as
reconstruction or compression [Pan2022ExtremeGI], especially for images. In
this scenario, these models completely change the perspectives concerning
classical machine learning approaches that usually require the original clean
signal to find the problem solution. Indeed, deep generative models can
regenerate content at the receiver in an unsupervised way, producing exciting
results in several tasks without requiring knowledge of the original signal
[Grassucci2023DiffusionMF].

Historically speaking, in his article accompanying Shannon’s Mathematical
Theory of Communication, Weaver identified three levels of communication
[shannon1949]: the technical level, the semantic level, and the effectiveness
level. Shannon deliberately focused on the technical level, concerned with the
encoding of the source message into a sequence of bits and the recovery of
this sequence after passing through a noisy channel, independently of the
meaning conveyed by the message. Very recently, a new paradigm has emerged for
wireless communications, where the attention is progressively moving to the
semantic level, devising the so-called semantic communication. This field is
built upon the idea that what matters is to regenerate, at the receiver side,
an information that is semantically equivalent to the transmitted one without
necessarily coinciding, at the bit (symbolic) level, with the sequence of
transmitted bits [strinati20216g], [barbarossa2023semantic]. This idea offers
many additional degrees of freedom that can be advantageously exploited in the
design of the system and in the resource allocation [Dai2021CommunicationBT].
In particular, this novel paradigm may unveil the big potential of deep
generative models by using them to regenerate content at the receiver side,
conditioned to only a very compact representation of the transmitted
information. The compression can be much higher than in conventional
communication encoding schemes, because what is required is not to recover a
sequence that is identical to the transmitted one, but one that is only
semantically equivalent to the transmitted one or that allows the destination
to fulfill the requirements of the task subsuming the exchange of information
[Grassucci2023GenerativeSC].

As a matter of fact, deep generative models are incredibly flexible to be
adopted with a plethora of different data types, including multimodal,
complex, and high-quality data with which classical machine learning
approaches have limitations. Moreover, they are versatile in solving multiple
tasks that may be directly related to the goal of communication, enabling
task-oriented communications. In addition, thanks to the unsupervised nature
of deep generative models, generative semantic communication frameworks do not
need to be trained end-to-end, enabling more flexible frameworks. Moreover,
the real, and still unveiled, breakthrough that deep generative models are
instead bringing to communication frameworks is opening the path to completely
new tasks and applications that were not even conceivable years ago, such as
remote control, 3D scene regeneration, or multimodal data transmission.

Figure 1: The three levels of Weaver’s model with generative models that lie
in the semantic level enabling new tasks and applications under the semantic
communication paradigm. The generated output will then be evaluated under
semantic metrics.

Together with exciting results, deep generative models such as diffusion or
large language models come with new challenges. To puzzle out their large
demand for computational resources during both training and inference,
captivating solutions have been unfolding and will be further developed in the
near future towards the evolution of generative semantic communication
frameworks.

The contributions of this paper are threefold and can be summarized as
follows.

  * •

We propose a unified perspective of deep generative models in semantic
communications and of their transformative role in the future sixth
generation.

  * •

We unveil the revolution these models bring to semantic communication,
enabling new tasks and unleashing completely novel applications that go beyond
classical communication problems.

  * •

We suggest some solutions to generative models’ challenges on high
computational resources demanding, running towards novel and efficient
generative semantic communication frameworks.

##  II Generative Models for Semantic Communication

###  II-A Communication Scenario

Fig. 1 depicts in simplified form the three levels envisioned by Weaver
[shannon1949]. The top level, the so called effectiveness level, identifies
the task that motivates the exchange of information between source and
destination. This level operates under specific requirements on key
performance indicators (KPI’s) that measure how well the (semantic) message
conveyed by the source allows the destination to fulfill the task in an
effective way. The middle level, the semantic level, includes a semantic
extractor, at the transmitter side, that extracts the features necessary at
the receiver side to regenerate a message that is semantically equivalent to
the transmitted one. At the receiver side, the semantic level uses a deep
generative model to regenerate message, based on a pre-trained generative
model. The scope of the transmitted bits is then only to condition the
generative model used at the receiver side to generate a message that fulfills
the requirements of the task to be executed. The bottom level represents the
technical level, where the relevant information is encoded using conventional
approaches, but that are aware of the higher levels. Hence, just to make an
example, if some of the received packets are detected to contain bit errors,
rather than requiring the retransmission of the erroneous packets, the system
can take advantage of the inpainting capabilities of the generative models to
reconstruct the whole message, with a clear saving in terms of data traffic
and delay. This is just an example of how the semantic and technical levels
can interact between each other to improve the effectiveness of the whole
system. One more illustrative example is in the context of autonomous driving,
where each car may have a webcam, a radar or a Lidar, that send information to
a road side unit (RSU) that collects information from possibly multiple cars
and takes decisions that are then sent back to each car to ensure a safe
driving. Two typical KPIs in this case are decision accuracy and delay,
because large delays may not be admissible to avoid car accidents. In such a
case, each car should limit the transmission rate by extrapolating only the
information that is strictly necessary to enable the road side unit to take
the right decision at the right time. One possibility is to use generative
models at the RSU side that run conditioned to the compressed information sent
by the cars. These simple examples suggest that generative models can play a
key role in the whole system and, of course, the interaction among the three
levels depends on the kind of generative model is being used.

Figure 2: Taxonomy of deep generative models for semantic communication.

###  II-B Generative Model Architectures

TABLE I: Generative models characteristics, with advantages and disadvantages. Model | Training | E2E | Sampling | Compression
---|---|---|---|---
VAEs | Stable | ✓ | Fast | Lossy
Flow-based | Stable | ✗ | Slow | Lossless
GANs | Unstable | ✗ | Fast | No
DDPMs | Stable | ✗ | Slow | Lossy

This section presents the most common methods among the wide range of deep
generative models and the advantages as well as the limitations of their
effective development in communication systems. A taxonomy of generative
semantic communication frameworks with the most common generative models is
shown in Fig. 2, while Table I reports a list of such generative models with
their advantages and disadvantages under different perspectives (training
stability, end-to-end (E2E) learning requirements, sampling speed, and
possibility to compress data).

Variational Autoencoders. Variational Autoencoders (VAEs) are among the first
deep generative models that demonstrated the ability to generate good-quality
multimedia content. They comprise two neural networks: the first one encodes
the mean and variance of the data that are then involved in adjusting a
Gaussian distribution to build a latent representation of the data. This
representation lies in a lower-dimensional space and includes the key features
of the content. During transmission, we can either transmit directly the built
latent representation, or we can transmit the extracted statistics and perform
the reparameterization with the Gaussian distribution at the receiver. The
latter option allows greater flexibility for the receiver in the case of bad
channel conditions in which the transmitted content may be corrupted or part
of the information may be lost. Indeed, during training the latent
representation is forced to be as close as possible to a standard Gaussian
distribution and, at inference time, if a portion of the transmitted
information is lost, it is possible to infer the learned statistics to
preserve the ideal distribution of the latent representation. The second
network, the decoder, can then take in input the latent vector and regenerate
the content according to the transmitted features. Therefore, in VAEs the
encoder acts as a semantic extractor at the transmitter and the decoder as a
generator at the receiver, in a stable end-to-end (E2E) training fashion. VAEs
can provide lossy compression and their sampling is fast, therefore useful in
practical applications.

Flow-based Models. Flow-based models such as Normalizing Flows are suitable
for directly estimating the likelihood of the data. They are based on a
sequence of invertible transformations that starts with a simple distribution
and turns it into a complex one (e.g., the data one). Differently from VAEs,
they usually do not have an encoder for learning semantic representations,
therefore, to employ flow models in semantic communication frameworks, we
should insert a semantic extractor at the transmitter, whose output will
condition the flow process in order to generate content with consistent
semantics. Flow-based models can provide lossless compression and their
training is very stable, but they suffer from slow sampling and sometimes
limited expressiveness.

Generative Adversarial Networks. Generative Adversarial Networks (GANs)
brought a real breakthrough in the generative modeling field, showing the
ability to generate a variety of content ranging from images to audio and 3D
structures. GANs are composed of two networks, a generator that takes in input
a noise vector and outputs the generated content, and a discriminator, whose
aim is to distinguish generated from real content and provide feedback to the
generator to improve the sample quality. At inference time, one can discard
the discriminator and involve only the generator to generate samples. However,
in a communication scenario, the unconditional generation may not be
sufficient and the generation process should be conditioned to the relevant
information sent by the source. To this end, we can build a semantic
communication framework by inserting a semantic extractor in the sender and
transmitting the lower-dimensional semantics to the receiver that will be
involved in guiding the generation process of the GAN generator. The semantic
extractor and the generator do not need end-to-end training, increasing the
flexibility of the framework. Furthermore, GANs have usually higher
expressiveness with respect to VAEs and Flows in generation abilities, and
GANs sampling process is very fast, although their training is highly unstable
and hyperparameter-dependent.

Diffusion Models. Denoising Diffusion Probabilistic Models (DDPMs) have
recently gained the floor among deep generative models due to their exciting
results in dealing with high-quality, multimedia, and multimodal data, beating
all the previous competitors. They are based on a Gaussian Markov chain that
progressively transforms the content into pure noise, training a neural
network to estimate the amount of added noise. The idea behind this approach
is that if the model learns this direction of the transformation from data to
noise, then once trained it will be able to perform the reverse process.
Therefore, at inference time, starting from a sample of pure noise the network
will progressively remove noise up to delineate a newly generated sample. As
for GANs, the generation process is unconditional and, in a communication
scenario, we are required to condition the process with the transmitted
information. Another similarity is that the semantic communication framework
involving DDPMs does not require end-to-end training and we can consider a
generic semantic extractor in the transmitter that sends over the channel the
semantic information that will be involved to guide the DDPMs generation
process. The use of DDPM in semantic communication for image transmission was
proposed in [Grassucci2023GenerativeSC, pezone2023semantic] .  Diffusion
models’ training is highly stable and generated samples are usually of high
quality, although they still suffer from slow sampling.

##  III Semantic-Conditioning Generative Models

Figure 3: Generative process guided from correct vs inaccurate semantic
conditioning. Inaccurate semantic conditioning leads the process far from the
transmitted content distribution, while correct semantic conditioning
conditions the process towards a generation closer to the original content.

As we have seen in Section II-B, the key element for generative semantic
communication frameworks is the semantic conditioning that guides the
generation process. In semantic communication, the semantics is often the only
information that the receiver has about the original content, therefore a
proper and powerful semantic extraction is crucial for content regeneration at
the receiver side. As Figure 3 shows, since the semantics will guide the
direction of the generation process, slightly inaccurate information may lead
to a completely incorrect generation far from the transmitted content.
Conversely, proper semantic information for conditioning will correctly lead
the generation process to the desired content, closer to the transmitted one.

However, extracting the proper semantic representation is not straightforward
since there is not a unique recipe to do it yet, as the semantic
representation may depend on the type of data or on the task the receiver has
to solve. There exist tasks or data for which extracting the semantics is
easier, such as segmenting an image to obtain the semantic map, or learning
the latent features as in VAEs. Conversely, there are tasks or data types for
which getting a semantic representation is tricky, such as multimodal data
like video in which one should ideally obtain a representation for both image
and audio combined.

Recently, several ways of semantically guiding diffusion models have been
proposed, starting from specific-semantic conditioning such as semantic maps
[Grassucci2023GenerativeSC] or textual descriptions
[Nam2023LanguageOrientedCW]. Concurrently, more generic approaches started to
appear in the literature, allowing multimodal conditioning to generate more
and more complex scenes and better align the semantic representations with the
desired output [Liu2021MoreCF, xu2022versatile].

##  IV Comparison Between Generative and Non-Generative Semantic Communication

###  IV-A Semantic Compression Beyond Perceptual Compression

Existing semantic communication transceivers such as deep joint source and
channel coding (DeepJSCC) [gunduz2023jsac] aim to remotely reproduce data that
are perceptually similar. These methods rely commonly on encoder-decoder
structured neural network models such as VAEs. As visualized in Fig. 2, the
semantic representations in a VAE are original data’s statistics $\mu$ and
$\sigma$ in a reduced dimension. In computer vision, obtaining these semantic
representations is referred to as perceptual compression during which the
reproduced outputs maintain perceptual similarities. Under encoder-decoder
architectures, any further compression leads to significant perceptual
dissimilarities, which is referred to as semantic compression.

Unlike DeepJSCC, generative semantic communication has the potential in
enabling semantic compression beyond perceptual compression. For instance,
receivers equipped with diffusion models can generate high-fidelity images by
feeding random noise. In this case, despite a zero transmit rate, distortion
may not always be high. Furthermore, sending additional semantic
representations and applying semantic conditioning can significantly reduce
the distortion, as shown in Fig. 3. These additional semantic representations,
such as text descriptions and semantic segmentation, are often in lower
dimensions than the original data, and therefore, their required rate can be
smaller than the rate for $\mu$ and $\sigma$ in perceptual compression.

###  IV-B Modular Architecture Beyond End-To-End Architecture

Deep learning approaches in communication have been successfully employed for
joint source-channel coding in an end-to-end framework [gunduz2023jsac].
However, they do not fully capture the meaning underlying the sequence of bits
to be transmitted. Furthermore, they typically require an overall
reformulation of the network architecture. Conventional schemes in fact need
to be able to check whether the received packets contain errors or not before
deciding to forward them to the upper layers in the usual OSI stack model.
Conversely, by employing deep generative models, we can still rely on the
classical hierarchical OSI model and handle the semantic aspect at the
application layer. In this way, we still exploit the robustness of current
networks, but we add many more degrees of flexibility because for example we
allow the receiver to drop packets whenever this action does not affect the
recovery of the semantic message. The distinguishing aspect of the new scheme
is that the application layer can interact directly with the source encoder to
decide what information to extract from the message and how to do that.
Differently from any other methods, the key feature of deep generative models
is that they have knowledge of the data distribution and this is the crucial
information that can be used to regenerate content even though the received
information is corrupted or contains missing data because, for example, of
erroneous packet. These models are not required to always receive the whole
information as they can exploit their knowledge of the data distribution to
consistently generate the missing parts. Similarly, even in the case of highly
corrupted received information, deep generative models can denoise it with the
help of their intrinsic knowledge of the data.

A class of standard problems in which deep generative models can unveil such
capabilities is solving inverse problems. The problem of content transmission
over a noisy channel can be modeled as two consequent processes, the forward
and inverse problems in which the first one is represented by the sender
encoding and the deteriorating process to solve is the transmission with
channel corruptions or distortions. Then, the model tries to recover the
original data or the key aspects of it from the received noisy and corrupted
information at the receiver side. Generative approaches take the corrupted
signal in input and directly estimate the clean one without requiring
supervision of the corresponding original clean signal. The key aspect is that
generative models can regenerate the missing portions or the degraded ones to
be consistent with the known parts of the signal thanks to their estimate of
the data distribution [Grassucci2023DiffusionMF]. This enables non-end-to-end
approaches by considering two separate models at receiver and transmitter,
increasing the flexibility of communication frameworks whose receivers can
interact with multiple sources without requiring joint training. Several
traditional inverse communication tasks can be excellently solved with deep
generative models such as denoising, compression, restoration, inpainting,
bandwidth extension, and many others.

Furthermore, in addition to improving perceptual results on the data, deep
generative models can have a broader impact on the whole network.
Communication systems can rely on deep generative models in the case of
traffic congestion or problems on the network, as we may transmit less
information to avoid congestion and then regenerate the missing one with the
generative model avoiding delays and reducing the latency.

##  V Applications in Generative Semantic Communication

###  V-A Semantic Decomposition for Channel-Adaptive Communication

Semantic communication may encounter challenges in transmission channels,
especially when dealing with fading channels in wireless communication setups.
Given the uncertainty of the channel condition before transmission, an
adaptive approach becomes crucial. In this context, joint source-channel
coding [gunduz2023jsac], incorporating superposition coding [Tian08], emerges
as an effective strategy to address unknown fading channel conditions.

Since the channel condition is unknown while no re-transmission is expected
due to low latency requirements, it is important to design the generative
model for semantic communication to be flexible. In particular, the encoder
needs to produce a set of tokens or prompts that allows the decoder to
reconstruct the input signal with variable quality. To this end, suppose that
the latent variables can be decomposed into a set of $L$ prompts, denoted by
${\cal Z}=\\{{\bf z}_{1},\ldots,{\bf z}_{L}\\}$, for a given input signal
${\bf x}$. That is, a generative model-based encoder can produce a
reconstruction of ${\bf x}$ using a subset of prompts. For convenience, let
$D_{l}=D({\bf x},\hat{\bf x}({\cal Z}_{l}))$ represent the distortion when the
decoder reconstructs ${\bf x}$ with a subset of prompts ${\cal Z}_{l}=\\{{\bf
z}_{1},\ldots,{\bf z}_{l}\\}$, which is denoted by $\hat{\bf x}({\cal
Z}_{l})$. Here, we assume that ${\cal Z}_{l}$ is more informative than ${\cal
Z}_{m}$ for $m=1,\ldots,l-1$. Superposition coding can be designed in such a
way that the receiver is able to decode more prompts as the channel condition
improves. In other words, the receiver can decode none of them or at most
${\bf z}_{1}$ when the channel condition is poor, while it can decode all the
prompts when the channel condition is good. Then, it is expected that ${\cal
D}_{l-1}\geq{\cal D}_{l}$, $l=1,\ldots,L$. In Fig. 4, we illustrate
superposition coding for semantic coding with generative models, where the
receiver can generate a cat image of different quality depending on the fading
channel condition.

Figure 4: An illustration of superposition coding for semantic coding with
generative models.

As the channel condition improves, the receiver’s ability to decode more
prompts implies that the quality of the reconstruction of the intended meaning
becomes more robust. The adaptive nature of the system ensures that the
fidelity of the reconstructed meaning depends on the prevailing channel
conditions. This relationship is particularly evident in semantic broadcast
communication, where a user with a good channel condition can have a clear
understanding, while a user with a poor channel condition may only grasp a
basic meaning. For example, if a user can only receive ${\bf z}_{1}$, he or
she can only generate an image of cat, while another user receiving all the
prompts can generate an image that is close the intended image.

###  V-B Multimodal Semantic Diversity for Reliable Communication

Multimodal semantic communication can effectively convey the meaning of a
message using various types of data, including video, speech, sketches, and
more. In [jiang2023large], a single semantic encoder is considered for
converting all types of data into unimodal text data. This process involves
encoding each type of multimodal data with dedicated encoders. The encoded
results are then processed by a conditional encoder based on the target
modality, which, in this case, is text. The output from the conditional
encoder is input into a text diffusion model to generate text data that
preserves semantic consistency with the original multimodal data. However,
this approach may have a few difficulties. For example, due to the use of
conditional encoding, if some of the text prompts are not correctly received,
the receiver may not be able to generate multimodal data with sufficient
accuracy. To avoid this problem, we can consider multimodal diversity. Each
type of data is encoded into unimodal text data, denoted by ${\bf z}_{m}$,
$m=1,\ldots,M$, where $M$ is the number of types of multimodal data. Then,
unlike [jiang2023large], rather than using conditional encoding sequentially,
conditional encoding can be carried out to recover when any $D$ out of $M$
encoded types can be successfully received, while each encoded type is
transmitted over an independent fading channel. It is noteworthy that $D$ can
be less than $M$ as each type of multimodal data can be correlated with the
others. For a large $M$, if $M(1-p_{\rm err})>D$, where $p_{\rm err}$ denotes
the decoding error probability when transmitting ${\bf z}_{m}$ over an
independent fading channel, we can expect that the full multimodal data can be
generated with a high probability thanks to multimodal diversity.

Figure 5: Schematic representation of different possibilities of emerging
applications with respective generative tasks that can be accomplished for
each application.

###  V-C Emerging Generative Applications

While deep generative models can be involved in classic communication
problems, they unleash their real potential when shaping novel applications
for semantic communications. Figure 5 shows some emerging applications
unveiled from generative models and the corresponding generative tasks that
can be accomplished to perform such applications. Besides data generation, the
most popular tasks in generative modeling are signal enhancement, style
transfer, domain adaptation, cross-modality translation, and content editing,
among others. In the following, we see how these tasks can be applied to
communication scenarios to give rise to new and emerging applications.

####  V-C1 Content Creation

One of the novelties that generative models allow us to bring to the world of
communications is to create information content at the receiver based on the
semantics extracted from the information transmitted. This enables the
realization of new goal-oriented communication paradigms. Unlike traditional
communication applications, generative models allow the translation of
transmitted information into a different representation related to some
semantic content. Tasks like signal-to-signal translation or modality
translation can be applied to enable such emerging applications. As an example
of content creation by modality translation, a text message can be translated
into an image rather than into a video or audio signal.

####  V-C2 Multi-User Communications

Multi-user communication may benefit from typical tasks of generative
modeling, such as domain adaptation, modality translation rather than style
transfer. In such scenarios, different information can be transmitted, each
subject to different conditions of the sender but also of the receiver. This
often leads to a loss of quality, inhomogeneity and non-uniformity of
communication. Based on the encoded semantics, it is possible not only to
generate new content, but also to adapt it to the conditions of the end user.
This has a significant impact on the quality of communication and resource
management.

Furthermore, in such a context, generative models can exploit the combination
of semantic spaces related to different users in transmission to provide new
content to the end user based on the combined semantics. As an example, we can
think of an image sent by a first user and a text by a second user. The
semantics related to the text can be used to edit the image sent by the first
user. As a result the end user directly receives the modified information
based on the combined semantics.

####  V-C3 Network Digital Twin

Network digital twin refers to the virtual representation of physical network
systems that can be used to study the behavior of specific network systems.
The semantic information in this case can be related not only to the
information content but also to the network system condition. Thus, if the
network model changes due to some different physical network condition, a
semantic-guided generative model can adapt the generated content to the new
domain of the network representation, without affecting the quality of the
communication.

####  V-C4 Multimodal Generation

Since semantic communication is no longer depends on data as traditional
communications but rather on its semantic representation, we may think of
sending a certain type of signal while receiving it endowed with additional
modalities. A typical generative modeling task in that case is represented by
cross-modality translation. One application example may be the semantic-aware
sonorization, where silent video can be received accompanied by music
generated based on the semantics of the video.

####  V-C5 Personalized Communication

Style transfer, along with the other possible tasks of generative modeling,
can be adopted for one of the most fascinating applications of future
communications that is the personalized communication. The transmitted
information could be personalized based on some semantics representing the
user taste rather than the subjective quality perceived by a single user.
Generative models can adapt the semantic information to the style desired by
the end user.

####  V-C6 LLM-Aided Communication

Large language models (LLMs) are envisaged to be integrated in future
communication systems to leverage their unique capabilities of reasoning and
in-context learning [lin2023pushing, Nam2023LanguageOrientedCW]. Generative
models can play a crucial role in translating various data modalities into
natural language compatible with LLMs. Moreover, they can also transform the
text outputs of LLMs into intended data modalities, maximizing their
effectiveness in specific tasks.

##  VI Challenges and Future Perspectives

Despite the exciting results and the novel tasks and applications that deep
generative models are unveiling, the new challenges such models are bringing
have yet to be completely overcome.

The first challenge to face surely regards computational efficiency and
sustainability. Deep generative models require a huge amount of resources in
terms of computational power, storage memory, and time both for training and
inference, caused by the computations on billions of parameters. Nevertheless,
there exist some pathways to undertake for reducing deep generative models’
impact on computational resources. Promising results are coming from the
extremely low-bit quantization of diffusion and large language models, in
which networks can be defined in 4 or even 2 bit, with consequent acceleration
of both training and inference process up to 10x, as well as a crucial
reduction of required runtime and storage memory [li2023qdm]. Merging low-bit
quantizations and energy management methods may lead to novel models
specifically tailored for communication frameworks that preserve the
performance of their large-scale counterparts while being more efficient.

The second challenge of deep generative models for semantic communication is
their reliability and trustworthiness [Fan2023OnTT]. Indeed, a critical
concern for employing these models in communication scenarios is ensuring
their robustness against attacks from malicious agents. In these cases, the
model may assume an unexpected behavior, and their outputs may no longer be
reliable. Similarly, studying privacy leakages in deep generative models is
emerging as a noteworthy research topic, as generative models may sometimes be
guided to reveal some private information used for the training process.
Recently, promising approaches have been presented to address both these
concerns: training the model to be robust to adversarial attacks, and avoiding
training over the same data may effectively alleviate data leakage at
inference time.

Finally, a third challenge to solve regards the explanations and the
interpretations we can give to models’ generated samples and to their
behavior. Indeed, in communication scenarios, we should be able to explain a
given output and what happens inside the model to produce that sample in order
to guarantee the quality and reliability of the generation. Moreover,
explaining the generation process and demystifying the “black box” belief
around deep networks may increase the confidence of users in these methods and
spread their adoption.

##  VII Conclusions

Deep generative models have suddenly entered our daily lives and are
revolutionizing several fields of application. In this paper, we present the
advantages of employing them to solve classic communication inverse problems,
nevertheless, since deep generative models unveil their real potential under
the new paradigm of semantic communication, we expound on the novel
possibilities, tasks, and applications these models are bringing to semantic
communication frameworks. Finally, we discuss the new challenges such methods
are posing and delineate some possible paths to solve them, especially from
the efficiency point of view.

## Acknowledgment

This work was partially supported by the European Union under the Italian
National Recovery and Resilience Plan (PNRR) of NextGenerationEU, partnership
on “Telecommunications of the Future” (PE00000001 - program RESTART), and in
part by the MSIT (Ministry of Science and ICT), Korea, under the ITRC
(Information Technology Research Center) support program
(IITP-2023-RS-2023-00259991) supervised by the IITP (Institute for Information
& Communications Technology Planning Evaluation).


figurec

# Learning Human Cognitive Appraisal Through Reinforcement Memory Unit

Yaosi Hu and Zhenzhong Chen  School of Remote Sensing and Information
Engineering, Wuhan University

###### Abstract

We propose a novel memory-enhancing mechanism for recurrent neural networks
that exploits the effect of human cognitive appraisal in sequential assessment
tasks. We conceptualize the memory-enhancing mechanism as Reinforcement Memory
Unit (RMU) that contains an appraisal state together with two positive and
negative reinforcement memories. The two reinforcement memories are decayed or
strengthened by stronger stimulus. Thereafter the appraisal state is updated
through the competition of positive and negative reinforcement memories.
Therefore, RMU can learn the appraisal variation under violent changing of the
stimuli for estimating human affective experience. As shown in the experiments
of video quality assessment and video quality of experience tasks, the
proposed reinforcement memory unit achieves superior performance among
recurrent neural networks, that demonstrates the effectiveness of RMU for
modeling human cognitive appraisal.

††Corresponding author: Zhenzhong Chen<EMAIL_ADDRESS>
##  1 Introduction

When interacting with environment, people’s affective action is dynamically
influenced by the external stimulations, namely Core Affect which is a
continuous assessment of one’s current state and affects other psychological
process accordingly [1]. The process of recursive and continuous assessment is
also the core of the appraisal evaluation [2]. Core affect depends on the
information possessed about the external cue, from its initial sensory
registration to cognitive appraisal. While major progress has been achieved on
the simulation of sensory registration in computer vision and natural language
processing communities, the modeling of cognitive appraisal is still at the
early stage of exploration. In this work, we aim to exploit the recursively
modeling method to simulate subjective appraisal variation under continuous
external stimuli, which is conducive to measuring and improving the prediction
of subjective assessment tasks.

In human appraisal evaluation, the input stimuli are intrinsically dynamic
that influence memory by proactive interference and retroactive interference.
When human sensory receives a sequence of stimuli, the affective experience is
varying continuously and affects the final decision making. Here, we consider
the reinforcement in behavioral psychology which can be strengthened
consciously or unconsciously elicited by the stimulus [3, 4]. The
reinforcement is classified into positive reinforcement and negative
reinforcement referring to the enhancement of behavior or memory. The positive
reinforcement occurs when receiving the favorable stimulus, which leads to a
tendency of giving a positive assessment. In contrast, the negative
reinforcement occurs given the opposite situation. In the sequential
assessment tasks, our working memory deploys its limited capacity for the
dynamic stimulus that maintains and manipulates the input information for such
goal-directed behavior [5]. The decision making in human assessment process
involves both conscious and unconscious thought [6], thus it is affected by
stimuli of both the received external information and internal mental
processes.

To study the human assessment process in a cognitive manner, some researchers
infer to psychological characteristics and experience. For example, Rimac-
Drlje and Seufert _et al._ [7, 8] utilize the recency effect in psychology and
assign higher weights to the last received sequence to stimulate the recency
effect. Seshadrinathan and Bovik [9] observe a hysteresis effect in the
subjective judgment of time-varying video quality, which means that subjects
react sharply to drops in video quality and provide poor quality for such time
interval, but react dully to improvements in video quality thereon. They also
propose a temporal pooling strategy accounting for the hysteresis effect. Kim
_et al._ [10] consider that the worst quality section has greater influence to
human assessment. So they propose a convolutional neural aggregation network
to learn frame weights and the overall quality is the weighted average of
frame quality scores. However, most methods only model the distribution of
time-dependent evaluation, instead of the recursive cognitive process. It is
often one-sided that only considering few psychological characteristics. Human
assessment is an iterative process and consists of continuous fluctuations in
core effect. Therefore, learning the effects of dynamic input on human
cognitive appraisal becomes essential for subjective assessment. This
motivated us to investigate the recurrent neural networks to mimic such a
procedure.

Figure 1: Inner structure of the reinforcement memory unit.

A variety of recurrent neural networks (RNNs) have been introduced to solve
sequential modeling problem and achieved state-of-the-art performance on
various tasks in the literature, including speech recognition [11, 12],
machine translation [13, 14], and video modeling [15]. To alleviate the issue
of gradients vanishing and exploding and learn long-range temporal
dependencies, some improved RNNs are designed and the especially successful
architecture among them are Long Short-Term Memory (LSTM) [16] and Gated
Recurrent Unit (GRU) [17] which control the information transmission through
well-designed gates. The design of LSTM and GRU are conducive to information
encoding in sensory registration phase and work memory modeling. However, more
appropriate modeling for the appraisal changing of the cognitive process is
still highly desired.

To simulate the human assessment in cognitive process, we design a
Reinforcement Memory Unit (RMU). The RMU contains a hidden state representing
the appraisal and positive and negative reinforcement memories. The appraisal
is adjusted by the competition of positive and negative reinforcement memories
which are updated by the forget gate and stimulus response. Therefore, RMU can
stimulate the response to the incoming stimuli, the decay of the impression,
and the appraisal variation under violent changes of stimuli. We validate RMU
on video quality assessment and quality of experience task to predict
retrospective assessment.

The main contributions of this work is threefold:

  * •

A Reinforcement Memory Unit (RMU) is proposed to estimate the human assessment
in cognitive process, which is easily combined with other methods and applied
to sequential user assessment tasks. It is validated on video quality
assessment task and quality of experience task, and achieves superior
performance.

  * •

The positive and negative reinforcement memories are designed to simulate the
proactive interference and retroactive interference on memory. The
reinforcement memories will decayed gradually affected by the forget gate and
be enhanced by stronger stimuli responses.

  * •

The appraisal state is designed to simulate subjective assessment which
constantly updated over time through the competition between positive and
negative reinforcement memories. It can generate recursive assessment under
continuous fluctuations of stimuli.

##  2 The Reinforcement Memory Unit

We propose RMU to simulate human assessment. The inner structure of RMU is
shown in Fig.1. We have the sequence of input signals
$\mathbf{X}=\left\\{x_{1},x_{2},\cdots,x_{T}\right\\}$, where $x_{t}$
represents a feature vector extracted from image, audio or text through neural
networks or hand-crafted descriptions.

In RMU, we first design two memory cells: positive reinforcement memory
$C_{t-1}^{+}$ and negative reinforcement memory $C_{t-1}^{-}$ which represent
the impression of positive and negative stimuli at timestep ${t-1}$,
respectively, whilst an appraisal hidden state $h_{t-1}$ to represent the
affective experience at timestep $t-1$. When received the current input
$x_{t}$, the stimulus response $s_{t}$ is computed which reflects the
direction and intensity of the stimulus of $x_{t}$ versus previous appraisal
$h_{t-1}$:

$s_{t}=\tanh\left(W_{s}h_{t-1}+U_{s}x_{t}+b_{s}\right)$ (1)

where $\tanh$ represents the hyperbolic tangent function thus the value of
stimulus response falls into the range $(-1,1)$. The plus/minus units of
$s_{t}$ represent positive/negative stimuli response compared to the previous
appraisal state. When the value of $s_{t}$ is closer to $1/-1$, the intensity
of the change becomes greater. The stimulus response may contain both plus and
minus units, as some reinforcement can be simultaneously positive and
negative.

As the previous stimuli gradually diminished over time, we attenuated the
reinforcement memories through a forget gate. Then these two memory states are
updated by the response of stronger stimulus through an element-wise maximal
operation, which can be described mathematically as:

$f_{t}^{+}=\sigma\left(W_{f}^{+}h_{t-1}+U_{f}^{+}x_{t}+b_{f}^{+}\right)$ (2)
$f_{t}^{-}=\sigma\left(W_{f}^{-}h_{t-1}+U_{f}^{-}x_{t}+b_{f}^{-}\right)$ (3)
$C_{t}^{+}=\max\left(f_{t}^{+}\odot
C_{t-1}^{+},\operatorname{ReLU}\left(s_{t}\right)\right)$ (4)
$C_{t}^{-}=\max\left(f_{t}^{-}\odot
C_{t-1}^{-},\operatorname{ReLU}\left(-s_{t}\right)\right)$ (5)

where the $\sigma$ and $\odot$ represent the logistic sigmoid function and the
Hadamard product, respectively. The ReLU stands for rectified linear unit and
is mathematically defined as $f\left(x\right)=\max\left(0,x\right)$. It
controls that negative stimuli can only update negative memory state and vice
versa. Finally, the appraisal $h_{t}$ is updated through the competition of
positive and negative reinforcement memories:

$h_{t}=\tanh\left(C_{t}^{+}-C_{t}^{-}\right).$ (6)

In order to better understand the practical meaning of RMU, we take the video
quality assessment task as an example. The input signal $\mathbf{X}$ can be
represented as the feature vectors consisting of different quality factors
extracted from frames. The appraisal hidden state $h_{t}$ can be mapped to
retrospective video quality score at $t$-th timestep. In this formula, if the
quality of $x_{t}$ is significantly worse than previous perceived video
quality $h_{t-1}$ and the intensity of change is larger than the negative
impression in the memory, the negative reinforcement memory $C_{t}^{-}$ will
be updated and strengthened. Thus the appraisal hidden state $h_{t}$ will
change towards negative stimuli and evaluation score will decrease through the
element-wise subtract operation between positive and negative reinforcement
memories. If the qualities of input sequence change repeatedly, the update of
assessment quality depends on the competition of positive and negative
impressions. If the qualities of input sequence are stable, the two memory
states will tend to be balanced, thereby maintain the stability of
retrospective evaluation $h_{t}$.

##  3 Experiments

We evaluate the performance of RMU on two sequential assessment tasks: Video
Quality Assessment (VQA) and Quality of Experience (QoE) prediction. We also
explore the changing characteristics of positive and negative reinforcement
memories, and appraisal hidden state through the visualization of continuous
modeling.

###  3.1 Model Establishment

To adapt RMU on predicting assessment results, we design a simple network as
shown in Figure 2. Given the input, the feature vectors $x$ are extracted with
batchsize $b$, sequence length $l$ and feature dimension $d_{1}$. Then a fully
connected layer is applied for feature transformation. After the temporal
modeling of RMU, we apply another fully connected layer to regress the hidden
state vectors to the output. The output can provide the continuous scores or
just the final evaluation, depending on the tasks.

To train RMU, the adaptive moment estimation optimizer (ADAM) [18] is adopted
with the learning rate $3\times 10^{-4}$. To reduce the overfitting during
training, we apply dropout [19] with rate of 0.5 on the input of RMU.

Figure 2: The illustration of network design.

###  3.2 Video Quality Assessment Task

Video Quality Assessment is to design an objective model predicting the
quality score of videos to approach human subjective evaluation. A challenge
issue in VQA is that the quality changes over time, which involves more
complicated cognitive process and could not be simply described as a temporal
pooling procedure. The stimuli of the changes of frame quality continuously
affects the users’ judgment on the video quality in the process of watching,
urging them to make the appropriate score finally. It could be regarded as a
cognitive appraisal process [20], which refers to a human thought process that
interprets and evaluates new situations and selects the appropriate reaction.
Therefore, learning the cognitive process plays a key role in building an
objective VQA model.

Dataset KoNViD-1k [21] aims at ’in the wild’ authentic distortions and is
created based on (Yahoo Flickr Creative Commons 100 Million) YFCC100m dataset.
The KoNViD-1k dataset contains 1,200 video sequences of resolution $960\times
540$ with a substantial content diversity. The video length is 8s and the Mean
Opinion Scores (MOS) ranges from 1.22 to 4.64. As there is no standard split
for training and testing set on KoNViD-1k, to better evaluate the cognitive
process in VQA, we split the dataset according to the variance of quality
change in temporal domain. The variance is obtained from frame qualities
calculated by the state-of-the-art VQA method TLVQM [22]. We use the 80%
videos of smaller variance as training set and the rest 20% as testing set.

Implementation Details In order to eliminate the influence of different
feature extraction neural networks, we extract hand-crafted features proposed
in comparison algorithms as the input $\mathbf{X}$ of RMU. The sampling rate
is set as 1 frame/second. As the VQA task only needs a quality evaluation for
the whole video, thus, the last appraisal state $h_{T}$ is used to generate
predicted quality score $y^{\prime}$ of the video. The loss function is
defined as the Mean Square Error (MSE) between predicted score $y^{\prime}$
and subjective MOS score $y$.

Performance Comparison To evaluate the performance of RMU fairly, we choose
four state-of-the-art NR-VQA and NR-IQA methods with publicly available
implementations to extract features, including BRISQUE [23], VBLINDS [24],
FRIQUEE [25] and TLVQM [22] with the feature dimension of 36, 46, 560 and 75,
respectively. We use the source code to extract features every second in
Matlab and then regress to quality score based on Pytorch. We compared three
recurrent neural networks: GRU, LSTM and RMU. All the configurations are kept
the same.

Table 1: Performance comparison of no-reference VQA on the KoNViD-1k dataset. Method |  PLCC |  SROCC |  KROCC |  RMSE
---|---|---|---|---
BRISQUE+LSTM |  0.5408 |  0.5521 |  0.3911 |  0.6092
BRISQUE+GRU |  0.5820 |  0.6083 |  0.4323 |  0.5868
BRISQUE+RMU |  0.6260 |  0.6321 |  0.4559 |  0.5788
VBLINDS+LSTM |  0.5948 |  0.6045 |  0.4274 |  0.5515
VBLINDS+GRU |  0.5831 |  0.5897 |  0.4186 |  0.5610
VBLINDS+RMU |  0.6288 |  0.6327 |  0.4474 |  0.5524
FRIQUEE+LSTM |  0.7859 |  0.7832 |  0.5869 |  0.4393
FRIQUEE+GRU |  0.7736 |  0.7693 |  0.5729 |  0.4890
FRIQUEE+RMU |  0.7751 |  0.7726 |  0.5759 |  0.4363
TLVQM+LSTM |  0.7686 |  0.7731 |  0.5710 |  0.4263
TLVQM+GRU |  0.7659 |  0.7735 |  0.5719 |  0.4263
TLVQM+RMU |  0.7871 |  0.7904 |  0.5884 |  0.4281

Four widely-used evaluation criteria are adopted for performance comparison:
Pearson linear correlation coefficient (PLCC), Spearman rank order correlation
coefficient (SROCC), Kendall’s rank-order correlation coeicient (KROCC) and
Root Mean Square Error (RMSE). The PLCC, SROCC and KROCC measure the
correlation between the predicted quality and ground truth, considering linear
dependence or monotonic. A better method will result in larger
PLCC/SROCC/KROCC and smaller RMSE.

The experimental results are shown in Table 1. It can be found that the
proposed RMU outperforms LSTM and GRU in most cases. In particular, RMU
surpasses LSTM and GRU on BRISQUE, VBLINDS and TLVQM, especially on PLCC,
SROCC and KROCC with a margin over 0.02, which shows that the predicted scores
of RMU have higher correlation with human subjective assessment. It is noted
that both RMU and GRU have slightly lower correlation when compared to LSTM in
the case of FRIQUEE. One possible reason is that the number of parameters of
RMU and GRU is less than LSTM, where FRIQUEE contains much more features
compared to other three methods.

###  3.3 Quality of Experience Task

Quality of Experience (QoE) refers to a viewer’s holistic perception and
satisfaction with a given content, communication network, or content-providing
service [26], where the dynamic nature of the perceived of streaming videos
results in continuous varying affective experience. Some QoE databases have
been developed by gathering both the continuous and retrospective QoE scores.
The continuous QoE score is composed of instantaneous subjective score which
reflects the quality changing of streaming videos, while the retrospective QoE
score is provided by subjects based the impression after the video is played.
Different from traditional video QoE prediction task which use QoE influencing
factors to predict continuous or retrospective QoE, we only consider the
subjective continuous QoE to predict retrospective QoE to compare the
performance of different RNNs on such a task. For this scenario, the influence
of different QoE factors are excluded, such that it is able to validate the
ability of learning the appraisal variation under violent changes of the
stimuli and simulating the human cognitive behavior.

Table 2: Performance comparison on QoE task. | Training Set | Testing Set
---|---|---
| RMSE | PLCC | RMSE | PLCC
LSTM | 0.4859 | 0.8320 | 0.4154 | 0.7883
GRU | 0.2369 | 0.9506 | 0.2962 | 0.7607
RMU | 0.2207 | 0.9569 | 0.2791 | 0.7984

Dataset LIVE-Netflix Video Quality of Experience database [27] gathered both
subjective continuous and retrospective QoE scores. The database consists of
112 distorted videos evaluated by over 55 human subjects on a mobile device.
The distorted videos were generated from 14 video contents by imposing a set
of 8 different playout patterns ranging from dynamically changing H.264
compression rates and re-buffering events to a mixture of compression and re-
buffering. The average number of continuous QoE scores per video is 1797. We
split the training set and testing set by 70% to 30%, which means that the
training set contains 80 videos from 10 video contents and the testing set
contains 32 videos from 4 video contents.

Figure 3: An example of the continuous variation of subjective instantaneous
QoE, two reinforcement memories and appearisal hidden state on retrospective
QoE prediction.

Implementation Details The size of hidden unit is set as 1 to better explore
the practical meaning and fitting ability of recurrent unit. The parameter $U$
in Equation 1 is initialized as -1.0 to fit the design of stimulus response,
while other parameters $W_{*},U_{*}$ are initialized as 1.0. The biases of
forget gates are initialized as 1.0 to avoid serious forgetting at the
beginning of training, and other biases are initialized as 0.

Performance Comparison Table 2 shows the RMSE and PLCC between predicted
endpoint scores and ground truth. The performance of RMU surpasses LSTM and
GRU on both training set and testing set. Under limited number of samples and
long sequence length over 1700, the fitting ability of RMU still shows
effectiveness and is better than LSTM and GRU as shown in Figure 4. Thus It
can be concluded that RMU can capture long-range dependencies and fit human
assessment data effectively.

Figure 4: The loss of LSTM, GRU and RMU based methods during training epochs
on retrospective QoE prediction.

To further validate the effect of positive and negative reinforcement unit, we
further show a visualization example of the variation of reinforcement
memories and appraisal state in Figure 3. Along with continuous external
stimuli, the two reinforcement memories are decayed or strengthened
recursively further to affect the appraisal state. Specifically, $i$) At the
beginning of video, the subjective instantaneous QoE is at a low level and the
negative reinforcement memory keeps active. With the gradual improvement of
subjective QoE, the positive reinforcement memory is aroused and suppresses
negative reinforcement memory at timestep $t_{1}$. $ii$) Between timestep
$t_{1}$ and $t_{2}$, the subjective instantaneous QoE becomes stable, thus the
positive reinforcement memory achieves balance between forgotten and new
stimuli responses. $iii$) From timestep $t_{2}$ to $t_{3}$, the quality of
video declines sharply, so does the subjective instantaneous QoE. The stimuli
responses turn into negative. Thus the positive reinforcement memory decays
under the effect of forget gate, instead, the negative reinforcement memory is
strengthened. $iv$) After timestep $t_{3}$ the subjective instantaneous begins
to rise again slowly. The negative reinforcement is attenuated until
suppressed by positive reinforcement memory. This experiment proves that the
reinforce and suppression of positive and negative reinforcement memories are
consistent with human psychological characteristics and experience. Besides,
the predicted appraisal state at final timestep is close to subjective
retrospective QoE which demonstrates that RMU can handle sequential user
assessment tasks effectively.

##  4 Characteristics Analysis

Architecture LSTM, GRU and RMU are all gated recurrent networks, thus we
compare their architectures and analyze the effects of different gates here.

The LSTM contains three gates: input gate $i_{t}$, forget gate $f_{t}$ and
output gate $o_{t}$ which influence the hidden state and cell memory through
the following mechanism:

$C_{t}=f_{t}\odot C_{t-1}+i_{t}\odot\widetilde{C}_{t},$ (7)
$h_{t}=o_{t}\odot\phi\left(C_{t}\right),$ (8)

where ${C}_{t}$ represents the candidate memory. The input gate aims at
controlling how much new information should be remembered, while the forget
gate controls how much information in the memory will be discarded. Then the
output gate decides which information will be output from memory.

Table 3: The comparison of computational resources of LSTM, GRU and RMU. Methods | Number of Parameters | Memory Consumption |
---|---|---|---
LSTM | $4\left(n_{1}n_{2}+n_{2}^{2}+n_{2}\right)$ | $n_{1}+6n_{2}+4\left(n_{1}n_{2}+n_{2}^{2}\right)$ |
GRU | $3\left(n_{1}n_{2}+n_{2}^{2}+n_{2}\right)$ | $n_{1}+4n_{2}+3\left(n_{1}n_{2}+n_{2}^{2}\right)$ |
RMU | $3\left(n_{1}n_{2}+n_{2}^{2}+n_{2}\right)$ | $n_{1}+6n_{2}+3\left(n_{1}n_{2}+n_{2}^{2}\right)$ |

Compared to LSTM, GRU combines the memory cell and hidden state, thus the
output gate is discarded. Besides, the reset gate $r_{t}$ and update gate
$z_{t}$ are applied which integrate the effect of input gate and forget gate:

$\tilde{h}_{t}=\phi\left(W_{h}\left(r_{t}\odot
h_{t-1}\right)+U_{h}x_{t}+b\right),$ (9) $h_{t}=z_{t}\odot
h_{t-1}+\left(1-z_{t}\right)\odot\tilde{h}_{t}.$ (10)

In this formula, the reset gate controls which information from the previous
hidden state will ignore and reset with the current input, and the update gate
controls how much information from the previous hidden state will carry over
to the current hidden state.

Compared to LSTM and GRU, RMU only keeps the forget gate as it is consistent
with human physiological mechanism and shows significant effectiveness on
information control [28, 29]. Different from the memory update mechanisms in
LSTM and GRU that remember external information intentionally, RMU transfers
external information into stimuli responses (Equation 1) and updates memories
with the response of stronger stimuli (Equation 4, 5). Thus, if the new
stimuli beyond the old reinforcement memories, the memories will be replaced
by the response of stronger stimuli which can be viewed as retroactive
interference. Otherwise, the memories will defeat new stimuli which can be
viewed as proactive interference.

Computational Resource Considering a recurrent unit with $n_{1}$ inputs and
$n_{2}$ hidden units, the recurrent units have
$\mathbf{x}\in\mathbb{R}^{n_{1}}$,
$\left\\{\mathbf{c}_{t},\mathbf{h}_{t},\mathbf{b}_{*}\right\\}\in\mathbb{R}^{n_{2}}$,
$\mathbf{W}_{*}\in\mathbb{R}^{n_{1}\times n_{2}}$,
$\mathbf{U}_{*}\in\mathbb{R}^{n_{2}\times n_{2}}$. We have $*=\\{i,o,f,c\\}$
for LSTM, $*=\\{r,z,h\\}$ for GRU and $*=\\{s,f^{+},f^{-}\\}$ for RMU. The
total number of parameters is shown in Table 3 which indicate that the
parameter number of RMU is three quarters of LSTM and equal to GRU.

After considering the memory consumption of inputs and hidden units, the
required memories of LSTM, GRU and RMU at each step are also shown in Table 3.
Since the value is dominated by the $4n_{2}^{2}$ term, it could be concluded
that the computational resource of RMU is comparable with GRU, and less than
LSTM.

Convergence Ability To compare the convergence ability of LSTM, GRU and RMU,
Figure 5 shows the loss and PLCC on training set and testing set on VQA task
with TLVQM as basis. The performance of LSTM and GRU are similar. From the
convergence of loss in Figure 5(a) it could be found that the training of RMU
has less disturbance than LSTM and GRU. Meanwhile, RMU also converges faster
than LSTM and GRU as shown in Figure 5(b). Thus it can be concluded that RMU
has a quicker and smoother convergence than LSTM and GRU.

##  5 Conclusion

We proposed a Reinforcement Memory Unit (RMU) to simulate the cognitive
appraisal process. We demonstrated the inner structure of RMU that involves
the decay of the impression, the response to new stimuli and the appraisal
variation under the competition between positive and negative reinforcement
memories. Additionally, we showed the success of applying RMU in user-oriented
tasks. The experiments on video quality assessment and quality of experience
task demonstrate that RMU can not only handle long-range dependencies and
temporal modeling, but also estimate the human assessment in cognitive
process.

(a) Loss

(b) PLCC

Figure 5: The loss and PLCC of LSTM, GRU and RMU based methods during training
epochs on VQA task.

## Broader Impact

This research proposes a novel RNN to model continuous assessment process, an
important step towards the simulation of human cognitive process. The
Reinforcement Memory Unit is a flexible module which is easily combined with
other methods. It provides a basic tool for many researchers to model time-
varying assessment task and analyze the memory characteristics. Therefore,
this work has the potential for impact in psychology, sentiment analysis,
video understanding, experience modeling, and many other sequential tasks.

## References

  * Russell [2003] James A. Russell.  Core affect and the psychological construction of emotion.  _Psychological Review_ , 110(1):145–172, 2003\.
  * Scherer [2010] Klaus R. Scherer.  The component process model: Architecture for a comprehensive computational model of emergent emotion.  In _Blueprint for affective computing: A sourcebook_ , chapter 2.1, pages 47–70. Oxford University Press, 2010.
  * Flora [2004] Stephen Ray Flora.  _The power of reinforcement_.  Albany, N.Y. : State University of New York Press, 2004.  ISBN 0791459152 (alk. paper).
  * Schultz [2015] Wolfram Schultz.  Neuronal reward and decision signals: From theories to data.  _Physiological Reviews_ , 95(3):853–951, 2015\.  doi: 10.1152/physrev.00023.2014.
  * Baddeley [2003] Alan Baddeley.  Working memory: looking back and looking forward.  _Nature Reviews Neuroscience_ , 10(4):829–839, 2003.
  * Dijksterhuis [2004] Ap Dijksterhuis.  Think different: the merits of unconscious thought in preference development and decision making.  _Journal of personality and social psychology_ , 87(5):586–598, 2004.
  * Rimac-Drlje et al. [2009] S. Rimac-Drlje, M. Vranjes, and D. Zagar.  Influence of temporal pooling method on the objective video quality evaluation.  In _IEEE International Symposium on Broadband Multimedia Systems and Broadcasting_ , pages 1–5, 2009.
  * Seufert et al. [2013] M. Seufert, M. Slanina, S. Egger, and M. Kottkamp.  To pool or not to pool: A comparison of temporal pooling methods for http adaptive video streaming.  In _Fifth International Workshop on Quality of Multimedia Experience_ , pages 52–57, 2013.
  * Seshadrinathan and Bovik [2011] K. Seshadrinathan and A. C. Bovik.  Temporal hysteresis model of time varying subjective video quality.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 1153–1156, 2011.
  * Kim et al. [2018] Woojae Kim, Jongyoo Kim, Sewoong Ahn, Jinwoo Kim, and Sanghoon Lee.  Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network.  In _the European Conference on Computer Vision_ , pages 219–234, 2018\.
  * Chorowski et al. [2015] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.  Attention-based models for speech recognition.  In _Advances in Neural Information Processing Systems 28_ , pages 577–585, 2015.
  * Graves et al. [2013] A. Graves, A. Mohamed, and G. Hinton.  Speech recognition with deep recurrent neural networks.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 6645–6649, 2013.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _ArXiv_ , abs/1609.08144, 2016.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems 27_ , pages 3104–3112, 2014.
  * Venugopalan et al. [2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko.  Sequence to sequence – video to text.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4534–4542, 2015.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780, 1997.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations_ , 2015\.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.
  * Cog [2013] Cognitive appraisal.  In Gerald F. Gebhart and Robert F. Schmidt, editors, _Encyclopedia of Pain_ , pages 691–691. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.  doi: 10.1007/978-3-642-28753-4_200399.
  * Hosu et al. [2017] V. Hosu, F. Hahn, M. Jenadeleh, H. Lin, H. Men, T. Szirányi, S. Li, and D. Saupe.  The konstanz natural video database (konvid-1k).  In _2017 Ninth International Conference on Quality of Multimedia Experience_ , pages 1–6, 2017.
  * Korhonen [2019] J. Korhonen.  Two-level approach for no-reference consumer video quality assessment.  _IEEE Transactions on Image Processing_ , 28(12):5923–5938, 2019.
  * Mittal et al. [2012] A. Mittal, A. K. Moorthy, and A. C. Bovik.  No-reference image quality assessment in the spatial domain.  _IEEE Transactions on Image Processing_ , 21(12):4695–4708, 2012.
  * Saad et al. [2014] M. A. Saad, A. C. Bovik, and C. Charrier.  Blind prediction of natural video quality.  _IEEE Transactions on Image Processing_ , 23(3):1352–1365, 2014.
  * Ghadiyaram and Bovik [2017] Deepti Ghadiyaram and Alan C Bovik.  Perceptual quality prediction on authentically distorted images using a bag of features approach.  _Journal of vision_ , 17(1):32–32, 2017.
  * Ghadiyaram et al. [2019] D. Ghadiyaram, J. Pan, and A. C. Bovik.  A subjective and objective study of stalling events in mobile streaming videos.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 29(1):183–197, 2019.
  * Bampis et al. [2017] C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, and A. C. Bovik.  Study of temporal effects on subjective video quality of experience.  _IEEE Transactions on Image Processing_ , 26(11):5217–5231, 2017.
  * Gers et al. [1999] F. A. Gers, J. Schmidhuber, and F. Cummins.  Learning to forget: continual prediction with lstm.  In _Ninth International Conference on Artificial Neural Networks_ , volume 2, pages 850–855, 1999.
  * van der Westhuizen and Lasenby [2018] Jos van der Westhuizen and Joan Lasenby.  The unreasonable effectiveness of the forget gate.  _CoRR_ , abs/1804.04849, 2018.