Data Storage for HEP Experiments in the Era of High-Performance Computing

Amit Bashyal and Peter van Gemmeren

Argonne National Laboratory

Saba Sehrish and Kyle Knoepfel

Fermi National Accelerator Laboratory

Suren Byna and Qiao Kang

Lawrence Berkeley National Laboratory

On Behalf of the HEP-CCE IOS Group

> ABSTRACT
>
>
>
>
> As particle physics experiments push their limits on both the energy and the
> intensity frontiers, the amount and complexity of the produced data are also
> expected to increase accordingly. With such large data volumes, next-
> generation efforts like the HL-LHC and DUNE will rely even more on both
> high-throughput (HTC) and high-performance (HPC) computing clusters. Full
> utilization of HPC resources requires scalable and efficient data-handling
> and I/O. For the last few decades, ROOT has been used by most HEP
> experiments to store data. However, other storage technologies like HDF5 may
> perform better in HPC environments. Initial explorations with HDF5 have
> begun using ATLAS, CMS and DUNE data; the DUNE experiment has also adopted
> HDF5 for its data-acquisition system. This paper presents the future outlook
> of the HEP computing and the role of HPC, and a summary of ongoing and
> future works to use HDF5 as a possible data storage technology for the HEP
> experiments to use in HPC environments.

##  1 Introduction

With the growing computational demands of the upcoming HEP experiments, the
role of high-performance computing (HPC) will also increase. HPC systems,
however, are different than the traditional grid-computing system with highly
customized architectures and hardware. To use HPC systems to meet the HEP
community’s computational demands, tools and technologies must be adopted to
utilize HPC resources. The organizational complexity of HEP data products and
workflows, however, makes complete adoption of such technologies non-trivial.
We thus outline an approach that motivates the use of HPC systems for
processing contexts where significant computational capabilities are required,
such as data reconstruction. However, for further-downstream contexts such as
data analysis, we support HEP experiments in their continued use of
technologies such as ROOT [1], which has been a workhorse of this community
for the last two decades.

This white paper discusses the role of HPC in light of the growing
computational needs of HEP experiments in section 2 . In section 3, we discuss
our exploratory work of using HDF5, one of the HPC-friendly I/O libraries, to
read/write HEP experiment data. In section 4, we discuss some of the avenues
that can be explored by the HEP community to introduce and eventually use HPC
systems in their workflows.

##  2 HEP Computing Context

The requirement to handle big data has pushed the HEP field to explore and
innovate data-computing and storage technology. The Open Science Grid [2] and
the Worldwide LHC Computing Grid (WLCG) [3] are examples of how the HEP
community (notably through Fermilab and CERN) is firmly committed to seeking
solutions to meets its computational and storage demands. These endeavors,
which utilize high-throughput computing (HTC), have been critical ingredients
in meeting experiments’ physics goals such as discovering the Higgs boson.

Up until Run II of the LHC, the WLCG system met data-processing needs at a
rate that was commensurate with those of data collection. By the end of Run 5,
however, the ATLAS and CMS experiments estimate that the collected data be 10
times larger than was collected during the first 3 runs [4]. Projections made
by ATLAS [4] and CMS [5] show that without a radical shift in the data-
management and -processing approach, the current computing infrastructure will
not be able to accommodate the storage and computing needs of these
experiments in future LHC runs. To illustrate, Figure 1 shows the projected
annual CPU consumption of the ATLAS experiment from 2020 to 2036. As can be
seen, with only modest improvements in the computing and software
infrastructure, the year-by-year CPU consumption rate will increase such that
the current infrastructure will not be able to sustain even the most
optimistic scenario. To comfortably accommodate the goals of future LHC runs,
a somewhat aggressive R&D program is thus required.

Figure 1: Projected CPU consumption rate from 2020 to 2036 for the ATLAS
experiment. The area between the blue and the red dot line shows the range of
CPU consumption estimation based on conservative (blue) to aggressive (red)
R&D in the computing infrastructure. Conservative scenario assumes the current
person power maintained at current level with the current level of expertise.
Aggressive R&D assumes increase in expertise either by allowing current person
power to increase the time in software R&D or bringing new people. The top
(bottom) black line shows the impact of 10% (20%) increase in budget annually
in the R&D. Figure taken from [4].  Figure 2: Projected peak CPU performance
of the various DOE super computers (red line) and the estimated HEP CPU
resource requirements assuming sustained performance (blue shade). Figure
taken from [6].

Intensity-frontier experiments have slightly different computing challenges
compared to those of the LHC. Although the DUNE experiment will be larger in
physical size than the LHC experiments (with up to four 10 kT active volumes
of liquid argon), DUNE’s far detector [7] will have fewer channels than those
of the LHC experiments. The data rates from DUNE’s channels, however, will be
significant–anywhere from a few GB for beam-induced readouts to over 100 TB
for supernova readouts. DUNE is thus a good candidate for utilizing the
parallel and accelerated data-processing capabilities of HPC resources [8].
Furthermore, as the US Department of Energy (DOE) is investing heavily in HPC
projects among its national labs, the HEP community should avail itself of the
HPC resources provided by the DOE.

The HPC ecosystem promises an efficient data-management and -computation
environment that could greatly benefit the HEP community. As shown in Figure
2, HPC resources could meet the HEP community’s CPU requirements through the
DUNE and LHC Run 5 eras. The implication, however, is that HEP experiments
would need to invest in the computing R&D efforts necessary for using HPC
systems. The HEP community would need to understand not only how to adjust its
current software for HPC systems, but also how that software would be
deployed, how to use the parallel I/O and parallel computing features of the
HPC machine, and what data storage mechanisms are best used in an HPC
environment.

###  2.1 High Performance I/O and Storage Requirements

HPC systems use parallel file systems that enable efficient data storage and
access. High-level parallel I/O libraries, such as HDF5 [9, 10, 11] and
PnetCDF [12], provide interfaces for applications to use parallel file systems
without exposing parallelism details to the user. In addition to the already
complex parallel I/O, there are several architectural changes expected in
upcoming HPC systems. Traditional storage sub-systems have been based on hard-
disk drives (HDD), which are much slower than the memory hardware of compute
nodes. The latency of HDD-based file systems is significantly high as a
result. To mitigate the impact of this latency on application performance, the
latest HPC systems are adding faster layers of storage (e.g., non-volatile
memory, solid-state drives) between a compute node’s memory and traditional
parallel file systems. These additional layers of storage and their file
systems further increase the complexity of using I/O on modern HPC systems.

Utilization of the HPC computing resources requires storing and accessing
these new types of storage systems efficiently. Below is a list of some HEP-
specific requirements for using HPC storage systems.

  * •

_Parallelism:_ Stored data should be able to be accessed by multiple parallel
processes. This requires multiple parallel processes to be able to do I/O
operations (e.g. opening a file, reading and writing buffers) on the data
concurrently.

  * •

_Data model:_ HEP data products are often complex C++ objects that cannot be
serialized easily (currently we can depend on ROOT to serialize the data-
products.) Either the storage software should provide tools to serialize the
complex data products (highly unlikely) or the data products should be modeled
to be friendly with the storage systems.

  * •

_Visibility:_ HPC-friendly data storage systems provide high level libraries
to fully utilize the HPC resources. However, users should be able to customize
and configure the parameters depending on the requirement of the HEP
experiment.

  * •

_Usability:_ The HEP community is accustomed to ROOT’s user-friendliness,
reasonably thorough code documentation, and robust user forums to share
issues, solutions, and learn of new features. The community will expect
similar attention to usability with any other storage format chosen for HPC
environments.

This is not an exhaustive list of requirements and the HEP community may
discover further requirements as the role of HPC increases in HEP workflows.

##  3 Using HDF5 Storage for HEP Data

To explore using HPC I/O libraries to store HEP data, we begin with the HDF5
library [9]. We work with existing HEP data models and frameworks to design an
approach that is experiment-agnostic. We also make basic assumptions regarding
the layout of experiment data:

  * •

The fundamental unit for processing HEP data is the _event_ , which contains
_data products_ —constructs that represent physically meaningful information
such as energy deposits (or hits), trigger times, and information related to
incident beams.

  * •

Events can be grouped into _subruns_ or _luminosity blocks_ , which in turn
are grouped into _runs_.

The representations of data products can be complex, which can be non-trivial
to map to a storage backend. Like other technologies, HDF5 does not provide
the near-automatic C++ type support the HEP community has relied upon through
ROOT over the last few decades. The following discusses our efforts in mapping
HEP data to HDF5.

###  3.1 Mapping HEP Data to HDF5

For a baseline result, we map HEP data to HDF5’s serial I/O capabilities. HDF5
provides the hierarchical data-model just like the directories, sub-
directories and files. This model is similar to that provided by the ROOT
framework, which groups stored data according to TBranchs, TTrees, and TFiles.
We are exploring two approaches for the HDF5-based design: one that is data-
product based and event based.

The data-product based approach is similar to how data is represented as
TBranch objects in a ROOT TTree. For our use case, the input data for HDF5 is
considered to be already serialized using ROOT serialization.

Each data-product is assigned a separate HDF5 dataset. This is similar to
writing different data-products in different TBranchs. Data can be aggregated
in batches before writing them into HDF5 datasets or written one at a time.

Event boundaries are identified by storing the size of each data-product in a
different HDF5 dataset.

This method requires $2N$ HDF5 datasets for $N$ data-products and at least one
additional HDF5 dataset to store the sizes of the data-products.

The event-based approach stores one complete event as a serialized byte stream
and product boundaries are stored in a different dataset. Hence this method
requires only three HDF5 datasets to store the data with all the relevant
information.
Data product based approach has better compression and allows access to
individual data-product without having to read the whole event. Typically HEP
experimental root files have large number of data-products. Since the data-
products are smaller in sizes, they need to be aggregated over multiple events
which requires more memory footprint.
Event based method requires only two HDF5 data-sets for each event. Compared
to the data-product based method, this method requires only two I/O process to
read/write each event. However, this also means that the users will have to
read the whole event to access necessary data-products.

Both event based and data-product based methods have pros and cons and we will
improve these methods and explore additional optimization strategies in the
upcoming days.

###  3.2 Proof of Concept

We have developed a multi-threaded test framework based on CMS’s framework as
part of this on-going work [13]. This framework supports any number of storage
formats; and can read from any supported format and write to any supported
format. Any new format can be easily added by providing the necessary base-
class interface implementation. This framework supports the study of scaling
of the I/O performance with different formats on different systems, memory
usage, file size and compression in both HPC and HPC environment. To study the
performance of HDF5 with HEP data, we have added event-based and data-product
based modules.
This frameowrk supports the data from ATLAS, CMS and the DUNE experiments so
far.

However, further performance studies and the comparison of HDF5 with other
other formats (including ROOT) are needed in the future. Furthermore,
additional exploration is needed in the areas like optimization of tuning
parameters and data layouts for HEP data, parallel I/O and compression in
HDF5.

###  3.3 Leveraging Parallel HDF5

HDF5 provides supports for the MPI [14] based parallel I/O which allows users
to effectively utilize the HPC resources. Users can utilize the collective I/O
libraries supported by HDF5 which allows multiple processes to write in a
single file saving additional computational resources needed to merge several
small files into few larger ones as done traditionally in the HEP field. HEP
frameworks are usually multi-threaded and leveraging on parallel HDF5 will
require further research and studies.

##  4 Future Directions

Over the last 2 decades, ROOT has been the work horse of the HEP experiments
in terms of data processing, storage and analysis. The tools developed by ROOT
has made the work flow of the HEP experiments much smoother and easier. Until
now, the data processing and storage of the HEP experiments overwhelmingly
depend upon the HTC systems in which ROOT performs best. However, as discussed
in this paper, the computational needs of the HEP fields over the next decade
can only be fulfilled by utilizing HPC systems which have much different
storage hardware and architectural design where other I/O libraries and file
format like HDF5 can perform better. HEP experiments need to identify the
areas where these HPC-friendly storage systems can be adopted directly or with
minimal work.
One of the challenges of the HEP experiment data is the serialization of data-
products. HEP data- products are often complex C++ objects that are hard to
couple directly with the HDF5 or other HPC-friendly I/O libraries. Currently,
we rely on ROOT to serialize these complex data-products which can then be
saved in the HDF5 format for our studies. However, we should identify the
areas where the data-products are simple enough to be directly stored in these
file formats. For example, DUNE directly stores its raw data into the HDF5
format. In the long run, HEP experiments should map their data-models to the
HPC-friendly file formats for the computationally intensive processes like
full detector simulation where parallel I/O performance is crucial.

##  5 Summary

HPC systems will be able to fulfill the HEP computational needs long into the
future and the HEP community will see the increased role of HPC in their
experiments. Even though ROOT will still be the preferred choice for the
custodial data-sets for the analysis, HPC-friendly I/O libraries like HDF5
maybe beneficial for the data-processing. HEP community needs to start
investigating the data storage formats that are optimized for the HPC
environment in terms of HEP requirements as discussed in this paper. At the
same time, the HEP experiments can try to identify the areas where HPC-
friendly libraries such as HDF5 can be adopted with or without utilizing ROOT-
serialization. In the long run, data generated from intensive computation
should be designed to be directly coupled with these HPC-friendly formats. The
computing requirements of the future HEP experiments can be fulfilled by
increasing the share of HPC in all the relevant areas. Fully utilizing the HPC
performance will require adopting the HPC-friendly data storage systems.
Exploring the HPC resources will require new or increased efforts from the HEP
community to identify the areas where the HPC resources can be adopted without
compromising the gains made in the last two decades. At the same time, HEP
community should increase the engagement with the HPC experts to integrate the
HPC related tools and technology into the HEP work-flow. The HEP experiments
will see increased role of HPC and the exploratory works (as discussed in this
paper) will help the HEP community to adopt these new resources and sustain
the computational requirements well into the future.

## References

  * [1] Rene Brun and Fons Rademakers.  ROOT - An Object Oriented Data Analysis Framework.  "https://root.cern.ch".
  * [2] Ruth Pordes, Don Petravick, Bill Kramer, Doug Olson, Miron Livny, Alain Roy, Paul Avery, Kent Blackburn, Torre Wenaus, Frank Würthwein, Ian Foster, Rob Gardner, Mike Wilde, Alan Blatecky, John McGee, and Rob Quick.  The open science grid.  In J. Phys. Conf. Ser., volume 78 of 78, page 012057, 2007\.
  * [3] Ian Bird.  Computing for the Large Hadron Collider.  Ann. Rev. Nucl. Part. Sci., 61:99–118, 2011.
  * [4] Zach Marshall, James Catmore, Alessandro Calafiura, Paolo Di Girolamo, and ATLAS Collaboration.  Atlas software and computing hl-lhc roadmap.  Technical report.
  * [5] O. Peckham.  CERN is betting big on Exa-Scale.  "https://www.hpcwire.com/2021/04/01/cern-is-betting-big-on-exascale/".
  * [6] Paolo Calafuira and Salman. Habib.  HEP computational requirements on behalf of HEP-CCE.
  * [7] Babak Abi et al.  Deep Underground Neutrino Experiment (DUNE), Far Detector Technical Design Report, Volume I Introduction to DUNE.  JINST, 15(08):T08008, 2020.
  * [8] P. Laycock.  DUNE Software and Computing Challenges.  "https://www.osti.gov/servlets/purl/1811451",.
  * [9] The HDF Group.  Hierarchical data format version 5, 2000-2010.  "http://www.hdfgroup.org/HDF5".
  * [10] Mike Folk, Gerd Heber, Quincey Koziol, Elena Pourmal, and Dana Robinson.  An overview of the HDF5 technology suite and its applications.  In EDBT/ICDT, 2011.
  * [11] Suren Byna, M. Breitenfeld, B. Dong, Q. Koziol, Elena Pourmal, et al.  ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems.  JCST, 2020.
  * [12] Jianwei Li, Wei keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale.  Parallel netCDF: A High-Performance Scientific I/O Interface.  In SC ’03: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, pages 39–39, 2003.
  * [13] Chris Jones and Saba Sehrish.  Root-serialization.  https://github.com/hep-cce2/root_serialization/tree/master, 2021\.
  * [14] MPICH.  "https://www.mpich.org".


2022

[1]Marko Ruman

1]Department of Adaptive Systems, Institute of Information Theory and
Automation, The Czech Academy of Sciences, Pod Vodárenskou věží 4, Prague, 182
00, Czech Republic

Conflict of Interest: Both authors declare that they have no conflict of
interest.

# Learning State Correspondence of Reinforcement Learning Tasks for Knowledge
Transfer

<EMAIL_ADDRESS>Tatiana V. Guy<EMAIL_ADDRESS>[

###### Abstract

Deep reinforcement learning has shown an ability to achieve super-human
performance in solving complex reinforcement learning tasks only from raw-
pixels. However, it fails to reuse knowledge from previously learnt tasks to
solve new, unseen ones. Generalizing and reusing knowledge are the fundamental
requirements for creating a truly intelligent agent. This work focuses on
transfer learning in reinforcement learning tasks. It offers a general method
for one-to-one transfer learning based on generative adversarial network model
tailored to reinforcement learning task.

###### keywords:

reinforcement learning, transfer learning, Markov decision process, deep
learning

##  1 Introduction

The field of deep reinforcement learning has received much attention recently.
Model-free methods for solving reinforcement learning (RL) tasks have shown
ability to achieve super-human performance in Atari games, dqn , advanceddqn ,
where the Deep $Q$-network (DQN) algorithm find optimal discrete actions.
Moreover, they are successfully used also e.g. in MuJoCo environments with
continuous actions, mujoco , or in the real-world robotic applications, realRL
. Advances were made also in model-based deep RL methods such as AlphaZero,
alphazero , or PlaNET, planet .

However, unlike humans, all the methods lack an important ability to
_generalize_ across different, but similar tasks. Humans have a remarkable
ability to learn motor skills by mimicking behaviors of others. For example,
developmental psychologists has shown that 18-month-old children are able to
infer the intentions and imitate behaviors of adults, psychology . Imitation
is not easy: children likely need to infer correspondences between their
observations and their internal representations, which effectively aligns the
two domains.

In machine learning, AI or robotics, using previously learnt knowledge in new,
different, but related tasks is called _transfer learning_ (TL), tlsurvey . In
the RL context, transfer learning is crucially important for developing agents
with lifelong learning, lifelongrl , for simulation-to-real knowledge transfer
used in robotics, sim2realgrasp , retinagan , rlcyclegan , or for developing
the general artificial intelligence, generalai .

The problems of transfer learning in RL and especially deep RL contexts often
stem from inability to generalize. For example, when deep learning was used
with image data, it was shown that the output of deep convolutional network
can be dramatically changed by 1-pixel perturbations of the input image,
onepixelworl . Naturally, as image data often form the observable states in RL
tasks, this problem occurs also in RL domain, for instance 1-pixel
perturbations lead to useless policies, onepixelrl . The methods fail to reuse
previously learnt knowledge even in very similar tasks, i.e. when the original
image is rotated, or some colors have been changed. It was also shown that
learning from scratch can be more optimal than fine-tuning the previously
gained model, Gamrian . All that is in huge contrast with human abilities to
generalize.

This work proposes a transfer learning approach with an emphasis on finding
correspondence between _unpaired_ data. It formalizes the problem of transfer
learning in RL tasks, provides survey of state-of-the-art methods and offers a
novel method for one-to-one transfer learning based on unsupervised data-
record correspondence search.

#### Related works

_Other approaches:_ Besides learning the task correspondence, some other
approaches for transfer learning in deep RL are being researched. One research
direction aims to learn general features of RL tasks transferable across
multiple tasks. Work midlevelfeatures  showed that learning policies on so-
called mid-level features can generalize better than learning policies
directly on image observations. In invariantfeaturesrl  the authors learnt
general features for two tasks with different dynamics, however, their method
requires paired image observations which are hard to obtain in practice. Work
barreto  achieved success in tasks differing in reward functions by
maintaining so-called successor features and by decoupling environment
dynamics and reward function.
Correspondence search: The pioneering works that used task correspondence were
based on unsupervised image-to-image translation models CycleGAN111Cycle
generative adversarial network, cyclegan , and UNIT222Unsupervised Image-to-
Image Translation Networks, unit . Work Gamrian  achieved partial results by
finding correspondence of states of two RL tasks. Their application potential
is rather limited as problems like mode-collapse were present. Works
rlcyclegan  and retinagan  improved the approach by introducing learnt
$Q$-function or object detection into the learning of the task correspondence.
One of the most recent work, dynamiccycle , introduces dynamics model in the
learning. This approach is strongly influenced by the video-to-video
translation model, videogan .

#### Contributions

  * $\bullet$

The work introduces a method for knowledge transfer between two similar RL
tasks based on RL-specific modification of CycleGAN.

  * $\bullet$

The work establishes a corresponding function finding similarity between the
data from different RL tasks. The method is able to successfully transfer the
knowledge between RL tasks while respecting their similarity.

  * •

The work proposes a new loss function consisting of several meaningful
components reflecting different type of losses. The modification introduces
$Q$-function and environment model into the learning. It is shown that each
component plays a significant role.

  * $\bullet$

The method is general and does not require any extra task-specific manual
engineering. It works with RL tasks regardless of data format of states (e.g.
images, sounds, numerical vectors, etc.).

Experiments with Atari game Pong demonstrated that the proposed method notably
speeds up the learning and increases the average reward. The proposed approach
is able to cope with the tasks for which standard approaches (GAN, CycleGAN)
fail and the most efficient way remains to learn from scratch.

The paper layout is as follows: Section 2 recalls the required background and
formulates the problem. Section 3 introduces the correspondence function and
proposes a novel method of its learning. Section 4 describes experimental
evaluation of the proposed method and compares it with baseline methods.
Section 5 provides concluding remarks and outlines the future research
directions.

##  2 Background and notation

This section briefly recalls RL formalism and introduces the considered
problem.

Throughout the text sets are denoted by bold capital letters (e.g. X),
functions by capital letters (e.g. X), variables by lower-case letters (e.g.
$x$). $\left\lVert x\right\lVert_{1}$ is the L1 norm of $x$. $x_{t}$ is the
value of $x$ at discrete time instant $t\in\textbf{{N}}$. $E_{p}[x]$ denotes
expected value of $x$ with respect to a probability density $p$ (if provided).
No notational distinction is made between a random variable and its
realization.

###  2.1 Reinforcement learning

Reinforcement learning (RL) considers an _agent_ purposefully interacting with
an _environment_ by selecting actions. This scenario is typically formulated
as a _Markov decision process_ (MDP), puterman .

######  Definition 1 (Markov decision process).

MDP is a tuple

$(\textbf{S},\textbf{A},T,R,\gamma),$ (1)

where S is a set of environment states, A is a set of actions,
$T(s_{t+1},s_{t},a_{t})$ is a _transition function_ from state
$s_{t}\in\textbf{{S}}$ to state $s_{t+1}\in\textbf{{S}}$ after taking action
$a_{t}\in\textbf{{A}}$; $R(s_{t+1},a_{t},s_{t})$ is a function that gives
_reward_ received by the transition
$(s_{t},a_{t})\rightarrow s_{t+1}$; and $\gamma\in[0,1]$ is a _discount
factor_.

The set $\textbf{S}\times\textbf{A}$ will be referred to as a _task domain_.

At each time step $t$, the agent observes environment state
$s_{t}\in\textbf{{S}}$ and takes action $a_{t}\in\textbf{{A}}$. Then the
environment transits to a new state $s_{t+1}$333observable by the agent and
the agent receives reward $r_{t}=R(s_{t+1},a_{t},s_{t})$. The _goal_ of the
agent is to learn policy $\pi:\textbf{S}\mapsto\textbf{A}$ that maximizes the
accumulated rewards. The solution of a MDP task is the _optimal_ $\pi^{*}$
such that:

$\pi^{*}=\underset{\pi\in\boldsymbol{\pi}}{\mathrm{\arg\max}}\hskip
1.9919ptE\left[\sum_{t=1}^{N}\gamma^{t}R(s_{t+1},\pi(s_{t}),s_{t})\right],$
(2)

where $N\in\mathbb{N}\cup\\{\infty\\}$ is decision horizon.

RL aims to train the agent to act optimally within MDP when either transition
function or reward (or both) are unknown. Reasonable RL algorithm modifies
$\pi$ over time to gradually get it closer to an optimal policy. Majority of
RL approaches are based on dynamic programming puterman  and use $Q$-function
(aka _action-value function_)

$Q(s,a)=E_{\pi^{*}}\left[\sum_{t=1}^{N}\gamma^{t}R_{t}(s_{t+1},\pi^{*}(s_{t}),s_{t})\mid
s_{1}=s,a_{1}=a\right].$ (3)

$Q$-function measures the expected future discounted reward the agent can
obtain by taking an action in a given state.

Algorithm 1 Q-learning

1:initial Q-function $Q_{0}(s,a)$, learning rate $\alpha\in(0,1)$, discount
factor $\gamma\in(0,1)$

2:get the initial state $s_{0}$

3:for $t=1,2,...,$ till convergence do

4: select action $a_{t}=\underset{a\in\textbf{{A}}}{\mathrm{argmax}}\hskip
1.9919ptQ(s_{t},a)$

5: get next state $s_{t+1}$ and reward $r_{t}$

6: if $s_{t}$ is terminal then

7: target $=r_{t}$

8: else

9: target $=r_{t}+\gamma\hskip
1.9919pt\underset{a\in\textbf{{A}}}{\mathrm{max}}Q(s_{t+1},a)$

10: end if

11: $Q_{new}(s_{t},a_{t})=(1-\alpha)Q(s_{t},a_{t})+\alpha($target$)$

12:end for

13:Q-function $Q(s,a)$

Equation (3) evaluates policy $\pi^{*}$, i.e. gives the expected value of
future discounted reward over the states induced by $\pi^{*}$ for given
starting state $s$ and action $a$.

The estimate of $Q$-function (3) for $s$ and $a$, denoted as $\hat{Q}(s,a)$,
can be gradually learnt on the stream of data records
$(s_{t},a_{t},r_{t},s_{t+1})$ using for instance temporal difference learning,
qlearning :

$\hat{Q}_{t+1}(s_{t},a_{t})=(1-\alpha)\hat{Q}_{t}(s_{t},a_{t})+\alpha(r_{t}+\gamma\hskip
1.9919pt\underset{a\in\textbf{{A}}}{\mathrm{max}}\hskip
1.9919pt\hat{Q}_{t}(s_{t+1},a)),$ (4)

where $\alpha\in(0,1)$ is a parameter called _learning rate_ ,
$\gamma\in(0,1)$ is a _discount factor_ and $r_{t}=R(s_{t+1},a_{t},s_{t})$.
The learning starts with initial estimate of the $Q$-function
$\hat{Q}_{0}(s,a)$. The learned optimal decision rule is then

$\pi^{*}\left(s\mid\hat{Q}\right)=\underset{a\in\textbf{{A}}}{\mathrm{argmax}}\hskip
1.9919pt\hat{Q}(s,a).$ (5)

###  2.2 Deep Q-learning

Whenever the state space is huge, for instance when state is given by video
frame, efficient learning of $Q$-function calls for approximation. The state-
of-the-art in function approximation points deep neural networks as a suitable
methodology, nn . Deep $Q$-Networks (DQN) use a standard off-policy
$Q$-learning algorithm qlearning  (see Algorithm 1) and deep neural networks
to estimate the $Q$-function (3).

DQN approximates $Q$-function by a deep neural network with parameters, which
can be trained similarly to the supervised learning, dqn . However, in the
supervised learning, i. i. d. input data are assumed. Moreover, output values
are expected to be the same for the same inputs, deepsupervisedlearning .
Neither of these assumptions is met in RL tasks. The consecutive states (e.g.
video frames) are usually highly correlated and thus very far from being i. i.
d. Also, output values contain learned $Q$-function which evolves during the
learning. This makes the learning process unstable. To enable data reuse and
stabilize the learning, DQN uses an experience replay technique to remove
correlations in the observed sequences, and employs additional target network
to stabilize the output values, see dqn  for details.

Experience replay considers that the last $n_{M}$ data records (so-called
_experience memory_ , denoted as M) are stored in a memory buffer. At each
learning step, mini-batch of data of length $n_{B}\in\mathbb{N}$ is randomly
sampled from the memory buffer and used for updating the neural network that
approximates $Q$-function. This makes the learning data closer to being i. i.
d.

Target network is an additional network serving for stabilizing the learning.
The idea is as follows. The parameters of the _original network_ are updated
at every learning step, while _target network_ is used to retrieve output
values and stays static, i.e. its parameters do not change in every learning
step. Every $n_{U}\in\mathbb{N}$ steps, parameters of the original network are
copied to the parameters of the target network, i.e. the original and target
networks are synchronised.

The DQN algorithm is summarized in Algorithm 2. $\theta$ denotes parameters of
the original network and $\theta^{T}$ are parameters of the target network.

Algorithm 2 DQN

1:initial parameters $\theta$ of the Q-function $Q(s,a,\theta)$, learning rate
$\alpha$, discount factor $\gamma$, exploration rate $\epsilon$, size of the
experience memory $n_{M}$, size of the learning mini-batch $n_{B}$, number of
steps for target network synchronization $n_{U}$

2:Initialize the experience memory to the size $n_{M}$

3:Set the parameters $\theta^{T}$ of the target network as $\theta^{T}=\theta$

4:for $t=1,2,...,$ till convergence do

5: With probability $\epsilon$ perform random action $a_{t}$ otherwise select
action $a_{t}=\underset{a\in\textbf{{A}}}{\mathrm{argmax}}\hskip
1.9919ptQ(s,a\mid\theta)$

6: Get next state $s_{t+1}$ and receive reward $r_{t}$

7: Check if the memory is full. If so, remove the oldest data record

8: Store $(s_{t},a_{t},r_{t},s_{t+1})$ in the experience memory M

9: Sample random mini-batch of size $n_{B}$ $(s_{j},a_{j},r_{j},s_{j+1})_{j\in
Random(n_{B})}\in\textbf{M}$

10: for every $j$ do

11: if $s_{j+1}$ is terminal then

12: $\text{target}_{j}=r_{j}$

13: else

14: $\text{target}_{j}=r_{j}+\gamma\hskip
1.9919pt\underset{a^{\prime}\in\textbf{{A}}}{\mathrm{max}}\hskip
1.9919ptQ(s_{j+1},a^{\prime}\mid\theta^{T})$

15: end if

16: end for

17: Perform a gradient descent step on
$\left(\left(\text{target}_{j}-Q(s_{j},a_{j}\mid\theta)\right)^{2}\right)_{j\in
Random(n_{B})}$ with Huber loss, huber , with respect to parameters $\theta$

18: Every $n_{U}$ steps set $\theta^{T}=\theta$

19:end for

20:Q-function $Q(s,a)$, experience memory M

###  2.3 Problem description

Assume two RL tasks on the same or similar environment. Generally the tasks
are related in some way, while may differ in state/action spaces, reward
function, environment dynamics, prior knowledge, etc. It is expected that
experience gained during solving one RL task will help to improve learning
performance in another RL task.

Let us consider one to be a source task with the domain
$\textbf{S}_{S}\times\textbf{A}_{S}$ and another be a target task with the
domain $\textbf{S}_{T}\times\textbf{A}_{T}$.

Our aim is to improve the results of target task learning via transfer of
knowledge from the source task. Clearly the success of the transfer will
depend on how similar these tasks are. Section 3 introduces correspondence
function that reflects their similarity and proposes how to learn it.

##  3 Transfer learning: the correspondence function

###  3.1 Problem definition

Consider a _source task_ and similar, but somehow different, _target task_.
The source task has been solved in the past by DQN algorithm (see Section
2.2). The obtained knowledge $\textbf{K}_{S}=(Q_{S},\textbf{M}_{S})$ consists
of learned $Q$-function $Q_{S}$ and experience memory
$\textbf{M}_{S}=\left(\left(s_{t},a_{t},s_{t+1},r_{t}\right)_{i=1}^{n_{M}}\right)$.

Now the agent needs to solve the target task. First the agent can apply a
random decision rule to the target task and collect experience memory
$\textbf{M}_{T}$. Our aim is to use knowledge $\textbf{K}_{S}$ gained by
solving the source task to leverage learning the target task.

Let there is a correspondence function $\mathcal{C}$ that establishes a
pairwise connection between similar instances of the target and source tasks.
It implies that for each state-action pair $(s_{T},a_{T})$ from the target
task function $\mathcal{C}$ gives a corresponding state-action pair
$(s_{S},a_{S})$ from the source task, i.e.
$(s_{S},a_{S})=\mathcal{C}(s_{T},a_{T})$. Intuitively: these pairs should
represent the same or very similar "agent-environment" state.444For example
pulling break levers ($a_{T}$) when seeing an animal on the road ($s_{T}$)
while driving a motorcycle corresponds to pressing the brake pedal ($a_{S}$)
when seeing a person crossing the road ($s_{S}$) while driving a car. Then
$Q$-function $Q_{T}$ of the target task can be written using $Q_{S}$ and
correspondence function $\mathcal{C}$ as follows:

$Q_{T}(s_{T},a_{T})=Q_{S}(\mathcal{C}(s_{T},a_{T})),\hskip
8.5359pt\forall(s_{T},a_{T})\in\left(\textbf{S}_{T}\times\textbf{A}_{T}\right).$
(6)

Hence the correspondence function can be used to transform state-action pairs
from the target task to the source task. Thus the agent can use $Q$-function
$Q_{S}$ from the source task to choose the optimal policy of the target task.
It is assumed that $Q$-function $Q_{T}$ (6) solves (at least partially) the
target task. Note that the correspondence function can be unknown to the
agent. The next section describes how to learn the it.

###  3.2 Correspondence function learning

Let us assume (for brevity) that the action spaces of source and target task
are identical, i.e. $\textbf{A}_{S}=\textbf{A}_{T}$, and that the mutually
corresponding actions are found by an identity mapping independently of the
current state. Thus, we only need to learn a mapping that finds the
corresponding states, i.e. correspondence function for states.

The proposed learning is inspired by CycleGAN, cyclegan , and extends it by
employing the environment dynamics and the optimal strategy for the
correspondence function learning. The approach is applicable for knowledge
transfer between the related RL tasks.

CycleGAN is based on GAN555Generative adversarial network, gan , and was
developed for learning image-to-image transformations in an unsupervised way.
The similar situation occurs here - there are two experience memories
$\textbf{M}_{S}$ and $\textbf{M}_{T}$ with mutually unpaired entries. The task
is to learn the correspondence function $\mathcal{C}$ that will help to match
them.

In CycleGAN, two mappings $G_{S}$ and $G_{T}$ called _generators_ are learnt,

$G_{S}:\hskip 2.84544pt\textbf{S}_{S}\rightarrow\textbf{S}_{T}\hskip
8.5359pt\text{and}\hskip 8.5359ptG_{T}:\hskip
2.84544pt\textbf{S}_{T}\rightarrow\textbf{S}_{S}.$ (7)

They are learnt as two GANs, thus generators $G_{S}$ and $G_{T}$ are learned
simultaneously with respective discriminators $D_{S}$ and $D_{T}$. The
generators learn to map the states from $\textbf{S}_{S}$ to $\textbf{S}_{T}$
and vice-versa, while the discriminators learn to _distinguish_ a real state
from a state mapped by a generator.

Mappings $G_{S}$, $G_{T}$, $D_{S}$ and $D_{T}$ are constructed as neural
networks with architecture depending on the data format, for instance for
image states convolutional layers are often used. Learning in CycleGAN
minimizes a loss consisting of two parts. The first part of the loss,
$\mathcal{L}_{GAN}$, comes from GAN and it is given by the adversarial loss
for generators $G_{S}$, $G_{T}$ and their respective discriminators $D_{S}$,
$D_{T}$ as follows:

$\displaystyle\mathcal{L}_{GAN}=\hskip 0.85355pt$ $\displaystyle
E_{p_{s_{S}}}\left[\mathrm{log}\hskip
0.85355ptD_{S}(s_{S})\right]+E_{p_{s_{T}}}\left[\mathrm{log}\left(1-D_{S}\left(G_{S}\left(s_{T}\right)\right)\right)\right]$
$\displaystyle+E_{p_{s_{T}}}\left[\mathrm{log}\hskip
0.85355ptD_{T}(s_{T})\right]+E_{p_{s_{S}}}\left[\mathrm{log}\left(1-D_{T}\left(G_{T}\left(s_{S}\right)\right)\right)\right],$
(8)

The adversarial training can learn mappings $G_{S}$ and $G_{T}$ (7) to produce
outputs undistinguishable from respective sets $\textbf{S}_{S}$ and
$\textbf{S}_{T}$. However, a network can learn to map the same set of input
images to any random permutation of images in the target domain while
minimizing the adversarial loss $\mathcal{L}_{GAN}$ at the same time. This
part of the loss alone thus does not guarantee finding the desired
correspondence function even if its global minimum is found.

To further reduce the search space of possible mappings $G_{S}$ and $G_{T}$,
CycleGAN introduces a so-called cycle-consistency requirement: every state
$s_{S}\in\textbf{S}_{S}$ must be recoverable after mapping it to
$\textbf{S}_{T}$, i.e. $G_{T}(G_{S}(s_{S}))\approx s_{S}$. The same
requirement applies to every state $s_{T}\in\textbf{S}_{T}$. This cycle-
consistency requirement can be expressed by minimising the second part of the
loss, $\mathcal{L}_{Cyc}$:

$\displaystyle\mathcal{L}_{Cyc}=\hskip 0.85355pt$ $\displaystyle
E_{p_{s_{S}}}\left[\left\lVert
G_{T}\left(G_{S}\left(s_{S}\right)\right)-s_{S}\right\rVert_{1}\right]+E_{p_{s_{T}}}\left[\left\lVert
G_{S}\left(G_{T}\left(s_{T}\right)\right)-s_{T}\right\rVert_{1}\right]$ (9)

(9) is referred to as cycle-consistency loss, cyclegan :

####  3.2.1 Relation to RL

Even direct application of CycleGAN to the states brought some success in
policy transfer, see for instance Gamrian . However, data records in
experience memories comprise additional (richer) yet unused information that
may be helpful for transfer of knowledge between tasks.

Respecting this information by including additional components into the loss
function that is minimized in CycleGAN learning will make the learned
correspondence even more relevant to RL tasks.

This work proposes to add other two components of the loss, that reflect

  * •

$Q$-function learned from the source task,

  * •

Environment model of the source task that reflect source task dynamics.

####  3.2.2 $\boldsymbol{Q}$ loss

The available $Q$-function, $Q_{S}$, should be incorporated in learning of the
correspondence $\mathcal{C}$ as it determines the optimal policy. The _cycle-
consistency_ loss (9) forces generators $G_{S}$ and $G_{T}$ to be inverses of
each other. By requiring that, the values of $Q$-function of state $s_{S}$ and
the state mapped to the other domain and back, $G_{T}(G_{S}(s_{S}))$, are the
same. So, it is assumed that the learning of the generators will focus on the
states that are relevant for the $Q$-function. Thus we propose to introduce
the loss $\mathcal{L}_{Q}$ associated with the $Q$-function in the following
form:

$\displaystyle\mathcal{L}_{Q}=\hskip 0.85355pt$ $\displaystyle
E_{p_{s_{S}}}\left[\left\lVert
Q_{S}\left(G_{T}\left(G_{S}\left(s_{S}\right)\right)\right)-Q_{S}\left(s_{S}\right)\right\rVert_{1}\right]$
(10)

The loss (10) will make the learned correspondence more suitable for
transferring knowledge between RL tasks as it indirectly indicates similarity
of the tasks goals.

####  3.2.3 Model loss

So far, all considered losses (8) - (10) are associated with state values.
However, any RL task is dynamic and the time sequence of states is important.
Consider states of the target and the source tasks at times $t$ and $t+1$. If
generator $G_{T}$ ensures mapping $s_{T\mathchar 24635\relax\;t}$ on
$s_{S\mathchar 24635\relax\;t+1}$ and generator $G_{S}$ maps $s_{S\mathchar
24635\relax\;t+1}$ back to $s_{T\mathchar 24635\relax\;t}$, then losses
$\mathcal{L}_{GAN},\mathcal{L}_{Cyc}$ and $\mathcal{L}_{Q}$ (8) - (10) are
minimal. However, it would not be of help for solving the target RL task.

Therefore, to reflect the underlying dynamics, it is also vital to consider
the loss respecting environment model $F$ of the source task:

$\displaystyle\mathcal{L}_{M}=\hskip 0.85355pt$ $\displaystyle
E_{p_{s_{T\mathchar 24635\relax\;t}a_{T\mathchar 24635\relax\;t}s_{T\mathchar
24635\relax\;t+1}}}\left[\left\lVert F\left(G_{T}\left(s_{T\mathchar
24635\relax\;t}\right),a_{T\mathchar
24635\relax\;t}\right)-G_{T}\left(s_{T\mathchar
24635\relax\;t+1}\right)\right\rVert_{1}\right]$ (11)

The correspondence function minimising (11) will respect the environment
dynamics. In particular, whenever action $a_{T\mathchar 24635\relax\;t}$ is
applied at state $s_{T\mathchar 24635\relax\;t}$ resulting in state
$s_{T\mathchar 24635\relax\;t+1}$, the triplet $(\bar{s}_{S\mathchar
24635\relax\;t},a_{T\mathchar 24635\relax\;t},\bar{s}_{S\mathchar
24635\relax\;t+1})$ should be observable in the source task, where
$\bar{s}_{S\mathchar 24635\relax\;t}=G_{T}\left(s_{T\mathchar
24635\relax\;t}\right)$ and $\bar{s}_{S\mathchar
24635\relax\;t+1}=G_{T}\left(s_{T\mathchar 24635\relax\;t+1}\right)$.

####  3.2.4 Final loss

The proposed loss comprises all the components (8), (9), (10) and (11) and has
the following form:

$\displaystyle\mathcal{L}=\hskip
0.85355pt\mathcal{L}_{GAN}+\lambda_{Cyc}\mathcal{L}_{Cyc}+\lambda_{Q}\mathcal{L}_{Q}+\lambda_{M}\mathcal{L}_{M},$
(12)

where $\lambda_{Cyc}$, $\lambda_{Q}$ and $\lambda_{M}$ are _loss parameters_
that define the relative influence (weight) of the respective component of the
loss.

The searched correspondence function $\mathcal{C}$ is then obtained as
follows:

$\mathcal{C}(s_{T},a_{T})=(G_{T}(s_{T}),I(a_{T})),\hskip
28.45274pt\forall(s_{T},a_{T})\in\textbf{S}_{T}\times\textbf{A}_{T},$ (13)

where $G_{T}$ is the generator from (7) mapping states from the _target task_
to states from the _source task_ and $I$ is an identity mapping.

##  4 Experimental part

To test efficiency of the proposed approach, two experiments on Atari game
Pong, atari , were conducted. Performance of the approach was evaluated based
on an average accumulated reward per game. GAN and CycleGAN were used as
baseline methods.

###  4.1 Domain description

Pong is a two-dimensional game simulating table tennis. There are 6 available
actions (do nothing, fire, move up, move down, move up fast, move down fast)
and the last 4 image frames served as a state. The agent learned to play the
game using the DQN algorithm from Section 2.2 and, thus, learned the
$Q$-function. To test the approach described in Section 3.2, the agent also
learned environment model $F$, see Section 3.2.1.

###  4.2 Experiment description

The proposed method for TL was tested in two experiments.

Experiment 1: The source and target tasks are the same, i.e. game Pong
(screenshot is shown on Fig. 1). The main aim of this experiment was to use
the proposed approach to find the identity transformation.

Experiment 2: The source task was the original Pong while the target task was
rotated Pong (see screenshot at Fig. 2). The game remained the same, but all
of the image frames were all rotated by 90 degrees.

The states in the experiments were the last 4 frames (images) of the game.

Each experiment included the following steps:

  1. 1)

The agent played the _source task_ (standard Pong), learned the optimal policy
by DQN and obtained the optimal $Q$-function $Q_{S}$, environment model $F$
and experience memory $\textbf{M}_{S}$ that consists of 10 000 data entries
collected at the end of the game.

  2. 2)

The agent played the _target task_ (standard Pong in Experiment 1, rotated
Pong in Experiment 2) using random actions and obtained experience memory
$\textbf{M}_{T}$ consisting of 10 000 data entries.

  3. 3)

The agent started learning the correspondence function $\mathcal{C}$ from the
_target task_ to the _source task_ using the method from Section 3 with the
$Q$-function $Q_{S}$, environment model $F$ and experience memories
$\textbf{M}_{S}$ and $\textbf{M}_{T}$,

  4. 4)

For every 1000 steps of correspondence function learning:

     * •

The agent played 5 games of the _target task_ using the correspondence
function that was being learned and the transformed $Q$-function from the
_source task_ $Q_{S}(\mathcal{C}(.,.))$ (6)

     * •

Average accumulated reward per game was computed.

  5. 5)

The agent played the _target task_ while using the learned
correspondence666the correspondence that achieved the best average accumulated
reward per game in the previous step was used here and $Q$-function
transferred from the _source_ task (while continuously fine-tuning the
$Q$-function with fixed correspondence function using DQN).

The _key metric_ to evaluate the success of the knowledge transfer was the
average accumulated reward per game while playing the _target task_ with the
transformed $Q_{S}(\mathcal{C}(.,.))$ with learned correspondence function
$\mathcal{C}$.

##### Baselines

The results are compared with two baselines - using GAN and CycleGAN methods,
gan , cyclegan , which brought some success for knowledge transfer in similar
settings, Gamrian .

Easy to see that GAN (CycleGAN) can be obtained from (12) for the loss
parameters set to the following values:

  * •

$\lambda_{Q}=\lambda_{M}=\lambda_{Cyc}=0$ (for GAN),

  * •

$\lambda_{Q}=\lambda_{M}=0$ (for CycleGAN).

Thus the proposed method is more general and includes baselines as particular
cases.

Figure 1: Standard Pong, atari  Figure 2: Rotated Pong by 90 degrees, atari

###  4.3 Experiment 1

This experiment tested transfer learning when _source_ and _target_ tasks are
the same.

The generators $G_{S}$ and $G_{T}$ (see Section 3.2) in this experiment were
constructed as neural networks with convolutional layers. Their specific
architecture was taken from generator_arch_resnet . The discriminators $D_{S}$
and $D_{T}$ were also constructed as neural networks with convolutional layers
with the same architecture as in discriminator_arch_resnet .

The parameters of all of the networks were initialized from Gaussian
distribution $N(0,0.02)$. The approach was tested for all the combinations of
the parameters: $\lambda_{Cyc}\in\\{0,1,10\\}$, $\lambda_{Q}\in\\{0,1\\}$ and
$\lambda_{M}\in\\{0,1,10\\}$ (12).

a) GAN
$\lambda_{Cyc}=\lambda_{Q}=\lambda_{M}=0$

b) CycleGAN
$\lambda_{Cyc}=10,\lambda_{Q}=\lambda_{M}=0$

c) $\lambda_{Cyc}=0$, $\lambda_{Q}=1$, $\lambda_{M}=0$

d) $\lambda_{Cyc}=0$, $\lambda_{Q}=0$, $\lambda_{M}=10$

e) $\lambda_{Cyc}=1$, $\lambda_{Q}=1$, $\lambda_{M}=1$

Figure 3:  Experiment 1: Average accumulated reward per game when playing 5
games with the transformed $Q$-function (6). The reward is computed each 1000
steps of correspondence function learning. The results are shown for different
values of loss parameters $\lambda_{Cyc}$, $\lambda_{Q}$ and $\lambda_{M}$.
Fig. 3a) and 3b) show the baselines using _GAN_ and _CycleGAN_ methods.

The results are presented in Fig. 3 \- Fig. 5.

Fig. 3 shows the average reward gained in 5 games that the agent played each
1000 steps of the learning of correspondence function for different
$\lambda_{Cyc}$, $\lambda_{Q}$ and $\lambda_{M}$. The maximum possible reward
in one game was 21 and Fig. 3 shows that the best results were achieved with
settings involving all components of the loss
$\lambda_{Cyc}=\lambda_{Q}=\lambda_{M}=1$. _GAN baseline_ (Fig. 3a) did not
produce good result. _CycleGAN_ performance (Fig. 3b) soon became unstable
though it provided good reward at the beginning.

Fig. 4 shows the progress correspondence function learning. It confirms that
the best results were achieved with all the loss components active,
$\lambda_{Cyc}=\lambda_{Q}=\lambda_{M}=1$ (12), while visually good results
were achieved also for the CycleGAN baseline with $\lambda_{Cyc}=10$,
$\lambda_{Q}=\lambda_{M}=0$ (although as Fig. 3 showed, the learned
correspondence function oscillated between good and a useless one). In other
settings no useful correspondence function has been learned.

Lastly, Fig. 5 compares the performance of the agent learning to play _Pong_
from scratch and the agent that learned the correspondence function and
employed the previously learned $Q$-function. Fig. 5 displays a moving average
of accumulated reward per game computed from the last 20 games. As Fig. 5
shows, when the target and source tasks are the same, the agent was able to
fully re-use the previously acquired knowledge.

a) GAN $\lambda_{Cyc}=0$, $\lambda_{Q}=0$, $\lambda_{M}=0$

b) CycleGAN $\lambda_{Cyc}=10,\lambda_{Q}=0$, $\lambda_{M}=0$

c) $\lambda_{Cyc}=0$, $\lambda_{Q}=1$, $\lambda_{M}=0$

d) $\lambda_{Cyc}=0,\lambda_{Q}=0$, $\lambda_{M}=10$

e) $\lambda_{Cyc}=1,\lambda_{Q}=1$, $\lambda_{M}=1$

Figure 4: Experiment 1: Progress of learning the correspondence function
$\mathcal{C}$ (13) after 0, 20000, 40000 and 60000 steps). The results are
shown for different values of loss parameters $\lambda_{Cyc}$, $\lambda_{Q}$
and $\lambda_{M}$ (12). The left parts of the pictures are frames serving as
states of the _target_ task, the right parts are the same states mapped by the
learned correspondence function, $\mathcal{C}$. Figure 5:  Moving average of
reward per game computed from the last 20 games depending on the number of
played Pong games. Orange line denotes the case hen the agent learned from
scratch, i.e. no TL was used. Blue line denotes the case with TL, i.e. when
the agent learned the correspondence function and used the transformed
$Q$-function (6). The agent was continuously learning the $Q$-function in both
cases.

###  4.4 Experiment 2

In this Experiment, the _target task_ is rotated Pong, i.e. the original Pong
with image frames rotated by 90 degrees (see Fig. 2).

Generators $G_{S}$ and $G_{T}$ (7) (see Section 3.2) were constructed as
neural networks. Two types of generators were tested in this experiment. The
architecture of the first one, referred to here as resnet generator, was taken
from generator_arch_resnet  and followed by a rotation layer as in spatial-
transformer . The second type, referred to as rotation generator, was composed
of the mentioned rotation layer only.

Discriminators $D_{S}$ and $D_{T}$ were constructed, same as in the Experiment
1 by neural networks with convolutional layers with the same architecture as
in discriminator_arch_resnet .

The method was tested for various settings of the loss parameters
$\lambda_{Cyc}$, $\lambda_{Q}$ and $\lambda_{M}$ (12). Settings providing the
best performance are presented (Fig. 6 \- Fig. 8) together with GAN and
CycleGAN baselines.

Fig. 6 shows the average reward gained in 5 games that the agent played each
1000 steps of the learning of correspondence function for both types of the
generators and for the best settings of $\lambda_{Cyc}$, $\lambda_{Q}$ and
$\lambda_{M}$. The maximum possible reward in one game was 21 and Fig. 7 shows
that the perfect correspondence was found with rotation generator. Learning of
correspondence function with resnet generator showed also some success and the
gained correspondence was subsequently used for fine-tuning the $Q$-function.
At the same time, the GAN and CycleGAN baselines did not achieve reasonable
result. Fig. 6a) and Fig. 6b) show that.

Fig. 7 shows the progress in correspondence function learning. It shows the
successful learning in the case of rotation generator (Fig. 7a), however the
correspondence learned in the case of resnet generator (Fig. 7b) also looked
visually close to the optimal one (especially after 50 000 steps).

Lastly, Fig. 8 shows agent’s performance when playing the _target task_ \-
rotated Pong. Fig. 8 displays a moving average of accumulated reward per game
computed from the last 20 games. Fig. 8 compares the agent’s results in the
target task in the following cases:

  1. 1.

When the agent played the _target task_ from scratch.

  2. 2.

The agent learned the correspondence function $\mathcal{C}$ using resnet
generator.

  3. 3.

The agent learned the correspondence function using rotation generator.

  4. 4.

The agent re-used the $Q$-function from the source task without respecting the
correspondence.

As Fig. 8 shows, the agent with rotation generator was able to fully reuse the
previously gained knowledge. Use of resnet generator gave significantly better
performance at the beginning compared with learning from scratch. Re-use of
the $Q$-function gave worse results even compare to the case of learning from
scratch.

a) Rotation generator
using GAN

a) Rotation generator
using CycleGAN

a) Rotation generator
with $\lambda_{Cyc}=\lambda_{Q}=0$, $\lambda_{M}=10$

b) Resnet generator
using GAN

b) Resnet generator
using CycleGAN

b) Resnet generator
with $\lambda_{Cyc}=1,\lambda_{Q}=0$, $\lambda_{M}=1$

Figure 6:  Average accumulated reward in 5 games when playing Rotated Pong
with the transformed $Q$-function (6). The reward is computed each 1000 steps
of the learning of correspondence function. The results are shown for rotation
and resnet generator with the best settings of the loss parameters in each
case as well as with using GAN and CycleGAN baselines.

a) Rotation generator using GAN

b) Rotation generator using CycleGAN

c) Rotation generator with $\lambda_{Cyc}=0$, $\lambda_{Q}=0$,
$\lambda_{M}=10$

d) Resnet generator using GAN

e) Resnet generator using CycleGAN

f) Resnet generator with $\lambda_{Cyc}=1,\lambda_{Q}=0$, $\lambda_{M}=1$

Figure 7: Progress in learning the correspondence function $\mathcal{C}$ (13).
The results are shown for rotation and resnet generators with the best
settings of the loss parameters $\lambda_{Cyc}$, $\lambda_{Q}$ and
$\lambda_{M}$ (12) as well as with using GAN and CycleGAN baselines. The left
parts of the pictures are frames representing the states of the _target_ task,
the right parts are the same states transformed by the correspondence function
$\mathcal{C}$.  Figure 8:  Moving average of reward per game computed from the
last 20 games depending on the number of played games for the game rotated
Pong for four different agents - an agent learning the game from scratch (blue
line), an agent using the correspondence function learned with resnet
generator (orange line), an agent using the correspondence function learned
with rotation generator (green line) and an agent reusing only the
$Q$-function without any correspondence function (red line). The agents were
continuously learning the $Q$-function.

##  5 Conclusion

The paper considers transfer learning in RL tasks. The work establishes a
correspondence function indicating similarity of the _source_ and _target
task_. The work proposes a novel correspondence function that respects RL
elements like $Q$-function and transfers model of the source task into
learning. It gives the agent an ability to gradually create a set of skills,
adapt and use them while interacting with the dynamically changing
environment.

The method was evaluated on simulated experiments involving 2-D Atari game
Pong and compared against two baselines using GAN and CycleGAN method.
Experiment 1 with identical source and target tasks served verification of the
proposed method. It was shown that the method succeeded and learned the tasks
correspondence. It also illustrated importance of all proposed loss components
as TL was the most successful when all the components were respected.
Comparing to baselines shows that using the loss relevant only to images, TL
was either unsuccessful (using only GAN loss) or unstable (using GAN and
Cycle-consistency loss).

In Experiment 2, the target task was different from the source task. Two types
of network architecture for generators in the learning of correspondence
function were tested. It was shown that choice of the architecture is crucial
for achieving good results. The generator composed of rotation and
convolutional layers yielded worse performance (but still usable) compare to
the generator with only rotation layer. It shows that the convolutional layers
traditionally used in image processing may not be good for this scenario. Thus
other architectures like transformers, transformers , should be investigated.

The proposed method was successful to learn the mapping between the states
from the _source task_ and the states from the _target task_ in both of the
experiments. Subsequently, TL was successful and the agent was able to play
the target task very well from the very beginning.

Many open problems remain unsolved, in particular: i) transfer learning for
tasks having low similarity, ii) using several source tasks, iii) choosing
only relevant source tasks similarly to evaluatingsourcetasks , iv) find a
better network architecture for the correspondence function learning, v)
testing the method on a real-world example.

Data availability statement: The datasets generated during and/or analysed
during the current study are available from the corresponding author on
reasonable request.

Ethical approval: This article does not contain any studies with human
participants performed by any of the authors.

## References

  * (1) Haitham Bou Ammar, Eric Eaton, José Marcio Luna, and Paul Ruvolo.  Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning.  In Twenty-fourth international joint conference on artificial intelligence, 2015.
  * (2) Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh.  Recycle-gan: Unsupervised video retargeting.  In Proceedings of the European conference on computer vision (ECCV), pages 119–135, 2018.
  * (3) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  Journal of Artificial Intelligence Research, 47:253–279, 2013.
  * (4) Bryan Chen, Alexander Sax, Gene Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto.  Robust policies via mid-level visual representations: An experimental study in manipulation and navigation.  arXiv preprint arXiv:2011.06698, 2020.
  * (5) Jeff Clune.  Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence.  arXiv preprint arXiv:1905.10985, 2019.
  * (6) André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P Hasselt and David Silver  Successor features for transfer in reinforcement learning.  Advances in neural information processing systems, 30, 2017
  * (7) Balázs Csanád Csáji et al.  Approximation with artificial neural networks.  Faculty of Sciences, Etvs Lornd University, Hungary, 24(48):7, 2001\.
  * (8) Shani Gamrian and Yoav Goldberg.  Transfer learning for related reinforcement learning tasks via image-to-image translation.  In International Conference on Machine Learning, pages 2063–2072. PMLR, 2019.
  * (9) Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * (10) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * (11) Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine.  Learning invariant feature spaces to transfer skills with reinforcement learning.  arXiv preprint arXiv:1703.02949, 2017.
  * (12) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In International conference on machine learning, pages 2555–2565. PMLR, 2019.
  * (13) Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai.  Retinagan: An object-aware approach to sim-to-real transfer.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10920–10926. IEEE, 2021.
  * (14) Peter J Huber.  Robust estimation of a location parameter.  In Breakthroughs in statistics, pages 492–518. Springer, 1992.
  * (15) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * (16) Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * (17) Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis.  Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12627–12637, 2019.
  * (18) Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * (19) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * (20) Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  Advances in neural information processing systems, 30, 2017.
  * (21) A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra.  Benchmarking reinforcement learning algorithms on real-world robots.  In Conference on robot learning, pages 561–591. PMLR, 2018.
  * (22) Andrew N Meltzoff.  Understanding the intentions of others: re-enactment of intended acts by 18-month-old children.  Developmental psychology, 31(5):838, 1995.
  * (23) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * (24) Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * (25) Martin L Puterman.  Markov decision processes.  Handbooks in operations research and management science, 2:331–434, 1990.
  * (26) Xinghua Qu, Zhu Sun, Yew-Soon Ong, Abhishek Gupta, and Pengfei Wei.  Minimalistic attacks: How little it takes to fool deep reinforcement learning policies.  IEEE Transactions on Cognitive and Developmental Systems, 13(4):806–817, 2020.
  * (27) Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari.  Rl-cyclegan: Reinforcement learning aware simulation-to-real.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11157–11166, 2020.
  * (28) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * (29) Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.  One pixel attack for fooling deep neural networks.  IEEE Transactions on Evolutionary Computation, 23(5):828–841, 2019\.
  * (30) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * (31) Christopher JCH Watkins and Peter Dayan.  Q-learning.  Machine learning, 8(3):279–292, 1992.
  * (32) Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang.  Learning cross-domain correspondence for control with dynamics cycle-consistency.  arXiv preprint arXiv:2012.09811, 2020.
  * (33) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * (34) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * (35) Sebastian Ruder.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1609.04747, 2016.
  * (36) Yueyue Hu, Shiliang Sun, Xin Xu and Jing Zhao.  International Journal of Machine Learning and Cybernetics.  International Journal of Machine Learning and Cybernetics, 11(11), 2020, pages 2461–2474.
  * (37) Marzieh Davoodabadi Farahani and Nasser Mozayani.  Evaluating skills in hierarchical reinforcement learning.  International Journal of Machine Learning and Cybernetics, 11(10), 2020, pages 2407–2420

##  6 Appendix - implementation details

###  6.1 Network architectures

##### CycleGAN generator architectures

The architectures of generators $G_{S}$ and $G_{T}$ in the experiment 1
(Section 4.3) and the resnet generators $G_{S}$ and $G_{T}$ in the experiment
2 (Section 4.4) were taken from cyclegan . The 9 residual blocks version was
used. Below, we follow the naming convention used in the mentioned work.

Let c7s1-f denote a $7\times 7$ Convolution-BatchNorm-ReLU layer with $f$
filters and stride 1. df denotes a $3\times 3$ Convolution-BatchNorm-ReLU
layer with $f$ filters and stride 2. Reflection padding was used to reduce
artefacts. Rf denotes a residual block that contains two $3\times 3$ con-
volutional layers with the same number of filters ($f$) on both layers. uf
denotes a $3\times 3$ fractional-strided-Convolution- BatchNorm-ReLU layer
with $f$ filters and stride 1 2.

The network architecture consisted of:

c7s1-64,d128,d256,R256,R256,R256,R256,R256,R256,R256,R256,
R256,u128,u64,c7s1-3

##### Discriminator architectures

For discriminator networks $D_{S}$ and $D_{T}$ in all the experiments,
$70\times 70$ PatchGAN was used, discriminator_arch_resnet . Let Cf denote a
$4\times 4$ Convolution-BatchNorm-LeakyReLU layer with $f$ filters and stride
2. After the last layer, a convolution to produce a 1-dimensional output was
used. Leaky ReLUs were used with a slope of 0.2.

The discriminator architecture was:

C64,C128,C256,C512

#####  $Q$-function architecture

$Q$-function had architecture taken from dqn . Let c-k-s-f denote a $k\times
k$ Convolution-ReLU layer with stride $s$ and $f$ filters and f-o is a Fully
connected-ReLU layer with $o$ outputs.

The $Q$-function architecture was:

c-8-1-32,c-4-2-64,c-3-1-64,f-512,f-6

##### Environment model architecture

The environment model $F$ had the same architecture as the generators $G_{S}$
and $G_{T}$ with one difference - the fifth residual block received one-hot
encoded actions as an additional input.

The architecture of environment model was then as follows:

c7s1-64,d128,d256,R256,R256,R256,R256,R262,R262,R262,R262,
R262,u128,u64,c7s1-3

###  6.2 Training

All the networks are trained from scratch with weights are initialized from a
Gaussian distribution $N(0,0.02)$.

The environment model $F$ was trained with Adam optimizer, adam , with
learning rate of 0.001, batch size of 16 and it was trained for 50 epochs.

For the training of $Q$-function, RMSprop optimizer, rmsprop , was used, the
learning rate was set to 0.0001 and batch size was set to 32. The other
parameters of $Q$-learning were identical as in dqn .

Generators $G_{S}$ and $G_{T}$ and discriminators $D_{S}$ and $D_{T}$ were
jointly trained using Adam optimizer with initial learning rate of 0.0002
which was linearly decayed to zero, the training took 4 epochs.