# Continuous Coordination As a Realistic Scenario for Lifelong Learning

Hadi Nekoei  Akilesh Badrinaaraayanan  Aaron Courville  Sarath Chandar

###### Abstract

Current deep reinforcement learning (RL) algorithms are still highly task-
specific and lack the ability to generalize to new environments. Lifelong
learning (LLL), however, aims at solving multiple tasks sequentially by
efficiently transferring and using knowledge between tasks. Despite a surge of
interest in lifelong RL in recent years, the lack of a realistic testbed makes
robust evaluation of LLL algorithms difficult. Multi-agent RL (MARL), on the
other hand, can be seen as a natural scenario for lifelong RL due to its
inherent non-stationarity, since the agents’ policies change over time. In
this work, we introduce a multi-agent lifelong learning testbed that supports
both zero-shot and few-shot settings. Our setup is based on Hanabi — a
partially-observable, fully cooperative multi-agent game that has been shown
to be challenging for zero-shot coordination. Its large strategy space makes
it a desirable environment for lifelong RL tasks. We evaluate several recent
MARL methods, and benchmark state-of-the-art LLL algorithms in limited memory
and computation regimes to shed light on their strengths and weaknesses. This
continual learning paradigm also provides us with a pragmatic way of going
beyond centralized training which is the most commonly used training protocol
in MARL. We empirically show that the agents trained in our setup are able to
coordinate well with unseen agents, without any additional assumptions made by
previous works. The code and all pre-trained models are available at
https://github.com/chandar-lab/Lifelong-Hanabi.

Lifelong Learning, Multi-agent Reinforcement Learning

##  1 Introduction

Deep reinforcement learning (RL) has shown an immense potential to achieve
superhuman performance (Mnih et al., 2013; Silver et al., 2018) on some narrow
and well-defined tasks. In contrast, humans can quickly and continually learn
new tasks while maintaining the skills to solve previously learned tasks. The
ability of an AI system to effectively update new information over time is
known as lifelong learning (LLL) or continual learning, and one can postulate
this as one of the fundamental ingredients of general AI. Balancing between
learning from recent experiences while not forgetting the knowledge acquired
from the past is a well-studied problem known as the stability-plasticity
dilemma (Carpenter & Grossberg, 1987). Catastrophic forgetting is a phenomenon
in which training the model with new information obstructs previously learned
knowledge. This is a common failure case in training neural networks to adapt
to new tasks or learning from non-stationary data streams (i.e. non-iid)
(McCloskey & Cohen, 1989). Alleviating catastrophic-forgetting is crucial to
enable real-world applications where input distributions can shift and where
retraining on past data or from scratch is infeasible. While lifelong learning
has been identified as an important and challenging problem decades ago
(Thrun, 1998; Ring, 1998), it has recently seen a surge of interest (Lopez-Paz
& Ranzato, 2017; Chaudhry et al., 2018b, 2019; Kirkpatrick et al., 2017;
Aljundi et al., 2018) with the success of deep learning.

Several standard benchmarks have been proposed to evaluate novel LLL
approaches, mostly for supervised learning settings such as Permuted MNIST
(Goodfellow et al., 2013), Split MNIST/CUB/CIFAR (Zenke et al., 2017; Chaudhry
et al., 2018b). One fundamental issue with using datasets like MNIST as a
source of data is the lack of resulting task complexity especially with the
large capacity of modern neural networks. Another issue with most current LLL
benchmarks is that the relation between tasks cannot be quantified easily.
Consequently, most of the evaluation efforts have focused mainly on mitigating
catastrophic forgetting, while an ideal LLL system should in addition measure
forward and backward transfer. Some recent works have shown limitations of LLL
benchmarks (Antoniou et al., 2020; Roady et al., 2020). For instance, it has
been shown that after continual training, the performance of a model trained
from scratch using only samples from the episodic memory at test-time, is
comparable to specifically designed LLL solutions for most of these benchmarks
(Prabhu et al., 2020). There have been efforts to address this by proposing
more challenging benchmarks like CORe50 (Lomonaco & Maltoni, 2017), CRIB
(Stojanov et al., 2019), OpenLoris (Shi et al., 2020), Stream51 (Roady et al.,
2020), and IIRC (Abdelsalam et al., 2020).

RL can be a natural fit for studying LLL as it provides an agent-environment
interaction paradigm wherein the agent is exposed to non-stationary streams of
data (Kaplanis et al., 2018, 2019). However, there is a dearth of well-
established benchmarks to study progress in lifelong RL. Most of these
benchmarks are hand-engineered customization to the standard RL environments
(Bellemare et al., 2013; Brockman et al., 2016) adding synthetic non-
stationarity to the environments (Henderson et al., 2017; Al-Shedivat et al.,
2017) or ordering some completely unrelated environments in a sequence (Xu et
al., 2020) to facilitate the evaluation of LLL performance (eg. a random
sequence of Atari used in (Kirkpatrick et al., 2017)). Designing overly-
tailored experiments for a specific lifelong RL problem can entail unwanted
bias in the study (Khetarpal et al., 2020).

In this work, we propose a new lifelong RL setup based on Hanabi (Bard et al.,
2020) called Lifelong Hanabi. Hanabi is a partially-observable, fully
cooperative multi-agent game that consists of 2-5 players. In our setup, one
agent (learner) is trained sequentially with a set of partners (tasks). The
learner and its partners are sampled from a large pool of pre-trained agents
($\geq$ 100). The pre-trained pool consists of agents trained with different
MARL methods such as Independent Q-learning (IQL) (Tan, 1993), Value
Decomposition Networks (VDN) (Sunehag et al., 2017), Simplified Action Decoder
(SAD) (Hu & Foerster, 2019), Other-Play (OP) (Hu et al., 2020) with different
architectures and seeds for each method that have shown good performance in
Hanabi. Bard et al. (2020) show that agents trained even with the same MARL
method but different seeds do not learn to cooperate in the zero-shot
scenario, thereby suggesting that these agents converge to different
strategies. This large strategy space of Hanabi makes it an ideal scenario for
LLL. How far-apart the agents are in the strategy space can be measured
through the cross-play (CP) matrix (Bard et al., 2020) that contains the
gameplay scores obtained by pairing the agents with one another. CP scores can
be used as a proxy for task-similarity to design tasks in Lifelong Hanabi.

Our contributions are as follows:

  * •

We propose a new lifelong reinforcement learning benchmark that has the
following desirable properties: (1) It is challenging for state-of-the-art
(SOTA) lifelong learning algorithms, (2) It is straightforward to quantify the
relation between tasks through the CP matrix, and (3) It is easily extendable
to long sequences of diverse tasks without any synthetic modifications.

  * •

We evaluate recent LLL algorithms on this benchmark in limited memory and
computation regimes and highlight their strengths and limitations.

  * •

We obtain comparable performance on zero-shot coordination in Hanabi even when
coordinating with agents trained with MARL methods different from that of the
learner, without any additional assumptions such as exploiting handcrafted
symmetries (Hu et al., 2020) or having access to other agent’s greedy action
or policy (Hu & Foerster, 2019).

##  2 Related work

In this section, we will provide an overview of existing lifelong RL
benchmarks. We will also review recent MARL algorithms since our benchmark is
based on a challenging MARL problem.

###  2.1 Lifelong Reinforcement Learning Benchmarks

With regard to lifelong RL benchmarks, Henderson et al. (2017) proposed 50 new
variations to OpenAI Gym environments through modifying some aspects of either
the environments or agents like gravity, morphology of the agent’s body, or
goal positions. Al-Shedivat et al. (2017) introduced RoboSumo — a 3D
environment based on MuJoCo physics simulator that allows pairs of agents to
compete against each other. The robots differ in anatomy: the number of legs,
their positions, and constraints on the thigh and knee joints. Lomonaco et al.
(2020) designed CRLMaze based on VizDoom (Kempka et al., 2016), an object-
picking LLL task that is composed of 4 scenarios (Light, Texture, Object, All)
of incremental difficulty and a total of 12 maps. While these are interesting
benchmarks, they still need synthetic modifications to either the environment
or the agent in order to introduce non-stationarity. Recently, Coinrun (Cobbe
et al., 2019) was proposed that is a procedurally generated environment having
different training and testing sets to measure generalization in RL. Jelly
Bean World (JBW) (Platanios et al., 2020) is a testbed introduced to develop
agents with never-ending learning capabilities. It provides support to create
non-stationary environments with wide range of tasks including multi-task and
multi-modal settings.

###  2.2 Multi-agent RL

In recent years, there has been rapid progress on novel MARL algorithms that
are based on centralized training with decentralized execution (Self-Play) as
a training paradigm (Sunehag et al., 2017; Foerster et al., 2017; Hong et al.,
2017; Foerster et al., 2019; Hu & Foerster, 2019). Sunehag et al. (2017) use a
Value Decomposition Network (VDN) to learn to decompose the joint state-action
value into agent-wise Q-values that depend on local observations of each
agent. Bayesian Action Decoder (BAD) (Foerster et al., 2019) and its
simplified version (SAD) (Hu & Foerster, 2019) propose public belief MDP and
use an approximate Bayesian update to tackle partially observable tasks. Hong
et al. (2017) aim to learn models of opponents by learning policy features
from raw observations of other agents. On the other hand, Omidshafiei et al.
(2017) is one of the few efforts toward decentralized multi-agent learning
which shows that optimistic learners can learn sample-efficient MARL policies.
There have also been advances towards developing more complex MARL challenges.
Stone et al. (2010) first introduced ad-hoc team-work challenge as a multi-
agent task where autonomous agents need to collaborate with previously unknown
teammates on tasks in which they are all individually capable of contributing
as team members as well as characteristics of a good ad-hoc player. More
recently, in the Hanabi challenge (Bard et al., 2020), the authors introduce
ad-hoc scenario as a setting with the objective of making the RL agents adapt
to play effectively with unknown partners or even humans. However, SP agents
learn brittle policies that fail to cooperate in this scenario (Bard et al.,
2020). Cooperative settings are more reflective of real-world scenarios such
as autonomous driving and are crucial for human-AI collaboration (Crandall et
al., 2018). The closest work to ours from the perspective of zero-shot
coordination is Other-Play (Hu et al., 2020), which exploits the symmetry in
the environment by training a self-play agent with shuffled observation space.
This simple but elegant idea when combined with adding an auxiliary task, has
been shown to be effective in training agents that are able to coordinate with
other agents trained with the same MARL methods. However, other-play is still
a self-play strategy and requires the knowledge of symmetry of the game
upfront. In this work, we aim to go beyond these restricting assumptions and
train a lifelong learner that is able to coordinate with unseen agents while
not forgetting to play well with previous partners.

Figure 1: Our Lifelong Hanabi setup consists of three phases: 1- Pre-training
(Optional): In this phase, a pool of agents are trained through SP, 2-
Continual training: The learner is taken from the pool ($\sim p_{train}$) and
trained sequentially with some partners ($\sim p_{train}$) and periodically
evaluated against all the partners, 3- Testing: The learner is evaluated with
a set of random agents excluding its partners ($\sim p_{test}$) to measure
generalization.

##  3 Multi-Agent RL and Lifelong Learning

MARL for LLL: Many machine learning algorithms make the assumption that the
observations in the dataset are independent and identically distributed
(i.i.d). However, in many real-world scenarios, this assumption is violated
because the underlying data distribution is non-stationary. Lifelong learning
tries to address this problem, where the non-stationarity of data is usually
described as a sequence of distinct tasks. On the other hand, MARL is
inherently non-stationary due to changing behavior of other agents present in
the environment. Therefore, MARL is a realistic scenario for LLL. Another
source of non-stationarity in MARL arises when the agent has to interact with
different agents over its lifetime, even if the other agents are fixed. For
example, in the game of Hanabi, we are interested in designing a single agent
that can learn to coordinate well with a sequence of agents it will see over
its lifetime. This is a lifelong learning problem.

LLL for MARL: Standard MARL methods typically focus on the centralized
training with decentralized execution setting where agents have access to
other agents’ policies and observations during training (Zhang et al., 2019).
Self-Play (SP) (Tesauro, 1995), the most common centralized training setting
involves training a single agent against itself without any extra supervision.
While this strategy works very well in competitive settings like playing the
game of Go (Silver et al., 2016), in cooperative settings it can produce
agents that establish highly specialized conventions that do not carry over to
novel partners they have not been trained with (Bard et al., 2020). In
particular, Bard et al. (2020) show that even though RL agents achieve a
decent score after training in the SP setting, their performance drops off
sharply in the zero-shot coordination scenario, with some agents scoring
essentially zero. Therefore, SP agents fail to learn robust strategies that
facilitate cooperation with other agents. Lifelong learning provides a natural
framework to transfer knowledge from previous experience to future scenarios.
Hence, in this paper, we consider lifelong learning as an alternative to self-
play in MARL with the hope that lifelong learning algorithms can learn to
coordinate well with unseen agents.

##  4 Lifelong Hanabi: A Benchmark for Lifelong Reinforcement Learning

In this section, we introduce our lifelong reinforcement learning benchmark
based on Hanabi. Hanabi (Bard et al., 2020) is a partially-observable, fully
cooperative multi-agent game that consists of 2-5 players. Each player can
observe other players’ hands except his/her own, making it a partially
observable game. The objective of the game is to form ordered stacks of cards
of respective colors (fireworks). The players can communicate with each other
implicitly through actions or explicitly through hints, which are limited in
number. Thus, Hanabi is a challenging game that requires the agent to possess
the theory of mind (Premack & Woodruff, 1978; Rabinowitz et al., 2018) in
order to cooperate effectively. The theory of mind is the ability of an agent
to see the world through the lens of other agents. Our objective is to design
a training paradigm that can learn zero-shot and few-shot coordination with
unseen agents. To this end, we take inspiration from the recent Invariant Risk
Minimization (IRM) (Arjovsky et al., 2019) that improves the Out-of-
Distribution (OOD) generalization by training an algorithm on multiple
environments. MARL provides such an environment naturally without any need to
hand-engineer different features to come up with a set of diverse
environments.

As shown in Figure 1, in our Lifelong RL setup, the learner ($\sim p_{train}$)
is trained sequentially on a set of tasks M = $\\{M_{t}\\}_{t=1}^{T}$ sampled
from a distribution $p_{train}$ over diverse strategies that perform well in
Hanabi. The objective of the learner is to learn to coordinate well with its
partners during continual training, with the ultimate goal of learning to
coordinate well with unseen agents at the end of the training. During the
testing phase, in order to measure generalization performance, the learner is
evaluated with some random agents sampled from $p_{test}$. Although we
consider a fully cooperative game in this work, our proposed lifelong RL setup
can be easily extended to other multi-agent scenarios (e.g. fully competitive,
mixed cooperative-competitive, etc.). Our proposed setup consists of three
phases: (1) Pre-training, (2) Continual-training, and (3) Testing. The
detailed description of each phase is as follows:

Pre-training: In this phase, agents are trained through SP to play the game of
Hanabi. We consider several recent MARL methods for training the agents
(IQL/VDN/OP/SAD, and their combinations) across different seeds and
architectures leading to a pool of agents having diverse strategies.

Continual-training: An agent sampled from the pool is chosen as the learner
and is trained sequentially with a set of agents (partners) for a fixed number
of games per partner. The learner is also periodically evaluated with all its
partners under both zero-shot and few-shot settings. In order to implement
memory-based LLL algorithms (ER, A-GEM, etc.), we also include an episodic
memory that is used to store some transitions from every task, which can be
then be used for replaying in the future tasks.

Testing: The learner is evaluated with $K$ random agents sampled from the pool
excluding its partners in order to measure the generalization performance.

###  4.1 Evaluation methods

We consider two modes of evaluation in our setup : (a) zero-shot and (b) few-
shot. In zero-shot setting, the learner is evaluated with another agent
without providing it a chance to make updates to its own policy through its
interaction with the other agent. On the other hand, in the few-shot setting,
the learner plays a few games with the other agent, thereby adapting its
policy through interaction, before being evaluated. We believe agents
performing well under both these evaluation settings are crucial to developing
AI systems that can adapt well to unknown partners not just in Hanabi but can
also facilitate effective collaboration with humans. Few-shot evaluation
setting also opens a door to explore recent advances in Meta-RL algorithms to
enable fast adaptation.

###  4.2 Metrics

We measure the learner’s performance during continual training with some
standard metrics from the LLL literature such as average score ($A$),
forgetting ($F$), and forward transfer ($FT$) (Lopez-Paz & Ranzato, 2017;
Chaudhry et al., 2018a). We also define a metric for measuring OOD
generalization in our setup called generalization improvement score ($GIS$),
inspired by (Zhang et al., 2018). To calculate these metrics, we map Hanabi
scores at the end of the game from [0, 25] to [0, 1] to have values more
consistent with the literature.

Average score ($A\in[0,1]$): Let $a_{i,j}$ be the score of the learner versus
the $j^{th}$ partner, after training it with the $i^{th}$ partner in
sequential training. The average score of the learner at task $t$ ($A_{t}$) is
defined as:

$A_{t}=\frac{1}{t}\sum_{j=1}^{t}a_{t,j}$ (1)

Forgetting($F\in[-1,1]$): Let $f_{j}^{t}$ represent the forgetting on task $j$
after the learner is trained on task $t$ and is computed as:

$f_{j}^{t}=\max_{l\in{1,...,t-1}}a_{l,j}-a_{t,j}\\\ $ (2)

The average forgetting at task $t$ is then defined as:

$F_{t}=\frac{1}{t-1}\sum_{j=1}^{t-1}f_{j}^{t}$ (3)

Average future score ($FT\in[0,1]$): Let $a_{i,j}$ be the score of the learner
versus the $j^{th}$ partner, after training it with the $i^{th}$ partner in
sequential training. The average future score (or forward transfer) of the
learner at task $t$ ($FT_{t}$) is defined as:

$FT_{t}=\frac{1}{T-t}\sum_{j=t+1}^{T}a_{t,j}$ (4)

Generalization Improvement Score ($GIS\in[0,1]$): Zhang et al. (2018) define
in-task generalization as the difference in the performance of an RL algorithm
between a set of training and testing trajectories all generated by the same
simulator. In this case, the only source of variation is through random seeds.
However, in our work, we are more interested in the out-of-task generalization
that can be defined as follows. Let $a_{0,k}$ and $a_{N,k}$ be the score of
the learner versus the $k_{th}$ random agent sampled from the pool (different
from its partners) before the start of LLL and at the end of continual
training, respectively ($T$ is number of tasks in LLL). The GIS is computed as
follows:

$GIS=\frac{1}{K}\sum_{k=1}^{K}(a_{T,k}-a_{0,k})$ (5)

where $K$ is the total number of unseen agents.

##  5 Experiments

Figure 2: CP scores before (left) and after continual training (right) –
$(i,j)_{th}$ element is the average score of agent $i$ paired with $j$. [A-C]
is before continual training – (A) Initial scores of the learner with its
partners, (B) Cross-play scores amongst the partners, low scores indicate they
are far apart in the strategy space, (C) Initial generalization scores with
some unseen agents. The learner is then trained continually with its partners
following the order indicated by the arrows. [D-E] is after continual training
– (D) Scores with the original learner and its partners, (E) Generalization
scores with the same unseen agents.

Figure 2 showcases how continual training can lead to improved scores and
better zero-shot coordination. An IQL agent (pre-trained with self-play) is
trained sequentially with 5 partners (ordering denoted by arrows in Figure 2),
and then evaluated with both its partners seen during continual training as
well as some unseen agents (SAD+AUX+OP, VDN+AUX). Sections (C) and (E) in
Figure 2 show the performance of the IQL agent on unseen agents before and
after continual training respectively, clearly indicating OOD generalization.
Likewise, sections (A) and (D) show the performance before and after training
respectively with its partners used in continual training.

As described in Section 4, we first pre-train a set of agents to play the game
of Hanabi through SP. Our RL agents are based on the R2D2 architecture
(Kapturowski et al., 2018) that are RNN-based DQN agents. The diversity in
strategies learned by these SP agents are controlled by varying the seed, the
MARL methods used for training (either IQL/VDN/OP/SAD and their combinations),
type of recurrence (LSTM/GRU), number and dimension of recurrent layers,
number and dimension of feed-forward layers before recurrence. The exact
architecture details are described in the Appendix B. A pool of more than 100
SP agents is created this way. A subset of 100 agents with 10 agents from each
of these 10 MARL methods are used to generate the cross-play (CP) matrix as
shown in Appendix-A. The entries in this matrix (diagonal entries indicate
self-play scores) are obtained through the gameplay of agents with each other
for 5k games, and then averaging the scores.

We propose two levels of tasks based on these scores that have different
difficulty levels: easy and hard. In both the settings, one of the agents is
used as learner and the rest as its partners that are fixed during continual
training i.e. they represent different tasks since these agents have different
strategies. Both the learner and its partners are initialized with weights of
the pre-trained agents as we found pre-training to be crucial to learn some
basic knowledge of Hanabi. The names of these tasks are self-explanatory in
the sense the learner in the hard version has to start from a much lower
cross-play score and learn to achieve a good final score (out of 25). During
continual training, the learner is trained sequentially with every partner for
a fixed number of epochs and evaluated periodically with all its partners
under both zero-shot and few-shot settings. In the zero-shot setting, the
learner is directly evaluated with all its partners, while in the few-shot
setting, the learner is fine-tuned with its partner for a few gradient steps
before evaluating against the same partner. In both these settings, scores are
reported based on average over 5k gameplay.

Figure 3: Zero-shot (top row) and Few-shot (bottom row) performance of
different LLL algorithms with Adam optimizer on hard task. From left to right:
current score ($\uparrow$), average score ($\uparrow$), forgetting
($\downarrow$), and average future score ($\uparrow$) respectively.
($\uparrow$ = higher better, $\downarrow$ = lower better).

Although the hard and easy settings consist of five tasks, our setup as such
can be effortlessly extended to any number of tasks by selecting a different
number of partners from the pre-trained pool and the pre-trained pool can
itself be expanded by training more agents through SP. For instance, we report
results on 10 tasks in section 5.3 as well as in Appendix C. Note that to
choose the partners, we excluded all the agents using either SAD or AUX from
the pre-trained pool as we wanted to compare the continually trained learner
with them in terms of zero-shot coordination. We also wanted to select
partners that have low CP scores so that the tasks are diverse.

R2D2 agents keep recent game transitions in a fixed-size prioritized replay
buffer (Schaul et al., 2015). At the end of every task, the replay-buffer is
sliced and stored in an episodic memory, which is then used for replay in
different memory-based LLL algorithms that we consider in our benchmark. The
learner can also start with random parameters (i.e. without pre-training),
albeit, this setting is very hard for the game of Hanabi.

We aim to answer the following questions through our experiments : (1) How
well standard LLL algorithms perform in our setup (section 5.1), (2) How well
do these LLL algorithms fare under constrained memory and compute settings
(section 5.2), (3) How lifelong RL methods perform in our setup (section 5.3),
(4) How well the agents trained in our setup do in zero-shot and few-shot
coordination scenarios in Hanabi (section 5.4), when compared to other recent
methods such as OP (Hu et al., 2020).

###  5.1 Lifelong learning benchmarking:

We implement some standard LLL algorithms that are both replay-based and
regularization-based.

Naive: This is the simplest algorithm in which the learner is trained
sequentially on subsequent tasks, starting with the learned parameters at the
end of the previous task, without any episodic memory or regularization.

Experience Replay (ER): We follow the procedure described in (Chaudhry et al.,
2019) for implementing ER. We sample a mini-batch $B_{k}$ from the current
task (in which the learner plays with partner $k$), and a mini-batch $B_{m}$
that consists of an equal number of samples from all previous tasks
collectively. These mini-batches are stacked and a single gradient step is
used to update the learner. Our implementation closely resembles the ring
buffer strategy described in (Chaudhry et al., 2019), as there’s equal
representation from all previous tasks when sampling $B_{m}$, although the
samples within each task itself are prioritized.

Averaged Gradient Episodic Memory (A-GEM): Minibatches $B_{k}$ and $B_{m}$ are
sampled as described in (Chaudhry et al., 2018b) and similar to ER. The
gradients corresponding to these mini-batches are first computed denoted by
$g$ and $g_{ref}$ respectively. If $g^{T}g_{ref}\geq 0$, then the gradient of
the current task $g$ is directly used to update the learner’s parameters,
whereas if $g^{T}g_{ref}<0$, $g$ is first projected such that $g^{T}g_{ref}=0$
before updating the learner. This projection ensures that the average loss
over the previous tasks does not increase.

Elastic Weight Consolidation (EWC): EWC is a regularization-based technique
proposed to alleviate catastrophic forgetting by selectively reducing the
plasticity of weights drawing inspiration from Bayesian methods (Kirkpatrick
et al., 2017). EWC uses Fisher information matrix as a surrogate for the
importance of learned weights and uses that for gradient updates. Offline EWC
uses one Fisher matrix per task, therefore the number of regularization terms
increases linearly with the number of tasks whereas Online EWC (Schwarz et
al., 2018) uses only one Fisher matrix that is computed based on all the
previous tasks. We consider both these variants in our benchmark.

Stable naive/ER/A-GEM/EWC: Mirzadeh et al. (2020) show that catastrophic
forgetting can be mitigated through careful design of training regimes such as
learning rate decay, batch size, dropout, and optimizer that can widen the
tasks’ local minima. The resulting model with these optimal choices is
referred to as “stable”. In particular, we consider if using larger batches,
exponential learning rate decay, dropout (either in feed-forward or recurrent
layers in R2D2), and SGD optimizer help improve continual training, thereby
leading to better final performance as well as generalization to unseen
agents.

Table 1: Benchmarking LLL methods on Hanabi. Average accuracy and forgetting
of LLL algorithms on hard task averaged over 5 runs with 5000 games.
($\uparrow$ = higher better, $\downarrow$ = lower better)
Method | Zero-shot |  | Few-shot
---|---|---|---
| $A_{T}\uparrow$ | $F_{T}\downarrow$ |  | $A_{T}\uparrow$ | $F_{T}\downarrow$
Naive Adam | $0.39$ | $0.60$ |  | $0.52$ | $0.44$
EWC off. Adam | $0.45$ | $0.45$ |  | $0.60$ | $0.27$
EWC on. Adam | $\mathbf{0.55}$ | $0.28$ |  | $\mathbf{0.63}$ | $0.17$
ER Adam | $0.44$ | $0.26$ |  | $0.53$ | $0.20$
AGEM Adam | $0.38$ | $0.62$ |  | $0.57$ | $0.38$
Naive SGD | $0.52$ | $0.15$ |  | $0.52$ | $0.14$
EWC off. SGD | $0.49$ | $0.12$ |  | $0.50$ | $0.11$
EWC on. SGD | $0.47$ | $0.12$ |  | $0.48$ | $0.11$
ER SGD | $0.50$ | $\mathbf{0.05}$ |  | $0.50$ | $\mathbf{0.04}$
AGEM SGD | $0.50$ | $0.13$ |  | $0.51$ | $0.12$
Multi-Task Adam | $0.70$ | $0$ |  | $0.77$ | $0$
Multi-Task SGD | $0.50$ | $0$ |  | $0.51$ | $0$

Multi-Task Learning (MTL): In this setting, there’s a common replay buffer
that contains the experiences of the learner interacting with all its
partners. Mini-batches sampled from this common replay buffer are used for
training the learner. This serves as an upper-bound on the achievable
performance in our benchmark.

We can observe from Figure 3 and Table 1 that online EWC with Adam has the
best average score in both the zero-shot and few-shot setting among the LLL
algorithms, while the forgetting is least for ER with SGD. Table 1 also shows
the effect of the optimizer on different LLL algorithms. LLL algorithms with
SGD tend to have comparatively less forgetting and better zero-shot
performance on average. We can infer from Figure 4 that using Adam helps in
fast adaptation to current task albeit at the expense of greater forgetting.
This effect can also be seen in higher average few shot scores (Table 1). As
one might expect, MTL with Adam achieves the highest average score in both
zero-shot and few-shot.

From the last column of Figure 3, we can observe that when the learner start
playing with a new partner, there is an increase in average future score
suggesting it has learned some useful skills from previous tasks that is
transferable to the other partners. However, with more training, the learner
possibly overfits to its current partner, leading to a drop in average future
scores.

To explore the effect of different training regimes on continual training, we
study the effect of larger batches (128 vs 32), learning rate decay with high
initial rates (0.2/0.02), and dropout. Our experiments suggest that Lifelong
Hanabi does not benefit greatly from the use of large batches as the gain in
scores is negligible. This observation aligns with the “stable” networks
(Mirzadeh et al., 2020) that suggests using small batches. In the case of EWC,
we find that there is a stark difference in performance with different
$\lambda$ values (the weight assigned to the Fischer term). Our experiments
indicate larger $\lambda$ is beneficial. Appendix- C contains the training
curves of LLL algorithms for other settings — hard task with SGD as well as
easy task with both Adam and SGD.

(a) Adam vs SGD (b) GIS (c) Episodic memory size (d) Few-shot gradient steps

Figure 4: More experiments: Generalization score with Inter-CP agents at the
end of every task during continual training. LLL algorithms using SGD as an
optimizer have better generalization performance compared to Adam. ER SGD has
the highest GIS.

###  5.2 Lifelong learning under constrained memory and compute:

####  5.2.1 Episodic memory size

In order to understand the effect of episodic memory on the performance of
memory-based LLL algorithms, we vary the episodic memory size ({2k, 8k, 32k}
$\times$ number of tasks) in the case of ER with both SGD and Adam as shown in
Figure 4. In both these cases, a larger episodic memory size results in a
better final performance.

####  5.2.2 Gradient updates for few-shot evaluation

To better understand the ability of the learner trained with different LLL
algorithms to adapt quickly to all its partners, we vary the number of
gradient steps used to update the learner in the few-shot evaluation scenario.
As it can be seen from Figure 4, there is a considerable difference between 10
and 50 gradient updates on the final performance, whereas the benefit reaped
beyond 50 updates is minimal.

###  5.3 Lifelong RL methods:

(Isele & Cosgun, 2018) propose some strategies for storing experiences in the
replay-buffer that has been shown to reduce catastrophic forgetting in RL. All
our methods use prioritized replay buffer that resembles the surprise strategy
(Isele & Cosgun, 2018). In addition, we also compare this with FIFO and Reward
strategies. For FIFO, we set the prioritization exponent $\alpha$ to 0 (Schaul
et al., 2015), which is equivalent to uniformly sampling. In case of Reward,
we do prioritized sampling that favors experiences based on the absolute value
of reward instead of TD-error as done in our default case. As can be seen in
Figure 5, ER with prioritized sampling performs best compared to Reward and
FIFO strategies in terms of both average score and average forgetting.
Implementing other sampling strategies such as Global Distribution Matching
and Coverage Maximization are left for future work.

Figure 5: Zero-shot (top row) and Few-shot (bottom row) performance of ER
methods with different types of episodic memory designed for lifelong RL
(Isele & Cosgun, 2018) with Adam optimizer on 10 tasks. Table 2: Comparison
with other MARL algorithms on self-play (SP), cross-play evaluation scores
within method (Intra-CP), and across different methods (Inter-CP). C:
centralized training, GA: agents share their greedy action along with their
standard action, L: true labels of cards needed, SYM: symmetries of the game
needed upfront, P: require access to some pre-trained agents in sequence, UP:
Having access to all the fixed pre-trained agents at the same time.
($\uparrow$ / $\downarrow$ = Difference in score after continual training,
red: pre-trained with MARL method, blue: trained continually with LLL method)
Training Method |  | SP |  | Intra-CP |  | Inter-CP | Limitations
---|---|---|---|---|---|---|---
SAD |  | $23.85\pm 0.03$ |  | $7.70\pm 0.69$ |  | $14.60\pm 0.24$ | C + GA
SAD \+ AUX |  | $23.57\pm 0.03$ |  | $20.97\pm 0.80$ |  | $18.51\pm 0.23$ | C + GA + L
SAD \+ OP |  | $24.14\pm 0.03$ |  | $10.10\pm 0.87$ |  | $16.09\pm 0.25$ | C + Sym + GA
SAD \+ AUX \+ OP |  | $23.40\pm 0.04$ |  | $21.23\pm 0.25$ |  |  17.77 $\pm$ 0.23 | C + Sym + L + GA
IQL \+ ER |  |  $20.91\pm 0.05$ ($\downarrow 2.98$) |  |  $15.73\pm 0.39$ ($\mathbf{\uparrow 7.06}$) |  |  $16.32\pm 0.21$ ($\mathbf{\uparrow 8.09}$) | P
IQL \+ AUX \+ ER |  |  $22.34\pm 0.06$ ($\downarrow 1.46$) |  |  $20.90\pm 0.06$ ($\downarrow 0.15$) |  |  $\mathbf{19.17\pm 0.22}$ ($\mathbf{\uparrow 1.33}$) | L + P
IQL \+ Multi-task |  |  $20.93\pm 0.09$ ($\downarrow 2.96$) |  |  $16.05\pm 0.30$ ($\mathbf{\uparrow 7.38}$) |  |  $\mathbf{17.88\pm 0.17}$ ($\mathbf{\uparrow 9.65}$) | UP

###  5.4 Zero-shot coordination:

We compare our best-performing LLL algorithms with recent MARL methods that
have shown good performance on Hanabi (Table 2). In addition to reporting
self-play evaluation scores, we evaluate each training method with two sets of
unseen partners under zero-shot coordination scenario: (1) Intra-CP \- a set
consisting of agents that are trained with the same MARL method as the
training method. For example, the SAD+OP agent is evaluated with other SAD+OP
agents only, but with different architectures and seeds. Similarly, in order
to evaluate agents trained in our setup, we evaluate them with other agents
that are trained with the same MARL method as the learner, (2) Inter-CP \- a
set containing 20 agents across all the MARL methods we consider.

As we can observe from Table 2, although recent MARL methods can achieve good
scores in SP and Intra-CP evaluations, they fail to achieve high scores in
Inter-CP highlighting their inability to coordinate effectively with other
MARL methods in the zero-shot scenario. We can observe that agents trained in
our setup have significant improvement in both Inter-CP and Intra-CP compared
to the agent at the start of continual learning, however, their SP scores are
lower than at the start. The difference in scores due to continual training is
indicated in brackets. It is also worth mentioning that IQL+AUX+ER achieve a
better Inter-CP score than even other MARL methods, although this comes at a
cost of slight reduction to Intra-CP score.

All the training methods in Table 2 have some limitations that we highlight
now. During training, SAD allows agents to have access to the greedy action of
their team mates in addition to the actual exploratory action chosen ($GA$).
AUX refers to having an auxiliary task that predicts the learner’s own hand
and hence requires ground truth labels for this ($L$). OP requires symmetries
of the game to be known beforehand ($SYM$). IQL+ER and IQL+AUX+ER require pre-
trained agents in sequence for LLL ($P$), while IQL + Multi-Task requires
access to all pre-trained agents simultaneously ($UP$). Figure 4 shows the
progression in generalization performance (Inter-CP) after every task during
continual training for several LLL algorithms. MTL (with Adam) and ER (with
SGD) have the best scores with the Inter-CP agents at the end of continual
training. However, MTL needs to interact with all the partners’ at the same
time which is not always a realistic assumption.

##  6 Conclusions and future work

In this work, we proposed Lifelong Hanabi as a new challenging benchmark for
lifelong RL. The non-stationarity in our benchmark was introduced through
agents having different strategies instead of synthetic modifications to the
environment or agent, while cross-play score served as an easy metric to
quantify the similarity between tasks. We analyzed the performance of some
well-known LLL algorithms on this benchmark. We also showed that an IQL agent
continually trained in our setup can zero-shot coordinate effectively with
unseen agents. We hope that the lifelong RL community adopt this as a standard
benchmark for evaluating algorithmic advances due to its ease of use.

Lifelong Hanabi aims to facilitate development of novel algorithms for
lifelong learning specific to RL (i.e. lifelong RL). This framework also
serves as a step towards thinking beyond centralized training in MARL. Some
interesting future directions are to understand the kind of policies learned
by the agents trained in our setup through policy visualization to see what
kind of conventions (if any) emerges. It is also valuable to evaluate our
trained agents with humans, as developing artificial agents capable of
coordinating effectively with humans is an important long-term goal of modern
AI. Moreover, exploiting recent advances in Meta-RL such as (Zintgraf et al.,
2019) for faster adaptation in the few-shot evaluation setting, instead of
naively fine-tuning can lead to agents that adapt well in the ad-hoc scenario.
We believe studying the effect of the order of partners that the learner
encounters and its effect on final performance is an interesting next step.
Currently, the non-stationarity across different strategies is what we only
exploit to design LLL tasks. This already resulted in an interesting trade-off
for the learner between adapting to new partners and not forgetting to
coordinate well with previous partners. Our preliminary experiments suggest
that extending our framework to learning partners is extremely difficult for
current methods, however, this could be an exciting future research direction.

Acknowledgment We would like to thank Gabriele Prato, Mahta Ramezanian, Khimya
Khetarpal, Matthew Riemer, Ankit Vani, Moksh Jain, and Salem Lahlou for
reviewing the paper and their valuable feedback. We are also grateful to
Hengyuan Hu for patiently answering queries regarding the Hanabi SAD code
repository, Darshan Patil and Rodrigo Chavez Zavaleta for reviewing our code
and Olexa Bilaniuk for helping us with engineering issues internally at Mila.
We would like to acknowledge Compute Canada and Calcul Quebec for providing
computing resources used in this work. SC is supported by a Canada CIFAR AI
Chair and an NSERC Discovery Grant.

## References

  * Abdelsalam et al. (2020) Abdelsalam, M., Faramarzi, M., Sodhani, S., and Chandar, S.  Iirc: Incremental implicitly-refined classification.  _arXiv preprint arXiv:2012.12477_ , 2020.
  * Al-Shedivat et al. (2017) Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., and Abbeel, P.  Continuous adaptation via meta-learning in nonstationary and competitive environments.  _arXiv preprint arXiv:1710.03641_ , 2017.
  * Aljundi et al. (2018) Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 139–154, 2018.
  * Antoniou et al. (2020) Antoniou, A., Patacchiola, M., Ochal, M., and Storkey, A.  Defining benchmarks for continual few-shot learning.  _arXiv preprint arXiv:2004.11967_ , 2020.
  * Arjovsky et al. (2019) Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Bard et al. (2020) Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., Parisotto, E., Dumoulin, V., Moitra, S., Hughes, E., et al.  The hanabi challenge: A new frontier for ai research.  _Artificial Intelligence_ , 280:103216, 2020.
  * Bellemare et al. (2013) Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Carpenter & Grossberg (1987) Carpenter, G. A. and Grossberg, S.  A massively parallel architecture for a self-organizing neural pattern recognition machine.  _Computer vision, graphics, and image processing_ , 37(1):54–115, 1987.
  * Chaudhry et al. (2018a) Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 532–547, 2018a.
  * Chaudhry et al. (2018b) Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M.  Efficient lifelong learning with a-gem.  _arXiv preprint arXiv:1812.00420_ , 2018b.
  * Chaudhry et al. (2019) Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_ , 2019.
  * Cobbe et al. (2019) Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J.  Quantifying generalization in reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1282–1289. PMLR, 2019.
  * Crandall et al. (2018) Crandall, J. W., Oudah, M., Ishowo-Oloko, F., Abdallah, S., Bonnefon, J.-F., Cebrian, M., Shariff, A., Goodrich, M. A., Rahwan, I., et al.  Cooperating with machines.  _Nature communications_ , 9(1):1–12, 2018.
  * Foerster et al. (2019) Foerster, J., Song, F., Hughes, E., Burch, N., Dunning, I., Whiteson, S., Botvinick, M., and Bowling, M.  Bayesian action decoder for deep multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1942–1951, 2019.
  * Foerster et al. (2017) Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I.  Learning with opponent-learning awareness.  _arXiv preprint arXiv:1709.04326_ , 2017.
  * Goodfellow et al. (2013) Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_ , 2013.
  * Henderson et al. (2017) Henderson, P., Chang, W.-D., Shkurti, F., Hansen, J., Meger, D., and Dudek, G.  Benchmark environments for multitask learning in continuous domains.  _arXiv preprint arXiv:1708.04352_ , 2017.
  * Hong et al. (2017) Hong, Z.-W., Su, S.-Y., Shann, T.-Y., Chang, Y.-H., and Lee, C.-Y.  A deep policy inference q-network for multi-agent systems.  _arXiv preprint arXiv:1712.07893_ , 2017.
  * Hu & Foerster (2019) Hu, H. and Foerster, J. N.  Simplified action decoder for deep multi-agent reinforcement learning.  _arXiv preprint arXiv:1912.02288_ , 2019.
  * Hu et al. (2020) Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J.  ” other-play” for zero-shot coordination.  _arXiv preprint arXiv:2003.02979_ , 2020.
  * Isele & Cosgun (2018) Isele, D. and Cosgun, A.  Selective experience replay for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Kaplanis et al. (2018) Kaplanis, C., Shanahan, M., and Clopath, C.  Continual reinforcement learning with complex synapses.  In _International Conference on Machine Learning_ , pp. 2497–2506. PMLR, 2018.
  * Kaplanis et al. (2019) Kaplanis, C., Shanahan, M., and Clopath, C.  Policy consolidation for continual reinforcement learning.  _arXiv preprint arXiv:1902.00255_ , 2019.
  * Kapturowski et al. (2018) Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W.  Recurrent experience replay in distributed reinforcement learning.  In _International conference on learning representations_ , 2018.
  * Kempka et al. (2016) Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaśkowski, W.  Vizdoom: A doom-based ai research platform for visual reinforcement learning.  In _2016 IEEE Conference on Computational Intelligence and Games (CIG)_ , pp. 1–8. IEEE, 2016.
  * Khetarpal et al. (2020) Khetarpal, K., Riemer, M., Rish, I., and Precup, D.  Towards continual reinforcement learning: A review and perspectives.  _arXiv preprint arXiv:2012.13490_ , 2020.
  * Kirkpatrick et al. (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Lomonaco & Maltoni (2017) Lomonaco, V. and Maltoni, D.  Core50: a new dataset and benchmark for continuous object recognition.  _arXiv preprint arXiv:1705.03550_ , 2017.
  * Lomonaco et al. (2020) Lomonaco, V., Desai, K., Culurciello, E., and Maltoni, D.  Continual reinforcement learning in 3d non-stationary environments.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 248–249, 2020.
  * Lopez-Paz & Ranzato (2017) Lopez-Paz, D. and Ranzato, M.  Gradient episodic memory for continual learning.  In _Advances in neural information processing systems_ , pp. 6467–6476, 2017.
  * McCloskey & Cohen (1989) McCloskey, M. and Cohen, N. J.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pp. 109–165. Elsevier, 1989.
  * Mirzadeh et al. (2020) Mirzadeh, S. I., Farajtabar, M., Pascanu, R., and Ghasemzadeh, H.  Understanding the role of training regimes in continual learning.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Omidshafiei et al. (2017) Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J.  Deep decentralized multi-task multi-agent reinforcement learning under partial observability.  In _International Conference on Machine Learning_ , pp. 2681–2690. PMLR, 2017.
  * Platanios et al. (2020) Platanios, E. A., Saparov, A., and Mitchell, T.  Jelly bean world: A testbed for never-ending learning.  _arXiv preprint arXiv:2002.06306_ , 2020.
  * Prabhu et al. (2020) Prabhu, A., Torr, P. H., and Dokania, P. K.  Gdumb: A simple approach that questions our progress in continual learning.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * Premack & Woodruff (1978) Premack, D. and Woodruff, G.  Does the chimpanzee have a theory of mind?  _Behavioral and brain sciences_ , 1(4):515–526, 1978.
  * Rabinowitz et al. (2018) Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M.  Machine theory of mind.  In _International conference on machine learning_ , pp. 4218–4227. PMLR, 2018.
  * Ring (1998) Ring, M. B.  Child: A first step towards continual learning.  In _Learning to learn_ , pp. 261–292. Springer, 1998.
  * Roady et al. (2020) Roady, R., Hayes, T. L., Vaidya, H., and Kanan, C.  Stream-51: Streaming classification and novelty detection from videos.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2020.
  * Schaul et al. (2015) Schaul, T., Quan, J., Antonoglou, I., and Silver, D.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015.
  * Schwarz et al. (2018) Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R.  Progress & compress: A scalable framework for continual learning.  In _International Conference on Machine Learning_ , pp. 4528–4537. PMLR, 2018.
  * Shi et al. (2020) Shi, X., Li, D., Zhao, P., Tian, Q., Tian, Y., Long, Q., Zhu, C., Song, J., Qiao, F., Song, L., Guo, Y., Wang, Z., Zhang, Y., Qin, B., Yang, W., Wang, F., Chan, R. H. M., and She, Q.  Are we ready for service robots? the openloris-scene datasets for lifelong slam, 2020.
  * Silver et al. (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Silver et al. (2018) Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Stojanov et al. (2019) Stojanov, S., Mishra, S., Thai, N. A., Dhanda, N., Humayun, A., Yu, C., Smith, L. B., and Rehg, J. M.  Incremental object learning from contiguous views.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 8777–8786, 2019.
  * Stone et al. (2010) Stone, P., Kaminka, G. A., Kraus, S., Rosenschein, J. S., et al.  Ad hoc autonomous agent teams: Collaboration without pre-coordination.  In _AAAI_ , pp. 6, 2010.
  * Sunehag et al. (2017) Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al.  Value-decomposition networks for cooperative multi-agent learning.  _arXiv preprint arXiv:1706.05296_ , 2017.
  * Tan (1993) Tan, M.  Multi-agent reinforcement learning: Independent vs. cooperative agents.  In _Proceedings of the tenth international conference on machine learning_ , pp. 330–337, 1993.
  * Tesauro (1995) Tesauro, G.  Temporal difference learning and td-gammon.  _Communications of the ACM_ , 38(3):58–68, 1995\.
  * Thrun (1998) Thrun, S.  Lifelong learning algorithms.  In _Learning to learn_ , pp. 181–209. Springer, 1998.
  * Xu et al. (2020) Xu, M., Ding, W., Zhu, J., Liu, Z., Chen, B., and Zhao, D.  Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes, 2020.
  * Zenke et al. (2017) Zenke, F., Poole, B., and Ganguli, S.  Continual learning through synaptic intelligence.  _Proceedings of machine learning research_ , 70:3987, 2017\.
  * Zhang et al. (2018) Zhang, A., Ballas, N., and Pineau, J.  A dissection of overfitting and generalization in continuous reinforcement learning.  _arXiv preprint arXiv:1806.07937_ , 2018.
  * Zhang et al. (2019) Zhang, K., Yang, Z., and Başar, T.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  _arXiv preprint arXiv:1911.10635_ , 2019.
  * Zintgraf et al. (2019) Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and Whiteson, S.  Fast context adaptation via meta-learning.  In _International Conference on Machine Learning_ , pp. 7693–7702. PMLR, 2019.

## Appendices

##  Appendix A Pool of Agents

Here is the cross-play matrix of all the 100 agents trained with different
MARL algorithms. There are five types of architectures (Table 3) with two
different seeds per MARL algorithm.

Figure 6: The pool of 100 agents pre-trained through Self-Play using different
MARL methods (IQL/VDN/OP/AUX/SAD, and their combinations). 10 agents having 5
different architectures with 2 seeds are generated with each of these MARL
methods. $(i,j)_{th}$ element is the average score of agent $i$ paired with
$j$ over 5k games. The diagonal entries indicate SP scores.

##  Appendix B List of agents

In this section, we present the exact type of agents that we use as the
learner and its partners in both easy and hard settings, as well as the set of
10 partners used in section 5.3 and in Appendix C. All these settings have an
IQL agent of Type-2 as the learner and a sequence of 5/10 agents (can be
extended to any number of agents) as its partners. Table 3 has details of the
exact architectures corresponding to these Types.

  * •

Easy : Learner — { IQL (Type-2) }
Partners — { IQL (Type-1), VDN (Type-3), VDN (Type-5), IQL+OP (Type-2), VDN+OP
(Type-5) }.

  * •

Hard : Learner — { IQL (Type-2) }
Partners — { VDN+OP (Type-3), VDN (Type-4), VDN (Type-5), IQL+OP (Type-3), VDN
(Type-3) }.
The partner agents in the Figure 2 are these hard agents.

  * •

10 agents: Learner — { IQL (Type-2) }
Partners — { VDN (Type-2), VDN (Type-3), IQL+OP (Type-2), VDN+OP (Type-5), IQL
(Type-4), VDN+OP (Type-1), VDN (Type-4), IQL+OP (Type-3), VDN+OP (Type-1), VDN
(Type-5) }.

The below are the set of 20 held-out agents that we use for across method
evaluation in Tables 6 and 7.

Inter-CP : { IQL (Type-1), IQL (Type-3), IQL+OP (Type-4), IQL+OP (Type-5),
VDN+AUX (Type-2), VDN+AUX (Type-3), SAD+OP (Type-3), SAD+OP (Type-1),
SAD+OP+AUX (Type-3), SAD+OP+AUX (Type-1), SAD+AUX (Type-3), SAD+AUX (Type-1),
SAD (Type-3), SAD (Type-2), IQL+AUX (Type-3), IQL+AUX (Type-1), VDN (Type-4),
VDN (Type-2), VDN+OP (Type-5), VDN+OP(Type-4) }.

Table 3: Exact architectures used in the pool.
Agent |  | RNN type |  | Num of feed-forward layers |  | Num of RNN layers |  | RNN hid dim
---|---|---|---|---|---|---|---|---
Type-1 |  | LSTM |  | 1 |  | 1 |  | 256
Type-2 |  | LSTM |  | 2 |  | 2 |  | 256
Type-3 |  | LSTM |  | 1 |  | 2 |  | 512
Type-4 |  | GRU |  | 1 |  | 2 |  | 256
Type-5 |  | GRU |  | 2 |  | 1 |  | 512

##  Appendix C LLL algorithms benchmarks

Figure 7: Zero-shot (top row) and Few-shot (bottom row) performance of
different LLL algorithms with Adam optimizer on hard task. From left to right:
current score, average score, forgetting and average future score
respectively. Figure 8: Zero-shot (top row) and Few-shot (bottom row)
performance of different LLL algorithms with Adam optimizer on easy task. From
left to right: current score, average score, forgetting and average future
score respectively. Figure 9: Zero-shot (top row) and Few-shot (bottom row)
performance of different LLL algorithms with SGD optimizer on hard task. From
left to right: current score, average score, forgetting and average future
score respectively. Figure 10: Zero-shot (top row) and Few-shot (bottom row)
performance of different LLL algorithms with SGD optimizer on easy task. From
left to right: current score, average score, forgetting and average future
score respectively. Figure 11: Zero-shot (top row) and Few-shot (bottom row)
performance of different LLL algorithms with Adam optimizer on 10 tasks. From
left to right: current score, average score, forgetting and average future
score respectively. Figure 12: Zero-shot (top row) and Few-shot (bottom row)
performance of different LLL algorithms with SGD optimizer on 10 tasks. From
left to right: current score, average score, forgetting and average future
score respectively.

For Figures 7-10, the learner is pre-trained with IQL method and is
continually trained with either hard or easy agents mentioned in section B.

The sequential order of partners were chosen at random from the pre-trained
pool in both easy and hard setting. Careful curation of the partner ordering
and its effect on lifelong learning is left as future work.

For Figures 11-12, the learner is pre-trained with IQL method and is
continually trained with 10 partners mentioned in section B.

##  Appendix D All hyperparameters and experiment details

Table 4: All common hyperparameters and their description.
hyperparameters |  | value |  | Description
---|---|---|---|---
$batchsize$ |  | 32 |  | batchsize used for both training
|  |  |  | and few-shot evaluation
$max\textunderscore train\textunderscore steps$ |  | 200m |  | maximum number of training steps per task
$max\textunderscore eval\textunderscore steps$ |  | 500k |  | maximum number of training steps
|  |  |  | during few-shot evaluation
$burn\textunderscore in\textunderscore frames$ |  | 10k |  | number of samples used to warm-up replay buffer
$eval\textunderscore burn\textunderscore in\textunderscore frames$ |  | 1k |  | number of samples used to warm-up
|  |  |  | evaluation replay buffer
$replay\textunderscore buffer\textunderscore size$ |  | 32768 |  | replay buffer size during continual training
$eval\textunderscore replay\textunderscore buffer\textunderscore size$ |  | 10000 |  | replay buffer size for few-shot evaluation
$epoch\textunderscore len\textunderscore size$ |  | 200 |  | number of gradient updates per epoch
$eval\textunderscore epoch\textunderscore len\textunderscore size$ |  | 50 |  | number of gradient updates for few-shot evaluation
$eval\textunderscore freq$ |  | 25 |  | Learner is evaluated after each 25 epochs
$num\textunderscore thread$ |  | 10 |  | number of threads used for R2D2 actors
$num\textunderscore game\textunderscore per\textunderscore thread$ |  | 80 |  | number of game per threads used for R2D2 actors
$eval\textunderscore num\textunderscore thread$ |  | 10 |  | number of threads used for R2D2 actors
|  |  |  | during few-shot evaluation
$eval\textunderscore num\textunderscore game\textunderscore per\textunderscore thread$ |  | 10 |  | number of games per threads used for R2D2 actors
|  |  |  | during few-shot evaluation
$sgd\textunderscore momentum$ |  | 0.8 |  | momentum for SGD optimizer
Table 5: Specific hyperparameters to each algorithm and their description
hyperparameters |  | value |  | Description
---|---|---|---|---
$ewc\textunderscore lambda$ |  | 50000 |  | EWC
$ewc\textunderscore gamma$ |  | 1 |  | EWC
$replay\textunderscore buffer\textunderscore size$ |  | 163840 |  | Multi-task

##  Appendix E MARL algorithms benchmarks

In order to obtain the Intra-CP scores for the existing MARL methods in the
Table 2 and Table 7 (referenced as BEST in caption), we take the agent from
each training method that performs best with the Inter-CP agents listed above
in section B and evaluate them with the other 9 agents of the same method from
the pretrained pool (Figure 6). However, in order to obtain the Intra-CP
scores for each MARL method in the Table 6 (referenced as AVG in caption), we
pick one agent, evaluate it with the rest (barring itself) and repeat the same
for all other agents. The average of these scores are reported. A similar
process is followed for reporting Inter-CP scores. The method of evaluating
our LLL methods remains consistent in all the Tables( 2, 6, 7). For IQL+ER, we
start with the IQL agent that has the least cross-play score and train it with
Hard agents sequentially using ER algorithm. In the case of IQL+AUX+ER, we
start with an IQL agent that is pre-trained with AUX and is continually
trained with the Hard agents using ER algorithm. This continually trained
agent is then evaluated with 9 other agents in either IQL or IQL+AUX
respectively in order to obtain Intra-CP scores. However, please note that the
auxiliary task is used only during pre-training and is not used during
continual training. Note that the middle row in the Table 7 is generated using
the latest models released by (Hu et al., 2020).

Table 6: AVG : Comparison with other MARL algorithms on self-play (SP), cross-
play evaluation scores within method (Intra-CP), and across different methods
(Inter-CP). C: centralized training, GA: agents share their greedy action
along with their standard action, L: true labels of cards needed, SYM:
symmetries of the game needed upfront, P: require access to some pre-trained
agents in sequence, UP: Having access to all the fixed pre-trained agents at
the same time. ($\uparrow$ / $\downarrow$ = Difference in score after
continual training red: pre-trained with MARL method, blue: trained
continually with LLL method)
Training method |  | SP |  | Intra-CP |  | Inter-CP |  | Limitations
---|---|---|---|---|---|---|---|---
SAD |  |  23.78 $\pm$ 0.03 |  |  4.38 $\pm$ 0.66 |  |  8.40 $\pm$ 0.23 |  | C+GA
SAD+AUX |  |  23.82 $\pm$ 0.02 |  |  21.15 $\pm$ 0.26 |  |  17.01 $\pm$ 0.22 |  | C+GA+L
SAD+OP |  |  23.67 $\pm$ 0.03 |  |  12.00 $\pm$ 0.86 |  |  12.79 $\pm$ 0.24 |  | C+Sym+GA
SAD+AUX+OP |  |  23.88 $\pm$ 0.03 |  |  22.01 $\pm$ 0.03 |  |  17.08 $\pm$ 0.22 |  | C+Sym+L+GA
IQL \+ ER |  |  20.91 $\pm$ 0.05 ($\downarrow$ 2.98) |  |  15.73$\pm 0.39$ ($\uparrow$ 7.06) |  |  16.32$\pm$0.21 ($\uparrow$ 8.09) |  | P
IQL+AUX \+ ER |  |  22.34$\pm$ 0.06 ($\downarrow 1.46$) |  |  20.90$\pm$ 0.06 ($\downarrow$0.15) |  |  19.17$\pm$0.22($\uparrow$1.33) |  | L+P
IQL \+ Multi-task |  |  20.93$\pm 0.09$ ($\downarrow 2.96$) |  |  16.05$\pm$ 0.30($\uparrow$ 7.38) |  | 17.88$\pm 0.17$ ($\uparrow$ 9.65) |  | UP
Table 7: BEST : Comparison with other MARL algorithms on self-play (SP),
cross-play evaluation scores within method (Intra-CP), and across different
methods (Inter-CP). C: centralized training, GA: agents share their greedy
action along with their standard action, L: true labels of cards needed, SYM:
symmetries of the game needed upfront, P: require access to some pre-trained
agents in sequence, UP: Having access to all the fixed pre-trained agents at
the same time. ($\uparrow$ / $\downarrow$ = Difference in score after
continual training, red: pre-trained with MARL method, blue: trained
continually with LLL method, ∗ : results obtained using models released by (Hu
et al., 2020))
Training Method |  | SP |  | Intra-CP |  | Inter-CP | Limitations
---|---|---|---|---|---|---|---
SAD |  | $23.85\pm 0.03$ |  | $7.70\pm 0.69$ |  | $14.60\pm 0.24$ | C + GA
SAD \+ AUX |  | $23.57\pm 0.03$ |  | $20.97\pm 0.80$ |  | $18.51\pm 0.23$ | C + GA + L
SAD \+ OP |  | $24.14\pm 0.03$ |  | $10.10\pm 0.87$ |  | $16.09\pm 0.25$ | C + Sym + GA
SAD \+ AUX \+ OP |  | $23.40\pm 0.04$ |  | $21.23\pm 0.25$ |  |  17.77 $\pm$ 0.23 | C + Sym + L + GA
${\color[rgb]{1,0,0}SAD}^{*}$ |  | $23.97\pm 0.04$ |  | $2.52\pm 0.0.34$ |  | $11.46\pm 0.35$ | C + GA
${\color[rgb]{1,0,0}SAD}+{\color[rgb]{1,0,0}AUX}^{*}$ |  | $24.09\pm 0.03$ |  | $17.65\pm 0.69$ |  | $17.60\pm 0.42$ | C + GA + L
${\color[rgb]{1,0,0}SAD}+{\color[rgb]{1,0,0}OP}^{*}$ |  |  23.93 $\pm$ 0.02 |  |  15.32 $\pm$ 0.65 |  |  17.50 $\pm$ 0.34 | C + Sym + GA
${\color[rgb]{1,0,0}SAD}+{\color[rgb]{1,0,0}AUX}+{\color[rgb]{1,0,0}OP}^{*}$ |  | $24.06\pm 0.02$ |  | $22.07\pm 0.11$ |  | $17.45\pm 0.38$ | C + Sym + L + GA
IQL \+ ER |  |  $20.91\pm 0.05$ ($\downarrow 2.98$) |  |  $15.73\pm 0.39$ ($\mathbf{\uparrow 7.06}$) |  |  $16.32\pm 0.21$ ($\mathbf{\uparrow 8.09}$) | P
IQL \+ AUX \+ ER |  |  $22.34\pm 0.06$ ($\downarrow 1.46$) |  |  $20.90\pm 0.06$ ($\downarrow 0.15$) |  |  $\mathbf{19.17\pm 0.22}$ ($\mathbf{\uparrow 1.33}$) | L + P
IQL \+ Multi-task |  |  $20.93\pm 0.09$ ($\downarrow 2.96$) |  |  $16.05\pm 0.30$ ($\mathbf{\uparrow 7.38}$) |  |  $\mathbf{17.88\pm 0.17}$ ($\uparrow$ 9.65) | UP


# Repulsive Casimir force between silicon dioxide and superconductor

Anh D. Phan  Department of Physics, University of South Florida, Tampa,
Florida 33620, USA<EMAIL_ADDRESS>N. A. Viet  Institute of Physics, 10
Daotan, Badinh, Hanoi, Vietnam

###### Abstract

We have presented a detailed investigation of the Casimir interaction between
the superconductor $Bi_{2}Sr_{2}CaCu_{2}O_{8+\delta}$ (BSCCO) and silicon
dioxide with the presence of bromobenzene in between. We found the dispersion
force is repulsive and the magnitude of the force can be changed by varying
the thickness of object and temperature. The repulsive force would provide a
method to deal with the stiction problems and bring much meaningful from
practical views.

###### pacs:

Valid PACS appear here

The Casimir force, one of the most important causes of stiction problems,
gives rise to critical impediments in fabrication and operation of nano/micro-
electromechanical (NEMS/MEMS) systems. In almost all cases in which the scale
is of hundreds of nanometers, the Casimir interactions produce such a
significant amount of friction as to draw the much attention of scientists
from a wide variety of research fields bib1 ; bib2 ; bib3 . In addition, the
theoretical understanding and measurements of the Casimir interactions have
grown substantially in the last ten years, allowing physicists to have a more
detailed understanding of fundamental physics, not only in nanophysics, but
particle physics and cosmology as well.

The simplest Casimir system was produced theoretically by Casimir in 1948 when
he developed a model describing the interaction between two parallel
conducting plates bib4 . Since then, the Casimir forces between real materials
such as metals bib5 , semiconductors bib6 , semimetals bib7  and high-Tc
superconductors bib8  have been extensively studied theoretically and
experimentally. It has been found that the presence of liquids between two
objects allow the sign of the Casimir force to switch bib1 ; bib2 ; bib3 .
These repulsive Casimir forces appear when the dielectric functions of object
1 and object 2 immersed in a medium 3 satisfy the relation
$\varepsilon_{1}(i\xi)<\varepsilon_{3}(i\xi)<\varepsilon_{2}(i\xi)$ over a
wide imaginary frequency range $\xi$. It is also possible to make the
repulsion using arrays of gold nanopillars on two plates bib9 . In addition,
metamaterials are promising candidates for creating these repulsive
interactions bib10 . The combination between experimental measurements and
theoretical calculations have provided essential information to help
researchers design nanoscale devices.

The well-known Lifshitz theory developed the generalization of the Casimir
force bib3 ; bib11 ; bib12 . In the theory, the force between uncharged
objects made of real materials was given by an analytical formula with the
frequency-dependent dielectric permittivity $\varepsilon(i\xi)$ . The
variation of the dielectric functions causes the change of the Casimir
interactions. The Casimir-Lifshitz force has been studied at for systems at
the thermal equilibrium between the atom-atom, plane-plane and atom-plane
configurations bib11 ; bib20 .

The cuprate superconductor, the high-Tc superconductor and the anisotropic
materials are widely used in various devices. It has been shown theoretically
that the Casimir force in the BSCCO-air-gold system is significantly affected
by the anisotropy in the dielectric functions. In the present paper, the
Casimir-Lifshitz force is calculated in the case of the perpendicular cleave
between the cuprate superconductor and silica with bromobenzene in between.
The force calculation take into account the thermal effect and the influence
of its thickness on the dispersion force. The force is repulsive and deals
with the sticking process in nano devices.

The general expression describing the Casimir interaction between two infinite
parallel plates is the Lifshitz formula. At a given separation $a$ and given
temperature $T$, the Casimir pressure between two plates is given bib11 ;
bib12

$\displaystyle
P(a)=-\dfrac{k_{B}T}{\pi}\sum_{n=0}^{\infty}\int_{0}^{\infty}qk_{\perp}dk_{\perp}\sum_{\alpha}\dfrac{r_{\alpha}^{(1)}r_{\alpha}^{(2)}}{e^{2qa}-r_{\alpha}^{(1)}r_{\alpha}^{(2)}},$
(1)

here $k_{B}$ is the Boltzmann constant, $k_{\perp}$ is the wave vector
component perpendicular to the plate, $\alpha=TM,TE$, $r_{TM}^{(1,2)}$ and
$r_{TE}^{(1,2)}$ denote the reflection coefficients of the transverse magnetic
(TM) and transverse electric (TE) field, respectively. The superscript (1) and
(2) correspond to the first body (silica) and the second body (BSCCO). In
addition, $q=\sqrt{k_{\perp}^{2}+\varepsilon_{3}\xi_{n}^{2}/c^{2}}$,
$\xi_{n}=2\pi nk_{B}T/\hbar$ are the Matsubara frequencies,$n$ is an integer,
and $\varepsilon_{3}\equiv\varepsilon_{3}(i\xi_{n})$ is the dielectric
function of medium in between two objects. In the calculation, bromobenzene is
medium. The dielectric function of the liquid can be described using the
oscillator model bib2

$\displaystyle\varepsilon_{3}(i\xi)=1+\sum_{i}\frac{C_{i}}{1+\xi^{2}/\omega_{i}^{2}},$
(2)

where parameters $C_{i}$ and $\omega_{i}$ were obtained by fitting with
experimental data in the large range of frequency bib2 .

It is important to note that for $n=0$, the prefactor of the integration is
$k_{B}T/(2\pi)$ instead of $k_{B}T/\pi$ for other values of $n$. In the case
of silicon dioxide, the reflection coefficients are presented bib2 ; bib3

$\displaystyle
r_{TM}^{(1)}=\frac{\varepsilon_{1}q-\varepsilon_{3}\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}}{\varepsilon_{1}q+\varepsilon_{3}\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}},$
(3) $\displaystyle
r_{TE}^{(1)}=\frac{q-\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}}{q+\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}},$
(4)

in which $\varepsilon_{1}\equiv\varepsilon_{1}(i\xi_{n})$ is the dielectric
function of silica. The dielectric fuction still has the form of an oscillator
model as in Eq.(2) and the parameters were generated in Ref.bib2 . Considering
the role of the thickness $D$ of the silica slab on the Casimir interaction,
the reflection coefficients $TM$ and $TE$ in Eq.(3) and Eq.(4) become bib13 ;
bib14

$\displaystyle r_{TM}^{(1)}\rightarrow
r_{TM}^{(1)}\frac{1-e^{-2\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}D}}{1-r_{TM}^{(1)2}e^{-2\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}D}},$
(5) $\displaystyle r_{TE}^{(1)}\rightarrow
r_{TE}^{(1)}\frac{1-e^{-2\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}D}}{1-r_{TE}^{(1)2}e^{-2\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}D}}.$
(6)

Because of the uniaxial property of BSCCO, its permittivity in the
perpendicular cleave is represented in the form of a tensor bib8

$\displaystyle\bar{\varepsilon_{2}}=\left[\begin{array}[]{ccc}\varepsilon_{2\perp}&0&0\\\
0&\varepsilon_{2\perp}&0\\\ 0&0&\varepsilon_{2||}\end{array}\right]$ (10)

where $\varepsilon_{2||}$ and $\varepsilon_{2\perp}$ are the dielectric
components along the optical axis and perpendicular to the optical axis.
Therefore, the expression of the reflection coefficients for BSCCO must be
modified. The TM and TE coefficients on the liquid-BSCCO interface are
performed bib8 ; bib15

$\displaystyle
r_{TM}^{(2)}=\frac{\varepsilon_{2\perp}q-\varepsilon_{3}\sqrt{\frac{\varepsilon_{2\perp}}{\varepsilon_{2||}}k_{\perp}^{2}+\varepsilon_{2\perp}\xi_{n}^{2}/c^{2}}}{\varepsilon_{2\perp}q+\varepsilon_{3}\sqrt{\frac{\varepsilon_{2\perp}}{\varepsilon_{2||}}k_{\perp}^{2}+\varepsilon_{2\perp}\xi_{n}^{2}/c^{2}}},,$
(11) $\displaystyle
r_{TE}^{(2)}=\frac{q-\sqrt{k_{\perp}^{2}+\varepsilon_{2\perp}\xi_{n}^{2}/c^{2}}}{q+\sqrt{k_{\perp}^{2}+\varepsilon_{2\perp}\xi_{n}^{2}/c^{2}}}.$
(12)

For BSCCO, the dielectric response $\varepsilon_{2||}$ and
$\varepsilon_{2\perp}$ are modeled based on the damped-multioscillator model.
Parameters corresponding to the resonance frequency, damping and oscillator
strength were given in Ref.bib8 .

As shown in Fig. 1, the Casimir forces are significantly influenced by the
thickness of the slab. The Casimir interactions in the real system are much
smaller than the force in the ideal case. It is clear that the presence of
bromodenzene makes the Casimir force in this case is repulsive. For $D>500$
$nm$, the effect of thickness nearly vanishes and the Casimir force in the
system can be modeled as an interactions between two plates. For metal such as
gold, when the thickness $D>30$ $nm$, a gold thin film can be treated as a
gold plate bib16 ; bib17 . The reason for the discrepancy between the two
cases is that the conductivity of metal is much larger than that of silicon
dioxide. When the thickness of a metal slab is reduced, it appears as if the
skin-depth effect occurs. This effect, however, is not present in the case of
silica.

Figure 1: (Color online) The relative Casimir pressures between a BSCCO plate
and a silica slab in the presence of bromobenzene, here $F_{0}(a)=\pi^{2}\hbar
c/(240a^{4})$ is the Casimir force between two ideal metal plates.

At small distances, there is not much change in the force with different
thicknesses. The reason is that at this range $D/a>>1$, so
$e^{-2\sqrt{k_{\perp}^{2}+\varepsilon_{1}\xi_{n}^{2}/c^{2}}D}<<1$. The
influence of thickness on the interaction disappears.

Figure 2: (Color online) The relative Casimir pressures are taken into account
the thermal effect with different values of thickness.

Bromobenzene molecules exist in liquid form over an important range of
temperature from $242$ $K$ to $429$ $K$. Fig. 2 shows the Casimir force in
this temperature range. It is evident that the interaction depends notably on
temperature. There are variations in the Casimir force at different
thicknesses. It is now possible to design a non-touching system because the
Casimir force is repulsive for entire range of distance. The Casimir force,
and the gravitational forces between the two bodies, with the earth lead to
the repulsive-attractive transition in our system and results in our system
reaching an equilibrium distance bib18 . Obviously, the stable position can be
varied by changing the sizes of bodies, the thickness and temperature. In
Ref.bib19 , authors presented the proposal for measuring the Casimir force
with the presence of a tiny spring in order to get a balanced position and the
oscillation frequency. However, in this case, the equilibrium positions exist
naturely. It is not necessary to attach a spring to the system to measure the
Casimir force. The force can be found via observation of the oscillation
frequencies. Because of the anisotropic property, the expressions of the
reflection coefficients TE and TM in the case of the parallel cleave
orientation are different from Eq.(11) and Eq.(12). This discrepancy is proof
that there is a difference between the Casimir forces in two orientations.

The thermal effect in the Casimir interaction plays an important role at long
distances bib12 . For short distances, this effect can be ignored. One can
used the double integration instead of summation and single integration as
Eq.(1) in calculations at short distances. The expression of the double
integration provides a good agreement with experiment.

This work presents a reliable anti-stiction method of NEMS/MEMS structures.
The presence of liquid can address the stiction issue causing catastrophic
failure in nanoscale devices. The temperature and thickness depedence of the
Casimir force allows control of the adhesion force between two surfaces. It is
currently difficult to measure properties in fluidic environments. However,
using liquid films in NEMS/MEMS devices with the range of the thickness of
liquid layer from 2 $nm$ to 70 $nm$ has been intensively investigated bib21 ;
bib22 . Moreover, authors in bib23  described the behavior of the tiny devices
in liquids. These research makes it possible to design nanostructures in the
microfluidic environment and our studies give an interesting view of what
happens physically in systems submerged in liquids.

###### Acknowledgements.

The work was partly funded by the Nafosted Grant No. 103.06-2011.51.

## References

  * (1) J. N. Munday, Federico Capasso and V. Adrian Parsegian, Nature 457, 170-173 (2009).
  * (2) P. J. van Zwol and G. Palasantzas, Phys. Rev. A 81, 062502 (2010).
  * (3) Anh D. Phan and N. A. Viet, Phys. Rev. A 84, 062503 (2011).
  * (4) H. B. G Casimir, Proc. K. Ned. Akad. Wet. 51, 793 (1948).
  * (5) P. J. van Zwol, G. Palasantzas, and J. Th. M. De Hosson, Phys. Rev. B 79, 195428 (2009).
  * (6) F. Chen, U. Mohideen, G. L. Klimchitskaya, and V. M. Mostepanenko, Phys. Rev. A 72, 020101 (2005).
  * (7) G. Torricelli, I. Pirozhenko, S. Thornton, A. Lambrecht, and C. Binns, EPL 93, 51001 (2011).
  * (8) Mark B. Romanowsky and Federico Capasso, Phys. Rev. A 78, 042110 (2008).
  * (9) Vassilios Yannopapas and Nikolay V. Vitanov, Phys. Status Solidi RRL 4, 19-21 (2010).
  * (10) Vassilios Yannopapas and Nikolay V. Vitanov, Phys. Rev. Lett. 103, 120401 (2009).
  * (11) G. L. Klimchitskaya, U. Mohideen, and V. M. Mostepanenko, Rev. Mod. Phys. 81, 1827-1885 (2009).
  * (12) V. Yannopapas and N. V. Vitanov, Phys. Rev. A 81, 042506 (2010).
  * (13) A. O. Sushkov, W. J. Kim , D. A. R. Dalvit and S. K. Lamoreaux , Nature 7, 230-233 (2011).
  * (14) Reza Matloob and Hossain Falinejad, Phys. Rev. A 64, 042102 (2001).
  * (15) M. S. Tomas, Phys. Rev. A 66, 052103 (2002).
  * (16) F. S. S. Rosa, D. A. R. Dalvit, and P. W. Milonni, Phys. Rev. A 78, 032117 (2008).
  * (17) Mariangela Lisanti, Davide Iannuzzi, and Federico Capasso, PNAS 102, 11989-11992 (2005).
  * (18) P. J. van Zwol, G. Palasantzas, and J. Th. M. DeHosson, Phys. Rev. E 79, 041605 (2009).
  * (19) Diego A. R. Dalvit and Roberto Onofrio, Phys. Rev. D 80, 064025 (2009).
  * (20) Gabriel Vidal, Jordi Agust , Francesc Torres, Gabriel Abadal, and Nuria Barniol , Phys. Status Solidi C 7, 2708-2711 (2010).
  * (21) Manuel Palacio and Bharat Bhushan, Adv. Mater. 20, 1194-1198 (2008).
  * (22) Yufei Mo, Fuchuan Huang, and Fei Zhao, Surf. Interface Anal. 43, 1006-1014 (2011).
  * (23) I. Etchart, H. Chen, P. Dryden, J. Jundt, C. Harrison, K. Hsu, F. Marty, and B. Mercier, Sensors and Actuators a-Physical 141, 266-275 (2008).