# Learning Modular Robot Locomotion from Demonstrations

Julian Whitman and Howie Choset Carnegie Mellon University, Pittsburgh PA,
USA<EMAIL_ADDRESS>
###### Abstract

Modular robots can be reconfigured to create a variety of designs from a small
set of components. But constructing a robot’s hardware on its own is not
enough– each robot needs a controller. One could create controllers for some
designs individually, but developing policies for additional designs can be
time consuming. This work presents a method that uses demonstrations from one
set of designs to accelerate policy learning for additional designs. We
leverage a learning framework in which a graph neural network is made up of
modular components, each component corresponds to a type of module (e.g., a
leg, wheel, or body) and these components can be recombined to learn from
multiple designs at once. In this paper we develop a combined reinforcement
and imitation learning algorithm. Our method is novel because the policy is
optimized to both maximize a reward for one design, and simultaneously imitate
demonstrations from different designs, within one objective function. We show
that when the modular policy is optimized with this combined objective,
demonstrations from one set of designs influence how the policy behaves on a
different design, decreasing the number of training iterations needed.

5in(0.55 in,0.5in) Preprint: Submitted to the 2023 International Conference on
Robotics and Automation

##  I Introduction

This paper introduces a framework for combination reinforcement and imitation
learning (RL+IL) to create policies for modular robots. Modular robots allow a
variety of designs to be constructed from a small set of inter-changeable
hardware components. But, constructing the robot is only the first step toward
deploying it. Each design needs a control policy, and it can be time-consuming
to develop a new policy for each new robot, especially given the large number
of the designs one could create.

We address this problem by training new robots to exhibit behaviors
demonstrated by different existing robots. That is, the novel feature of our
method is that given demonstrations from multiple robots with different
designs, we can train a policy to control a robot with yet a different design
from those shown in the demonstrations. This is where our RL+IL work differs
from prior efforts– related work assumes the robot in demonstrations and the
robot trained have the same designs [1, 2, 3, 4], for instance, using
demonstrations from a robot with four legs to train a robot with four legs. We
constrain our methods to robots composed from combinations of a set of
modules, but allow robots to have varying numbers and types of limbs.

Figure 1:  This work presents a new combination reinforcement and imitation
learning algorithm for modular robots. With our method, demonstrations can
come from robots with multiple different designs, and can be different yet
than the design the policy is trained to control. A modular policy (top)
consists of neural network components used by each module, represented by
brain icons. All modules of a given type use the same neural network, e.g.,
all wheels use the same blue “brain” even when they are placed in different
locations on a single robot or placed in different robots. The policy can be
applied to different designs by connecting the components into a graph, where
the graph structure corresponds to the connectivity of modules in the robot.
In this example, data from a hexapod (bottom left) and car-like robot (bottom
center) are used to train a robot with both legs and wheels (bottom right).
The demonstrations are used in the imitation learning part of training loss
indicated by the unfilled arrows, and the leg-wheel robot is used in the
reinforcement learning part of the loss, indicated by the filled arrow.

The RL+IL method introduced here, as its name suggests, optimizes the policy
by summing two parts: a reinforcement learning (RL) part that maximizes the
reward received from robot-environment interactions, and an imitation learning
(IL) part that minimizes the difference between the actions output by the
policy and the actions shown through demonstrations of a system executing a
task. The demonstrations can be represented as a dataset of trajectories,
which do not have an associated reward, and in fact, we may not even know how
“good” the demonstrations are. We leverage the demonstrations to bootstrap RL
for new designs, beyond those designs present in the demonstrations, such that
the demonstrations act effectively as a “prior” for the policy.

The ability to “mix and match” between the design that the policy is trained
to control (for the RL part) and the designs in the demonstrations (for the IL
part) stems from the fact that we use modular policies. Our prior work [5]
introduced the concept of modular policies111Our use of the term “modular
polices” shares similarities with the “shared modular policies” of [6]. whose
intrinsic structure allows them to train with multiple robot designs. In the
modular policy framework, the policy is divided up into inter-changable
components, each of which corresponds to a type of module (e.g., a leg, a
wheel, or a body) (Fig. 1). To apply the policy to a robot, first the robot is
represented as a graph, where modules represent nodes and edges represent
electromechanical connections between them. Then, the components of the
modular policy are reconfigured into a graph matching the structure of the
physical robot. By reconfiguring the policy during training, it can learn from
and apply to a variety of designs at once. Previously, modular policies were
trained with RL. We develop a RL+IL method with modular policies, where the
policy is reconfigured to apply to different designs in the two parts of the
training objective function.

We apply our method to train a locomotion policy for robots with different
combinations of legs and wheels. Our experiments show how including
demonstrations, even from robots with different designs than the robot being
trained, can accelerate learning, by causing the policy to converge in fewer
iterations than when learning without demonstrations.

##  II Background

###  II-A Reinforcement and imitation learning

RL can train a control policy, often implemented as a deep neural network,
using data collected iteratively from an agent’s interaction with its
environment [7]. RL methods have been developed recently to control a variety
of robots including articulated locomoting robots [7, 8, 9, 10, 11, 12].
However, the most frequently-used RL formulations assume that robot behavior
must be discovered without prior knowledge.

Prior knowledge can be inserted into RL methods most often through one of two
approaches. Firstly, a specific motion can be encouraged via reward shaping,
e.g., rewarding a specific footfall pattern [5, 13]. Secondly, motions can be
forced by constructing action spaces which superpose policy outputs onto a
hand-crafted periodic pattern, e.g., perturbing a nominal trot gait output by
an open-loop trajectory generator [14, 15, 16, 17, 18]. Both of these methods
require experts to manually identify and specify desired motions for the
robots being trained, and alter the rewards or actions accordingly.

A policy can also be taught from prior knowledge using IL. In IL, the policy
is trained to reproduce the actions taken within a dataset of demonstrations.
For example, the most long-standing IL method known as “behavioral cloning”
trains a policy with supervised regression [19, 20]. Expert demonstrations
typically come either from humans [21, 22] or a model-predictive controller
[23, 24]. When the demonstrations are not sufficient on their own to learn a
global policy, RL+IL can be combined to leverage a dataset of demonstrations
without being limited to states and actions contained therein [2, 3, 1, 4,
25]. For example, [2] combined RL and IL by training a value function
approximation network from state transitions in a dataset. [3] used expert
demonstrations within a variant of hindsight experience replay [26]. [1]
constrained the states visited by the policy to be near the states in the
dataset. A limitation of these methods is that they require the robot design
in the demonstrations be similar to the robot being trained, for instance,
having the same number and type of joints and limbs.

###  II-B Modular policies

Our past work [5] introduced the modular policy framework, geared toward
systems where multiple designs are constructed from a set of modules. Modular
policies leverage the fact that the structure of a modular robot can be
represented as a design graph, with nodes as modules and edges as connections
between them. A design graph is used to create a policy graph with the same
structure (Fig. 1). Within a policy graph, each type of module has a neural
network associated with it– i.e., there is one network used to control all of
the “leg” modules, and a different one used to control all of the “wheel”
modules. Each node in the policy graph uses the neural network associated with
its corresponding module type to compute a module’s actions from its
observations. For example, a single neural network for leg-type modules is
used to compute actions for each of the legs on a hexapod, one leg at a time.

Modular policies are implemented using message-passing graph neural networks
(GNN) [27, 28], which allows modules to learn to modulate their outputs via a
communication procedure in which they exchange information over the graph
edges. As a result, the policy produces different behaviors for a module
depending on its location within the robot, and the presence of other modules
within the robot. In this work, we use the policy architecture of [5], but
train it with a new RL+IL algorithm.

##  III RL+IL for modular robots

We are not the first to use RL [5, 29, 30, 16] nor RL+IL [1, 2, 3] for robot
locomotion, and as such our methods use concepts from these prior methods. The
main novelty of our method lies in the combination of modular policies with an
RL+IL algorithm. Recall our prior work [5] used an RL approach to train one
modular policy for multiple designs at once. In this work, we add an IL part
to the training, which influences the behaviors the policy learns with the RL
part, such that a new robot can learn from behaviors demonstrated by other
robots with different designs. We next pose the policy optimization problem
addressed, then describe how RL and IL objectives are formulated, followed by
training algorithm details.

Figure 2: A flow chart depicting the RL+IL algorithm phases. Each block in
this diagram is described in Sec. III, where the subsection label is shown in
blue in the corner of each block.

###  III-A Problem formulation

The design $d_{\textrm{train}}$, which the policy is being trained to control
is first chosen by the user. The objective we optimize, as is conventionally
used in deep RL, maximizes the reward a policy would accumulate when applied
to that design. This maximization problem is,

$\displaystyle\theta^{*}=$
$\displaystyle\operatorname*{argmax}_{\theta}\mathbb{E}_{a\sim\pi_{\theta}}\sum_{t=1}^{T}r(s_{t},a_{t},d_{\textrm{train}})$
(1) $\displaystyle\textrm{s.t.}\quad
s_{t+1}=f(s_{t},a_{t},d_{\textrm{train}})\qquad\textrm{(Dynamics)}$ (2)
$\displaystyle a\sim\pi_{\theta}(a|o,d)\qquad\textrm{(Policy)}$ (3)
$\displaystyle o\sim O(o|s)\qquad\textrm{(Observation function)}$ (4)
$\displaystyle s_{o}\sim p(s)\qquad\textrm{(Initial state distribution.)}$ (5)

The policy $\pi_{\theta}$ is a distribution over actions, with parameters
$\theta$, conditioned on a design and observations. The reward $r:S\times
A\times\mathcal{D}\to\mathbb{R}$ is summed over a time horizon $T$. The
observation function $O$ adds noise to, or removes, parts of the state vector.
We choose the same observation function as [31], which removes the x/y
position, yaw, and the linear velocity of the body from the state, with
Gaussian noise added to the remaining joint and IMU sensor readings.

We solve (1) by developing a model-based reinforcement learning (MBRL) method.
We choose this class of policy optimizer over others possible, such as model-
free reinforcement learning (MFRL), primarily because MBRL has been shown by
past studies to be more data-efficient than MFRL [32, 33, 30, 34]. An MBRL
algorithm learns an approximate model $\tilde{f}_{\phi}$ with parameters
$\phi$ and then optimizes the policy parameters with that model as a
differentiable stand-in for the true dynamics. Our algorithm is inspired in
particular by the algorithms of [5] and [30], and similarly iterate between
collecting data, training the model, and training the policy. The phases of
the algorithm, which are depicted in flow-chart form in Fig. 2, and psuedocode
in Algorithm 1.

###  III-B Gather trajectories with random actions

First, the robot $d_{\text{train}}$ is randomly initialized at states
perturbed from its nominal position. We create random actions by sampling $10$
values from a normal distribution for each degree-of-freedom, and fitting
splines to create smooth joint commands over $100$ time steps, similar to the
methods used by [29, 5]. These actions are applied to obtain an initial
dataset of trajectories $\mathcal{T}_{\textrm{train}}$.

###  III-C Train approximate dynamics model

Next, the dataset $\mathcal{T}_{\textrm{train}}$ is used to fit an approximate
dynamics model for $d_{\textrm{train}}$, with the same supervised model
learning methods described by [5]. The model $\tilde{f}_{\phi}$ is a modular
model with the same structure as the modular policy, i.e., $\tilde{f}_{\phi}$
is implemented as a graph neural network with body, leg, and wheel module node
types. The body node takes as input the state of the body (world frame
orientation and velocity), and the leg and wheel nodes take in the joint
angles and joint velocities from their respective modules. The model outputs
an estimated change in state given a state and action input.

###  III-D Gather demonstration datasets

The set $D_{\text{demo}}$ can contain multiple designs, each of which has
multiple trajectories that form a demonstration dataset. We define a
demonstration trajectory $\tau$ as a sequence of observations and actions for
a single design over an $H$-step horizon. Each design has a set of $L$
trajectories, $\mathcal{T}^{i}$, for a design
$i\in\\{1\dots|D_{\text{demo}}|\\}$. The collection of these trajectories make
up the full demonstration dataset $\mathcal{T}_{\text{demo}}$,

$\begin{split}\tau=\\{(o_{0},a_{0}),\dots(o_{H},a_{H})\\},\quad\mathcal{T}^{i}=\\{\tau^{i}_{1}\dots\tau^{i}_{L}\\},\\\
\mathcal{T}_{\text{demo}}=\\{\mathcal{T}^{1},\dots\mathcal{T}^{|D_{\text{demo}}|}\\}\end{split}$
(6)

These demonstrations do not need to contain full states nor rewards, and can
be created via an existing policy such as open-loop hand-crafted behaviors, or
human tele-operation.

###  III-E Pre-train policy with behavioral cloning

Before the RL phase of policy optimization begins, we use IL to “pre-train” a
policy, that is, to warm-start the policy for the later RL+IL phase. In policy
pre-training, we employ behavioral cloning using the demonstration dataset.

To encourage the policy to imitate the actions in the dataset we use a log
likelihood loss,

$\mathcal{L}_{\textrm{IL}}=-\sum_{d\in
D_{\textrm{demo}}}\sum_{(o,a)\in\mathcal{T}_{\textrm{demo}}}\log(\pi_{\theta}(a|o,d)).$
(7)

This IL loss is the joint log likelihood that the policy will produce the
actions from the dataset when given the observations from the dataset. This
loss is different from a conventional behavioral cloning loss because it
includes a summation over designs. Note each design is made up from a
different combination of modules, each of which may have a different number of
degrees-of-freedom (and thus a different action and observation
dimensionality). We can compute this loss using a modular policy because the
modular policy can apply to multiple designs.

The policy trained from this loss is not effective globally– it does not know
how to recover to from arbitrary random positions drawn from an initial state
distribution $p(s)$. It also only uses $D_{\textrm{demo}}$, and is not applied
at this phase to $d_{\textrm{train}}$. However, it serves to initialize the
policy for further training. This phase can also be conducted in parallel with
model learning (Sec. III-C).

###  III-F Train policy using model and RL

The policy is optimized to minimize objective cost (maximize reward) with
respect to the model. In this phase, model parameters are frozen, and the
policy is used to compute a series of states, actions, and observations. The
RL loss function derived from (1) is

$\mathcal{L}_{\textrm{RL}}=-\mathbb{E}_{a\sim\pi_{\theta}}\sum_{t=1}^{T}r(\tilde{s}_{t},a_{t},d_{\textrm{train}}).$
(8)

The policy parameters are optimized with respect to roll-outs with the
approximate dynamics model, where the approximate state evolves according to
the model
$\tilde{s}_{t+1}=\tilde{f}_{\phi}(\tilde{s}_{t},a_{t},d_{\textrm{train}})$.
The initial state $\tilde{s}_{t}=s_{0}$ is taken from the simulation. Actions
are sampled according to the policy,
$a_{t}\sim\pi_{\theta}(a_{t}|o_{t},d_{\textrm{train}})$, and observations from
the observation function $o_{t}\sim O(\tilde{s}_{t})$.

Over the time horizon length $T$, the policy is applied to the approximate
state, a reward is computed, and the state is advanced according to the model.
The total reward is accumulated in $\mathcal{L}_{\textrm{RL}}$. After a
$T$-step roll-out of forward passes using the model and policy, the policy
parameter gradients can be computed by back-propagating through the end-to-end
differentiable sequence of states and actions. These forward and backward
passes are repeated for $K$ steps with random mini-batches of
$N_{\textrm{batch}}$ initial states. This process takes advantage of the fact
that the a neural network-based model is differentiable and can be applied in
large parallel batches.

The main term in the reward function $r$ rewards the locomotion forward in the
$+x$ direction. Other terms with smaller relative weights penalize roll,
pitch, yaw, deviation from $y=0$, control effort, and distance from a nominal
pose. The reward does not include motion style terms, and the action space is
commanded joint velocities without the need for any open-loop user-defined
trajectory.

###  III-G Train policy using model and RL+IL

On its own, (1) searches for a policy without prior knowledge. We next turn to
incorporating behavioral priors derived from a set of trajectories from
designs $D_{\text{demo}}$.

To optimize the policy parameters so that they achieve both the RL and IL
objectives, we can add the IL loss (7) to the RL loss (8) with weight
$\lambda\in\mathbb{R}^{+}$,

$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{\textrm{RL}}+\lambda\mathcal{L}_{\textrm{IL}}.$
(9)

and optimize policy parameters with respect to this combined loss, with
stochastic gradient descent on $d\mathcal{L}/d\theta$.

Where in the earlier pre-training phase, the policy was optimized only with
respect to the IL loss, in this phase the policy is optimized with respect to
the combined RL and IL loss (9). At each iteration in the policy optimization
phase, a batch of trajectories are sampled from $\mathcal{T}_{\textrm{demo}}$,
used to compute the IL loss, and added to the RL loss. As a result, the policy
is encouraged to reproduce the actions in the demonstrations, without being
limited to those behaviors. With this loss, the policy can learn recovery
behaviors and proceed from initial states outside of those contained in the
demonstration dataset.

A novel aspect of our RL+IL method is that demonstration designs in
$\mathcal{L}_{\textrm{IL}}$ do not need to be the same as the training designs
in $\mathcal{L}_{\textrm{RL}}$. This is possible because the modular policy
enables one set of neural network parameters $\theta$ to apply to multiple
different designs. As long as $d_{\text{train}}$ and designs in
$D_{\text{demo}}$ are constructed with combinations of modules, our RL+IL
method allows for robots with different designs to learn from one another.

###  III-H Gather trajectories with policy actions

After the policy training phase, the policy is used to gather additional
trajectories in simulation. Policy optimization can exploit model bias [35] to
produce a policy that will be low-cost under the model but could be high-cost
in the simulation environment. To correct model bias near states created by
the policy, we apply the policy to gather trajectories, and use those
trajectories to fine-tune the model. At the next iteration of policy
optimization the model will be more accurate, causing a virtuous cycle in
which the model improves near states visited by the policy, and the policy is
optimized with respect to the refined model. The policy is applied without
added noise to the simulation environment. Then, it is applied with time-
correlated noise where the noise is created in the same manner as in Sec.
III-B, but with a lower noise variance. The resulting trajectories are added
to $\mathcal{T}_{\textrm{train}}$. The model is fine-tuned with
$\mathcal{T}_{\textrm{train}}$, and used for the next policy optimization
phase.

###  III-I Reduce IL loss weight

The drawback to using demonstrations to bootstrap policy learning is that
those demonstrations may not be optimal with respect to the RL objective, such
that the IL loss (7) could limit the minimum RL loss (8) that could be
achieved without IL. To account for this possibility, we set $\lambda$ in (9)
to decay at each iteration. By decaying the IL term, we observe the policy
initially imitates the demonstrations, then as the algorithm iterations
between phases, the same motion style (e.g. alternating tripod-like gaits)
qualitatively persist even when as the behavior may otherwise diverge from the
demonstrations.

###  III-J Implementation details

A few additional implementation details helped the algorithm to converge
reliably. Firstly, where the policy in our prior work [5] was not recurrent
over time, the policy in this work is recurrent over time. We found that a
recurrent policy trained more reliably than a non-recurrent policy, due to
vanishing gradients over the $T$-step roll-out with the policy and model. We
apply truncated back-propagation through time [36] to train the recurrent
policy to operate for more than $T$ time steps. Each time a batch of initial
states for the policy roll-out are sampled, half of those states come from the
simulation, and half of the initial states come from states approximated by
applying the policy to the model. The approximated states are associated with
recurrent hidden state vectors, which are used as the initial hidden recurrent
vector when those approximate states are used as initial states in a
subsequent batch.

Secondly, with a fixed or decaying learning rate, the policy optimization
would sometimes diverge. We used a simple adaptive learning rate rule to
stabilize policy optimization. At the start of each policy optimization phase,
we compute the net reward from applying the policy for $T$ steps to a large
set (e.g. 10x the batch size) of initial states. This “validation reward” acts
similarly to a validation loss in a supervised learning problem. If the
current validation reward is lower than the initial validation reward, the
policy parameters revert to the last point when the validation reward was
computed, the learning rate reduces, and continues.

Occasionally, we encountered a poor-quality local minimum that can occur when
the RL and IL loss conflict during early iterations. The likelihood of the
policy imitating the demonstrations could be low and have a large variance,
while the reward obtained is high. To prevent this situation, we penalize the
entropy of the actions output by the policy when given observations from the
demonstration dataset, that is, an additional loss term of the form
$\sum_{o\in\mathcal{T}_{\textrm{demo}}}H(\pi_{\theta}(a|o)$ where $H(\cdot)$
is the entropy function. This term effectively rewards the policy for being
“certain” (low variance) about the actions output in the IL loss calculation.
We found this additional loss term stabilized the initial RL+IL iterations.

Algorithm 1 MBRL with IL for modular robots.

1: Train policy $\pi_{\theta}$ with behavioral cloning from
$\mathcal{T}_{\textrm{demo}}$

2: Collect dataset $\mathcal{T}_{\textrm{train}}$ from random action
trajectories

3: for $i=1\dots$ Num. iterations do

4: Model learning phase:

5: Train model $\tilde{f}_{\phi}$ from $\mathcal{T}_{\textrm{train}}$ with
behavioral cloning

6: Policy optimization phase:

7: Initialize buffer $\mathcal{B}$ with random initial robot states and zero-
valued policy hidden-state vectors.

8: for $k=1\dots K$ do

9: Sample a batch of initial states $s_{0}$ with hidden-state vectors from
$\mathcal{B}$, and set hidden state of $\pi_{\theta}$

10: $R=0$, $\tilde{s}_{0}=s_{0}$

11: Roll out policy with approx. dynamics

12: for $t=1\dots T$ do

13: $a_{t}\sim\pi_{\theta}(a_{t}|O(\tilde{s}_{t}))$,
$\tilde{s}_{t+1}=\tilde{f}_{\phi}(\tilde{s}_{t},a_{t})$

14: $R=R+r(\tilde{s}_{t},a_{t})$

15: end for

16: $\mathcal{L}_{\textrm{RL}}=-R$

17: if IL weight $\lambda>0$ then

18: Sample a batch of $(o,a)$ from $\mathcal{T}_{demo}$

19: Compute $\mathcal{L}_{\textrm{IL}}$ per (7),
$\mathcal{L}=\mathcal{L}_{\textrm{RL}}+\lambda\mathcal{L}_{\textrm{IL}}$

20: end if

21: Update $\pi_{\theta}$ via gradient descent with $d\mathcal{L}/d\theta$

22: if $k\%\textrm{update rate}==0$ then

23: Create a batch of new random initial states, and apply the policy for
$T/2$ steps

24: Overwrite part of $\mathcal{B}$ with intermediate states and policy
hidden-state vectors, i.e.,
$\mathcal{B}^{N_{\textrm{batch}}/2:N_{\textrm{batch}}}\leftarrow\\{s_{T/2,i},h_{T/2}\\}_{i=N_{\textrm{batch}}/2}^{N_{\textrm{batch}}}$

25: end if

26: end for

27: Data collection:

28: Apply $\pi_{\theta}$ to collect data $\mathcal{T}_{\textrm{new}}$

29:
$\mathcal{T}_{\textrm{train}}\leftarrow\mathcal{T}_{\textrm{train}}\cup\mathcal{T}_{\textrm{new}}$

30: Decrease IL weight $\lambda$

31: end for

Figure 3:  We test our RL+IL method with experiments in which we train robots
with different combinations of legs and wheels to locomote. Demonstrations
come from car-like and hexapod robots (left column), and the policy is trained
for robots with either the same or different designs (right column). These
plots show the results of experiments described in Sec. IV, and compare the
outcome of training with and without demonstration data (RL vs. RL+IL). In
each plot, the blue curve corresponds to the average over three trials when
training with RL+IL method, and the red curve to the RL method without IL. The
shaded regions indicate the maximum and minimum value for the trials. (a) A
hexapod trained using demonstrations of a hexapod walking with an alternating
tripod gait, (b) A car trained using demonstrations of a car driving with skid
steering, (c) a robot with four legs and two wheels (leg-leg-wheel) trained
using demonstrations from both a hexapod and a car, and (d) a robot with two
legs and two wheels (leg-none-wheel) trained using demonstrations from both a
hexapod and a car. RL+IL required fewer iterations than RL to converge in (a)
from unconverged to two iterations, (c) from three to two iterations, and (d)
from four to one iteration. RL+IL decreased the variance in convergence rate
in all four cases.

##  IV Experiments

We conducted experiments to show the benefit of imitating demonstrations from
modular designs using our RL+IL algorithm. The demonstration trajectories
$\mathcal{T}_{demo}$ can be derived from robots with either the same designs,
or different designs, as the designs the policy is trained to control, as long
as all designs involved are made up of different combinations of a set of
modules. We hypothesize that the inclusion of demonstration trajectories can
improve policy convergence speed. We further hypothesize that the improvement
in convergence speed will be more noticeable when the training designs are the
same as the demonstration designs, but that there will still be an effect when
the training and demonstration designs are disjoint sets, i.e., when the
design the policy learns to control are not shown by the demonstration
dataset.

In these experiments, we use a module set with legs, wheels, and a body. We
create demonstration data for two designs: the four-wheel car (eight D.o.F.)
and the six-leg hexapod (eighteen D.o.F.) by hand-crafting behaviors based on
expert knowledge. The hexapod demonstrations show alternating tripod gait
trajectories, where the left and right side step sizes are modulated to steer
the body yaw towards zero while walking forward. The car demonstrations show
differential drive skid-steering, in which the left and right side wheel
velocities are modulated to steer the body yaw towards zero while driving
forward. For both the car and hexapod, we start the robot in 50 different
randomly perturbed initial conditions, and create trajectories of 100 time
steps showing the robot turning toward zero yaw and moving forward. These
trajectories form the dataset $\mathcal{T}_{\textrm{demo}}.$ We used the
NVIDIA IsaacGym simulator [37] with a time step of $1/60$ s, and each action
is applied for five time steps.

We test two cases: firstly, where demonstrations are from the training designs
($D_{\textrm{demo}}=\\{d_{\textrm{train}}\\}$), and secondly, where
demonstrations are from different designs ($d_{\text{train}}\notin
D_{\text{demo}}$). In each test, we run the algorithm three times for five
iterations, average the results, then plot the max, min, and mean of the
distance travelled in 16.7 s (200 simulation time steps). Each condition
trained for about 2 hours on a computer with an 8-core AMD Ryzen 7 2700X
processor and a NVIDIA GTX 1070 GPU. The results of these tests are shown in
Fig. 3.

In the first two tests, we set $d_{\textrm{train}}$ to be only either the
hexapod (six-leg) or car (four-wheel) design, and measure the effect of
including demonstrations in training. We found that including hexapod
demonstrations improved the convergence speed for the hexapod. However, since
the car is able to learn to locomote nearly optimally in a single iteration,
the car demonstrations had little effect on the car policy convergence. This
result suggests that when the learning task is less complex (lower
dimensional, smoother), the demonstrations have less of an effect, but do not
impede the RL+IL algorithm.

In the second two tests, we set $d_{\textrm{train}}$ to be a design with four
legs and two wheels, then a design with two legs and two wheels, and measure
the effect of including both hexapod and car demonstrations in training. The
inclusion of demonstrations increased the speed of convergence and decreases
the variance, even when the designs learning to locomote are different than
the designs shown in the dataset. The supplementary video shows robots in
simulation using the policy trained with and without demonstrations.

##  V Conclusions

We introduced a novel reinforcement and imitation learning paradigm, where
different designs can learn from each other using modular policies. This
method allows demonstrations from one set of designs can act as a sort of
“prior” over behaviors for different designs. Our experiments show that
including demonstrations from modular designs can accelerate learning, even
when those demonstrations are not from the same robot designs that are being
trained, i.e., given a policy for a robot with legs, other robots with a
different number of legs can learn from the first robot.

These methods come with some limitations. Firstly, the robot design
$d_{\text{train}}$ and those in $D_{\text{demo}}$ must all be made up from
different combinations of a set of modules. At present, the method cannot use
arbitrary robots, or for instance, demonstrations recorded directly from a
human body. Secondly, we have not yet validated the output policies on real
robots, though our past work [5] demonstrated that modular policies can
control real robots. Lastly, we do not yet have a baseline for how well the
policy performs relative to other policy optimization methods. To the best of
our knowledge, our method is the first to train one design using data from
multiple different designs, so we primarily focused on comparing our method
with and without IL. This comparison is warranted because the algorithm both
with and without IL converges to the same average distance travelled, although
for the hexapod, the version without IL requires more than the five iterations
plotted to reach its full performance level. Comparisons to other methods such
as MFRL, for instance adapting the methods of [6] to include demonstrations,
could help establish how well our policies perform relative to policies
trained with other methods.

One direction for future work would be to apply RL+IL to train the policy for
multiple designs at once. The formulation and algorithm used here can be
readily adapted to include training with a set of multiple designs at once in
the RL loss. Our method would allow for arbitrary combinations of training and
demonstration designs. The RL+IL methods we developed could also be combined
with other learning objectives, such as learning to climb using vision, as is
the subject of our concurrent work. The policy learned on flat ground could be
used to warm-start a climbing policy. Alternatively, demonstrations from
experts could show the robot how to adapt its behavior to different
environments.

Future work can also develop additional baseline comparisons, other network
architectures, designs, and environments. While we have focused on a narrow
class of locomoting robots in this work, we hope our methods will prove
general enough to be applied to different types of robots and settings. For
robots to become more versatile and capable in the real world, we believe they
could benefit from learning from other robots that have different designs.

## References

  * [1] X. B. Peng, E. Coumans, T. Zhang, T.-W. E. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” in _Robotics: Science and Systems_ , 07 2020.
  * [2] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband _et al._ , “Deep q-learning from demonstrations,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [3] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp, “Goal-conditioned imitation learning,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [4] Z. Xie, P. Clary, J. Dao, P. Morais, J. Hurst, and M. Panne, “Learning locomotion skills for cassie: Iterative design and sim-to-real,” in _Conf. on Robot Learning_ , 2020, pp. 317–329.
  * [5] J. Whitman, M. Travers, and H. Choset, “Learning modular robot control policies,” _arXiv preprint arXiv:2105.10049_ , 2021.
  * [6] W. Huang, I. Mordatch, and D. Pathak, “One policy to control them all: Shared modular policies for agent-agnostic control,” in _ICML_ , 2020.
  * [7] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: lessons we have learned,” _The International Journal of Robotics Research_ , vol. 40, no. 4-5, pp. 698–721, 2021.
  * [8] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, M. Riedmiller _et al._ , “Emergence of locomotion behaviours in rich environments,” _arXiv preprint arXiv:1707.02286_ , 2017.
  * [9] R. Hafner, T. Hertweck, P. Kloeppner, M. Bloesch, M. Neunert, M. Wulfmeier, S. Tunyasuvunakool, N. Heess, and M. Riedmiller, “Towards general and autonomous learning of core skills: A case study in locomotion,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1084–1099.
  * [10] B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp, K. Chua, F. Hutter, and R. Calandra, “On the importance of hyperparameter optimization for model-based reinforcement learning,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2021, pp. 4015–4023.
  * [11] B. Amos, S. Stanton, D. Yarats, and A. G. Wilson, “On the model-based stochastic value gradient for continuous reinforcement learning,” in _Learning for Dynamics and Control_. PMLR, 2021, pp. 6–20.
  * [12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [13] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in _Conference on Robot Learning_. PMLR, 2022, pp. 91–100.
  * [14] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” in _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.
  * [15] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke, “Policies modulating trajectory generators,” in _Conference on Robot Learning_. PMLR, 2018, pp. 916–926.
  * [16] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani, “Data efficient reinforcement learning for legged robots,” in _Conf. on Robot Learning_. PMLR, 2020, pp. 1–10.
  * [17] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, 2019.
  * [18] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” _Science robotics_ , vol. 5, no. 47, 2020.
  * [19] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” _Advances in neural information processing systems_ , vol. 1, 1988.
  * [20] E. Kaufmann, A. Loquercio, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza, “Deep drone acrobatics,” in _Robotics: Science and Systems_ , July 2020.
  * [21] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [22] J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah _et al._ , “Urban driving with conditional imitation learning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 251–257.
  * [23] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots, “Agile autonomous driving using end-to-end deep imitation learning,” in _Robotics: Science and Systems_ , 2018.
  * [24] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots, “Imitation learning for agile autonomous driving,” _The International Journal of Robotics Research_ , vol. 39, no. 2-3, pp. 286–302, 2020.
  * [25] G. Sartoretti, J. Kerr, Y. Shi, G. Wagner, T. S. Kumar, S. Koenig, and H. Choset, “Primal: Pathfinding via reinforcement and imitation multi-agent learning,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2378–2385, 2019.
  * [26] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba, “Hindsight experience replay,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5048–5058.
  * [27] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 1, pp. 61–80, 2008.
  * [28] T. Wang, R. Liao, J. Ba, and S. Fidler, “Nervenet: Learning structured policy with graph neural networks,” in _Int. Conf. on Learning Representations_ , 2018.
  * [29] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning,” in _2018 IEEE Int. Conf. on robotics and Automation (ICRA)_. IEEE, 2018, pp. 7559–7566.
  * [30] M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 12 519–12 530.
  * [31] J. Whitman, M. Travers, and H. Choset, “Modular mobile robot design selection with deep reinforcement learning,” in _Machine Learning for Engineering Modeling, Simulation, and Design Workshop, at Neural Information Processing Systems_ , 2020.
  * [32] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 4754–4765.
  * [33] B. Amos, I. D. J. Rodriguez, J. Sacks, B. Boots, and J. Z. Kolter, “Differentiable mpc for end-to-end planning and control,” in _Proceedings of the 32nd Int. Conf. on Neural Information Processing Systems_ , 2018, pp. 8299–8310.
  * [34] A. Rajeswaran, I. Mordatch, and V. Kumar, “A game theoretic framework for model based reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 7953–7963.
  * [35] M. Deisenroth and C. E. Rasmussen, “Pilco: A model-based and data-efficient approach to policy search,” in _Proceedings of the 28th International Conference on machine learning (ICML-11)_. Citeseer, 2011, pp. 465–472.
  * [36] P. J. Werbos, “Backpropagation through time: what it does and how to do it,” _Proceedings of the IEEE_ , vol. 78, no. 10, pp. 1550–1560, 1990.
  * [37] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa _et al._ , “Isaac gym: High performance gpu based physics simulation for robot learning,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.


# Giant diffusion of nanomechanical rotors in a tilted washboard potential

L. Bellando1, M. Kleine1, Y. Amarouchene1, M. Perrin1, Y. Louyer1 1Université
de Bordeaux, CNRS, LOMA, UMR 5798, F-33405 Talence, France

###### Abstract

We present an experimental realization of a biased optical periodic potential
in the low friction limit. The noise-induced bistability between locked
(torsional) and running (spinning) states in the rotational motion of a
nanodumbbell is driven by an elliptically polarized light beam tilting the
angular potential. By varying the gas pressure around the point of maximum
intermittency, the rotational effective diffusion coefficient increases by
more than 3 orders of magnitude over free-space diffusion. These experimental
results are in agreement with a simple two-state model that is derived from
the Langevin equation through using timescale separation. Our work provides a
new experimental platform to study the weak thermal noise limit for diffusion
in this system.

Figure 1:  a) Illustration of the energy landscape for the rotation of the
Brownian dimer around the optical axis for three different tilts. When the
polarization is linear (blue dotted line), the potential is not tilted and the
orientation of the dimer remains in the locked state. Conversely, for a
circular polarization (solid blue line) the potential wells disappear. For
elliptical polarization (dashed line), the rotational motion can change from a
running state (spinning) to a locked state (torsional) and vice versa. b)
Surface plot in log-log scales of the angular vibration and translational
power spectral densities as a function of the gas pressure for a beam
ellipticity of $\phi_{\lambda/4}=25^{\circ}$ and an optical power of $180$ mW.
The solid and dashed lines (red) represent the local maxima of the PSD
computed by Langevin simulations of Eq. (2). The three grey horizontal lines
at 0.41 mbar, 0.51 mbar and 0.61 mbar outline the two-state coexistence region
where the middle line denotes the maximum of intermittency.

Figure 2:  Time traces of the dimer angular velocity $\dot{\varphi}$ obtained
by short-time Fourier transform for the three pressures that are shown as gray
horizontal lines in Fig. 1(b).

Thermal diffusion in a tilted periodic (also named washboard) potential
constitutes an archetypal model of transport phenomena in nonequilibrium
statistical physics (Kramers, 1940; Risken, 1989). It describes a diverse
range of systems, including Josephson junctions (Longobardi  _et al._ , 2011),
molecular motors Hayashi  _et al._ (2015), synchronization phenomena Strogatz
(2003), and diffusion on crystal surfaces Frenken and van der Veen (1985), to
name only a few.

Brownian particles moving in the tilted potential at non-zero temperature
exhibit two distinct well-characterized limiting behaviors: locked and running
states. The first emerges when the potential wells are deeper than the thermal
energy scale, the diffusing particle remains in a local minimum. In the
latter, a sufficient tilt of the potential, lowering its energy barrier,
allows the particle to flow down the potential wells. Depending on the system
parameters (friction, temperature and tilt), both solutions may coexist, and
stochastic transitions between locked and running states may occur. It has
been shown, both experimentally Lee and Grier (2006); Hayashi  _et al._
(2015); Evstigneev  _et al._ (2008) and theoretically (Reimann  _et al._ ,
2001; Costantini and Marchesoni, 1999), that in this two-state coexistence
regime, the effective diffusion coefficient can be enhanced by several orders
of magnitude relative to the free diffusion coefficient. However, this has
only been experimentally observed in the overdamped regime which is by now a
fully understood stochastic process (Risken, 1989).

To our knowledge, an experimental study of giant diffusion in the strongly
inertial (underdamped) regime is still missing. The theoretical problem is
also more demanding in the underdamped and weak noise limits because the usual
expansion of the matrix continued fraction (Risken, 1989) fails to converge
well (Lindner and Sokolov, 2016). On these grounds, this nonlinear stochastic
system in the low-noise and low-damping limits is still attractive (Lindner
and Sokolov, 2016).

We propose a simple experimental setup based on the optical trapping of a
nanodumbbell in a moderate vacuum. A special feature of our experiment is that
the friction coefficient, being linearly proportional to the gas pressure $P$,
can be tuned over several orders of magnitude. Therefore, for a nearly
constant temperature, Brownian motion can move from the overdamped to the
underdamped regime. This makes the optical trapping setup a suitable platform
to study stochastic dynamics in the low-friction regime Amarouchene  _et al._
(2019); Mangeat  _et al._ (2019); Rondin  _et al._ (2017).

The main focus of this Letter is the analysis of the rotational diffusivity in
the bistability region for an underdamped stochastic mechanical nanorotor
travelling in a tilted periodic potential. We provide experimental
confirmation of a giant diffusivity by tracking the rotational motion of a
single silica nanodimer. Due to the experimentally achieved timescale
separation, we show that the effective diffusion is well described by a two-
state model for this nearly one-dimensional rotational motion.

The experimental setup consists of a vacuum optical tweezer trapping a
dielectric silica dumbbell made of two nanospheres of nominal radius $68\pm 7$
nm (Ahn  _et al._ , 2018; Reimann  _et al._ , 2018). The continuous wave
trapping laser beam (wavelength, $\lambda=1064~{}$nm and power, $180~{}$mW)
passes through an objective lens with a $0.8$ numerical aperture, the
polarization’s ellipticity of which can be changed by a quarter-wave plate. A
detailed description of the experimental setup, calibration and processing
procedures is given in the Supplemental Information (SI, ). The shape and the
size of the dimer are assessed under linear polarization, and correspond to an
aspect ratio $L/D=1.8,$ with two spheres of radius $R_{p}=65.7~{}$nm, where
$D=2R_{p}$ and $L$ is the major axis length of the dimer. Our detection system
is able to simultaneaously record the particle’s center of mass motion and its
angular displacement.

The nanodimer experiences both a trapping force and a torque, both of which
are induced by elliptically polarized light. In particular, when such an
asymmetric Rayleigh scatterer lies in the transverse ($\hat{x},\hat{y}$)
plane, corresponding to $\theta=\pi/2$, the torque reads

$M=M_{C}-M_{L}\sin(2\varphi),$ (1)

where $\varphi$ is the angle between the dimer main axis and the lab’s
horizontal $\hat{x}$ axis. Its time evolution is related to the dimer rotation
around the $\hat{z}$ optical axis. $M_{C}$ and $M_{L}$ are respectively the
circular and linear contributions of the optical torque which depend on the
dimer polarizability tensor and the laser beam polarization (SI, ).

In a linearly polarized optical tweezer, where $M_{C}=0$, the long axis of the
dimer (i.e., in the direction of greatest polarizability) will tend to align
with the laser beam polarization. This is because the polarizability of the
dimer along its long axis is greater than the polarizability perpendicular to
it. As a result, the dimer acts as a torsion balance with a linear restoring
torque, $-2M_{L}\varphi$. Note that regardless of the gas pressure, the dimer
oscillates about $\varphi=0$ with a torsional frequency $\Omega_{\rm
L}=\sqrt{2M_{L}/I_{\perp}}$, where $I_{\perp}$ is the moment of inertia. In
contrast, for a circularly polarized trap, where $M_{L}=0$, the asymmetric
particle undergoes a constant torque, whatever its orientation $\varphi$. As a
result of the continuous transfer of angular momentum, the particle will spin
about the $\hat{z}$ axis at constant frequency, as given by $\Omega_{\rm
R}=M_{C}/I_{\perp}\Gamma_{\varphi}$, where $\Gamma_{\varphi}$ is the rotation
or torsional vibration damping rate about the $\hat{z}$ optical axis (see (SI,
) for its determination). Note that the rotation frequency can exceed GHz,
which allows us to study material stresses due to centrifugal forces Ashkin
and Dziedzic (1977); Hümmer  _et al._ (2020), measure ultraweak torques Bruce
_et al._ (2021); Rashid  _et al._ (2018), and examine quantum features of
rotation Stickler  _et al._ (2021, 2018). Figure 1(a) illustrates the
potential landscape and orientational dynamics for different polarization
states. Considering the general case, for elliptical polarization, the
potential has a washboard shape:
$U(\varphi)=M_{L}\sin^{2}(\varphi)-M_{C}\varphi$, and one can observe running
and locked states. Below, we only consider elliptical polarization which is
held constant at $\phi_{\lambda/4}=25^{\circ}$, the angle of the quarter
waveplate.

The dynamics of an asymmetric Brownian particle trapped under elliptically
polarized light is 6-dimensional with translational and rotational degrees of
freedom coupled to each other. This generally leads to a highly nonlinear
problem Rashid  _et al._ (2018); Bang  _et al._ (2020); Arita  _et al._
(2013). However, in the following, we show that the giant diffusion phenomenon
is well described by one-dimensional rotational dynamics. Comparison between
the experiment and numerical simulations is made quantitative provided that an
additional term is added to the usual extinction optical torque
($\propto\Re[\textbf{p}\times\textbf{E}_{\textbf{\rm inj}}]$, with p the
dipole moment and $\textbf{E}_{\textbf{\rm inj}}$ the incident electric
field). This extra term, called scattering torque
($\propto\Im[\textbf{p}\times\textbf{p*}]$), is due to the interference of the
fields scattered by the particle (Nieto-Vesperinas, 2015; SI, ). This
$\varphi$ angle dynamics reproducing the stochastic jumps between torsion and
continuous rotation states is described by the Langevin equation

$I_{\perp}\ddot{\varphi}=-I_{\perp}\Gamma_{\varphi}\dot{\varphi}+M+M_{\rm
th},$ (2)

where $\Gamma_{\varphi}$ is the rotational damping rate and $M_{\rm
th}=\sqrt{2k_{B}TI_{\perp}\Gamma_{\varphi}}\zeta(t)$ is the thermal torque
with $\zeta(t)$ a gaussian noise of zero mean value. $M$ is given by Eq. (1).

Figure 1(b) shows the position and orientation power spectral densities as a
function of pressure, along with the numerical Langevin simulations (solid red
lines). For consistency, we also show the frequencies associated to
translational motions in the $X,Y$ and $Z$ directions that are independent of
gas pressure. The observed non-degeneracy of the transverse frequencies $X$,
$Y$ is expected for a non-circularly polarized laser beam. In the high
friction limit, the torsional dynamics dominate the dimer orientation motion
and their resonance frequencies remain constant in relation to pressure.
Conversely, the rotational dynamics is dominant in the low friction regime,
i.e., at low pressure, and its angular frequency, $\Omega_{\rm R}$, rises as
the inverse of the gas damping.

The red solid lines in Fig. 1(b) show a good agreement between the experiment
and one-dimensional simulations of Eq. (2). One can also clearly see that the
bistability region is quantitatively well-described by the model. The two
locked states observed in the spectral densities are also found in the
numerical simulations, and correspond to the fundamental peak and its
overtone. Although not within the scope of this study, we do not preclude
contributions with other angles leading to coupling with extra motional
degrees of freedom. The experimental determination of the dimer size and shape
by the calibration procedure (SI, ) is used for calculating the moments of
inertia and polarizabilities. The latter are further supported numerically
using Comsol Multiphysics. We find that $I_{\perp}=2\times 10^{-32}$
kg$\cdot$m2, $M_{C}$ and $M_{L}$ are respectively of the order of $2\times
10^{-22}$ N$\cdot$m and $6\times 10^{-20}$ N$\cdot$m, in good agreement with
both experimental and numerical results.

As mentioned above, in the calculation of optical torques the contribution of
scattering is included in addition to the usual extinction contribution Nieto-
Vesperinas (2015). Since the polarization of the trapping beam is not linear,
this scattering torque is crucial to reproduce quantitatively the experimental
results, particularly when the dimer spins. In many previous works, the
scattering torque was ignored because the rotational Langevin equation was
only qualitatively discussed. It is worth noting that if the scattering torque
is not taken into account, the particle’s rotation speed is overestimated by
typically an order of magnitude for the actual dimer SI  and much more for a
spheroid (data not shown). Some quantitative deviations between theory and
experiment appear at low pressure but without greatly affecting the giant
diffusion effect.

Figure 3: a) Probability distribution function of the occupation time from the
locked to running states $P(\tau_{L})$, red symbols, and the running to locked
states $P(\tau_{R})$, blue symbols, for various pressures where the
bistability is observed. Exponential behaviors suggest that stochastic
transitions between locked and running states are thermally activated,
following an Arrhenius-type law. b) Pressure dependence of the mean occupation
times in the locked and running states [symbols: experiment, solid line:
Langevin simulations, Eq. (2)].

We now consider the quantitative study of this stochastic nonequilibrium
phenomenon. Fig. 2 shows the time traces of the angular velocity
$\dot{\varphi}$ obtained by short time Fourier transforms in the bistability
region. In the locked state, the mean angular velocity is
$\langle\dot{\varphi}\rangle_{L}\equiv 0$. While the mean rotational speed,
$\langle\dot{\varphi}\rangle_{R}$, in the running state increases continuously
with the pressure drop. The most noticeable feature of the time traces is the
two-state noise-induced transitions, where the maximum of intermittency occurs
in the middle of the bistability region at about $P\approx 0.56$ mbar [Fig.
1(b)].

Figure 4: a) Measured power spectral densities of the dimer angular velocity
(dots) and double Lorentzian fit (Eq. (3), solid lines) at the boundaries of
the bistable region ($P=0.653$ mbar in blue, $P=0.408$ mbar in black), and at
the pressure around which the maximum of giant diffusion exists ($P=0.560$
mbar in red). The dashed line represents the free-space diffusion coefficient
$D_{0}$ at $P=0.653$ mbar, the double arrow displays the excess of diffusion
$D_{\text{eff}}/D_{0}$, while both the total escape rate $\Gamma_{\rm T}$ and
the rotational damping rate $\Gamma_{\varphi}$ are marked by vertical arrows.
b) On the left axis. Pressure dependence of the effective diffusion
coefficient $D_{\text{eff}}$ in units of $D_{0}$ estimated by two methods: (i)
data from Fig. 4(a) are fitted by using Eq. (3), blue open circles, while (ii)
transition rates obtained from Fig. (3) are used to calculate $D_{\text{eff}}$
from Eq. (4), green filled circles. The black solid line results from the
Langevin simulations [Eq. (2)]. On the right axis. Proportion of the mean
occupancy in the locked state (red squares: experiment and black dashed line:
Langevin simulations).

We use the time traces of angular velocities to determine the mean occupation
times in the locked and running states. Fig. 3(a) shows the distributions
$P(\tau/\langle\tau\rangle)$ of the mean occupation times, recorded for
different gas pressures. These distributions decrease exponentially as
expected for a Kramers-like problem. Deeper insight into two-state coexistence
can be gained if we plot the pressure dependence of the average occupancy
times in the locked and running states, as shown in Fig. 3(b), both for
experimental and numerical data. It can be evidenced that when
$\Gamma_{\varphi}\ll\Omega_{L}$ (as in this case), the transition rate $r_{L}$
out of a locked state (regardless of the final state, locked or running) is
independent of the friction coefficient, i.e., of the pressure. This is in
agreement with our observations (see Fig. 3(b) red squares and solid line).
Besides, this value is well approximated by the well-known Arrhenius law,
$1/\tau_{L}=r_{L}\sim\Omega_{L}\exp(-E_{\rm b}/k_{\rm B}T),$ $E_{\rm b}$ being
the barrier height given by the potential difference between a minimum and the
lower neighboring maximum Büttiker and Landauer (1984). This offers insight in
how a temperature change affects the stability of the torque state. A
different behavior is observed for transitions from a running to a locked
state for low damping and finite temperatures. This behaviour is explained by
the fact that the rate $r_{R}$ is such that
$1/\tau_{R}=r_{R}\sim\Gamma_{\varphi}\propto P$ Büttiker  _et al._ (1983).
Note that some differences occurring at low pressure can be observed betwee*n
the one-dimensional model and the experiment [as seen in Fig. 3(b)]. They are
not due to a time windowing effect, but could arise from the coupling between
other rotational degrees of freedom, as seen below 0.45 mbar with the
appearance of an additional resonance in the spectral densities [Fig. 1(b)].
In the rest of this Letter, we focus on the measurement of the effective
rotational diffusion coefficient, which is defined by the Kubo relation

$D_{\rm
eff}=\int^{\infty}_{0}dt^{\prime}\langle[\dot{\varphi}(t^{\prime})-\langle\dot{\varphi}\rangle][\dot{\varphi}(0)-\langle\dot{\varphi}\rangle]\rangle.$

This coefficient satisfies the usual Einstein relation
$\lim_{t\rightarrow\infty}\langle\varphi^{2}(t)\rangle-\langle\varphi(t)\rangle^{2}=2D_{\rm
eff}t$, where $D_{\rm eff}$ can in principle be enhanced by orders of
magnitude over the free coefficient diffusion $D_{0}$ at the crossover from
locked to running states. Interestingly, outside the region of bistability,
the power spectral density of $\dot{\varphi}$ allows for a measurement of both
$D_{0}=k_{\rm B}T/\Gamma_{\varphi}I_{\perp}$ and its related timescale
$1/\Gamma_{\varphi}$. Surprisingly, we can derive a simple two-state
stochastic noise model to fit our experimental results in the bistability
region (see details in Sec. V.B of the Supplemental Material SI ). Since the
parameter range we consider allows large timescale separations ($1/\Omega_{\rm
R}\ll 1/\Gamma_{\varphi}\ll{\tau_{L},\tau_{R}}$), the spectral density of the
rotation speed in the region of bistability reads

$S_{\dot{\varphi}\dot{\varphi}}(\omega)=\frac{D_{0}}{1+\frac{\omega^{2}}{\Gamma_{\varphi}^{2}}}+\frac{D_{\text{eff}}}{1+\frac{\omega^{2}}{\Gamma_{T}^{2}}}$
(3)

where $\Gamma_{\rm T}=r_{R}+r_{L},$ is the total escape rate, corresponding to
the longest timescale of the system. From the time traces of the angular
velocity $\dot{\varphi}$ (Fig. 2), we compute the power spectral densities of
the rotation velocity (defined as the Fourier transform of the rotational
velocity autocorrelation function) in the coexistence regime, as shown in Fig.
4(a). Using then Eq. (3), we can determine the effective diffusion $D_{\rm
eff}$ relative to $D_{0}$, that is displayed in Fig. 4(b) as a function of
pressure. Note that $\Gamma_{\varphi}$ and $\Gamma_{\rm T}$ are also measured,
allowing to corroborate the values of the nanodimer size and its aspect ratio.
To go further into the two-state model, we use the transition rates $r_{R}$
and $r_{L}$ deduced from Fig. 3 to calculate $D_{\rm eff}$. In our parameter
regime, the two-state model gives an expression that bears much similarities
with those developed in (Lindner and Sokolov, 2016; Machlup, 1954)

$D_{\text{eff}}=\langle\dot{\varphi}\rangle_{{}_{\rm
R}}^{2}~{}\frac{r_{L}r_{R}}{(r_{L}+r_{R})^{3}},$ (4)

where $\langle\dot{\varphi}\rangle_{{}_{\rm R}}$ is the average angular
velocity taken only over the running states of the whole temporal trace.
Comparison of the filled and open circles in Fig. 4(b) indicates that when Eq.
(4) is used with the results obtained from occupancy times, good agreement is
obtained with the diffusion estimate by spectral densities by using Eq. (3).
The occupancy rate in the locked state defined as
$N_{-}=\langle\tau_{L}\rangle/(\langle\tau_{L}\rangle+\langle\tau_{R}\rangle)$
is represented by red squares in Fig. 4(b). We observe that the giant increase
in the effective diffusion coefficient is roughly maximal at a pressure where
the states are evenly distributed.

In conclusion, we have observed in a very small pressure range a giant
increase of the diffusion coefficient in the underdamped and weak noise limit.
Carefully calculating the optical torque, a reasonable quantitative agreement
has been obtained using a one-dimensional Langevin model. This supports
evidence that even though the real system dynamics has 6 dimensions, it can be
reduced to a one-dimensional system within at least some parameter regimes.
Cooling the translational (and possibly rotational) degrees of freedom would
mitigate the rotranslational coupling (Reimann  _et al._ , 2018; Ahn  _et al._
, 2018), enabling ideally over a wide parameter regime to reach the true one-
dimensional Brownian nanorotor as described in Eq. (2). Thus, an optically
levitated stator-rotator constitutes an outstanding platform where its
dynamics can be controlled in many ways up to the quantum ground state at room
or cryogenic temperature (Delic  _et al._ , 2020; Magrini  _et al._ , 2021;
Tebbenjohanns  _et al._ , 2021).

The authors acknowledge financial support from the Region Nouvelle-Aquitaine
(grant 2018-1R50304) and the Agence Nationale pour la Recherche (grant
ANR-21-CE30-0006). M. K. acknowledges the financial support from the EUR Light
S&T Graduate Program (PIA3 Program “Investment for the Future”,
ANR-17-EURE-0027).

## References

  * Kramers (1940) H. Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, Physica 7, 284 (1940).
  * Risken (1989) H. Risken,  _The Fokker-Planck Equation: Methods of Solution and Applications Second Edition_ (Springer, 1989).
  * Longobardi  _et al._ (2011) L. Longobardi, D. Massarotti, G. Rotoli, D. Stornaiuolo, G. Papari, A. Kawakami, G. P. Pepe, A. Barone, and F. Tafuri, Thermal hopping and retrapping of a brownian particle in the tilted periodic potential of a NbN/MgO/NbN josephson junction, Phys. Rev. B 84, 184504 (2011).
  * Hayashi  _et al._ (2015) R. Hayashi, K. Sasaki, S. Nakamura, S. Kudo, Y. Inoue, H. Noji, and K. Hayashi, Giant acceleration of diffusion observed in a single-molecule experiment on F1-ATPase, Phys. Rev. Lett. 114, 248101 (2015).
  * Strogatz (2003) S. H. Strogatz,  _Sync: The Emerging Science of Spontaneous Order_ (Hachette Books, 2003).
  * Frenken and van der Veen (1985) J. W. M. Frenken and J. F. van der Veen, Observation of surface melting, Phys. Rev. Lett. 54, 134 (1985).
  * Lee and Grier (2006) S.-H. Lee and D. G. Grier, Giant colloidal diffusivity on corrugated optical vortices, Phys. Rev. Lett. 96, 190601 (2006).
  * Evstigneev  _et al._ (2008) M. Evstigneev, O. Zvyagolskaya, S. Bleil, R. Eichorn, C. Bechinger, and P. Reimann, Diffusion of colloidal particles in a tilted periodic potential: Theory versus experiment, Phys. Rev. E 77, 041107 (2008).
  * Reimann  _et al._ (2001) P. Reimann, C. Van den Broeck, H. Linke, P. Hänggi, J. M. Rubi, and A. Pérez-Madrid, Giant acceleration of free diffusion by use of tilted periodic potentials, Phys. Rev. Lett. 87, 010602 (2001).
  * Costantini and Marchesoni (1999) G. Costantini and F. Marchesoni, Threshold diffusion in a tilted washboard potential, Europhysics Letters (EPL) 48, 491 (1999).
  * Lindner and Sokolov (2016) B. Lindner and I. M. Sokolov, Giant diffusion of underdamped particles in a biased periodic potential, Phys. Rev. E 93, 042106 (2016).
  * Amarouchene  _et al._ (2019) Y. Amarouchene, M. Mangeat, B. Vidal Montes, L. Ondic, T. Guerin, D. S. Dean, and Y. Louyer, Nonequilibrium dynamics induced by scattering forces for optically trapped nanoparticles in strongly inertial regimes, Phys. Rev. Lett. 122, 183901 (2019).
  * Mangeat  _et al._ (2019) M. Mangeat, Y. Amarouchene, Y. Louyer, T. Guerin, and D. S. Dean, Role of nonconservative scattering forces and damping on brownian particles in optical traps, Phys. Rev. E 99, 052107 (2019).
  * Rondin  _et al._ (2017) L. Rondin, J. Gieseler, F. Ricci, R. Quidant, C. Dellago, and L. Novotny, Direct measurement of kramers turnover with a levitated nanoparticle, Nature Nanotech. 12, 1130 (2017).
  * Ahn  _et al._ (2018) J. Ahn, Z. Xu, J. Bang, Y.-H. Deng, T. M. Hoang, Q. Han, R.-M. Ma, and T. Li, Optically levitated nanodumbbell torsion balance and ghz nanomechanical rotor, Phys. Rev. Lett. 121, 033603 (2018).
  * Reimann  _et al._ (2018) R. Reimann, M. Doderer, E. Hebestreit, R. Diehl, M. Frimmer, D. Windey, F. Tebbenjohanns, and L. Novotny, Ghz rotation of an optically trapped nanoparticle in vacuum, Phys. Rev. Lett. 121, 033602 (2018).
  * (17) See Supplementary Materials for a complete description of our theoretical approach and the numerical methods used in the present paper. Detailed description of the experimental apparatus and also for the particle’s size and shape caracterisation procedures are also provided.
  * Ashkin and Dziedzic (1977) A. Ashkin and J. M. Dziedzic, Feedback stabilization of optically levitated particles, Applied Physics Letters 30, 202 (1977), https://doi.org/10.1063/1.89335 .
  * Hümmer  _et al._ (2020) D. Hümmer, R. Lampert, K. Kustura, P. Maurer, C. Gonzalez-Ballestero, and O. Romero-Isart, Acoustic and optical properties of a fast-spinning dielectric nanoparticle, Phys. Rev. B 101, 205416 (2020).
  * Bruce  _et al._ (2021) G. D. Bruce, P. Rodríguez-Sevilla, and K. Dholakia, Initiating revolutions for optical manipulation: the origins and applications of rotational dynamics of trapped particles, Advances in Physics: X 6, 1838322 (2021), https://doi.org/10.1080/23746149.2020.1838322 .
  * Rashid  _et al._ (2018) M. Rashid, M. Toroš, A. Setter, and H. Ulbricht, Precession motion in levitated optomechanics, Phys. Rev. Lett. 121, 253601 (2018).
  * Stickler  _et al._ (2021) B. A. Stickler, K. Hornberger, and M. S. Kim, Quantum rotations of nanoparticles, Nature Reviews Physics 3, 589 (2021).
  * Stickler  _et al._ (2018) B. A. Stickler, B. Papendell, S. Kuhn, B. Schrinski, M. Millen, J. andArndt, and K. Hornberger, Probing macroscopic quantum superpositions with nanorotors, New J. Phys. 20, 122001 (2018).
  * Bang  _et al._ (2020) J. Bang, T. Seberson, P. Ju, J. Ahn, Z. Xu, X. Gao, F. Robicheaux, and T. Li, Five-dimensional cooling and nonlinear dynamics of an optically levitated nanodumbbell, Phys. Rev. Research 2, 043054 (2020).
  * Arita  _et al._ (2013) Y. Arita, M. Mazilu, and K. Dholakia, Laser-induced rotation and cooling of a trapped microgyroscope in vacuum, Nature Communications 4, 2374 (2013).
  * Nieto-Vesperinas (2015) M. Nieto-Vesperinas, Optical torque: Electromagnetic spin and orbital-angular-momentum conservation laws and their significance, Phys. Rev. A 92, 043843 (2015).
  * Büttiker and Landauer (1984) M. Büttiker and R. Landauer, Escape-energy distribution for particles in an extremely underdamped potential well, Phys. Rev. B 30, 1551 (1984).
  * Büttiker  _et al._ (1983) M. Büttiker, E. P. Harris, and R. Landauer, Thermal activation in extremely underdamped josephson-junction circuits, Phys. Rev. B 28, 1268 (1983).
  * Machlup (1954) S. Machlup, Noise in semiconductors: Spectrum of a two-parameter random signal, Journal of Applied Physics 25, 341 (1954), https://doi.org/10.1063/1.1721637 .
  * Delic  _et al._ (2020) U. Delic, M. Reisenbauer, K. Dare, D. Grass, V. Vuletic, N. Kiesel, and M. Aspelmeyer, Cooling of a levitated nanoparticle to the motional quantum ground state, Science 367, 892 (2020).
  * Magrini  _et al._ (2021) L. Magrini, P. Rosenzweig, C. Bach, A. Deutschmann-Olek, S. G. Hofer, S. Hong, N. Kiesel, A. Kugi, and M. Aspelmeyer, Real-time optimal quantum control of mechanical motion at room temperature, Nature 595, 373 (2021).
  * Tebbenjohanns  _et al._ (2021) F. Tebbenjohanns, M. L. Mattana, M. Rossi, M. Frimmer, and L. Novotny, Quantum control of a nanoparticle optically levitated in cryogenic free space, Nature 595, 378 (2021).