# Landmark Guided Active Exploration with
Stable Low-level Policy Learning

Fei Cui, Jiaojiao Fang, Mengke Yang, Guizhong Liu The authors are with the
School of Electronic and Information Engineering, Xi’an Jiaotong University,
Xi’an 710049, China. Correspondence to: Guizhong Liu<EMAIL_ADDRESS>
###### Abstract

Goal-conditioned hierarchical reinforcement learning (GCHRL) decomposes long-
horizon tasks into sub-tasks through a hierarchical framework and it has
demonstrated promising results across a variety of domains. However, the high-
level policy’s action space is often excessively large, presenting a
significant challenge to effective exploration and resulting in potentially
inefficient training. Moreover, the dynamic variability of the low-level
policy introduces non-stationarity to the high-level state transition
function, significantly impeding the learning of the high-level policy. In
this paper, we design a measure of prospect for subgoals by planning in the
goal space based on the goal-conditioned value function. Building upon the
measure of prospect, we propose a landmark-guided exploration strategy by
integrating the measures of prospect and novelty which aims to guide the agent
to explore efficiently and improve sample efficiency. To address the non-
stationarity arising from the dynamic changes of the low-level policy, we
apply a state-specific regularization to the learning of low-level policy,
which facilitates stable learning of the hierarchical policy. The experimental
results demonstrate that our proposed exploration strategy significantly
outperforms the baseline methods across multiple tasks.

###### Index Terms:

Hierarchical reinforcement learning (HRL), subgoal, exploration-exploitation,
state-specific regularization.

##  I Introduction

Deep Reinforcement Learning (DRL) is a powerful approach for solving
sequential decision-making problems, such as video games[1, 2] and robot
navigation[3, 4, 5]. DRL models these problems as partially observable Markov
decision processes (POMDPs), and learns the optimal policies by maximizing the
cumulative discounted reward. However, in many complex tasks, agents often
struggle to collect sufficient high-reward trajectories. Hierarchical
reinforcement learning decomposes complex, long-horizon decision-making tasks
into sub-tasks of different time scales and is a promising method for solving
such long-horizon tasks. Goal-conditioned hierarchical reinforcement
learning[6, 7, 8, 9] is a two-level hierarchical reinforcement learning
paradigm, where the high-level policy decomposes the original task into a
series of subgoals, and the low-level policy guides the agent to achieve these
subgoals. The learning objective of the hierarchical policy is still to learn
a decision function that maximizes the cumulative expected reward, so the
learning of the high-level policy depends on the external rewards, while the
learning of the low-level policy depends on the intrinsic rewards defined
through the subgoals.

Effective subgoals are crucial for achieving good performance and efficiency
in goal-conditioned hierarchical reinforcement learning. Selecting reasonable
subgoals that capture the task’s semantics provides meaningful guidance for
low-level policy learning. Pre-defined subgoal space[10, 11] and task-specific
subgoal representation space[12, 13, 14, 15] learned online can be employed to
better represent the action space of the high-level policy. Pre-defined
subgoals can quickly supervise low-level policy learning via intrinsic
rewards, while learned subgoal representations can be optimized for specific
tasks. However, sampling actions in a large subgoal space can lead to
inadequate exploration and inefficient training of the agent’s high-level
policy.

To guide the agent to explore efficiently, some approaches[10, 11] reduce the
complexity of the high-level action space by using adjacency constraints,
which promotes the agent to explore reasonable states. To further avoid
introducing additional non-stationarity in HRL, HESS[15] proposes an active
exploration strategy that considers a combined measure of novelty and
potential for subgoals after stabilizing the learning of subgoal
representation space. The novelty measure aims to enhance the agent’s ability
to explore new states, while the potential measure guides the agent to explore
in the direction that is more likely to expand the explored area. HESS is
effective in guiding the agent to unexplored areas, but it ignores the impact
of the final goal on exploration. Therefore, relying solely on the measures of
novelty and potential may not always guide the agent to the most promising
areas.

Our insight is that the agent not only needs to expand the exploration area
but also needs to pay attention to the areas that are more likely to achieve
the final task goal. To this end, we design a prospect measure for subgoals
through landmark-based planning in the subgoal representation space. This
measure can reflect the likelihood of exploring in the direction of a subgoal
leading the agent to states closer to achieving the final task goal.
Considering the measure of prospect and novelty for subgoals, we propose a
Landmark-guided active Exploration strategy with Stable low-level Policy
learning (LESP). The strategy incorporates the measure of prospect to guide
the agent to explore subgoals that are more likely to lead to the final task
goal. Additionally, In goal-conditioned hierarchical reinforcement learning,
the high-level state transition function is dependent not only on the physical
environment but also on the online-learned low-level policy. Due to the
dynamic changes of the low-level policy, the non-stationarity of the high-
level state transition function greatly hinders the learning of the high-level
policy. Some prior works[16, 17] have attempted to re-label the actions of the
high-level state transitions (i.e., subgoals) to make them adaptable to the
dynamic changes in the low-level policy. In this paper, we apply state-
specific regularization to the learning of low-level policy to alleviate the
non-stationarity caused by the dynamic changes in the low-level policy and
facilitate the learning of the high-level policy. The proposed LESP strategy
effectively balances the exploration-exploitation trade-off, and enables
efficient learning of goal-conditioned hierarchical policies.

We compare the proposed method LESP with the state-of-the-art baselines in the
Mujoco experimental environment[18]. The experimental results demonstrate that
LESP, which takes into account the guidance of the task goal for exploration,
outperforms the baseline methods. Additionally, we conduct ablation
experiments to verify the roles of different components of LESP.

The article is organized as follows: Section II covers the relevant works.
Section III discusses the essential preliminary concepts. Section IV describes
the proposed method LESP. In Section V, experiments and results are discussed.
Finally, this article is concluded in Section VI.

##  II Related Work

When dealing with long horizon decision-making tasks, how to guide the agent
to explore promising trajectories is a crucial problem. Hierarchical
reinforcement learning (HRL)[13, 19, 20, 21, 22, 23] improves sample
efficiency by decomposing complex tasks. Selecting reasonable subgoals during
interaction with the environment can also avoid blind exploration. Related
works are described from these two aspects.

###  II-A Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning divides the original task into sub-tasks
at different timescales using a hierarchical structure. The high-level policy
communicates with the low-level policy through subgoals, and the signal passed
from the high-level policy to the low-level policy can vary across different
tasks, ranging from using discrete value for option[24, 25, 26] to employing
pre-defined subgoal space[10, 11, 16] or subgoal representation space learned
online[12, 13, 14, 15, 27, 28]. The use of discrete-valued options naturally
reduces the complexity of the high-level action space, but the limited rules
decrease the adaptability of the solution to different tasks. On the other
hand, learning subgoal representation often results in a high-dimensional
action space, which can hinder the agent’s ability to explore effectively. To
explore effectively, HRAC[10] restricts the high-level action space within the
$k$-step adjacency area through adjacency constraints, while HIGL[11] samples
landmarks through coverage-based sampling and novelty-based sampling in the
replay buffer, restricting the actions of the high-level policy within the
domain of the most urgent landmark, but introduces additional non-
stationarity. In order to choose appropriate subgoals to guide exploration,
HESS[15] proposes an exploration strategy by considering the measure of
novelty and potential for subgoals but ignores the guidance of the task final
goal for exploration. In contrast, our approach plans landmarks in the subgoal
representation space according to the task goal and designs a measure of
prospect that considers the influence of the task’s final goal, proposing a
more efficient hierarchical exploration strategy.

###  II-B Subgoal Selection

When employing deep reinforcement learning to solve complex sequential
decision-making tasks, selecting appropriate subgoals for the agent can be an
effective strategy. Several studies[29, 30, 31, 32, 33] focus on learning
goal-conditioned value functions and rationally design reward functions to
make the value function reflect the reachability between two states. These
approaches allow for planning in the goal space based on reachability, with
states on the planned path selected as subgoals. With good perception of the
environment map, some rule-based methods[3, 34, 35] utilize heuristic search
method to find an optimal trajectory, and sample subgoals along the trajectory
based on physical priors. After utilizing a value function to learn the
reachability between states in the state space, L3P[32] clusters the candidate
states based on their reachability, with each cluster center representing a
potential landmark. Then, a sparse topological graph is constructed with the
potential landmarks as nodes and reachability as edges. Finally, subgoals are
generated by conducting graph search algorithm on the constructed graph. On
the other hand, HIGL[11] uses coverage-based sampling and novelty to sample
landmarks from the replay buffer. To facilitate effective exploration with
reasonable subgoals, the latest work HESS[15] samples candidate landmarks in
the neighborhood of the agent’s current state and selects subgoals based on
the measures of novelty and potential. This active exploration strategy avoids
introducing additional non-stationarity and speeds up the learning process of
the hierarchical policy. Like previous works[11, 32], we also utilize a goal-
conditioned value function to plan landmarks in the goal space. We designed a
measure of prospect based on the landmarks. Considering measures of novelty
and prospect to select subgoals, we proposed an active hierarchical
exploration strategy.

##  III Preliminaries

Reinforcement learning formulates the sequential decision-making problem as a
Markov decision process (MDP)[36], defined as a tuple
$M=<\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma>$ where $\mathcal{S}$ denotes
the state space, $\mathcal{A}$ denotes the action space, $\mathcal{P}$
represents the state transition function that reflects the dynamics of the
environment, $r$ is the reward function typically designed by human experts
for the task, and $\gamma\in[0,1)$ is a discount factor. A policy $\pi(a|s)$
maps a given state $s$ to a probability distribution over the action $a$. The
goal of reinforcement learning is to learn a policy that maximizes the
expected cumulative discounted reward
$\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}]$, where $r_{t}$ is the
immediate reward that the agent receives from the environment after taking
action $a_{t}$ at state $s_{t}$. Reinforcement learning is mainly divided into
two categories: the value-based methods[37, 38] and the policy gradient
methods[39, 40, 41]. Value-based methods compute the state-action value
function and choose actions greedily based on the computed value function.
Policy gradient methods optimize the policy directly by the policy gradient
computed on the value function. To encourage the agent to explore the state
space, we adopt the Soft Actor-Critic (SAC)[39] algorithm for both the high-
level and the low-level policies in our experiments. In SAC, the standard
value loss function is:

$\begin{split}L_{Q}(\theta)=\mathbb{E}_{s_{t},a_{t},s_{t+1}\sim\mathcal{B},a_{t+1}\sim\pi_{\phi}}[\frac{1}{2}(Q_{\theta}(s_{t},a_{t})-(r(s_{t},a_{t})\\\
+\gamma
Q_{\theta}(s_{t+1},a_{t+1})-\alpha\textrm{log}(\pi_{\phi}(a_{t+1}|s_{t+1}))))^{2}]\end{split}$
(1)

where $\gamma$ is the discount factor, $\alpha$ denotes the temperature
coefficient, $\mathcal{B}$ is the replay buffer, and $\pi_{\psi}$ represents
the policy. The policy loss function can be expressed as follows:

$\begin{split}L_{\pi}(\psi)=\mathbb{E}_{s_{t}\sim\mathcal{B},a_{t}\sim\pi_{\psi}}[\textrm{log}\pi_{\psi}(a_{t}|s_{t})-\frac{1}{\alpha}Q_{\theta}(s_{t},a_{t})]\end{split}$
(2)

Goal-conditioned hierarchical reinforcement learning models long-horizon
decision-making tasks as a goal-conditioned Markov decision process,
$M=<\mathcal{S},\mathcal{G},\mathcal{A},\mathcal{P},r,\gamma>$, where
$\mathcal{G}$ represents the goal space. As illustrated in Figure 1, goal-
conditioned reinforcement learning is a hierarchical framework consisting of
two policies. The high-level policy $\pi_{h}(g|s)$ operates at a lower
temporal resolution, sampling a high-level action every c time steps (i.e.,
$t\equiv 0(\textrm{mod}c)$. When $t\not\equiv 0(\textrm{mod}c)$, a predefined
function such as the identity function is used to specify a subgoal for the
low-level policy. Given the current state $s_{t}$ and subgoal $g_{t}$, the
low-level policy $\pi_{l}(a_{t}|s_{t},g_{t})$ produce low-level actions that
are executed by the agent to interact with the environment. During training,
since the goal of the low-level policy is to enable the agent to achieve the
subgoals specified by the high-level policy, the low-level policy is optimized
using intrinsic rewards $-\left\|\phi(s_{t+1})-g_{t}\right\|_{2}$ computed
from the sub-goal, where $\phi$ is the sub-goal representation function that
maps the state to the goal space. Similar to the standard reinforcement
learning paradigm, the learning objective of the hierarchical structure is
still to enable the agent to interact efficiently with the external
environment, so the reward of the high-level policy is defined as the sum of
$c$ external rewards $\sum_{t=0}^{c}r_{t}^{env}$ after executing a high-level
action.

Figure 1: The framework of goal-conditioned hierarchical reinforcement
learning (GCHRL, [6, 7, 8, 9]). $\phi$ is the subgoal representation function
that maps the state to the goal space. The hierarchical framework consists of
a high-level policy and a low-level policy. The reward of the high-level
policy is a sum of c (the low-level policy length) external rewards, while the
reward of the low-level policy is the negative distance between the state and
subgoal in the latent space.

In the goal-conditioned reinforcement learning, the high-level policy and the
low-level policy can be trained simultaneously in an end-to-end manner. The
high-level policy provides real-time subgoals to the low-level policy and
guides its learning through intrinsic rewards. The dynamic low-level policy
learned online influences the stationarity of the high-level state
transitions. Therefore, a well-designed hierarchical exploration strategy can
significantly enhance the learning efficiency of the hierarchical policy.

##  IV Method

In this section, we propose LESP: Landmark guided active Exploration with
Stable Low-level Policy learning. We describe LESP from three aspects:
measures for subgoals, stable low-level policy learning and hierarchical
exploration strategy.

###  IV-A Measures for Subgoals

Figure 2: landmark selection and prospect calculation process. The calculation
of prospect involves four stages: 1) Sampling: An adequate number of sample
points are randomly selected from the state space. Then, the FPS algorithm is
employed to sample $n_{cov}$ landmark points. 2) Building a graph: The sampled
landmarks, current position, and goal are used as nodes to build a graph. The
edges of the graph represent the reachability between two nodes. 3) Path
planing: Using the shortest path planning algorithm, a feasible path from the
current position to the goal is determined based on the constructed graph. 4)
Calculation: Landmark is sampled along the trajectory and selected as
$l_{sel}$. The prospect of the subgoals (within the neighborhood of the
current position) is then calculated based on the selected landmark.

In goal-conditioned hierarchical reinforcement learning, effectively guiding
the agent’s exploration is crucial for improving the algorithm performance.
Previous count-based exploration methods[11, 15] define the novelty of
subgoals based on the number of visits to states. However, solely relying on
visit counts may not always guide the agent to explore promising states. The
state-of-the-art exploration approach[15] introduces the measure of potential
for subgoals, aiming to effectively guide the agent to explore the unexplored
region. Our insight is that reasonable subgoals should not only guide the
agent towards expanding the exploration area but also guide the agent to
explore regions that are likely to lead to the ultimate goal. To address this,
we design the measure of prospect for subgoals, which reflects the subgoal’s
positivity towards achieving the final goal.

The prospect measure requires specifying the exploration direction for the
agent. Following prior work[11, 32], we choose to plan landmarks in the
subgoal representation space. We aim to sample landmarks that cover a wide
range of the goal space. To achieve this, we employ the Farthest Point
Sampling (FPS)[43] algorithm to sample $n_{cov}$ landmarks from the goal
space. Starting with an initial candidate set, we iteratively add the farthest
landmark to the sampled landmark set until a sufficient number of landmarks
are sampled where the distance between two states is measured by the Euclidean
distance in the goal space. This sampling process ensures that the sampled
landmarks are spread out and cover a diverse range of region in the goal
space. To search for a promising path from the current state to the goal, we
build a graph consisting of the current state, the goal, and the sampled
landmarks as nodes; following previous works[11, 29, 30, 32] and [42], the
edges between nodes are weighted based on the goal-conditioned value function
$V(s_{1},\phi(s_{2})$. This value function reflects the reachability between
two states $s_{1}$ and $s_{2}$. Once the graph is built, we can plan the
landmarks on the graph. By applying a shortest path planning algorithm, we can
find a feasible path from the current state to the goal, selecting the
landmark $l_{sel}$ on the path that is closest to the current state as the
region the agent should explore. Algorithm 1 describes the process of landmark
selection.

Algorithm 1 Landmark Selection

Input:  current state $s_{t}$, goal $g$, threshold $\tau$, $\phi(s)$

and goal-conditioned value function $V(s,g)$

Output:  selected lanmark $l_{sel}$

Initial $n$ transitions $T=(s,a,s^{\prime})$ from buffer $B_{pre}$

$N\leftarrow\textbf{FPS}(S=\left\\{s\parallel(s,a,s^{\prime})\in
T\right\\})\cup\left\\{g\right\\}$

$W_{i,j}\leftarrow\infty$

for  $\forall(n_{i},n_{j})\in N\times N$ do

$w_{i,j}\leftarrow[-V(n_{i},\phi(n_{j}))]$

if $w_{i,j}\leq\tau$ then

$W_{i,j}=w_{i,j}$

end if

end for

$Trajectory\ K\leftarrow\textbf{Shortest Path Planing}(N,W)$

$k_{sel}\leftarrow\text{argmin}_{k_{i}\in K}-V(s_{t},\phi(k_{i}))$

$l_{sel}=\phi(k_{sel})$

return  $l_{sel}$

In practice, before training the hierarchical policy, we allow the agent
randomly walk in the environment and collect trajectories stored in buffer
$B_{pre}$ (not the replay buffer $\mathcal{B}$). We then use these
trajectories to train the goal-conditioned value function $V(s,g)$. For the
current state $s_{t}$, after selecting landmark $l_{sel}$ with Algorithm 1,
the exploration strategy considers selecting a subgoal $g_{t}$ near the
current state $s_{t}$, where the prospect of the subgoal $g_{t}$ is defined as
follows:

$\begin{split}P(g_{t})=-\left\|g_{t}-l_{sel}\right\|_{2}\end{split}$ (3)

The landmark selection and prospect calculation process are depicted in Figure
2. The prospect measure takes into account the influence of the goal on the
subgoal selection by considering the feasible path to the goal in the latent
space. In contrast to previous work, the prospect measure not only encourages
the agent to explore unexplored region but also guides the agent to explore
region that have a positive impact on achieving the task goal. To maintain the
agent’s ability to explore new states, similar to previous work[11, 15], we
also consider the novelty measure for subgoals. The novelty of a subgoal is
measured by discrete counting in the replay buffer $\mathcal{B}$, defined as:

$\begin{split}N(s_{i})=\mathbb{E}_{s_{i+jc}\sim\mathcal{B}}[-\sum_{j=0}^{\left\lfloor(T-i)/c\right\rfloor}\gamma^{j}n(\phi(s_{i+jc}))]\end{split}$
(4)

where $c$ denotes the horizon for the low-level policy, $\gamma$ is the
discount factor and $n(\phi(s))$ indicates the immediate count of $s$ in the
replay buffer $\mathcal{B}$. Following HESS[15], In order to reduce the
computational cost, we discretize the state space into cells and estimate
$n(\phi(s))$ by counting the number of times each cell is visited.

###  IV-B Stable Low-level Policy Learning

Goal-conditioned hierarchical reinforcement learning is an effective paradigm
for solving complex sequential decision-making tasks. However, the dynamic
changes in low-level policy introduce non-stationarity in the high-level state
transitions. Over time, taking the same high-level action in the same state
can lead to completely different state transitions. This is because high-level
state transitions are not solely determined by the environmental dynamics. To
mitigate the non-stationarity in learning the low-level policy, inspired by
the stable learning of subgoal representations in HESS[15], we introduce a
state-specific regularization term $L_{r}$ in the loss of the low-level
policy. $L_{r}$ is designed to limit the estimation differences of the low-
level Q-function at different time steps without compromising learning
efficiency. The definition of $L_{r}$ is as follows:

$\begin{split}L_{r}(\theta)=\mathbb{E}_{s,g\sim\mathcal{B},a\sim\pi_{l}}[\lambda(s,g)\left\|Q_{\theta}(s,g,a)-Q_{\theta_{old}}(s,g,a)\right\|_{2}]\end{split}$
(5)

where $\mathcal{B}$ is the replay buffer, $\pi_{l}$ is the low-level policy
and $\lambda(s,g)$ is the regularization weight of the state-subgoal pair
$(s,g)$. In practice, we first calculate the loss $L_{Q}(\theta)$ for
different state-subgoal pairs using Equation 1. Then, we set $\lambda$ to 1
for the $k\%$ smallest state-subgoal pairs and 0 for the rest. This is to
limit the variation of the value function for states where the agent has
already explored sufficiently, while maintaining the learning efficiency of
the low-level policy in unknown states. So the overall loss for the value
function of the low-level policy is represented by
$L_{Q}(\theta)+L_{r}(\theta)$.

###  IV-C Hierarchical Exploration Strategy

Active exploration strategy can directly influence the behavioral policy,
avoiding the introduction of additional non-stationarity through intrinsic
rewards. Selecting subgoals with high potential and novelty can facilitate the
intelligent agent in expanding its exploration area. However, blindly
exploring unknown region may lead to accumulation of ineffective experiences.
In order to further enhance the efficiency of training the hierarchical
policy, we propose an active hierarchical exploration strategy that takes into
account the influence of the final goal. The proposed exploration strategy
aims to maintain the agent’s ability to explore unknown regions while
prioritizing subgoal $g_{t}$ with high prospect, specified as follows:

$\begin{split}g_{t}=\mathop{\mathrm{argmax}}\limits_{\phi(s)}N(\phi(s))+\alpha
P(\phi(s))\\\ \mathrm{subject\ to}\begin{cases}D(\phi(s),\phi(s_{t}))\leq
r_{g}\\\ s\in\mathcal{B}\end{cases}\end{split}$ (6)

where $N$ and $P$ are normalized measures of novelty and prospect,
respectively, $D$ represents the Euclidean distance, $\mathcal{B}$ is the
replay buffer, $\phi$ is the subgoal representation function learned online,
and $\alpha$ is the balancing coefficient. Algorithm 2 describes our active
hierarchical exploration strategy. In LESSON[13], it has been demonstrated
that the triplet loss based on slow features can capture the relative
positional relationships in the state space. Therefore, following LESSON, we
train the subgoal representation function $\phi$ using the triplet loss, as
follows:

$\begin{split}L_{\phi}=\mathbb{E}_{(s_{t},s_{t+1},s_{t+c})\sim\mathcal{B}}[\left\|\phi(s_{t})-\phi(s_{t+1})\right\|_{2}\\\
+max(0,\delta-\left\|\phi(s_{t})-\phi(s_{t+c})\right\|_{2})]\end{split}$ (7)

where $\mathcal{B}$ is the replay buffer, $\delta$ is the margin parameter of
triplet loss, $c$ is the low level policy length of the hierarchical
framework.

Algorithm 2 LESP algorithm

Initialize: $\pi_{h}(g|s),\pi_{l}(a|s,g)$ and $\phi(s)$

for $i=1..episodeNum$ do

for  $t=0..T-1$ do

if $t\equiv 0(\textrm{mod}c)$ then

Uniformly sample a number $n$ in the range (0,1)

if $n<p\,(p\in(0,1))$ then

Select landmark $l_{sel}$ with Algorithm 1

Generate a candidate set of subgoals

Calculate the prospect and novelty of sub-

goals by Eq.3 and Eq.4

Select subgoal $g_{t}$ by Eq.6

else

Execute $g_{t}\sim\pi_{h}(\cdot|s_{t})$

Update $\pi_{h}$

else

$g_{t}=g_{t-1}$

end if

Execute $a_{t}\sim\pi(\cdot|s_{t},g_{t})$

Store experiences in replay buffer $\mathcal{B}$

Update $\pi_{l}$ by Eq.1 and Eq.5

end for

if $i\equiv 0(\textrm{mod}I)$ then

Update $\phi$ using the triplet loss defined by Eq.7

end if

end for

return: $\pi_{h},\pi_{l},\phi$

##  V Experiments

In order to evaluate the performance of the landmark-guided hierarchical
exploration scheme, we conduct experiments on challenging sparse reward tasks
in the Mujoco environment. The experimental design primarily aims to answer
the following questions:

1) How does the landmark-guided active exploration strategy compare to the
state-of-the-art active exploration method HESS in terms of performance?

2) How does the Prospect measure guide exploration and improve performance?

3) What is the importance of each component in the proposed hierarchical
exploration strategy, as well as the impact of the hyper-parameters on the
experiments?

###  V-A Experimental Setup

We conduct experiments on several long-horizon decision making tasks based on
the Mujoco engine, as illustrated in Figure 3.

(a)

(b)

(c)

(d)

(e)

Figure 3: The Mujoco environments we used for hierarchical exploration
experiment. (a) Point Maze. (b) Ant Maze. (c) Ant Maze (W-shape). (d) Ant
Push. (e) Ant FourRooms. In our experiments, all the task environments are
designed with sparse reward. This setup adds to the challenge of the task, as
the agents must explore and discover effective strategies to achieve the
desired goal despite the scarcity of rewards.

Point Maze: Simulate a ball initially positioned in the bottom-left corner of
a U-shaped maze, with the objective of reaching the top-left corner of the
maze.

Ant Maze: Similar to Point Maze, but the agent is a simulated ant navigating
through the maze environment.

Ant FourRooms: The task environment is a maze with four rooms. A simulated ant
starts from the bottom-left corner and aims to reach a distant goal located in
the top-right corner.

Ant Push: In this task, there is a movable block obstructing the path of the
ant towards the target. The ant needs to move the block aside in order to
reach the goal.

Ant Maze (W-shape): In this environment, the maze structure is larger, 32 ×
32. The start position is (14,0), and the goal is (14, 14).

In the experiments, all of these task environments are designed with sparse
rewards, making the tasks more challenging and suitable for evaluating the
exploratory capabilities of the algorithms. The adoption of sparse rewards
ensures that the agents must actively explore and discover effective
strategies to achieve their goals. This setup provides a rigorous evaluation
of the algorithms’ performance in handling tasks with limited feedback and
encourages the development of efficient exploration techniques.

In all the experiments, the actor network for both the low-level and the high-
level policies is a Multi-Layer Perceptron (MLP) with two hidden layers of
dimension 256. The Critic network structure is the same as that of the actor
network. The subgoal representation function is a MLP with one hidden layer of
dimension 100. The activation function of the MLP is the ReLU function. All
the experiments are carried out on NVIDIA GTX 2080 Ti GPU and optimized using
the Adam optimizer. In all the experiments, the radius $r_{g}$ of subgoal
selection is set to 20. The discount factor $\gamma$ is set to 0.99. The
subgoal representation function is updated every 100 episodes. The batch size
for both level policies is set to 128.

###  V-B Comparative Analysis

We conduct a comparative analysis of our proposed algorithm with several
existing hierarchical reinforcement learning methods, including HISM, HSR,
HESS, LESSON, and SAC. (i) H-ICM utilizes curiosity as intrinsic rewards to
guide the agent’s exploration in the environment and learn useful skills. (ii)
HESS introduces a potential measure for subgoal and proposes an active
exploration strategy with stable subgoal representation learning. (iii) HSR
introduces a count-based exploration method that utilizes implicit state visit
counting to guide the exploration process. (iv) LESSON utilizes triplet loss
to learn subgoal representations online and achieves effective hierarchical
exploration by selecting slow features as the subgoal space representation.
(v) SAC is a reinforcement learning algorithm based on the actor-critic
mechanism. In the experiment, both the high-level policy and low-level policy
of the hierarchical framework are trained by the SAC algorithm.

The experimental results depicted in Figure 4 demonstrate that, our proposed
exploration strategy LESP outperforms all the baseline methods in handling
hard-exploration tasks. The superiority of LESP can be attributed to its
comprehensive consideration of both the novelty and the prospect in the
exploration strategy. The calculation of prospect involves planning feasible
trajectory based on reachability between landmarks, which takes into account
the influence of task goal on exploration. Unlike the potential-based
exploration approach in HESS, which only focuses on expanding the exploration
area, LESP effectively guides the agent towards regions that have a positive
impact on reaching the task goal. Additionally, we find that, in more complex
tasks such as Ant FourRooms and Ant Maze (W-shape), which place greater
demands on the exploration capability of the agent, the advantage of LESP in
terms of sample utilization efficiency becomes more pronounced. It can be
observed that, HICM and HSR fall short compared to our proposed method. HICM’s
performance is dependent on the learning error of the dynamic model; the
instability of the intrinsic rewards restricts the learning of the high-level
policy. HSR calculates state visit counts using the $L_{1}$ norm of the
successor representation, but relying solely on the successor representation,
proves inadequate in promoting effective exploration. The experimental results
demonstrate that, SAC exhibits the poorest performance when compared to the
goal-conditioned hierarchical framework LESSON, which incorporates online
learning of subgoal representations. This finding serves as strong evidence
supporting the effectiveness of the goal-conditioned hierarchical framework.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4: Learning curves of LESP and baselines on all environments. (a) Point
Maze. (b) Ant Maze. (c) Ant Maze (W-shape). (d) Ant Push. (e) Ant FourRooms.
(f) Ant Maze (Images). The x-axis represents the training time steps, while
the y-axis represents the average success rate over 50 episodes. The
experiments are evaluated for each algorithm using five different random
seeds. The shaded area represents the $95\%$ confidence interval.

###  V-C Qualitative Analysis on Measures for Subgoals

To assess the impact of subgoal measures on subgoal selection, we visualize
the Prospect and Novelty measures in the AntMaze task, as shown in Figure 5.
The visualization reveals that, subgoals with high Novelty are distributed
throughout the candidate subgoal set, indicating that the agent can explore in
various directions to expand the explored area. Figure 5 provides insightful
observations regarding the Prospect measure. It indicates that subgoals
guiding the agent closer to the goal exhibit higher Prospect. In contrast,
subgoals that do not contribute significantly to task completion have lower
Prospect. This suggests that the Prospect measure effectively captures the
potential impact of subgoals on the agent’s ability to reach the ultimate
goal. As depicted in Figure 5, after considering both novelty and prospect
measures comprehensively, the exploration strategy tends to prioritize
subgoals with high Prospect. Meantime, the exploration strategy also maintains
the ability to explore unknown region, thereby striking a balance between
exploiting promising subgoals and continuing to explore uncharted areas. After
training 300,000 time steps, the agent has made progress and could reach
positions closer to the target. Furthermore, as the buffer expands, the
novelty measure based on counting becomes more accurate. Subgoals with high
novelty tend to concentrate in directions where exploration is insufficiently
conducted.

(a)

(b)

Figure 5:  The visualization of subgoal measures in the AntMaze task. (a)
visualization at time steps 200000. (b) visualization at time steps 300000.
The circular markers represent the candidate subgoal set sampled by the agent.
The color intensity of the markers, ranging from red to blue, indicates the
corresponding measure values.

###  V-D Ablation Studies

Ablative analysis of various components: In order to investigate the
importance of different components in LESP, we conduct an ablation study on
the proposed exploration strategy. We evaluate the performance of the
algorithm under various experimental settings, including (i) the original
exploration strategy (LESP) proposed in our work. (ii) LESP without stable
value function, which means employing Equation 2 as the loss function to
update the parameters of the value network in the low-level policy, without
incorporating state-specific regularization. (iii) LESP without prospect,
which means that the weight coefficient in Equation 6 is set to 0. (iv)
replace prospect with potential, which means the prospect measure is replaced
with the potential measure used in HESS and (v)HESS, state-of-the-art active
exploration method. We conduct experiments in three different tasks: AntMaze
(Images), AntMaze (W-shape), and AntFourRooms. The experimental results are
depicted in Figure 6. As observed in Figure 6, replacing the prospect measure
with potential leads to a significant decline in the performance of the
exploration strategy. This highlights the superiority of the prospect measure
over potential. While potential measure encourages the agent to expand the
exploration area, prospect effectively guides the agent towards regions that
positively contribute to accomplishing the task. In the experimental setting
without prospect, the algorithm performs poorly across multiple complex tasks.
This emphasizes the crucial role of the prospect measure in the exploration
process. LESP demonstrates better performance compared to the setting without
a stable value function, especially in the AntMaze (W-shape) task, LESP
discovers effective trajectories at an earlier stage. Furthermore, even after
replacing prospect with potential, the performance of the exploration strategy
remains superior to that of HESS. This further highlights the importance of
stable learning of the low-level state-action value function in promoting the
stability of hierarchical policies.

Figure 6:  Ablation studies on the components of proposed exploration
strategy. In the conducted ablation experiments, three task scenarios are
examined, namely: (a) Ant Maze (Images). (b) Ant Maze (W-shape). (c) Ant
FourRooms. For each of these tasks, the experiments are evaluated using five
different random seeds.

Ablation studies on the hyper-parameters selection: To evaluate the
effectiveness of the hyper-parameters, including the number of landmark
samples ($n_{cov}$), the balance coefficient ($\alpha$), and the low-level
policy length ($c$), we conduct an ablation study. All tests are performed in
the challenging Ant FourRooms task, and the experimental results are presented
in Figure 7.

(a)

(b)

(c)

Figure 7: Ablation studies on the hyper-parameters. (a) Balance coefficient
$\alpha$. (b) Number of landmarks $n_{cov}$. (c) Low level policy length $c$.
All the ablation experiments regarding hyper-parameters are conducted
specifically on the Ant FourRooms task

Balance coefficient $\alpha$: In the proposed exploration strategy, the
balance coefficient $\alpha$ is used to balance the importance of novelty and
prospect measure. A larger $\alpha$ value indicates that the exploration
strategy prioritizes subgoals that have a more positive impact on guiding the
agent towards the goal, while potentially diminishing the ability to explore
new states. In complex scenarios, choosing an appropriate balance coefficient
is crucial. From Figure 7 (a), it can be observed that larger values of
$\alpha$ in a reasonable range tend to yield better results. For all the
experiments in Section 5.2, we set $\alpha$ to 0.1

Number of landmark samples $n_{cov}$: LESP utilizes the FPS algorithm to
sample $n_{cov}$ candidate landmarks from the initial set of landmarks. It
then selects the nearest landmark on the planned trajectory as the $l_{sel}$
(selected landmark). The number of sampled landmarks, $n_{cov}$, influences
the efficiency of trajectory planning. If $n_{cov}$ is too small, it may
result in the inability to plan feasible path to the goal. On the other hand,
if $n_{cov}$ is too large, the selected $l_{sel}$ may be too close to the
current state $s_{t}$. This can hinder the effectiveness of prospect in
guiding the exploration process. So selecting an appropriate $n_{cov}$ is
indeed crucial for choosing subgoals that provide meaningful guidance during
exploration. In the experiments conducted in Section 5.2, for the Ant
FourRooms task, we set $n_{cov}$ to 60, indicating a larger number of
candidate landmarks to ensure comprehensive coverage of the environment. For
the Ant Maze (W-shape) task, $n_{cov}$ is set to 40. For other tasks,
$n_{cov}$ is set to 20.

Low-level policy length $c$: Hierarchical reinforcement learning (HRL)
decomposes long-horizon tasks into sub-tasks with finite horizon, denoted as
$c$. Selecting an appropriate value for $c$ is advantageous as it allows for
proper allocation of task difficulty between the high-level policy and low-
level policy. In active exploration approaches, the selected sub-goals are fed
to the low-level policy with a finite horizon. Therefore, selecting an
inappropriate value for $c$ can result in the selected sub-goals being unable
to effectively guide exploration, leading to the failure of the active
exploration strategy. The experimental results are shown in Figure 7 (c). For
all the experiments in Section 5.2,we set $c$ to 20.

##  VI Conclusion

Goal-conditioned hierarchical reinforcement learning (HRL) is a paradigm that
effectively addresses complex long-horizon problems. However, improving sample
efficiency is a crucial and yet unresolved issue in reinforcement learning. To
tackle challenging sparse reward tasks, we proposed an active exploration
strategy LESP, which takes into account prospect and novelty measures for
subgoals. We designed a prospect measure for subgoals in LESP. LESP generated
promising trajectories by planning landmarks in the goal space. It then
computed prospect measures for subgoals based on the selected landmark. Unlike
HIGL, which guided exploration by constraining the high-level policy, LESP
sampled subgoals in the vicinity of the current state to guild exploration.
This active exploration approach avoided introducing additional non-
stationarity to the high-level policy. In addition, to mitigate the impact of
dynamic changes in the low-level policy on the high-level state transitions,
we incorporated state-specific regularization into the training of the low-
level policy. Experimental results demonstrated that LESP outperformed the
state-of-the-art approach HESS. Additionally, some people may point out that
LESP requires extra buffer to store samples of prior state space, as well as a
goal-condition value function that reflects the reachability between two
states. Our view on this is that, since an expert policy is not required for
randomly walk in the environment and the goal-conditioned value function only
focuses on the neighboring states, the additional computation cost brought by
LESP is acceptable. For some complex, long-term tasks such as robot navigation
and robotic arm control, LESP is of significant importance for efficient
interaction between agents and the environment.

## References

  * [1] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel _et al._ , “Mastering atari, go, chess and shogi by planning with a learned model,” _Nature_ , vol. 588, no. 7839, pp. 604–609, 2020.
  * [2] V. Micheli, E. Alonso, and F. Fleuret, “Transformers are sample efficient world models,” _arXiv Preprint arXiv:2209.00588_ , 2022.
  * [3] L. Kästner, X. Zhao, T. Buiyan, J. Li, Z. Shen, J. Lambrecht, and C. Marx, “Connecting deep-reinforcement-learning-based obstacle avoidance with conventional global planners using waypoint generators,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 1213–1220.
  * [4] D. Dugas, J. Nieto, R. Siegwart, and J. J. Chung, “Navrep: Unsupervised representations for reinforcement learning of robot navigation in dynamic human environments,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 7829–7835.
  * [5] H. Li, X. Yang, Y. Yang, S. Mei, and Z. Zhang, “Memonav: Selecting informative memories for visual navigation,” _arXiv Preprint arXiv:2208.09610_ , 2022\.
  * [6] P. Dayan and G. E. Hinton, “Feudal reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 5, 1992, pp. 271–178.
  * [7] O. Nachum, S. Gu, H. Lee, and S. Levine, “Near-optimal representation learning for hierarchical reinforcement learning,” _arXiv Preprint arXiv:1810.01257_ , 2018.
  * [8] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum, “Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016, pp. 1–9.
  * [9] V. H. Wang, J. Pajarinen, T. Wang, and J.-K. Kämäräinen, “Hierarchical reinforcement learning with adversarially guided subgoals.” _CoRR_ , 2022.
  * [10] T. Zhang, S. Guo, T. Tan, X. Hu, and F. Chen, “Generating adjacency-constrained subgoals in hierarchical reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 21579–21590.
  * [11] J. Kim, Y. Seo, and J. Shin, “Landmark-guided subgoal generation in hierarchical reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021, pp. 28336–28349.
  * [12] A. Péré, S. Forestier, O. Sigaud, and P.-y. Oudeyer, “Unsupervised learning of goal spaces for intrinsically motivated goal exploration,” in _International Conference on Learning Representations_ , 2018, pp. 1–26.
  * [13] S. Li, L. Zheng, J. Wang, and C. Zhang, “Learning subgoal representations with slow dynamics,” in _International Conference on Learning Representations_ , 2021, pp. 1–18.
  * [14] S. Sukhbaatar, E. Denton, A. Szlam, and R. Fergus, “Learning goal embeddings via self-play for hierarchical reinforcement learning,” _arXiv Preprint arXiv:1811.09083_ , 2018.
  * [15] S. Li, J. Zhang, J. Wang, Y. Yu, and C. Zhang, “Active hierarchical exploration with stable subgoal representation learning,” in _International Conference on Learning Representations_ , 2022, pp. 1–18.
  * [16] O. Nachum, S. S. Gu, H. Lee, and S. Levine, “Data-efficient hierarchical reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018, pp. 1–11.
  * [17] A. Levy, G. Konidaris, R. Platt, and K. Saenko, “Learning multi-level hierarchies with hindsight,” in _International Conference on Learning Representations_ , 2019, pp. 1–15.
  * [18] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2012, pp. 5026–5033.
  * [19] M. Eppe, C. Gumbsch, M. Kerzel, P. D. Nguyen, M. V. Butz, and S. Wermter, “Intelligent problem-solving as integrated hierarchical reinforcement learning,” _Nature Machine Intelligence_ , vol. 4, no. 1, pp. 11–20, 2022\.
  * [20] G. Wan, S. Pan, C. Gong, C. Zhou, and G. Haffari, “Reasoning like human: Hierarchical reinforcement learning for knowledge graph reasoning,” in _Proceedings of the Twenty-ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 1926–1932.
  * [21] Y. Takubo, H. Chen, and K. Ho, “Hierarchical reinforcement learning framework for stochastic spaceflight campaign design,” _Journal of Spacecraft and rockets_ , vol. 59, no. 2, pp. 421–433, 2022.
  * [22] M. Rohmatillah and J.-T. Chien, “Hierarchical reinforcement learning with guidance for multi-domain dialogue policy,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 748–761, 2023.
  * [23] S. Notter, F. Schimpf, G. Müller, and W. Fichter, “Hierarchical reinforcement learning approach for autonomous cross-country soaring,” _Journal of Guidance, Control, and Dynamics_ , vol. 46, no. 1, pp. 114–126, 2023.
  * [24] K. Gregor, D. J. Rezende, and D. Wierstra, “Variational intrinsic control,” _arXiv Preprint arXiv:1611.07507_ , 2016.
  * [25] R. Fox, S. Krishnan, I. Stoica, and K. Goldberg, “Multi-level discovery of deep options,” _arXiv Preprint arXiv:1703.08294_ , 2017.
  * [26] J. Duan, S. Eben Li, Y. Guan, Q. Sun, and B. Cheng, “Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data,” _IET Intelligent Transport Systems_ , vol. 14, no. 5, pp. 297–305, 2020.
  * [27] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu, “Feudal networks for hierarchical reinforcement learning,” in _International Conference on Machine Learning_ , 2017, pp. 3540–3549.
  * [28] D. Ghosh, A. Gupta, and S. Levine, “Learning actionable representations with goal conditioned policies,” in _International Conference on Learning Representations_ , 2019, pp. 1–18.
  * [29] V. Pong, S. Gu, M. Dalal, and S. Levine, “Temporal difference models: Model-free deep rl for model-based control,” in _International Conference on Learning Representations_ , 2018, pp. 1–14.
  * [30] S. Nasiriany, V. Pong, S. Lin, and S. Levine, “Planning with goal-conditioned policies,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019, pp. 1–12.
  * [31] E. Chane-Sane, C. Schmid, and I. Laptev, “Goal-conditioned reinforcement learning with imagined subgoals,” in _International Conference on Machine Learning_ , 2021, pp. 1430–1440.
  * [32] L. Zhang, G. Yang, and B. C. Stadie, “World model as a graph: Learning latent landmarks for planning,” in _International Conference on Machine Learning_ , 2021, pp. 12611–12620.
  * [33] J. Li, C. Tang, M. Tomizuka, and W. Zhan, “Hierarchical planning through goal-conditioned offline reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 10216–10223, 2022.
  * [34] L. Kästner, T. Buiyan, L. Jiao, T. A. Le, X. Zhao, Z. Shen, and J. Lambrecht, “Arena-rosnav: Towards deployment of deep-reinforcement-learning-based obstacle avoidance into conventional autonomous navigation systems,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 6456–6463.
  * [35] L. Kästner, X. Zhao, Z. Shen, and J. Lambrecht, “Obstacle-aware waypoint generation for long-range guidance of deep-reinforcement-learning-based navigation approaches,” _arXiv Preprint arXiv:2109.11639_ , 2021.
  * [36] R. S. Sutton and A. G. Barto, _Reinforcement earning: An introduction_. MIT press, 2018.
  * [37] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016, pp. 2094–2100.
  * [38] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in _International Conference on Machine Learning_ , 2016, pp. 1995–2003.
  * [39] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv Preprint arXiv:1812.05905_ , 2018.
  * [40] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv Preprint arXiv:1707.06347_ , 2017\.
  * [41] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International Conference on Machine Learning_ , 2016, pp. 1928–1937.
  * [42] Z. Huang, F. Liu, and H. Su, “Mapping state space using landmarks for universal goal reaching,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019, pp. 1942–1952.
  * [43] D. Arthur and S. Vassilvitskii, “K-means++ the advantages of careful seeding,” in _Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms_ , 2007, pp. 1027–1035.


# Forensic Iris Image Synthesis

Rasel Ahmed Bhuiyan Adam Czajka
Department of Computer Science and Engineering, University of Notre Dame
384 Fitzpatrick Hall of Engineering, Notre Dame, 46556, Indiana, USA
<EMAIL_ADDRESS>
###### Abstract

Post-mortem iris recognition is an emerging application of iris-based human
identification in a forensic setup, able to correctly identify deceased
subjects even three weeks post-mortem. This technique thus is considered as an
important component of future forensic toolkits. The current advancements in
this field are seriously slowed down by exceptionally difficult data
collection, which can happen in mortuary conditions, at crime scenes, or in
“body farm” facilities. This paper makes a novel contribution to facilitate
progress in post-mortem iris recognition by offering a conditional StyleGAN-
based iris synthesis model, trained on the largest-available dataset of post-
mortem iris samples acquired from more than 350 subjects, generating – through
appropriate exploration of StyleGAN latent space – multiple within-class (same
identity) and between-class (different new identities) post-mortem iris
images, compliant with ISO/IEC 29794-6, and with decomposition deformations
controlled by the requested PMI (post mortem interval). Besides an obvious
application to enhance the existing, very sparse, post-mortem iris datasets to
advance – among others – iris presentation attack endeavors, we anticipate it
may be useful to generate samples that would expose professional forensic
human examiners to never-seen-before deformations for various PMIs, increasing
their training effectiveness. The source codes and model weights are made
available with the paper.

Figure 1: Synthesis of PMI-conditioned same- and different-identity forensic
iris images, with a desired iris image quality (controlled by matching the ISO
quality metrics), and with the desired genuine/impostor score distributions
(mimicking score distributions obtained for authentic forensic samples).

##  1 Introduction

The immeasurable appetite of modern machine learning for data is unfading,
despite a success of various few-shot and transfer learning approaches. This
is especially true in emerging domains, in which we are just discovering the
domain’s specificity, often solely through sparse and insufficient data. Post-
mortem iris recognition is one of such domains. It gains more attention,
however, after discovery that it may be considered in forensic applications
[1], and after seeing the first post-mortem-specific and human-interpretable
recognition methods being proposed [2, 3]. Although it sounds gruesome, it
also constitutes yet another presentation attack type, and receives an
attention in the presentation attack detection area.

The question arises how to progress this research knowing that collections of
new post-mortem iris images will always be exceptionally complex and not
creating well-balanced datasets in terms of the PMI (Post-Mortem Interval),
environmental factors and/or subjects’ demographical characteristics. We
hypothesise that modern generative models, which proved useful in generating
realistic biometric samples on non-existing identities, may partially fill
this gap. However, an accurate modeling of post-mortem changes to the eye is
virtually impossible due to multiple environmental, demise-reason-related, and
subject’s health factors that contribute to post-demise deformations of iris
texture. This work proposes the first forensic iris synthesis model that
generates samples matching selected statistics of authentic post-mortem
samples, such as comparison score distributions and iris image quality
metrics, as illustrated in Fig. 1. More specifically, the main contribution of
this work is the conditional StyleGAN-based post-mortem iris synthesis model,
which:

  * •

generates full-resolution ($640\times 480$ pixels) iris post-mortem samples
for a given post-mortem interval (up to 1674 hours),

  * •

in addition to generation of new-identity images, also – for a given non-
existing identity – generates an infinite number of samples considered as
same-identity images; this was achieved by sampling from a vicinity of the
conditional StyleGAN’s latent vector (representing a given new identity) in
order to mimic authentic genuine score distribution,

  * •

generates post-mortem iris images with ISO quality metrics [4] mimicking those
for authentic forensic iris images.

As an additional contribution, facilitating the speed of adoption of this
work, this paper offers a dataset of 180,000 synthetic post-mortem iris
images, composed of 10,000 samples in each of the 18 PMI ranges (from 0 up to
1674 hours). For each PMI range there are 1,000 identities, and for each
identity there are 10 same-identity (genuine) samples. We do not control the
laterality (_i.e_. left vs right iris image) when synthesizing samples. The
source codes, model weights and instructions how to acquire a copy of the
dataset are offered with this paper111https://github.com/CVRL/Forensic-Iris-
Image-Synthesis.

Two immediate applications of the results from this study include: (a)
enhancement of the existing datasets for post-mortem iris recognition and iris
presentation attack detection, and (b) supporting human forensic iris
examiners’ training, by generating variants of the same forensic sample to
expose the experts to never-seen-before deformations of iris tissues.

##  2 Related Works

The concept of synthesizing biometric iris images was, to our knowledge, first
proposed by Cui _et al_. [5], who leveraged principal component analysis and
super-resolution techniques to create synthetic samples. Their approach set
the stage for subsequent developments in iris image generation. Shah and Ross
[6] harnessed Markov random fields to generate iris images. Zuo _et al_. [7]
introduced an anatomy-based model aimed at enhancing the realism of generated
iris samples. Wei _et al_. [8] explored patch-based sampling as a means of
generating iris images. Venugopalan and Savvides [9] ventured into generating
iris images from iris code templates, offering an alternative avenue for iris
synthesis belonging to a family of “inverse biometrics” approaches. There have
been several alternative methods proposed in the literature for the generation
of iris images from binary templates [10, 11, 12]. However, these methods
often struggle to replicate the visual realism of generated iris images,
resulting in synthetic samples that can appear artificial and thus can be
easily detected by anomaly detection techniques.

In recent years, the Generative Adversarial Networks (GAN) have gained
popularity for its success in realistic image generation, especially in tasks
like face generation [13, 14, 15]. Some GAN-based methods have been explored
for iris image generation. Minaee _et al_. [16] utilized a Deep Convolutional
GAN (DCGAN) to create realistic iris images, while Kohli _et al_. [17] used
DCGAN and iris quality metrics to study impact of synthetic irises on
presentation attack detection. Yadav _et al_. [18] employed a relativistic
average standard GAN (RaS-GAN) with Frechet Inception Distance to synthesize
high-quality iris images. Yadav and Ross [19] introduced a cyclic image
translation GAN (CIT-GAN) for multi-domain style transfer in iris images.

While the above GAN-based approaches have advanced the generation of realistic
iris images, they still exhibit occasional distortions and blurry textures.
Moreover, their primary use lies in anti-spoofing rather than iris
recognition, as they cannot generate same-identity iris samples. To address
this limitation, Kakani _et al_. [20] proposed an iris synthesis model that is
able to generate same-identity live iris samples. Maureira _et al_. [21]
utilized StyleGAN2 to generate synthetic periocular iris images. Wang _et al_.
[22] harnessed the advanced StyleGAN2 architecture to create highly realistic
iris images. To gain precise control over the generated attributes, they
implemented contrastive learning, effectively disentangling identity vectors
and enabling identity control for synthetic iris images.

To our knowledge, synthesis of both same- and different-identity forensic iris
samples, for a given PMI, and matching both the comparison score and ISO image
quality metrics to those of authentic post-mortem iris samples, have never
been explored before. We hope that the model and approach offered in this
paper will fill this gap.

##  3 Methodology

###  3.1 Dataset Composition and Splitting

There are currently three publicly available datasets of post-mortem iris
images, and we have acquired all of them to conduct this research:

#### Warsaw BioBase Post Mortem Iris v2.0 [1]

which contains 1,787 visible (VIS) and 1200 near-infrared (NIR) images from 37
deceased subjects. This dataset was collected in 2018 by conducting 1 to 13
sessions from 5 to 814 hours after death in the hospital mortuary. Most
environmental conditions, such as where cadavers were kept before entering the
cold storage, air pressure, and humidity, are unknown. However, the mortuary
room temperature was approximately $None$ Celsius ($None$ Fahrenheit).

#### Warsaw BioBase Post Mortem Iris v3.0 [23]

which contains 785 VIS and 1094 NIR images from 42 cadavers collected in 2020
over a time horizon of up to 369 hours since demise. The environmental
conditions were the same as in case of Warsaw BioBase Post Mortem Iris v2.0
set.

#### NIJ-2018-DU-BX-0215 [24]

which is the latest and the largest forensic iris dataset, gathered at the
Dutchess County Medical Examiner’s Office (DCMEO) from 269 individuals who had
passed away. A total of 53 data-gathering sessions were conducted and resulted
in 5,770 NIR images and 4,643 VIS images.

In this study, we only used NIR images and combined these three datasets into
a cohesive set comprising 8,064 post-mortem iris images representing 338
subjects. The combined dataset is split into 18 classes, with each class
gathering samples acquired within a 24-hour post-mortem interval (PMI). For
example, Class 1 gathers samples acquired within the first 24 hours after
death (so PMI $\in(0,24)$), Class 2 includes samples acquired from 25 to 48
hours after demise (so PMI $\in(25,48)$), and so on. This pattern continues up
to Class 17, which includes samples with the PMI $\in(385,408)$. Due to
limited availability of samples with PMI $>$ 408 hours, data for PMIs ranging
from 409 to 1674 hours (max PMI for our combined dataset) were consolidated
into Class 18, representing a more extended time since death. Table 1 shows
training data distribution across PMI classes, and Fig. 3(a) shows example
authentic samples from this combined dataset.

Table 1: Number of training samples for each PMI range. Class # | Number of | Class # | Number of
---|---|---|---
(PMI range) | samples | (PMI range) | samples
1 (0-24h) | 2490 | 10 (217h-240h) | 116
2 (25h-48h) | 1542 | 11 (241h-264h) | 125
3 (49h-72h) | 1094 | 12 (265h-288h) | 93
4 (73h-96h) | 484 | 13 (289h-312h) | 45
5 (97h-120h) | 494 | 14 (313h-336h) | 55
6 (121h-144h) | 222 | 15 (337h-360h) | 64
7 (145h-168h) | 328 | 16 (361h-384h) | 48
8 (169h-192h) | 174 | 17 (385h-408h) | 54
9 (193h-216h) | 238 | 18 (409h-1674h) | 327

###  3.2 Synthetic Image Generator

Our approach is built on the StyleGAN2-ADA [25] architecture, a model tailored
for training generative adversarial networks with limited data. StyleGAN2-ADA
employs non-leaking augmentations applied solely to the discriminator to
diversify the data it sees, while the generator remains exposed only to the
original samples, focusing on the genuine data distribution. Additionally, an
adaptive mechanism dynamically controls the augmentation intensity, balancing
between overfitting prevention and sample quality preservation. This strategy
enables StyleGAN2-ADA to generate high-quality samples, even with limited
training data effectively. The consideration of conditional information is
made possible in the official implementation of
StyleGAN2-ADA222https://github.com/NVlabs/stylegan2-ada, which is used by us
to generate forensic synthesis iris images with specific PMI range, given as
the condition during synthesis.

The synthesis process initiates from a given latent code $\mathbf{z}$ in the
latent space $\mathcal{Z}$. The mapping network in the original StyleGAN2-ADA
generator [15], represented as $f:\mathcal{Z}\to\mathcal{W}$, performs a
projection of $\mathbf{z}$ into the intermediate latent code $\mathbf{w}$,
where $\mathbf{w}\in\mathcal{W}$. In the conditional training scenario, $f$
possesses the capability to project $\mathbf{z}$ into $\mathbf{w}$ taking into
account an embedded condition vector $\mathbf{y}$, which allows the generator
to produce images with specific attributes based on the given condition. In
our context, the condition vector represents the PMI range, which categorizes
the time elapsed since death into bins, as described in Sec. 3.1. This allows
for learning the nature of iris texture deformations associated with different
PMIs. The system thus enables creating synthetic iris images that
statistically mimic the changes in iris appearance that occur over time after
death.

Due to very sparse data available for training, a number of augmentations have
been applied, including geometric and color adjustments, filtering, and image
corruption techniques. Geometric transformations involve x-flips,
translations, isotropic and anisotropic scaling, and arbitrary rotations. For
intensity transformations, the pipeline adjusts brightness and contrast.
Image-space filtering splits the frequency content into four bands and
amplifies or weakens their amplitudes.

We trained the model in the original setting from scratch for a batch size of
32 and with authentic iris images rescaled to $256\times 256$. During the
generation of synthetic images, to synthesize same-identity iris images, we
introduce additional noise to a randomly selected latent vector $\mathbf{w}$.
Namely, for a given $\mathbf{w}$ we generate
$\mathbf{w}^{\prime}=\mathbf{\varepsilon w}$, where
$\varepsilon\in(0,\varepsilon_{\max})$ and $\varepsilon_{\max}$ is the radius
of a hypersphere in the latent space $\mathbf{w}$, found experimentally for a
given dataset of post-mortem iris images, guarantying that resulting samples –
when matched to each other – follow the same genuine score distribution as for
authentic post-mortem samples. In each generation of same-identity sample,
$\varepsilon$ is drawn from a uniform distribution in a range of
$(0,\varepsilon_{\max})$. Fig. 2 shows the genuine score distributions
calculated for samples synthesized when drawing from hyperspheres in
$\mathbf{w}$ of various radii $\varepsilon_{\max}$.

Figure 2: Genuine score distributions obtained for authentic post-mortem
samples and synthesized post-mortem samples for different sizes of the
hypersphere (defined by radius $\varepsilon_{\max}$, shown in parentheses)
defining the same-identity manifold in the StyleGAN’s latent space
$\mathbf{w}$. The plots suggest that $\varepsilon_{\max}=0.09$ allows for
mimicking the authentic genuine score distribution on average for all PMIs.

###  3.3 ISO/IEC 29794-6 Iris Image Quality Metrics

In this study, we have implemented and calculated twelve iris image quality
metrics defined in ISO/IEC FDIS 29794-6:2014(E) [4] to assess the usefulness
of synthesized iris images for iris recognition and presentation attack
detection. The selected metrics are:
`USABLE_IRIS_AREA`,
`IRIS_SCLERA_CONTRAST`,
`IRIS_PUPIL_CONTRAST`,
`PUPIL_BOUNDARY_CIRCULARITY`,
`GREY_SCALE_UTILIZATION`,
`IRIS_RADIUS`,
`PUPIL_IRIS_RATIO`,
`IRIS_PUPIL_CONCENTRICITY`,
`MARGIN_ADEQUACY`,
`SHARPNESS`,
`MOTION_BLUR`, and
`OVERALL_QUALITY`.

Except for `GREY_SCALE_UTILIZATION` (measured in bits), `IRIS_RADIUS`
(measured in pixels), and `MOTION_BLUR` (measured as the major-to-minor axis
ratio of the ellipse approximating the point spread function estimated from
the image), all other metrics are in the range of (0,100). The 255 value means
that a given image quality metric could not be calculated.

###  3.4 Iris Recognition Method

To calculate the comparison scores, we employed an open-source academic
solution specially designed for post-mortem iris recognition, and based on
human-driven binarized image features (HDBIF) [26]. HDBIF combines deep
learning-based iris segmentation and human-inspired domain-specific feature
extraction to make iris biometrics more robust against post-mortem changes.
This method uses domain-specific filters, derived via Independent Component
Analysis (similarly to BSIF filters proposed by Kannala and Rahtu [27]), from
iris image patches collected in an eye tracking experiment, in which subjects
were solving a post-mortem iris recognition task. The resulting HDBIF filters
are applied to normalized iris images and the convolutional results are
binarized into a code, as in Daugman’s approach [28]. Finally, the comparison
score is calculated as the Hamming distance between codes representing non-
occluded iris portions. The Hamming distance is simply the fraction of bits
that the two iris codes disagree. This method is, to our knowledge, the state-
of-the-art approach to post-mortem iris recognition [3], hence its selection
for this study

(a) Example authentic post-mortem iris images originating from the Warsaw
BioBase Post Mortem Iris v2.0 [1] and v3.0 [23] datasets (with approval).

(b) Example synthetic samples, generated by the proposed model.

Figure 3: Examples of authentic and synthetically-generated images in the PMI
ranges considered in this work. Note a remarkable realism of the generated
synthetic images, and its capability to capture intricate details, such as the
deformed (due to decomposition) iris texture, irregular specular highlights
from the drying cornea (progressing when the PMI progresses), and even the
metal eye retractors. Figure 4: A set of synthetic images generated for a
starting latent vector $\mathbf{w}$ (leftmost image) and latent vectors
$\mathbf{w}^{\prime}=\mathbf{\varepsilon w}$ sampled from the same-identity
hyperspheres centered at $\mathbf{w}$ and of radii
$\varepsilon=0.01,0.03,0.05,0.07,0.09$ (from the second left to the rightmost
image, respectively).

(a) PMI: 0–24 hours

(b) PMI: 25–48 hours

(c) PMI: 49–72 hours

(d) PMI: 73–96 hours

(e) PMI: 97–120 hours

(f) PMI: 121–144 hours

(g) PMI: 145–168 hours

(h) PMI: 169–192 hours

(i) PMI: 193–216 hours

(j) PMI: 217–240 hours

(k) PMI: 241–264 hours

(l) PMI: 265–288 hours

(m) PMI: 289–312 hours

(n) PMI: 313–336 hours

(o) PMI: 337–360 hours

(p) PMI: 361–384 hours

(q) PMI: 385–408 hours

(r) PMI: 409–1674 hours

Figure 5: Comparison score distributions obtained within all PMI ranges for
authentic and synthetic post-mortem iris images.

(a) USABLE_IRIS_AREA

(b) IRIS_SCLERA_CONTRAST

(c) IRIS_PUPIL_CONTRAST

(d) PUPIL_BOUNDARY_CIRCULARITY

(e) GREY_SCALE_UTILISATION

(f) IRIS_RADIUS

(g) PUPIL_IRIS_RATIO

(h) IRIS_PUPIL_CONCENTRICITY

(i) MARGIN_ADEQUACY

(j) SHARPNESS

(k) MOTION_BLUR

(l) OVERALL_QUALITY

Figure 6: Distributions of the ISO/IEC 29794-6 iris image quality metrics
calculated for authentic and synthesized post-mortem iris samples.

##  4 Experimental Analysis

###  4.1 Evaluation Samples

To evaluate the trained model in terms of its capabilities to generate ISO-
compliant same- and different-identity post-mortem iris samples, we have
generated synthetic forensic iris images representing all PMI ranges, and
matching the number of samples in each PMI category with the numbers of
authentic samples in the training set for each PMI class. Fig. 3 shows
examples of authentic and synthetic images for each PMI class, where Fig. 3(a)
represents authentic samples and Fig. 3(b) features generated synthetic
samples. A visual inspection suggests that samples generated by the proposed
model include all intricate details specific to forensic iris images, such
post-mortem-deformed iris tissue and cornea, metal retractors, eyelid shape
due to application of such retractors, and deformed specular highlights caused
by drying cornea.

###  4.2 Matching the Comparison Score Distributions

As described in Sec. 3.2, to generate synthetic images that closely mimic
authentic images, we sampled from hyperspheres of different sizes, defined by
a radius $\varepsilon_{\max}$ centered at a latent vector representing a given
new identity. As we can see in Figure 4, depending on the distance from the
starting latent vector, the synthesis process introduces variations in the
iris texture, pupil size, and level of the decomposition-related deformations.
The obvious question is thus about the radius $\varepsilon_{\max}$ to not go
“too far” if one wants to generate genuine samples.

To assess the $\varepsilon_{\max}$ value in a quantitative manner, we
calculated the genuine comparison score distributions (using the HDBIF
matcher, Sec. 3.4) for both the authentic and synthetic samples to see which
$\varepsilon_{\max}$ levels allow to align the authentic and synthetic genuine
score distributions. An interesting observation was that, as shown in Fig. 2,
$\varepsilon_{\max}=0.09$ allows for a close alignment of these two
distributions for the entire dataset. However, when we consider the
distribution of genuine scores on a class-wise basis, setting a PMI-specific
$\varepsilon_{\max}$ may be necessary. For instance $\varepsilon_{\max}=0.05$
is a better choice for PMIs up to 312h, and $\varepsilon_{\max}=0.09$ allows
for a better score distribution alignment for the remaining PMI ranges
(313h-336h, 337h-360h, 361h-384h, 384h-408h, and 409h-1674h). This phenomenon
can be explained by large and unpredictable dynamics of post-mortem changes
for eyes with larger PMI values. Fig. 7 in the supplementary materials
illustrates all genuine score distributions calculated for authentic images
and synthetic samples generated for various $\varepsilon_{\max}$ and for each
PMI class.

For simplicity and illustration of the model’s operation, a single
$\varepsilon_{\max}=0.05$ was used to generate samples for the final
evaluation, illustrated for all PMI classes in Fig. 5.

###  4.3 Matching the ISO/IEC 29794-6 Image Quality Metrics

To assess the iris recognition-specific quality of the generated images, we
analyzed how eleven ISO/IEC 29794-6 iris image quality metrics align between
the authentic and synthetic images. These metrics offer a multifaceted
evaluation, considering aspects such as iris pattern visibility, pupil-to-iris
size ratio, pupil shape regularity, gray-scale utilization, and sharpness. In
addition, the overall quality metric, combining those eleven properties, has
been calculated. A visual representation of the distributions of ISO metrics
allowing to assess the quality of the synthesized images in relation to the
authentic samples is shown in Fig. 6. The synthetic images closely mirrored
the quality properties of the authentic images. This congruence in
distributions highlights the model’s proficiency in replicating various
quality attributes. However, two discernible differences were found. The first
one in the `GREY_SCALE_UTILIZATION` metric, where the synthetic images
exhibited a higher score for most of the generated samples compared to
authentic images, indicating a more efficient utilization of the grayscale
spectrum by the trained model. These variations may be partially attributed to
the extensive color transformations applied within the adaptive discriminator
augmentation pipeline of the StyleGAN2-ADA model. The second difference can be
noted in `SHARPNESS` values, being a bit smaller for synthetic samples. This
can be attributed to up-scaling carried out after the synthesis is finished.
Knowing that iris recognition usually employs lower-pass filters to encode
identity features, this should not have a significant impact on the usability
of the generated samples.

##  5 Summary

This study offers a conditional StyleGAN-based generative method to synthesize
post-mortem iris images, with a hope to advance both forensic iris recognition
(including human examination) and iris presentation attack detection, by
offering the first-of-this-kind tool to generate never-seen-before, realistic,
ISO-compliant, identity-preserving samples exhibiting tissue deformations
typical for a given post-mortem interval. The same-identity samples were
possible to be generated by an appropriate definition of latent space sub-
manifolds “storing” representations of the same identities. These sub-
manifolds were discovered by matching the genuine score distributions
calculated for synthetic and authentic images. In addition to the sources
codes and model weights, this paper also offers a ready-to-use dataset of
180,000 synthetic, post-mortem iris images for 18 different PMI ranges.

## References

  * [1] Mateusz Trokielewicz, Adam Czajka, and Piotr Maciejewicz.  Iris recognition after death.  IEEE Transactions on Information Forensics and Security, 14(6):1501–1514, 2018.
  * [2] Andrey Kuehlkamp, Aidan Boyd, Adam Czajka, Kevin Bowyer, Patrick Flynn, Dennis Chute, and Eric Benjamin.  Interpretable deep learning-based forensic iris segmentation and recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 359–368, 2022.
  * [3] Aidan Boyd, Daniel Moreira, Andrey Kuehlkamp, Kevin Bowyer, and Adam Czajka.  Human saliency-driven patch-based matching for interpretable post-mortem iris recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 701–710, 2023.
  * [4] Information Technology.  Biometric Sample Quality — Part 6: Iris Image Data, 2014.
  * [5] Jiali Cui, Yunhong Wang, JunZhou Huang, Tieniu Tan, and Zhenan Sun.  An iris image synthesis method based on pca and super-resolution.  In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 4, pages 471–474. IEEE, 2004.
  * [6] Samir Shah and Arun Ross.  Generating synthetic irises by feature agglomeration.  In 2006 international conference on image processing, pages 317–320. IEEE, 2006.
  * [7] Jinyu Zuo, Natalia A Schmid, and Xiaohan Chen.  On generation and analysis of synthetic iris images.  IEEE Transactions on Information Forensics and Security, 2(1):77–90, 2007.
  * [8] Zhuoshi Wei, Tieniu Tan, and Zhenan Sun.  Synthesis of large realistic iris databases using patch-based sampling.  In 2008 19th International Conference on Pattern Recognition, pages 1–4. IEEE, 2008.
  * [9] Shreyas Venugopalan and Marios Savvides.  How to generate spoofed irises from an iris code template.  IEEE Transactions on Information Forensics and Security, 6(2):385–395, 2011.
  * [10] Javier Galbally, Arun Ross, Marta Gomez-Barrero, Julian Fierrez, and Javier Ortega-Garcia.  Iris image reconstruction from binary templates: An efficient probabilistic approach based on genetic algorithms.  Computer Vision and Image Understanding, 117(10):1512–1525, 2013\.
  * [11] Javier Galbally, Marios Savvides, Shreyas Venugopalan, and Arun A Ross.  Iris image reconstruction from binary templates.  Handbook of Iris Recognition, pages 469–496, 2016.
  * [12] Lakin Wecker, Faramarz Samavati, and Marina Gavrilova.  Iris synthesis: a reverse subdivision application.  In Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, pages 121–125, 2005.
  * [13] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [14] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [15] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [16] Shervin Minaee and Amirali Abdolrashidi.  Iris-gan: Learning to generate realistic iris images using convolutional gan.  arXiv preprint arXiv:1812.04822, 2018.
  * [17] Naman Kohli, Daksha Yadav, Mayank Vatsa, Richa Singh, and Afzel Noore.  Synthetic iris presentation attack using idcgan.  In 2017 IEEE International Joint Conference on Biometrics (IJCB), pages 674–680. IEEE, 2017.
  * [18] Shivangi Yadav, Cunjian Chen, and Arun Ross.  Synthesizing iris images using rasgan with application in presentation attack detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  * [19] Shivangi Yadav and Arun Ross.  Cit-gan: Cyclic image translation generative adversarial network with application in iris presentation attack detection.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2412–2421, 2021.
  * [20] Vijay Kakani, Cheng-Bin Jin, and Hakil Kim.  Segmentation-based id preserving iris synthesis using generative adversarial networks.  Multimedia Tools and Applications, pages 1–29, 2023.
  * [21] Jose Maureira, Juan Tapia, Claudia Arellano, and Christoph Busch.  Synthetic periocular iris pai from a small set of near-infrared-images.  arXiv preprint arXiv:2107.12014, 2021.
  * [22] Chen Wang, Zhaofeng He, Caiyong Wang, and Qing Tian.  Generating intra-and inter-class iris images by identity contrast.  In 2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1–7. IEEE, 2022.
  * [23] Mateusz Trokielewicz, Adam Czajka, and Piotr Maciejewicz.  Post-mortem iris recognition with deep-learning-based image segmentation.  Image and Vision Computing, 94:103866, 2020.
  * [24] Adam Czajka, Dennis J. Chute, Arun Ross, Patrick J. Flynn, and Kevin W. Bowyer.  Software tool and methodology for enhancement of unidentified decedent systems with post-mortem automatic iris recognition.  New York, 2019-2021. Inter-university Consortium for Political and Social Research [distributor], 2023.
  * [25] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  Advances in neural information processing systems, 33:12104–12114, 2020.
  * [26] Adam Czajka, Daniel Moreira, Kevin Bowyer, and Patrick Flynn.  Domain-specific human-inspired binarized statistical image features for iris recognition.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 959–967. IEEE, 2019.
  * [27] Juho Kannala and Esa Rahtu.  Bsif: Binarized statistical image features.  In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pages 1363–1366. IEEE, 2012.
  * [28] John Daugman.  New methods in iris recognition.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 37(5):1167–1175, 2007.

Forensic Iris Image Synthesis
Supplementary materials: PMI-specific comparison score distributions

Fig. 7 shows genuine score distributions obtained for authentic post-mortem
samples and synthesized post-mortem samples for different sizes of the
hypersphere (defined by radius $\varepsilon_{\max}$, shown in parentheses)
defining the same-identity manifold in the StyleGAN’s latent space
$\mathbf{w}$, broken by the PMI range (StyleGAN condition). The plots suggest
that setting different $\varepsilon_{\max}$ for different PMI ranges may be
beneficial.

(a) PMI: 0–24 hours

(b) PMI: 25–48 hours

(c) PMI: 49–72 hours

(d) PMI: 73–96 hours

(e) PMI: 97–120 hours

(f) PMI: 121–144 hours

(g) PMI: 145–168 hours

(h) PMI: 169–192 hours

(i) PMI: 193–216 hours

(j) PMI: 217–240 hours

(k) PMI: 241–264 hours

(l) PMI: 265–288 hours

(m) PMI: 289–312 hours

(n) PMI: 313–336 hours

(o) PMI: 337–360 hours

(p) PMI: 361–384 hours

(q) PMI: 385–408 hours

(r) PMI: 409–1674 hours

Figure 7: Same as in Fig. 2, except for breaking the plots by the PMI range.