# Online Loss Function Learning

Christian Raymond  Qi Chen  Bing Xue  Mengjie Zhang

###### Abstract

Loss function learning is a new meta-learning paradigm that aims to automate
the essential task of designing a loss function for a machine learning model.
Existing techniques for loss function learning have shown promising results,
often improving a model’s training dynamics and final inference performance.
However, a significant limitation of these techniques is that the loss
functions are meta-learned in an offline fashion, where the meta-objective
only considers the very first few steps of training, which is a significantly
shorter time horizon than the one typically used for training deep neural
networks. This causes significant bias towards loss functions that perform
well at the very start of training but perform poorly at the end of training.
To address this issue we propose a new loss function learning technique for
adaptively updating the loss function online after each update to the base
model parameters. The experimental results show that our proposed method
consistently outperforms the cross-entropy loss and offline loss function
learning techniques on a diverse range of neural network architectures and
datasets.

Machine Learning, ICML

##  1 Introduction

Figure 1: An example of an adaptive meta-learned loss function generated by
AdaLFL on the CIFAR-10 dataset, where color represents the current gradient
step.

When applying deep neural networks to a given learning task, a significant
amount of time is typically allocated towards performing manual tuning of the
hyper-parameters to achieve competitive learning performances (Bengio, 2012).
Selection of the appropriate hyper-parameters is critical for embedding the
relevant inductive biases into the learning algorithm (Gordon & Desjardins,
1995). The inductive biases control both the set of searchable models and the
learning rules used to find the final model parameters. Therefore, the field
of meta-learning (Schmidhuber, 1987; Vanschoren, 2018; Peng, 2020; Hospedales
et al., 2022), as well as the closely related field of hyper-parameter
optimization (Bergstra et al., 2011; Feurer & Hutter, 2019), aim to automate
the design and selection of a suitable set of inductive biases (or a subset of
them) and have been long-standing areas of interest to the machine learning
community.

One component that has only very recently been receiving attention in the
meta-learning context is the loss function. The loss function (Wang et al.,
2022) is one of the most central components of any gradient-based supervised
learning system, as it determines the base learning algorithm’s learning path
and the selection of the final model (Reed & MarksII, 1999). Furthermore, in
deep learning, neural networks are typically trained through the
backpropagation of gradients that originate from the loss function (Rumelhart
et al., 1986; Goodfellow et al., 2016). Given this importance, a new and
emerging subfield of meta-learning referred to as Loss Function Learning
(Gonzalez & Miikkulainen, 2020; Bechtle et al., 2021; Raymond et al., 2022;
Collet et al., 2022) aims to attempt the difficult task of inferring a highly
performant loss function directly from the given data.

Loss function learning aims to meta-learn a specialized task-specific loss
function, which yields improved performance capabilities when utilized in
training compared to handcrafted loss functions on one or many related tasks,
i.e., a task distribution (Hospedales et al., 2022). Initial approaches to
loss function learning have shown promise at enhancing various aspects of deep
neural network training, such as improving the convergence and sample
efficiency (Gonzalez & Miikkulainen, 2020; Bechtle et al., 2021), as well as
the generalization (Gonzalez & Miikkulainen, 2021; Liu et al., 2020; Li et
al., 2022; Leng et al., 2022), and model robustness (Gao et al., 2021, 2022).
However, one prevailing limitation of the existing approaches to loss function
learning is that they have thus far exclusively focused on learning a loss
function in the offline meta-learning settings.

In offline loss function learning, training is prototypically partitioned into
two phases. In the first phase, the base loss function is meta-learned via
iteratively updating the loss function by performing one or a few base
training steps to approximate the performance. Second, the base model is
trained using the learned loss function, which is now fixed, and is used in
place of the conventional handcrafted loss function. Unfortunately, this
methodology is prone to a severe short-horizon bias (Wu et al., 2018) towards
loss functions which are performant in the early stages of training but often
have poor performance in the later stages.

To address the limitation of offline loss function learning, we propose a new
technique for online loss function learning called Adaptive Loss Function
Learning (AdaLFL). In the proposed technique, the learned loss function is
represented as a small feed-forward neural network that is trained
simultaneously with the base learning model. Unlike prior methods, AdaLFL can
adaptively transform both the shape and scale of the loss function throughout
the learning process to adapt to what is required at each stage of the
learning process, as shown in Figure 1. In offline loss function learning, the
central goal is to improve the performance of a model by specializing the loss
function to a small set of related tasks. Online loss function learning
naturally extends this general philosophy to instead specialize the loss
function to each individual gradient step on a single task.

###  1.1 Contributions

  * •

We propose a method for efficiently learning adaptive loss functions via
online meta-learning by utilizing online unrolled differentiation to update
the meta-learned loss function after each update to the base model.

  * •

We address shortcomings in the design of neural network-based loss function
parameterizations, which previously caused learned loss functions to be biased
toward overly flat shapes resulting in poor training dynamics.

  * •

Empirically, we demonstrate that models trained with our method has enhanced
convergence capabilities and inference performance compared to handcrafted
loss functions and offline loss function learning methods.

  * •

Finally, we analyze the meta-learned loss functions, highlighting several key
trends to explore why our adaptive meta-learned loss functions are so
performant in contrast to traditional handcrafted loss functions.

##  2 Online Loss Function Learning

In this work, we aim to automate the design and selection of the loss function
and improve upon the performance of supervised machine learning systems. This
is achieved via meta-learning an adaptive loss function that transforms both
its shape and scale throughout the learning process. To achieve this, we
propose Adaptive Loss Function Learning (AdaLFL), an efficient task and model-
agnostic approach for online adaptation of the base loss function.

###  2.1 Problem Setup

In a prototypical supervised learning setup, we are given a set of $N$
independently and identically distributed (i.i.d.) examples of form
$\mathcal{D}=\\{(x_{1},y_{1}),\dots,(x_{N},y_{N})\\}$, where $x_{i}\in X$ is
the $i$th instance’s feature vector and $y_{i}\in Y$ is its corresponding
class label. We want to learn a mapping between $X$ and $Y$ using some base
learning model, e.g., a classifier or regressor,
$f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$, where $\theta$ is the base
model parameters. In this paper, similar to others (Finn et al., 2017; Bechtle
et al., 2021; Raymond et al., 2022), we constrain the selection of the base
models to those amenable to a stochastic gradient descent (SGD) style training
procedures such that optimization of model parameters $\theta$ occurs via
optimizing some task-specific loss function $\mathcal{L}_{\mathcal{T}}$ as
follows:

$\theta_{t+1}=\theta_{t}-\alpha\nabla_{\theta_{t}}\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{t}}(x))$
(1)

where $\mathcal{L}_{\mathcal{T}}$ is a handcrafted loss function, typically
the cross entropy between the predicted label and the ground truth label in
classification or the squared error in regression. The principal goal of
AdaLFL is to replace this conventional handcrafted loss function
$\mathcal{L}_{\mathcal{T}}$ with a meta-learned adaptive loss function
$\mathcal{M}_{\phi}$, where the meta-parameters $\phi$ are learned
simultaneously with the base parameters $\theta$, allowing for online
adaptation of the loss function. We formulate the task of learning $\phi$ and
$\theta$ as a non-stationary bilevel optimization problem, where $t$ is the
current time step

$\displaystyle\begin{split}\phi_{t+1}&=\operatorname*{arg\,min}_{\phi}\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{t+1}}(x))\\\
s.t.\;\;\;\;\;\theta_{t+1}(\phi_{t})&=\operatorname*{arg\,min}_{\theta}\mathcal{M}_{\phi_{t}}(y,f_{\theta_{t}}(x)).\end{split}$
(2)

The outer optimization problem aims to meta-learn a performant loss function
$\mathcal{M}_{\phi}$ that minimizes the error on the given task. The inner
optimization problem directly minimizes the learned loss value produced by
$\mathcal{M}_{\phi}$ to learn the base model parameters $\theta$.

###  2.2 Loss Function Representation

Algorithm 1 Loss Function Initialization (Offline)

Input: $\mathcal{L}_{\mathcal{T}}\leftarrow$ Task loss function (meta-
objective)

1: $\mathcal{M}_{\phi_{0}}\leftarrow$ Initialize parameters of meta learner

2: for $i\in\\{0,...,\mathcal{S}_{init}\\}$ do

3: $\theta_{0}\leftarrow$ Reset parameters of base learner

4: for $j\in\\{0,...,\mathcal{S}_{inner}\\}$ do

5: $X$, $y$ $\leftarrow$ Sample from $\mathcal{D}_{train}$

6:
$\mathcal{M}_{learned}\leftarrow\mathcal{M}_{\phi_{i}}(y,f_{\theta_{j}}(X))$

7:
$\theta_{j+1}\leftarrow\theta_{j}-\alpha\nabla_{\theta_{j}}\mathcal{M}_{learned}$

8: end for

9: $X$, $y$ $\leftarrow$ Sample from $\mathcal{D}_{valid}$

10:
$\mathcal{L}_{task}\leftarrow\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{j+1}}(X))$

11: $\phi_{i+1}\leftarrow\phi_{i}-\eta\nabla_{\phi_{i}}\mathcal{L}_{task}$

12: end for

In AdaLFL, the choice of loss function parameterization is a small feedforward
neural network, which is chosen due to its high expressiveness and design
flexibility. Our meta-learned loss function parameterization inspired by
(Bechtle et al., 2021) is a small feedforward neural network denoted by
$\ell_{\phi}$ with two hidden layers and 40 hidden units each, which is
applied class/output-wise.

$\textstyle\mathcal{M}_{\phi}(y,f_{\theta}(x))=\frac{1}{\mathcal{C}}\sum_{i=0}^{\mathcal{C}}\ell_{\phi}(y_{i},f_{\theta}(x)_{i})$
(3)

Crucially, key design decisions are made regarding the activation functions
used in $\ell_{\phi}$ to enforce desirable behavior. In (Bechtle et al.,
2021), ReLU activations are used in the hidden layers, and the smooth Softplus
activation is used in the output layer to constrain the loss to be non-
negative, i.e., $\ell{\phi}:\mathbb{R}^{2}\rightarrow\mathbb{R}_{0}^{+}$.
Unfortunately, this network architecture is prone to unintentionally
encouraging overly flat loss functions, see Appendix A.1. Generally, flat
regions in the loss function are very detrimental to training as uniform loss
is given to non-uniform errors. Removal of the Softplus activation in the
output can partially resolve this flatness issue; however, without it, the
learned loss functions would violate the typical constraint that a loss
function should be at least $\mathcal{C}^{1}$, i.e., continuous in the zeroth
and first derivatives.

Alternative smooth activations, such as Sigmoid, TanH, SoftPlus, ELU, etc.,
can be used in the hidden layers instead; however, due to their range-bounded
limits, they are also prone to encouraging loss functions that have large flat
regions when their activations saturate. Therefore, to inhibit this behavior,
the unbounded leaky ReLU (Maas et al., 2013) is combined with the smooth ReLU,
i.e., SoftPlus (Dugas et al., 2000), as follows:

$\varphi_{hidden}(x)=\tfrac{1}{\beta}\log(e^{\beta x}+1)\cdot(1-\gamma)+\gamma
x$ (4)

This smooth leaky ReLU activation function with leak parameter $\gamma$ and
smoothness parameter $\beta$ has desirable characteristics for representing a
loss function. It is smooth and has linear asymptotic behavior necessary for
tasks such as regression, where extrapolation of the learned loss function can
often occur. Furthermore, as its output is not bounded, it does not encourage
flatness in the learned loss function. See Appendix A.2 for more details.

###  2.3 Loss Function Initialization

One challenge for online loss function learning is achieving a stable and
performant initial set of parameters for the learned loss function. If $\phi$
is initialized poorly, too much time is spent on fixing $\phi$ in the early
stages of the learning process, resulting in poor base convergence, or in the
worst case, $f_{\theta}$ to diverge. To address this, offline loss function
learning using Meta-learning via Learned Loss (ML3) (Bechtle et al., 2021) is
utilized to fine-tune the initial loss function to the base model prior to
performing online learning. The initialization process is summarized in
Algorithm 1, where $\mathcal{S}_{init}=2500$. In AdaLFL’s initialization
process one step on $\theta$ is taken for each step in $\phi$, i.e., inner
gradient steps $\mathcal{S}_{inner}=1$. However, if $\mathcal{S}_{inner}<1$,
implicit gradients (Lorraine et al., 2020; Gao et al., 2022) can instead be
utilized to reduce the initialization process’s memory footprint and
computational overhead.

###  2.4 Online Meta-Optimization

Algorithm 2 Loss Function Adaptation (Online)

Input: $\mathcal{M}_{\phi}\leftarrow$ Learned loss function (base-objective)

Input: $\mathcal{L}_{\mathcal{T}}\leftarrow$ Task loss function (meta-
objective)

1: $\theta_{0}\leftarrow$ Initialize parameters of base learner

2: for $i\in\\{0,...,\mathcal{S}_{train}\\}$ do

3: $X$, $y$ $\leftarrow$ Sample from $\mathcal{D}_{train}$

4:
$\mathcal{M}_{learned}\leftarrow\mathcal{M}_{\phi_{i}}(y,f_{\theta_{i}}(X))$

5:
$\theta_{i+1}\leftarrow\theta_{i}-\alpha\nabla_{\theta_{i}}\mathcal{M}_{learned}$

6: $X$, $y$ $\leftarrow$ Sample from $\mathcal{D}_{valid}$

7:
$\mathcal{L}_{task}\leftarrow\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{i+1}}(X))$

8: $\phi_{i+1}\leftarrow\phi_{i}-\eta\nabla_{\phi_{i}}\mathcal{L}_{task}$

9: end for

Figure 2: Computational graph of AdaLFL, where $\theta$ is updated using
$\mathcal{M}_{\phi}$ in the inner loop (Base Update). The optimization path is
tracked in the computational graph and then used to update $\phi$ based on the
meta-objective in the
outer loop (Meta Update). The dashed lines show the gradients for $\theta$ and
$\phi$ with respect to their given objectives.

To optimize $\phi$, unrolled differentiation is utilized in the outer loop to
update the learned loss function after each update to the base model
parameters $\theta$ in the inner loop, which occurs via vanilla
backpropagation. This is conceptually the simplest way to optimise $\phi$ as
all the intermediate iterates generated by the optimizer in the inner loop can
be stored and then backpropagate through in the outer loop (Maclaurin et al.,
2015). The full iterative learning process is summarized in Algorithm 2 and
proceeds as follows: perform a forward pass $f_{\theta_{t}}(x)$ to obtain an
initial set of predictions. The learned loss function $\mathcal{M}_{\phi}$ is
then used to produce a base loss value

$\mathcal{M}_{learned}=\mathcal{M}_{\phi_{t}}(y,f_{\theta_{t}}(x)).$ (5)

Using $\mathcal{M}_{learned}$, the current weights $\theta_{t}$ are updated by
taking a step in the opposite direction of the gradient of the loss with
respect to $\theta_{t}$, where $\alpha$ is the base learning rate.

$\begin{split}\theta_{t+1}&=\theta_{t}-\alpha\nabla_{\theta_{t}}\mathcal{M}_{\phi_{t}}(y,f_{\theta_{t}}(x))\\\
&=\theta_{t}-\alpha\nabla_{\theta_{t}}\mathbb{E}_{X,y}\big{[}\mathcal{M}_{\phi_{t}}(y,f_{\theta_{t}}(x))\big{]}\end{split}$
(6)

which can be further decomposed via the chain rule as shown in Equation (7).
Importantly, all the intermediate iterates generated by the (base) optimizer
at the $t^{th}$ time-step when updating $\theta$ are stored in memory.

$\theta_{t+1}=\theta_{t}-\alpha\nabla_{f}\mathcal{M}_{\phi_{t}}(y,f_{\theta_{t}}(x))\nabla_{\theta_{t}}f_{\theta_{t}}(x)$
(7)

$\phi_{t}$ can now be updated to $\phi_{t+1}$ based on the learning
progression made by $\theta$. Using $\theta_{t+1}$ as a function of
$\phi_{t}$, compute a forward pass using the updated base weights
$f_{\theta_{t+1}}(x)$ to obtain a new set of predictions. The instances can
either be sampled from the training set or a held-out validation set. The new
set of predictions is used to compute the task loss
$\mathcal{L}_{\mathcal{T}}$ to optimize $\phi_{t}$ through $\theta_{t+1}$

$\mathcal{L}_{task}=\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{t+1}}(x))$ (8)

where $\mathcal{L}_{\mathcal{T}}$ is selected based on the respective
application. For example, the squared error loss for the task of regression or
the cross-entropy loss for classification. The task loss is a crucial
component for embedding the end goal task into the learned loss function.
Optimization of the current meta-loss network loss weights $\phi_{t}$ now
occurs by taking the gradient of $\mathcal{L}_{\mathcal{T}}$, where $\eta$ is
the meta learning rate.

$\begin{split}\phi_{t+1}&=\phi_{t}-\eta\nabla_{\phi_{t}}\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{t+1}}(x))\\\
&=\phi_{t}-\eta\nabla_{\phi_{t}}\mathbb{E}_{X,y}\big{[}\mathcal{L}_{\mathcal{T}}(y,f_{\theta_{t+1}}(x))\big{]}\end{split}$
(9)

where the gradient computation is decomposed by applying the chain rule as
shown in Equation (10) where the gradient with respect to the meta-loss
network weights $\phi_{t}$ requires the updated model parameters
$\theta_{t+1}$ from Equation (6).

$\phi_{t+1}=\phi_{t}-\eta\nabla_{f}\mathcal{L}_{\mathcal{T}}\nabla_{\theta_{t+1}}f_{\theta_{t+1}}\nabla_{\phi_{t}}\theta_{t+1}$
(10)

This process is repeated for a fixed number of gradient steps $S_{train}$,
which is identical to what would typically be used for training $f_{\theta}$.
An overview and summary of the full associated data flow between the inner and
outer optimization of $\theta$ and $\phi$, respectively, is given in Figure 2.

###  2.5 Implicit Tuning of Learning Rate Schedule

In offline loss function learning, it is known from (Gonzalez & Miikkulainen,
2021; Raymond et al., 2022) that there is implicit initial learning rate
tuning of $\alpha$ when meta-learning a loss function since

$\exists\alpha\exists\phi:\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}}\approx\theta-\nabla_{\theta}\mathcal{M}_{\phi}.$
(11)

Consequently, an emergent behavior, unique to online loss function learning,
is that the adaptive loss function generated by AdaLFL implicitly embodies
multiple different learning rates throughout the learning process hence often
causing a fine-tuning of the fixed learning rate or of a predetermined
learning rate schedule.

##  3 Related Work

The method that we propose in this paper addresses the general problem of
meta-learning a (base) loss function, i.e. loss function learning. Existing
loss function learning methods can be categorized along two key axes, loss
function representation and meta-optimization. Frequently used representations
in loss function learning include parametric (Gonzalez & Miikkulainen, 2020;
Raymond et al., 2022) and nonparametric (Liu et al., 2020; Li et al., 2022)
genetic programming expression trees. In addition to this, alternative
representations such as truncated Taylor polynomials (Gonzalez & Miikkulainen,
2021; Gao et al., 2021, 2022) and small feed-forward neural networks (Bechtle
et al., 2021) have also been recently explored. Regarding meta-optimization,
loss function learning methods have heavily utilized computationally expensive
evolution-based methods such as evolutionary algorithms (Koza et al., 1994)
and evolutionary strategies (Hansen & Ostermeier, 2001). While more recent
approaches have made use of gradient-based approaches unrolled differentiation
(Maclaurin et al., 2015), and implicit differentiation (Lorraine et al.,
2020).

A common trait among these methods is that, in contrast to AdaLFL, they
perform offline loss function learning, resulting in a severe short-horizon
bias and sub-optimal performance at the end of training. This short-horizon
bias arises from how the various approaches compute their respective meta-
objectives. In offline evolution-based approaches, the fitness, i.e., meta-
objective, is typically calculated by computing the performance at the end of
a partial training session, e.g., $\leq 1000$ gradient steps (Gonzalez &
Miikkulainen, 2021; Raymond et al., 2022). A truncated number of gradient
steps are required to be used as evolution-based methods have to evaluate the
performance of a large number of candidate solutions, typically $L$ loss
function over $K$ iterations/generations, where $25\leq L,K\leq 100$.
Therefore, performing full training sessions, which can be hundreds of
thousands or even millions of gradient steps for each candidate solution, is
infeasible.

Regarding the existing gradient-based approaches, offline unrolled
optimization requires the whole optimization path to be stored in memory; in
practice, this significantly restricts the number of inner gradient steps
before computing the meta-objective to only a small number of steps. Methods
such as implicit differentiation can obviate these memory issues; however, it
would still require a full training session in the inner loop, which is a
prohibitive number of forward passes to perform in tractable time.
Furthermore, the dependence of the model-parameters on the meta-parameters
increasingly shrinks and eventually vanishes as the number of steps increases
(Rajeswaran et al., 2019).

###  3.1 Online vs Offline Loss Function Learning

The key algorithmic difference of AdaLFL from prior offline gradient-based
methods (Bechtle et al., 2021; Gao et al., 2022) is that $\phi$ is updated
after each update to $\theta$ in lockstep in a single phase as opposed to
learning $\theta$ and $\phi$ in separate phases. This is achieved by not
resetting $\theta$ after each update to $\phi$ (Algorithm 1, line 3), and
consequently, $\phi$ has to adapt to each newly updated timestep such that
$\phi=(\phi_{0},\phi_{1},\dots,\phi_{S_{train}})$. In offline loss function
learning, $\phi$ is learned separately at meta-training time and then is fixed
for the full duration of the meta-testing phase where $\theta$ is learned and
$\phi=(\phi_{0})$. Another crucial difference is that in online loss function
learning, there is implicit tuning of the learning rate schedule, as mentioned
in Section 2.5.

###  3.2 Alternative Paradigms

Although online loss function learning has not been explored in the meta-
learning context, some existing research outside the subfield has previously
explored the possibility of adaptive loss functions, such as in (Li et al.,
2019) and (Wang et al., 2020). However, we emphasize that these approaches are
categorically different in that they do not learn the loss function from
scratch; instead, they interpolate between a small subset of handcrafted loss
functions, updating the loss function after each epoch. Furthermore, in
contrast to loss function learning which is both task and model-agnostic,
these techniques are restricted to being task-specific, e.g., face recognition
only. Finally, this class of approaches does not implicitly tune the base
learning rate $\alpha$, as is the case in loss function learning.

##  4 Experimental Evaluation

In this section, the experimental setup for evaluating AdaLFL is presented. In
summary, experiments are conducted across four open-access datasets and
multiple well-established network architectures. The performance of the
proposed method is contrasted against the handcrafted cross-entropy loss and
AdaLFL’s offline counterpart ML3 Supervised (Bechtle et al., 2021). The
experiments were implemented in PyTorch (Paszke et al., 2017), and Higher
(Grefenstette et al., 2019), and the code for reproducing the experiments can
be found at github.com/*redacted*.

###  4.1 Benchmark Tasks

Following the established literature on loss function learning (Gonzalez &
Miikkulainen, 2021; Bechtle et al., 2021; Raymond et al., 2022), MNIST (LeCun
et al., 1998) is initially used as a simple domain to illustrate the
capabilities of the proposed method. Following this, the more challenging
tasks of CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), and SVHN (Netzer et
al., 2011), are employed to assess the performance of AdaLFL to determine
whether the results can generalize to larger, more challenging tasks.

The original training-testing partitioning is used for all four datasets, with
10% of the training instances allocated for validation. In addition, standard
data augmentation techniques consisting of normalization, random horizontal
flips, and cropping are applied to the training data of CIFAR-10, CIFAR-100,
and SVHN during meta and base training.

###  4.2 Benchmark Models

A diverse set of commonly used and well-established benchmark architectures
are utilized to evaluate the performance of AdaLFL. For MNIST, logistic
regression (McCullagh et al., 1989), a simple two hidden layer multi-layer
perceptron (MLP) taken from (Baydin et al., 2018), and the LeNet-5 (LeCun et
al., 1998) architecture is used. Following this experiments are conducted on
CIFAR-10, VGG-16 (Simonyan et al., 2014), AllCNN-C (Springenberg et al.,
2014), ResNet-18 (He et al., 2016), and SqueezeNet (Iandola et al., 2016) are
used. For the remaining datasets, CIFAR-100 and SVHN, WideResNet 28-10 and
WideResNet 16-8 (Zagoruyko et al., 2016) is employed, respectively.

(a) MNIST + Logistic (b) MNIST + MLP (c) MNIST + LeNet-5 (d) CIFAR-10 + VGG-16
(e) CIFAR-10 + AllCNN-C (f) CIFAR-10 + ResNet-18 (g) CIFAR-10 + SqueezeNet (h)
CIFAR-100 + WRN 28-10 (i) SVHN + WRN 16-8

Figure 3: Mean learning curves across 10 independent executions of each
algorithm on each task + model pair, showing the training error rate (y-axis)
against gradient steps (x-axis). Best viewed in color.

##  5 Results and Analysis

The results in Figure 3 show the average training learning curves of AdaLFL
compared with the baseline cross-entropy loss and ML3 across 10 executions of
each method on each dataset + model pair. The results show that AdaLFL makes
clear and consistent gains in convergence speed compared to the baseline and
offline loss function learning method ML3, except on CIFAR-100 where there was
difficulty in achieving a stable initialization. Furthermore, the errors
obtained by AdaLFL at the end of training are typically better (lower) than
both of the compared methods, suggesting that performance gains are being made
in addition to enhanced convergence and training speeds.

Another key observation is that AdaLFL improves upon the performance of the
baseline on the more challenging tasks of CIFAR-10, CIFAR-100, and SVHN, where
offline loss functions learning method ML3 consistently performs poorly.
Improved performance on these datasets is achieved via AdaLFL adaptively
updating the learned loss function throughout the learning process to the
changes in the training dynamics. This is in contrast to ML3, where the loss
function remains static, resulting in poor performance on tasks where the
training dynamics at the beginning of training vary significantly from those
at the end of training.

###  5.1 Final Inference Testing Performance

Table 1: Results reporting the mean $\pm$ standard deviation of final
inference testing error rates across 10 independent
executions of each algorithm on each task + model pair (using no base learning rate scheduler). Task | Model | Baseline | ML3 (Offline) | AdaLFL (Online)
---|---|---|---|---
MNIST | Logistic (McCullagh et al., 1989) | 0.0766$\pm$0.0009 | 0.0710$\pm$0.0010 | 0.0697$\pm$0.0010
| MLP (Baydin et al., 2018) | 0.0203$\pm$0.0006 | 0.0185$\pm$0.0004 | 0.0184$\pm$0.0006
| LeNet-5 (LeCun et al., 1998) | 0.0125$\pm$0.0007 | 0.0094$\pm$0.0005 | 0.0091$\pm$0.0004
CIFAR-10 | VGG-16 (Simonyan et al., 2014) | 0.1036$\pm$0.0049 | 0.1027$\pm$0.0062 | 0.0903$\pm$0.0032
| AllCNN-C (Springenberg et al., 2014) | 0.1030$\pm$0.0062 | 0.1015$\pm$0.0055 | 0.0835$\pm$0.0050
| ResNet-18 (He et al., 2016) | 0.0871$\pm$0.0057 | 0.0883$\pm$0.0041 | 0.0788$\pm$0.0035
| SqueezeNet (Iandola et al., 2016) | 0.1226$\pm$0.0080 | 0.1162$\pm$0.0052 | 0.1083$\pm$0.0049
CIFAR-100 | WRN 28-10 (Zagoruyko et al., 2016) | 0.3046$\pm$0.0087 | 0.3108$\pm$0.0075 | 0.2668$\pm$0.0283
SVHN | WRN 16-8 (Zagoruyko et al., 2016) | 0.0512$\pm$0.0043 | 0.0500$\pm$0.0034 | 0.0441$\pm$0.0014

The corresponding final inference testing results reporting the average error
rate across 10 independent executions of each method are shown in Table 1. The
results show that AdaLFL’s meta-learned loss functions produce superior
inference performance when used in training compared to the baseline on all
the tested problems. A further observation is that the gains achieved by
AdaLFL are consistent and stable. Notably, in most cases, lower variability
than the baseline is observed, as shown by the relatively small standard
deviation in error rate across the independent runs.

Contrasting the performance of AdaLFL to ML3, similar performance is obtained
on the MNIST experiments, suggesting that the training dynamics at the
beginning of training are similar to those at the end; hence the modest
difference in results. While on the more challenging tasks of CIFAR-10,
CIFAR-100, and SVHN, AdaLFL produced significantly better results than ML3,
demonstrating the scalability of the newly proposed loss function learning
approach.

The results attained by AdaLFL are are promising given that the base models
tested were designed and optimized around the cross-entropy loss. We
hypothesize that larger performance gains may be attained using networks
designed specifically around meta-learned loss function, similar to the
results shown in (Kim et al., 2018; Elsken et al., 2020; Ding et al., 2022).
Thus future work will explore learning the loss function in tandem with the
network architecture.

###  5.2 Inner Gradient Steps

Table 2: Results reporting the mean $\pm$ standard deviation
of testing error rates when using an increasing number of inner gradient steps $\mathcal{S}_{inner}$ with ML3. Method | CIFAR-10 + AllCNN-C
---|---
ML3 ($\mathcal{S}_{inner}=1$) | 0.1015$\pm$0.0055
ML3 ($\mathcal{S}_{inner}=5$) | 0.0978$\pm$0.0052
ML3 ($\mathcal{S}_{inner}=10$) | 0.0985$\pm$0.0050
ML3 ($\mathcal{S}_{inner}=15$) | 0.0989$\pm$0.0049
ML3 ($\mathcal{S}_{inner}=20$) | 0.0974$\pm$0.0061
AdaLFL (Online) | 0.0835$\pm$0.0050

In ML3, (Bechtle et al., 2021) suggested taking only one inner step, i.e.,
setting $\mathcal{S}_{inner}=1$ in Algorithm 1. A reasonable question to ask
is whether increasing the number of inner steps to extend the horizon of the
meta-objective past the first step will reduce the disparity in performance
between ML3 and AdaLFL. To answer this question, experiments are performed on
CIFAR-10 AllCNN-C with ML3 setting $\mathcal{S}_{inner}=\\{1,5,10,15,20\\}$.
The results reported in Table 2 show that increasing the number of inner steps
in ML3 up to the limit of what is feasible in memory on a consumer GPU does
not resolve the short horizon bias present in offline loss function learning.
Furthermore, the results show that increasing the number of inner steps only
results in marginal improvements in the performance over
$\mathcal{S}_{inner}=1$. Hence, offline loss function learning methods that
seek to obviate the memory issues of unrolled differentiation to allow for an
increased number of inner steps, such as (Gao et al., 2022), which uses
implicit differentiation, are still prone to a kind of short-horizon bias.

###  5.3 Run-time Analysis

Table 3: Average run-time of the entire learning process for each benchmark
method. Each algorithm is run on a single Nvidia RTX A5000, and results are
reported in hours.

Task and Model | Baseline | Offline | Online
---|---|---|---
MNIST + Logistic | 0.06 | 0.31 | 0.55
MNIST + MLP | 0.06 | 0.32 | 0.56
MNIST + LeNet-5 | 0.10 | 0.38 | 0.67
CIFAR-10 + VGG-16 | 1.50 | 1.85 | 5.56
CIFAR-10 + AllCNN-C | 1.41 | 1.72 | 5.53
CIFAR-10 + ResNet-18 | 1.81 | 2.18 | 8.38
CIFAR-10 + SqueezeNet | 1.72 | 2.02 | 7.88
CIFAR-100 + WRN 28-10 | 8.81 | 10.3 | 50.49
SVHN + WRN 16-8 | 7.32 | 7.61 | 24.75

Figure 4: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where
each row represents
a loss function, and the color represents the current gradient step.

The average run-time of the entire learning process of all benchmark methods
on all tasks is reported in Table 3. Notably, there are two key reasons why
the computational overhead of AdaLFL is not as bad as it may at first seem.
First, the time reported for the baseline does not include the implicit cost
of manual hyper-parameter selection and tuning of the loss function, as well
as the initial learning rate and learning rate schedule, which is needed prior
to training in order to attain reasonable performance (Goodfellow et al.,
2016). Second, a large proportion of the cost of AdaLFL comes from storing a
large number of intermediate iterates needed for the outer loop. However, the
intermediate iterates stored in this process are identical to those used in
other popular meta-learning paradigms (Andrychowicz et al., 2016; Finn et al.,
2017). Consequently, future work can explore combining AdaLFL with other
optimization-based meta-learning methods with minimal overhead cost, as is the
case in methods such as MetaSGD (Li et al., 2017), where both the initial
parameters and a parameter-wise matrix of learning rate terms are learned
simultaneously.

###  5.4 Visualizing Learned Loss Functions

To better understand why the meta-learned loss functions produced by AdaLFL
are so performant, two of the learned loss functions are highlighted in Figure
4, where the learned loss function is plotted at equispaced intervals
throughout the training. See Appendix C for further examples of the diverse
and creative loss function meta-learned by AdaLFL.

Analyzing the learned loss functions, it can be observed that the loss
functions change significantly in their shape throughout the learning process.
In both cases, the learned loss functions attributed strong penalties for
severe misclassification at the start of the learning process, and than
gradually pivoted to a more moderate or minor penalty as learning progressed.
This behavior enables fast and efficient learning early on, and reduces the
sensitivity of the base model to outliers in the later stages of the learning
process. A further observation is that the scale of the learned loss function
changes, confirming that implicit learning rate tuning, as noted in Section
3.1, is occurring.

###  5.5 Implicit Early Stopping

A unique property observed in the loss functions generated by AdaLFL is that
often once base convergence is achieved the learned loss function will
intentionally start to flatten or take on a parabolic form, see Figures 7 and
12 in Appendix. This is implicitly a type of early stopping, also observed in
related paradigms such as in hypergradient descent (Baydin et al., 2018),
which meta-learns base learning rates. In hypergradient descent the learned
learning rate has previously been observed to oscillate around 0 near the end
of training, at times becoming negative, essentially terminating training.
Implicit early stopping is beneficial as it is known to have a regularizing
effect on model training (Yao et al., 2007); however, if not performed
carefully it can also be detrimental to training due to terminating training
prematurely. Therefore, in future work, we aim to further investigate and
explore regulating this behavior, as a potential avenue for further improving
performance.

##  6 Conclusion

In this work, the first fully online approach to loss function learning is
proposed. The proposed technique, AdaLFL, infers the base loss function
directly from the data and adaptively trains it with the base model parameters
simultaneously using unrolled differentiation. The results showed that models
trained with our method have enhanced convergence capabilities and inference
performance compared with the de facto standard cross-entropy loss and offline
loss function learning method ML3. Further analysis on the learned loss
functions identified common patterns in the shape of the learned loss
function, as well revealed unique emergent behavior present only in adaptively
learned loss functions. Namely, implicit tuning of the learning rate schedule
as well as implicit early stopping. While this work has solely set focus on
meta-learning the loss function in isolation to better understand and analyze
its properties, we believe that further benefits can be realized upon being
combined with existing optimization-based meta-learning techniques.

## References

  * Andrychowicz et al. (2016) Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and De Freitas, N.  Learning to learn by gradient descent by gradient descent.  _Advances in neural information processing systems_ , 29, 2016.
  * Antoniou et al. (2018) Antoniou, A., Edwards, H., and Storkey, A.  How to train your maml.  _arXiv preprint arXiv:1810.09502_ , 2018.
  * Baydin et al. (2018) Baydin, A. G., Cornish, R., Rubio, D. M., Schmidt, M., and Wood, F.  Online learning rate adaptation with hypergradient descent.  In _Sixth International Conference on Learning Representations_ , 2018\.
  * Bechtle et al. (2021) Bechtle, S., Molchanov, A., Chebotar, Y., Grefenstette, E., Righetti, L., Sukhatme, G., and Meier, F.  Meta-learning via learned loss.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pp. 4161–4168. IEEE, 2021.
  * Bengio (2012) Bengio, Y.  Practical recommendations for gradient-based training of deep architectures.  In _Neural networks: Tricks of the trade_ , pp. 437–478. Springer, 2012.
  * Bergstra et al. (2011) Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ , 24, 2011.
  * Collet et al. (2022) Collet, A., Bazco-Nogueras, A., Banchs, A., and Fiore, M.  Loss meta-learning for forecasting, 2022.  URL https://openreview.net/forum?id=rczz7TUKIIB.
  * Ding et al. (2022) Ding, Y., Wu, Y., Huang, C., Tang, S., Yang, Y., Wei, L., Zhuang, Y., and Tian, Q.  Learning to learn by jointly optimizing neural architecture and weights.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 129–138, 2022.
  * Dugas et al. (2000) Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R.  Incorporating second-order functional knowledge for better option pricing.  _Advances in neural information processing systems_ , 13, 2000.
  * Elsken et al. (2020) Elsken, T., Staffler, B., Metzen, J. H., and Hutter, F.  Meta-learning of neural architectures for few-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12365–12375, 2020.
  * Feurer & Hutter (2019) Feurer, M. and Hutter, F.  Hyperparameter optimization.  In _Automated machine learning_ , pp. 3–33. Springer, Cham, 2019\.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pp. 1126–1135. PMLR, 2017.
  * Gao et al. (2021) Gao, B., Gouk, H., and Hospedales, T. M.  Searching for robustness: Loss learning for noisy classification tasks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6670–6679, 2021.
  * Gao et al. (2022) Gao, B., Gouk, H., Yang, Y., and Hospedales, T.  Loss function learning for domain generalization by implicit gradient.  In _International Conference on Machine Learning_ , pp. 7002–7016. PMLR, 2022.
  * Gonzalez & Miikkulainen (2020) Gonzalez, S. and Miikkulainen, R.  Improved training speed, accuracy, and data utilization through loss function optimization.  In _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pp. 1–8. IEEE, 2020.
  * Gonzalez & Miikkulainen (2021) Gonzalez, S. and Miikkulainen, R.  Optimizing loss functions through multi-variate taylor polynomial parameterization.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 305–313, 2021.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., and Courville, A.  _Deep Learning_.  MIT press, 2016.
  * Gordon & Desjardins (1995) Gordon, D. F. and Desjardins, M.  Evaluation and selection of biases in machine learning.  _Machine learning_ , 20(1):5–22, 1995.
  * Grefenstette et al. (2019) Grefenstette, E., Amos, B., Yarats, D., Htut, P. M., Molchanov, A., Meier, F., Kiela, D., Cho, K., and Chintala, S.  Generalized inner loop meta-learning.  _arXiv preprint arXiv:1910.01727_ , 2019.
  * Hansen & Ostermeier (2001) Hansen, N. and Ostermeier, A.  Completely derandomized self-adaptation in evolution strategies.  _Evolutionary computation_ , 9(2):159–195, 2001\.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hospedales et al. (2022) Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A.  Meta-learning in neural networks: A survey.  _IEEE Transactions on Pattern Analysis; Machine Intelligence_ , 44(09):5149–5169, 2022.
  * Iandola et al. (2016) Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size.  _arXiv preprint arXiv:1602.07360_ , 2016.
  * Kim et al. (2018) Kim, J., Lee, S., Kim, S., Cha, M., Lee, J. K., Choi, Y., Choi, Y., Cho, D.-Y., and Kim, J.  Auto-meta: Automated gradient based meta learner search.  _arXiv preprint arXiv:1806.06927_ , 2018.
  * Koza et al. (1994) Koza, J. R. et al.  _Genetic programming II_ , volume 17.  MIT press Cambridge, 1994.
  * Krizhevsky & Hinton (2009) Krizhevsky, A. and Hinton, G.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Leng et al. (2022) Leng, Z., Tan, M., Liu, C., Cubuk, E. D., Shi, X., Cheng, S., and Anguelov, D.  Polyloss: A polynomial expansion perspective of classification loss functions.  _Tenth International Conference on Learning Representations_ , 2022\.
  * Li et al. (2019) Li, C., Yuan, X., Lin, C., Guo, M., Wu, W., Yan, J., and Ouyang, W.  Am-lfs: Automl for loss function search.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8410–8419, 2019.
  * Li et al. (2022) Li, H., Fu, T., Dai, J., Li, H., Huang, G., and Zhu, X.  Autoloss-zero: Searching loss functions from scratch for generic tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1009–1018, 2022.
  * Li et al. (2017) Li, Z., Zhou, F., Chen, F., and Li, H.  Meta-sgd: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_ , 2017.
  * Liu et al. (2020) Liu, P., Zhang, G., Wang, B., Xu, H., Liang, X., Jiang, Y., and Li, Z.  Loss function discovery for object detection via convergence-simulation driven search.  In _ICLR_ , 2020.
  * Lorraine et al. (2020) Lorraine, J., Vicol, P., and Duvenaud, D.  Optimizing millions of hyperparameters by implicit differentiation.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1540–1552. PMLR, 2020.
  * Maas et al. (2013) Maas, A. L., Hannun, A. Y., Ng, A. Y., et al.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30, pp. 3. Atlanta, Georgia, USA, 2013\.
  * Maclaurin et al. (2015) Maclaurin, D., Duvenaud, D., and Adams, R.  Gradient-based hyperparameter optimization through reversible learning.  In _International conference on machine learning_ , pp. 2113–2122. PMLR, 2015.
  * McCullagh et al. (1989) McCullagh, P., Nelder, J. A., et al.  _Generalized linear models_.  Routledge, 1989.
  * Netzer et al. (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Paszke et al. (2017) Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A.  Automatic differentiation in pytorch.  2017\.
  * Peng (2020) Peng, H.  A comprehensive overview and survey of recent advances in meta-learning.  _arXiv preprint arXiv:2004.11149_ , 2020.
  * Rajeswaran et al. (2019) Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S.  Meta-learning with implicit gradients.  _Advances in neural information processing systems_ , 32, 2019.
  * Raymond et al. (2022) Raymond, C., Chen, Q., Xue, B., and Zhang, M.  Learning symbolic model-agnostic loss functions via meta-learning.  _arXiv preprint arXiv:2209.08907_ , 2022.
  * Reed & MarksII (1999) Reed, R. and MarksII, R. J.  _Neural smithing: supervised learning in feedforward artificial neural networks_.  Mit Press, 1999.
  * Rumelhart et al. (1986) Rumelhart, D. E., Hinton, G. E., and Williams, R. J.  Learning representations by back-propagating errors.  _nature_ , 323(6088):533–536, 1986.
  * Schmidhuber (1987) Schmidhuber, J.  _Evolutionary Principles in Self-Referential Learning_.  PhD thesis, Technische Universität München, 1987.
  * Simonyan et al. (2014) Simonyan, K., Zisserman, A., et al.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Springenberg et al. (2014) Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M.  Striving for simplicity: The all convolutional net.  _arXiv preprint arXiv:1412.6806_ , 2014.
  * Vanschoren (2018) Vanschoren, J.  Meta-learning: A survey.  _arXiv preprint arXiv:1810.03548_ , 2018.
  * Wang et al. (2022) Wang, Q., Ma, Y., Zhao, K., and Tian, Y.  A comprehensive survey of loss functions in machine learning.  _Annals of Data Science_ , 9(2):187–212, 2022\.
  * Wang et al. (2020) Wang, X., Wang, S., Chi, C., Zhang, S., and Mei, T.  Loss function search for face recognition.  In _International Conference on Machine Learning_ , pp. 10029–10038. PMLR, 2020.
  * Wu et al. (2018) Wu, Y., Ren, M., Liao, R., and Grosse, R.  Understanding short-horizon bias in stochastic meta-optimization.  _arXiv preprint arXiv:1803.02021_ , 2018.
  * Yao et al. (2007) Yao, Y., Rosasco, L., and Caponnetto, A.  On early stopping in gradient descent learning.  _Constructive Approximation_ , 26(2):289–315, 2007.
  * Zagoruyko et al. (2016) Zagoruyko, S., Komodakis, N., et al.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.

##  Appendix A Loss Function Representation

The representation of the learned loss function under consideration in AdaLFL
is a simple feed-forward neural network. We consider the general case of a
feed-forward neural network with one input layer, $L$ hidden layers, and one
output layer. A hidden layer refers to a feed-forward mapping between two
adjacent layers $h_{\phi^{(l)}}$ such that

$h_{\phi^{(l)}}\big{(}h_{\phi^{(l-1)}}\big{)}=\varphi^{(l)}\big{(}\phi^{(l)\mathsf{T}}h_{\phi^{(}l-1)}\big{)},\forall
l=1,\dots L,$ (12)

where $\varphi(\cdot)^{(l)}$ refers to the element-wise activation function of
the $l^{th}$ layer, and $\phi^{(l)}$ is the matrix of interconnecting weights
between $h_{\phi^{(l-1)}}$ and $h_{\phi^{(l)}}$. For the input layer, the
mapping is defined as $h_{\phi^{(0)}}(y_{i},f_{\theta}(x)_{i})$, and for the
output layer as $h_{\phi^{(out)}}(h_{\phi^{(L)}})$. Subsequently, the meta-
learned loss function $\ell{\phi}$ parameterized by the set of meta-parameters
$\phi=\\{\phi_{0},\dots,\phi_{l},\phi_{out}\\}$ can be defined as a
composition of feed-forward mappings such that

$\ell{\phi}\big{(}y_{i},f_{\theta}(x)_{i}\big{)}=h_{\phi^{(out)}}\Big{(}h_{\phi^{(L)}}\Big{(}\dots\Big{(}h_{\phi^{(0)}}\big{(}y_{i},f_{\theta}(x)_{i}\big{)}\Big{)}\dots\Big{)}\Big{)}$
(13)

which is applied output-wise across the $\mathcal{C}$ output channels of the
ground truth and predicted labels, e.g., applied to each index of the one-hot
encoded class vector in classification, or to each continuous output in
regression. The loss value produced by $\ell{\phi}$ is then summed across the
output channel to reduce the loss vector into its final scalar form.

$\mathcal{M}_{\phi}(y,f_{\theta}(x))=\frac{1}{\mathcal{C}}\sum_{i=0}^{\mathcal{C}}\ell_{\phi}(y_{i},f_{\theta}(x)_{i})$
(14)

###  A.1 Network Architecture

Figure 5: Four example loss functions generated by AdaLFL using the network
architecture proposed in (Bechtle et al., 2021), which uses a softplus
activation in the output layer, causing flattening behavior degrading learning
performance.

The learned loss function used in our experiments has $L=2$ hidden layers and
40 hidden units in each layer, inspired by the network configuration utilized
in Meta-Learning via Learned Loss (ML3 Supervised) (Bechtle et al., 2021). We
found no consistent improvement in performance across our experiments by
increasing or decreasing the number of hidden layers or nodes. However, it was
found that the choice of non-linear activations used in ML3, was highly prone
to encouraging poor-performing loss functions with large flat regions, as
shown in Figure 5.

In ML3, rectified linear units, $\varphi_{ReLU}(x)=max(0,x)$, are used in the
hidden layers and the smooth SoftPlus $\varphi_{softplus}=\log(e^{\beta x}+1)$
is used in the output layer to enforce the optional constraint that
$\mathcal{M}_{learned}$ should be non-negative, i.e., $\forall y\forall
f_{\theta}(x)\mathcal{M}_{\phi_{t}}(y,f_{\theta}(x))\geq 0$. An adverse side-
effect of using the softplus activation in the output is that all negative
inputs to the output layer go to 0, resulting in flat regions in the learned
loss. Furthermore, removal of the output activation does not resolve this
issue, as ReLU, as well as other common activations such as Sigmoid, TanH, and
ELU, are also bounded and are prone to causing flatness when their activations
saturate, a common occurrence when taking gradients through long unrolled
optimization paths (Antoniou et al., 2018).

###  A.2 Smooth Leaky ReLU

To inhibit the flattening behavior of learned loss functions, a range
unbounded activation function should be used. A popular activation function
that is unbounded (when the leak parameter $\gamma<0$) is the Leaky ReLU (Maas
et al., 2013)

$\displaystyle\varphi_{leaky}(x)$ $\displaystyle=\max(\gamma\cdot x,x)$ (15)
$\displaystyle=\max(0,x)\cdot(1-\gamma)+\gamma x$ (16)

However, it is typically assumed that a loss function should be at least
$\mathcal{C}^{1}$, i.e., continuous in the zeroth and first derivatives.
Fortunately, there is a smooth approximation to the ReLU, commonly referred to
as the SoftPlus activation function (Dugas et al., 2000), where $\beta$
(typically set to 1) controls the smoothness.

$\varphi_{smooth}(x)=\tfrac{1}{\beta}\cdot\log(e^{\beta x}+1)$ (17)

The leaky ReLU is combined with the smooth ReLU by taking the term $max(0,x)$
from Equation (16) and substituting it with the smooth SoftPlus defined in
Equation (17) to construct a smooth approximation to the leaky ReLU

$\varphi_{hidden}(x)=\tfrac{1}{\beta}\log(e^{\beta x}+1)\cdot(1-\gamma)+\gamma
x$ (18)

where the derivative of the smooth leaky ReLU with respect to the input $x$ is

$\displaystyle\varphi_{hidden}^{\prime}(x)$
$\displaystyle=\frac{d}{dx}\left[\frac{\log(e^{\beta
x}+1)\cdot(1-y)}{\beta}+\gamma x\right]$ (19)
$\displaystyle=\frac{\frac{d}{dx}[\log(e^{\beta
x}+1)]\cdot(1-y)}{\beta}+\gamma$ (20)
$\displaystyle=\frac{\frac{d}{dx}[e^{\beta x}+1]\cdot(1-y)}{\beta\cdot
e^{\beta x}+1}+\gamma$ (21) $\displaystyle=\frac{e^{\beta
x}\cdot\beta\cdot(1-y)}{\beta\cdot e^{\beta x}+1}+\gamma$ (22)
$\displaystyle=\frac{e^{\beta x}(1-\gamma)}{e^{\beta x}+1}+\gamma$ (23)
$\displaystyle=\frac{e^{\beta x}(1-\gamma)}{e^{\beta
x}+1}+\frac{\gamma(e^{\beta x}+1)}{e^{\beta x}+1}$ (24)
$\displaystyle=\frac{e^{\beta x}+\gamma}{e^{\beta x}+1}$ (25)

The smooth leaky ReLU and its corresponding derivatives are shown in Figure 6.
Early iterations of AdaLFL learned $\gamma$ and $\beta$ simultaneously with
the network weights $\phi$, however; empirically, we found that setting
$\gamma=0.01$ and $\beta=10$ gave adequate inference performance across our
experiments.

Figure 6: The proposed activation function and its corresponding derivatives
when shifting $\gamma$ are shown in (a) and (b), respectively. In (c) and (d)
the activation function and its derivatives when shifting $\beta$ are shown.
Finally, in (c), the smooth leaky ReLU is contrasted with the original smooth
and leaky variants ReLU.

##  Appendix B Experimental Setup

To initialize $\mathcal{M}_{\phi}$, $\mathcal{S}_{init}=2500$ steps are taken
in offline mode with a meta learning rate of $\eta=1e-3$. In contrast, in
online mode, a meta learning rate of $\eta=1e-5$ is used (note, a high meta
learning rate in online mode can cause a jittering effect in the loss
function, which can cause training instability). The popular Adam optimizer is
used in the outer loop for both initialization and online adaptation.

In the inner-loop, all base models are trained using stochastic gradient
descent (SGD) with a base learning rate $\alpha=0.01$, and on CIFAR-10,
CIFAR-100, and SVHN, Nesterov momentum 0.9, and weight decay 0.0005 are used.
The remaining base-model hyper-parameters are selected using their respective
values from the literature in an identical setup to (Gonzalez & Miikkulainen,
2021).

All experimental results reported show the average results across 10
independent executions on different seeds for the purpose of analysing
algorithmic consistency. Importantly, our experiments control for the base
initializations such that all methods get identical initial parameters across
the same random seed; thus, any difference in variance between the methods can
be attributed to the respective algorithms and their loss functions.
Furthermore, the choice of hyper-parameters between ML3 and AdaLFL has been
standardized to allow for a fair comparison.

##  Appendix C Learned Loss Functions (Extended)

Figure 7: Loss functions generated by AdaLFL on the MNIST dataset, where each
row represents
a loss function, and the color represents the current gradient step.

Figure 8: Loss functions generated by AdaLFL on the MNIST dataset, where each
row represents
a loss function, and the color represents the current gradient step.

Figure 9: Loss functions generated by AdaLFL on the MNIST dataset, where each
row represents
a loss function, and the color represents the current gradient step.

Figure 10: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where
each row represents
a loss function, and the color represents the current gradient step.

Figure 11: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where
each row represents
a loss function, and the color represents the current gradient step.

Figure 12: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where
each row represents
a loss function, and the color represents the current gradient step.

Figure 13: Loss functions generated by AdaLFL on the CIFAR-10 dataset, where
each row represents
a loss function, and the color represents the current gradient step.