introduces a reward function sensitive to transaction costs and risk.

The reward $r_{t}$ is realized at the end of the period $(t-1,t]$ and includes
the return of the position ${a}_{t-1}$ held during that interval. The
objective of financial trading is generally to maximize future returns, or in
more vernacular terms; to buy when the price is low and sell when the price is
high. The multiplicative return of a financial instrument at time $t$ is
defined as the relative change in price from time $t-1$ to $t$

$y_{t}=\frac{p_{t}}{p_{t-1}}-1$ (5.5)

Multiplicative returns, unlike additive returns, have the advantage that they
are insensitive to the size of the capital traded. Logarithmic returns
$\log{(y_{t}+1)}$ are typically used in algorithmic trading for their
symmetric properties [JXL17, Hua18, ZZW+20]. The gross log return realized at
time $t$ is

${r}^{gross}_{t}=\log{\left(y_{t}+1\right)}{a}_{t-1}$ (5.6)

At the end of the period $(t-1,t]$, due to price movements $y_{t}$ in the
market, the weight ${a}_{t-1}$ evolve into

${a}^{\prime}_{t}=\frac{a_{t-1}\frac{p_{t}}{p_{t-1}}}{a_{t-1}y_{t}+1}$ (5.7)

where $a^{\prime}_{t}\in\mathbb{R}$. At the start of the next period $t$, the
agent must rebalance the portfolio from its current weight ${a}^{\prime}_{t}$
to its chosen weight ${a}_{t}$. As noted in section 2.1, the subsequent trades
resulting from this rebalancing are subject to transaction costs. The size of
the required rebalancing at time $t$ is represented by
$||{a}_{t}-{a}^{\prime}_{t}||$. The log-return net of transaction costs at
time $t$ is defined as

${r}^{net}_{t}={r}^{gross}_{t}-\lambda_{c}||{a}_{t-1}-{a}^{\prime}_{t-1}||$
(5.8)

where $\lambda_{\eta}\in[0,1]$ is the transaction cost fraction that is
assumed to be identical for buying and selling.

The log-return net of transaction costs assumes that the trader is risk-
neutral, which is rarely true. The Sharpe ratio is the most common measure of
risk-adjusted return; however, as noted in section 2.6, directly optimizing
the Sharpe ratio might not be optimal. Instead, this thesis adopts the
variance over returns [ZZW+20] as a risk term

$\sigma^{2}(r^{net}_{i}|i=t-L+1,...,t)=\sigma^{2}_{L}(r^{net}_{t})$ (5.9)

where $L\in\mathbb{N}_{+}$ is the lookback window to calculate the variance of
returns. In this thesis, the lookback window is $L=60$. In conclusion,
subtracting the risk term defined in equation 5.9 from the net returns defined
in equation 5.8 gives the risk-adjusted log-return net of transaction costs
$r_{t}$, defined as

$r_{t}={r}^{net}_{t}-\lambda_{\sigma}\sigma^{2}_{L}({r}^{net}_{t})$ (5.10)

where $\lambda_{\sigma}\geq 0$ is a risk-sensitivity term that can be
considered a trade-off hyperparameter for the stochastic gradient descent
optimizer. If $\lambda_{\sigma}=0$, the agent is risk-neutral. The
reinforcement learning agents are optimized using the reward function defined
in equation 5.10.

###  6 Reinforcement learning algorithm

This chapter presents two model-free reinforcement learning algorithms that
solve the trading MDP defined in chapter 5. There are three types of
reinforcement learning algorithms: critic-based, actor-based, and actor-
critic-based. Despite the popularity of critic-based algorithms, such as
Q-learning, they are unsuitable in this context due to their inability to
handle high-dimensional or continuous action spaces. Actor-based and actor-
critic-based methods, known as policy gradient methods (4.6), are appropriate
since they can handle continuous action and state spaces. Furthermore, policy
gradient methods are suitable for continuing tasks like trading. As both
actor-based and actor-critic-based methods have advantages and disadvantages,
it remains to be determined which methodology is most appropriate for this
problem. Actor-based RL methods like REINFORCE are generally successful in
stochastic continuous action spaces and have been applied to both single
instrument trading and portfolio optimization [JXL17, ZZR20]. However, actor-
based RL suffers high variance in learning and tends to be unstable and
inconvenient in online learning. Actor-critic methods like Deep Deterministic
Policy Gradient (DDPG) [LHP+15] have become popular lately and have been
applied to several RL trading and portfolio optimization problems [LXZ+18,
YLZW20]. Deterministic policies can be appropriate for financial trading, and
off-policy learning combined with replay memory can be practical for online
learning. However, training two neural networks is generally deemed to be
unstable. Thus, the selection of a reinforcement learning algorithm is non-
trivial. This chapter presents an actor-based algorithm (6.2) and an actor-
critic-based algorithm (6.3) for solving the trading MDP.

####  6.1 Immediate reward environment

The zero market impact assumption in chapter 5.1 implies that the agent’s
participation in the market will not affect future prices $\mathbf{p}$. In
other words, the zero market assumption implies that the agent’s actions will
not affect the future external state of the environment. However, actions
performed at the start of period $t$ affect the transaction costs paid by the
agent at the start of the subsequent period $t+1$. The reward $r_{t+1}$
depends on transaction costs incurred at time $t$, and thus the agent’s
previous action $\mathbf{a}_{t-1}$ will affect the following action. In this
framework, this influence is encapsulated by adopting the recursive mechanism
introduced by Moody et al. [MWLS98] of considering the past action as a part
of the internal state of the environment. Consequently, large position changes
are discouraged.

The goal of the policy gradient agent is to find the policy parameters
$\theta$ that maximize the average rate of reward per time step. All rewards
are equally important to the final return through commutativity. Since the
agent does not affect the subsequent state of the environment, the goal is to
maximize the expected immediate reward $\mathbb{E}[r_{t+1}]$, exactly
expressed in equation 5.10 as the expected cumulative logarithmic return net
of transaction costs and the risk-sensitivity term. Therefore, the action-
value of the action ${a}_{t}$ is its immediate reward $r_{t+1}$, i.e.,

$Q(s_{t},a_{t})=r_{t+1}$ (6.1)

$\forall s_{t}\in\mathcal{S},a_{t}\in\mathcal{A}(s_{t})$. As an immediate
reward process, the reward function can be directly optimized by the policy
gradient from rewards.

The actor-based direct policy gradient method introduced in section 6.2
optimizes the policy by using the immediate reward directly. In contrast, the
actor-critic method introduced in section 6.3 optimizes the policy using
critique from a Q-network optimized to minimize the loss to the immediate
reward.

####  6.2 Direct policy gradient

The first actor-based reinforcement learning algorithm is a direct policy
gradient method inspired by the REINFORCE algorithm. Instead of computing
learned probabilities for each action, the direct policy gradient method
stochastically samples actions from a Gaussian distribution. Let
$\pi_{\theta,\epsilon}:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ be the
stochastic policy parameterized by the weights
$\theta\in\mathbb{R}^{d^{\prime}}$. The policy is defined as a normal
probability density over a real-valued scalar action

$\pi_{\theta,\epsilon}({a}|\mathbf{s})=\frac{1}{\epsilon\sqrt{2\pi}}e^{\left(-\frac{\left({a}-\mu_{\theta}(\mathbf{s})\right)^{2}}{2\epsilon^{2}}\right)}$
(6.2)

where the mean is given by a parametric function approximator
$\mu_{\theta}(\mathbf{s}):\mathbb{R}^{|\mathbf{s}|}\rightarrow[-1,1]$ that
depends on the state and outputs an independent mean for the Gaussian
distribution. The standard deviation is given as an exploration rate
$\epsilon$. The exploration rate $\epsilon\in\mathbb{R}$ is positive and
decays at $\lambda_{\epsilon}\in[0,1]$ to encourage exploration of the action
space in early learning epochs. The rate has a minimum $\epsilon_{min}\geq 0$
such that $\epsilon\geq\epsilon_{min}$, $\forall t$. After each episode, the
exploration rate updates according to the following update rule

$\epsilon\leftarrow\max{(\lambda_{\epsilon}\epsilon,\epsilon_{min})}$ (6.3)

At every step $t$, the agent samples an action $a_{t}\sim\pi_{\theta}$ from
the policy and clips the action to the interval $[-1,1]$.

The novel idea of using the exploration rate $\epsilon$ as a controlled,
decaying standard deviation of the stochastic policy represents progress in
the research area. As $\epsilon$ approaches $0$, the policy becomes more or
less deterministic to the mean given by the parametric function approximation
$\mu_{\theta}$, which is advantageous in critical domains such as financial
trading. However, being a stochastic policy, the stochastic sampling required
for the REINFORCE update is still available, blending the best of both worlds
for an algorithmic trading agent in an immediate reward environment.

###### Optimization

As the model should be compatible with pre-trade training and online learning,
optimization is defined in an online stochastic batch learning scheme.
Trajectories are divided into mini-batches $[t_{s},t_{e}]$, where
$t_{s}<t_{e}$. The policy’s performance measure on a mini-batch is defined as

$J(\pi_{\theta,\epsilon})_{[t_{s},t_{e}]}=\mathbb{E}_{\pi_{\theta,\epsilon}}\left[\sum_{t=t_{s}+1}^{t_{e}}r_{t}\right]$
(6.4)

i.e., the expected sum of immediate rewards during the mini-batch
$[t_{s},t_{e}]$ when following the policy $\pi_{\theta,\epsilon}$. Using the
policy gradient theorem, the gradient of the performance measure $J$ with
respect to the parameter weights $\theta$ is defined as

$\nabla_{\theta}J(\pi_{\theta,\epsilon})_{[t_{s},t_{e}]}=\mathbb{E}_{\pi_{\theta,\epsilon}}\left[\sum_{t=t_{s}+1}^{t_{e}}r_{t}\nabla_{\theta}\log\pi_{\theta,\epsilon}(a_{t}|s_{t})\right]$
(6.5)

This expectation is empirically estimated from rollouts under
$\pi_{\theta,\epsilon}$. The parameter weights are updated using a stochastic
gradient ascent pass

$\theta\leftarrow\theta+\alpha\nabla_{\theta}J(\pi_{\theta,\epsilon})_{[t_{s},t_{e}]}$
(6.6)

###### Pseudocode

The pseudocode for the actor-based algorithm is given in Algorithm 2.

Algorithm 2 Actor-Based Algorithm for Trading

Input: a differentiable stochastic policy parameterization
$\pi_{\theta,\epsilon}(a|s)$

Algorithm parameters: learning rate $\alpha^{\theta}>0$, mini-batch size
$b>0$, initial exploration rate $\epsilon\geq 0$, exploration decay rate
$\lambda_{\epsilon}\in[0,1]$, exploration minimum $\epsilon_{min}\geq 0$

Initialize: empty list $\mathcal{B}$ of size $b$

repeat

Receive initial state of the environment $s_{0}\in\mathcal{S}$

repeat

for t = 0,1,…,T-1 do

Sample action $a_{t}\sim\pi_{\theta,\epsilon}(\cdot|s_{t})$

Execute action $a_{t}$ in the environment and observe $r_{t}$ and $s_{t+1}$

Store pair of reward $r_{t}$ and log-probabilities
$\nabla_{\theta}\ln\pi_{\theta,\epsilon}(a_{t}|s_{t})$ in $\mathcal{B}$

if $|\mathcal{B}|==b$ or $s_{t}$ is terminal then

Update the policy $\pi_{\theta,\epsilon}$ by one step of gradient ascent
using:
$\nabla_{\theta}J(\pi_{\theta,\epsilon})\approx\sum_{\mathcal{B}}r_{t}\nabla_{\theta}\ln{\pi_{\theta,\epsilon}(a_{t}|\mathbf{s}_{t})}$

Reset $\mathcal{B}$ to empty list

end if

end for

until terminal state

Update the exploration rate
$\epsilon=\max{(\epsilon\lambda_{\epsilon},\epsilon_{min})}$

until convergence

####  6.3 Deterministic actor-critic

There are several different actor-critic algorithms available, like
asynchronous advantage actor-critic (A3C) [MBM+16] and proximal policy
optimization (PPO) [SWD+17]. However, due to the critical nature of the
problem, actor-critic algorithms that utilize deterministic policies are of
interest, as stochasticity may negatively affect the model’s performance.
Thus, the second RL algorithm is an off-policy deterministic actor-critic
algorithm inspired by the deep deterministic policy gradient algorithm
[LHP+15].

Let $\mu_{\theta}:\mathcal{A}\rightarrow\mathcal{S}$ be the deterministic
policy parameterized by $\theta\in\mathbb{R}^{d^{\prime}}$. This is the same
function approximator used to generate the mean for the Gaussian action
selection in the direct policy gradient algorithm 6.2. However, the function
approximator is now used to map states to actions directly and not as the mean
in a probability distribution over the actions. Instead of optimizing the
policy using the log probabilities of the sampled action scaled by the reward,
the deterministic policy is optimized using a learned action-value critic
relying on the deterministic policy gradient theorem. Let
$Q_{\phi}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ be the
Q-network critic parameterized by $\phi\in\mathbb{R}^{b^{\prime}}$. Although
exploring the state space is unnecessary in this problem, as the external
environment is not affected by the agent’s actions, it can be advantageous to
explore the action space in the early stages of learning to provide the agent
with training examples from the entire action space. To ensure the sufficient
exploration of the action space, the algorithm is trained off-policy with an
exploration policy $\mu^{\prime}_{\theta}$ defined as

$\mu^{\prime}_{\theta}(s)=\mu_{\theta}(s)+\epsilon\mathcal{W}$ (6.7)

where $\mathcal{W}\sim\mathcal{U}_{[-1,1)}$ is sampled noise from an uniform
distribution. The exploration parameters
$\epsilon,\epsilon_{min},\lambda_{\epsilon}$ are defined similarly for the
direct policy gradient algorithm 6.2. Clipping agents’ actions to the interval
$[-1,1]$ prevents them from taking larger positions than their available
capital.

###### Optimization

Both the actor and critic networks are updated using randomly sampled mini-
batches $\mathcal{B}$ from a replay memory $\mathcal{D}$. The replay memory
provides random batches in sequential order for stateful RNNs, and random
batches not in sequential order that minimize correlation between samples for
non-stateful DNNs. The exploration policy $\mu^{\prime}_{\theta}$ explores the
environment and generates transitions $\tau$ stored in the replay memory
$\mathcal{D}$.

The objective function $J$ for the deterministic policy $\mu_{\theta}$ is
defined as

$J(\mu_{\theta})=\mathbb{E}_{s\sim\mathcal{B}}[Q_{\phi}(s,\mu_{\theta}(s))]$
(6.8)

and its gradient is given as

$\nabla_{\theta}J(\mu_{\theta})=\mathbb{E}_{s\sim\mathcal{B}}[\nabla_{\theta}\mu_{\theta}(s)\nabla_{a}Q_{\phi}(s,a)|_{a=\mu_{\theta}(s)}]$
(6.9)

Since the environment is an immediate reward environment, the target for the
Q-network updates is the immediate reward, i.e., $y=r$. MSE loss is used as a
loss function as the outliers are of critical importance to the success of the
trading agent. The loss function $L(\phi)$ for the Q-network $Q_{\phi}$ is
defined as

$L(Q_{\phi})=\mathbb{E}_{s,a,r\sim\mathcal{B}}[(Q_{\phi}(s,a)-r)^{2}]$ (6.10)

and its gradient is given as

$\nabla_{\phi}L(Q_{\phi})=\mathbb{E}_{s,a,r\sim\mathcal{B}}[(Q_{\phi}(s,a)-r)\nabla_{\phi}Q_{\phi}(s,a)]$
(6.11)

###### Pseudocode

The pseudocode for the deterministic actor-critic algorithm is given in
Algorithm 3.

Algorithm 3 Actor-Critic Algorithm for Trading

Input: a differentiable deterministic policy parameterization
$\mu_{\theta}(s)$

Input: a differentiable state-action value function parameterization
$Q_{\phi}(s,a)$

Algorithm parameters: learning rates $\alpha^{\theta}>0$, $\alpha^{\phi}>0$,
mini-batch size $b>0$, replay memory size $d\geq b$, initial exploration rate
$\epsilon\geq 0$, exploration decay rate $\lambda_{\epsilon}\in[0,1]$,
exploration minimum $\epsilon_{min}\geq 0$

Initialize empty replay memory cache $\mathcal{D}$

repeat

Receive initial state of the environment $s_{0}\in\mathcal{S}$

for t = 1,…,T do

Select action $a_{t}=\mu_{\theta}(s_{t})+\epsilon\mathcal{W}$ from the
exploration policy

Execute $a_{t}$ in the environment and observe $r_{t}$ and $s_{t+1}$

Store transition $\tau_{t}=(s_{t},a_{t},r_{t})$ in the replay memory
$\mathcal{D}$

Sample a random mini-batch $\mathcal{B}$ of $|\mathcal{B}|$ transitions $\tau$
from $\mathcal{D}$

Update the Q-network by one step of gradient descent using
$\nabla_{\phi}\frac{1}{|\mathcal{B}|}\sum_{(s,a,r)\in\mathcal{B}}(Q_{\phi}(s,a)-r)^{2}$

Update the policy by one step of gradient ascent using
$\nabla_{\theta}\frac{1}{|\mathcal{B}|}\sum_{s\in\mathcal{B}}Q_{\phi}(s,\mu_{\theta}(s))$

end for

Update the exploration rate
$\epsilon=\max{(\epsilon\cdot\lambda_{\epsilon},\epsilon_{min})}$

until convergence

###  7 Network topology

The reinforcement learning algorithms introduced in chapter 6 utilize function
approximation to generalize over a continuous state and action space. Section
2.5 introduced function approximators for extracting predictive patterns from
financial data, where empirical research suggested the superiority of deep
learning methods. Thus, the function approximators introduced in this chapter
rely on deep learning techniques introduced in 3. In the research presented in
section 2.5, the function approximators based on convolutional neural networks
(3.3.10) and those based on the LSTM (3.3.11) consistently performed well.
Thus, this section introduces two function approximators based on CNNs and
LSTMs, respectively. The sequential information layer, presented in section
7.4, leverages these techniques to extract predictive patterns from the data.
Furthermore, the decision-making layer that maps forecasts to market
positions, presented in section 7.5, employs the recursive mechanism
introduced by Moody et al. [MWLS98], enabling the agent to consider
transaction costs.

The direct policy gradient algorithm presented in chapter 6.2 is an actor-
based RL algorithm that only uses a parameterized policy network. The
deterministic actor-critic algorithm presented in chapter 6.3 uses a
parameterized policy network and a parameterized critic network. This chapter
outlines these function approximators, which fortunately consist of many of
the same components. Section 7.2 describes the policy network, while section
7.3 describes the Q-network. The last section 7.6 describes the optimization
and regularization of the networks.

####  7.1 Network input

The first step is to specify the input into the networks. Section 5.3 defined
the agent state $\mathbf{s}_{t}^{a}$ of the partially observable environment.
This section describes the modified version of the agent state
$\mathbf{s}^{a^{\prime}}_{t}$, which both the policy and critic agents receive
as input. The modified agent state applies two forms of processing to the
network input; the first is extracting the relevant observations from the
agent state, which ensures that the network input is of fixed size, and the
second is normalizing the network input, which is advantageous for the non-
linear function approximators introduced in this chapter. This thesis adopts
the philosophy of technical traders of the price reflecting all necessary
information, and therefore the past price is used to represent the agent
state. The primary reason for selecting the price series alone as the state
representation is to examine the ability of a general deep reinforcement
learning model to extract patterns from raw, relatively unprocessed data.
Although additional data sources could aid the agent in discovering predictive
patterns, that is beyond the scope of this thesis.

Adopting the ideas of Jiang et al. [JXL17], the agent state is down-sampled by
extracting the three most relevant prices from a period; the closing price,
the highest price, and the lowest price. Thus, the price tensor used to
represent the external agent state at time $t$ is defined as

$\hat{\mathbf{p}}_{t}=\left[{p}_{t},{p}_{t}^{high},{p}_{t}^{low}\right]$ (7.1)

Normalizing input data for neural networks speeds up learning [GBC16] and is
beneficial for reinforcement learning as well [ARS+20]. However, normalizing
the whole time series ex-ante is a form of lookahead. The normalization scheme
can only use data up to time $\leq t$ for the observation $\mathbf{p}_{t}$
$\forall t$. The choice of instrument weights depends on relative log returns
rather than absolute price changes. The price tensor $\hat{\mathbf{p}}_{t}$ is
normalized using the log-returns from the previous closing price $p_{t-1}$.
Additionally, adopting the ideas from Zhang et al. [ZZR20], the input is
further normalized by dividing by a volatility term defined as

$\sigma^{2}\left(\log{\left(\frac{p_{i}}{p_{i-1}}\right)}|i=t-L+1,...,t\right)\sqrt{L}=\sigma^{2}_{L,t}\sqrt{L}$
(7.2)

where $L\in\mathbb{N}_{+}$ is the lookback window to calculate the volatility
of the closing price, which is set to $L=60$ similarly as [ZZR20]. The
normalized price tensor at time $t$ is thus defined as

$\bar{\mathbf{p}}_{t}=\log{\left(\hat{\mathbf{p}}_{t}\oslash
p_{t-1}\right)}\oslash\sigma^{2}_{L,t}\sqrt{L}$ (7.3)

As a precaution against outliers in volatile markets, which can be detrimental
to the performance of DNNs, the normalized price tensor $\bar{\mathbf{p}}_{t}$
is clipped to the interval $[-1,1]$.

Stacking the past $n$ observations produces the approximated environment
state. Thus, the final price tensor is adjusted to contain the $n$ most recent
observations $\bar{\mathbf{p}}_{t-n+1:t}\in\mathbb{R}^{3\times n}$. The
stacked price tensor is considered the external agent state and defined as
$\mathbf{x}^{S}_{t}=\bar{\mathbf{p}}_{t-n+1:t}$ The networks also adopt the
recursive mechanism introduced by Moody et al. [MWLS98] of considering the
past action as a part of the internal environment, allowing the agent to take
the effects of transaction costs into account. The instrument weight from the
previous period $a_{t-1}$ is inserted into the final decision-making layer
after extracting the sequential features in the sequential information layer.
The modified agent state thus approximates the state of the environment

$\mathbf{s}^{a^{\prime}}_{t}=(\mathbf{x}^{S}_{t},a_{t-1})$ (7.4)

The policy networks and Q-network receive this modified agent state
$\mathbf{s}^{a^{\prime}}_{t}$ as input. As an action-value function, the
Q-network also takes the current action $a_{t}$ as input.

####  7.2 Policy network

The deterministic policy 6.3 and the mean-generating parametric function
approximator in the stochastic policy 6.2 are the same function approximator
$\mu_{\theta}:\mathbb{R}^{|\mathcal{S}|}\rightarrow[-1,1]$ parameterized by
$\theta\in\mathbb{R}^{d^{\prime}}$, and will in this chapter be referred to as
the policy network. The policy network consists of a sequential information
layer, a decision-making layer, and a $\tanh$ function. The input to the
policy network is the modified agent state $\mathbf{s}^{a^{\prime}}_{t}$. The
external part of the agent state $\mathbf{x}^{S}_{t}$, i.e., the price tensor
of stacked observations, is input into the sequential information layer. The
sequential information layer output is concatenated with the previous action
$a_{t-1}$ to produce input into the decision-making layer. The output from the
decision-making layer maps to a $\tanh$ function that produces the action
constrained to the action space $[-1,1]$.

Figure 7.1: Policy network architecture

####  7.3 Q-network

The Q-network
$Q_{\phi}:\mathbb{R}^{|\mathcal{S}|}\times\mathbb{R}^{|\mathcal{A}|}\rightarrow\mathbb{R}$
is a function approximator parameterized by $\phi\in\mathbb{R}^{b^{\prime}}$.
It is an action-value function that assigns the value of performing a specific
action in a specific state and thus takes two arguments, the modified agent
state $\mathbf{s}^{a^{\prime}}_{t}$ and the action $a_{t}$. Other than that,
there are two differences between the critic and policy networks. Firstly, the
Q-network has an additional layer before the sequential information net that
concatenates the agent state $\mathbf{s}^{a}_{t}$ and the current action
$a_{t}$ and maps it through a fully-connected layer into a leaky-ReLU
activation function with negative slope $0.01$ and dropout with probability
$0.2$. The second difference is that the output after the decision-making
layer does not map to a $\tanh$ function since the Q-network outputs action-
values, which are not constrained to any specific interval.

Figure 7.2: Q-network architecture

####  7.4 Sequential information layer

In essence, an algorithmic trading agent places bets on the relative price
change, or returns, of financial instruments. The agent’s success ultimately
depends on its ability to predict the future. However, doing so in highly
competitive and efficient markets is non-trivial. To remain competitive in
continuously evolving markets, the agent must learn to recognize patterns and
generate rules based on past experiences. The sequential information layer
extracts the sequential features from the input data and is arguably the most
integral part of the model. Let $\mathbf{x}^{I}_{t}$ be the input into the
sequential information net (for the policy network
$\mathbf{x}^{I}_{t}=\mathbf{x}^{S}_{t}$). The sequential information layer is
a parametric function approximator that takes the input $\mathbf{x}^{I}_{t}$
and outputs a feature vector $\mathbf{g}_{t}$, defined as

$f^{S}(\mathbf{x}^{I}_{t})=\mathbf{g}_{t}$ (7.5)

The choice of the appropriate function approximator for this task is non-
trivial. The inductive bias of the model must align with that of the problem
for the model to generalize effectively. Therefore, selecting a model that
captures the problem’s underlying structure while also being efficient and
scalable is imperative. Research on financial time series forecasting found
that deep learning models, specifically those based on the CNN and LSTM
architecture, consistently outperformed traditional time series forecasting
methods such as the ARIMA and GARCH [XNS15, MRC18, SNN18, SGO20]. The
universal approximation theorem (3.3.8) establishes that there are no
theoretical constraints on feedforward networks’111111Of arbitrary width or
height. expressivity. However, feedforward networks are not as naturally well-
suited to processing sequential data as CNNs and LSTMs. Therefore, they may
not achieve the same level of performance, even though it is theoretically
possible. Additionally, feedforward networks may require significantly more
computing power and memory to achieve the same performance as CNNs or LSTMs on
sequential data. Transformers were also considered due to their effectiveness
in forecasting time series [MSA22]. Transformers employ an encoder-decoder
architecture and rely on attention mechanisms to capture long-term
dependencies. Thus, they do not require a hidden state, like RNNs, and are
relatively easy to parallelize, enabling efficient training on large datasets.
A variant called decision transformers [CLR+21] has been applied to offline
reinforcement learning. However, it is unclear how to apply the transformer in
its conventional encoder-decoder topology to online reinforcement learning.
Therefore, the transformer is, for the moment, unsuitable for this problem.
The gated recurrent unit (GRU) is a newer version of the recurrent neural
network that is less computationally complex than the LSTM. However, LSTMs are
generally considered superior for forecasting financial data [SGO20].

This section defines two distinct DNN topologies for the sequential
information layer; the first is based on convolutional neural networks, while
the second is based on recurrent neural networks, specifically the LSTM. The
two sequential information topologies both consist of two hidden layers, which
is enough for the vast majority of problems. Performance is usually not
improved by adding additional layers.

#####  7.4.1 Convolutional neural network

The CNN-based sequential information layer topology includes two 1-dimensional
convolutional layers. In the absence of established heuristics, determining
the parameters for a CNN can be challenging. Thus, the parameters chosen for
these layers are partly informed by research on CNNs in financial time series
forecasting [SGO20] and partly determined through experimentation. The first
convolutional layer has kernel size $3$ and stride $1$ and processes each of
the $3$ columns in the input $\mathbf{x}^{I}_{t}$ as separate channels of size
$1\times n$, where $n$ is the number of stacked observations. It outputs $32$
feature maps of size $1\times n-2$. The second convolutional layer has kernel
size $3$ and stride $1$ and outputs $32$ feature maps of size $1\times n-4$.
Batch norm is used after both convolutional layers on the feature maps to
stabilize and speed up learning. The CNN uses the Leaky-ReLU activation
function with a negative slope of $0.01$ after the batch norm layers to
generate the activation maps. Dropout with probability $p=0.2$ is used between
the layers. Max pooling with kernel size $2$ and stride $2$ is applied after
the final convolutional layer to down-sample the output before all activation
maps are concatenated into one big activation map.

Figure 7.3: Convolutional sequential information layer architecture

#####  7.4.2 Long Short-Term Memory

The second sequential information network topology introduces memory through a
recurrent neural network to solve the partially observable MDP. Long Short-
term Memory (LSTM) is the go-to solution for environments where memory is
required and are good at modeling noisy nonstationary data. The LSTM
sequential information net architecture consists of two stacked LSTM layers.
Both LSTM layers have $128$ units in the hidden state, which was chosen
experimentally. Following both LSTM layers, the network employs dropout with
dropout-probability $p=0.2$. The LSTM cell contains three sigmoid functions
and one hyperbolic tangent function, so inserting an activation function after
the LSTM layer is superfluous. Batchnorm is incompatible with RNNs, as the
recurrent part of the network is not considered when computing the
normalization statistic and is, therefore, not used.

Figure 7.4: LSTM sequential information layer architecture

####  7.5 Decision-making layer

In the decision-making layer, the previous action $a_{t-1}$ is concatenated
into the features $\mathbf{g}_{t}$, i.e., the output from the sequential
information layer, and mapped to a single output value. The previous action
weight $a_{t-1}$ allows the agent to consider transaction costs when making
trading decisions (policy network) or when assigning value to actions in
states (Q-network). The mapping comprises a fully-connected layer between the
features and the previous action to a single output value. This output value
is the action (or Gaussian mean) for the policy (after mapping it to a $\tanh$
function) or the action-value for the Q-network. The input to the decision-
making layer is defined as

$\mathbf{x}_{t}^{D}=(\mathbf{g}_{t},a_{t-1})$ (7.6)

The decision-making layer is a dot product between a weight vector
$\mathbf{w}^{D}\in\mathbb{R}^{|\mathbf{x}_{t}^{D}|}$ and the input
$\mathbf{x}_{t}^{D}$, defined as

$f_{D}(\mathbf{x}_{t}^{D})=(\mathbf{w}^{D})^{\top}\mathbf{x}^{D}_{t}$ (7.7)

####  7.6 Network optimization

The weights $\theta$ and $\phi$ of the policy network $\mu_{\theta}$ and
Q-network $Q_{\phi}$ are initialized using Kaiming initialization which
ensures that the initial weights of the network are not too large and have a
small variance, which helps to prevent the network from getting stuck in local
minima and allows the network to generalize better. Additionally, Kaiming
initialization considers the type of activation function used in the network,
contrary to conventional initialization schemes. The weight initialization
scheme centers the initial output distribution of the networks around $0$ with
a small standard deviation regardless of the input. The weights are updated
using the Adam stochastic gradient descent algorithm on mini-batches, allowing
the network to update the weights more efficiently and accurately than other
SGD algorithms. The gradient norm for each mini-batch is clipped to $1$ to
prevent exploding gradients. There are many potential activation functions for
neural networks, including the default recommendation, the ReLU. To combat the
“dying ReLU problem”, the leaky-ReLU activation function is used in the
networks. The negative slope, or the “leak”, is set to the standard value of
$0.01$.

#####  7.6.1 Regularization

Machine learning research generally focuses on problems with complex
structures and high signal-to-noise ratios, such as image classification. For
these problems, complicated non-linear models like neural nets have
demonstrated their effectiveness. However, in a high-noise environment such as
financial forecasting, where the R-squared is often of order $10^{-4}$
[Isi21], anything beyond linear regression poses a significant overfitting
risk. An overfitted network will likely perform well on the training set but
generalize poorly out-of-sample. An algorithmic trading agent that performs
well on the training set is of little use, and it is imperative to reduce the
generalization error. Therefore, regularization is needed to mitigate the risk
of overfitting and reduce the generalization error. A description of the
regularization techniques used for these networks is provided in section
3.3.6.

For ML models to be generalizable, they must learn data patterns rather than
individual data points to identify a bigger picture agnostic of noisy details.
Regularization techniques such as weight decay limit the capacity of the
networks by adding a parameter norm penalty to the objective function. Weight
decay uses the $L^{2}$ norm; other norms, such as the $L^{1}$ norm, can also
be used. The $L^{2}$ norm is appropriate since it punishes outliers harsher
and is easier to optimize with gradient-based methods. The parameter
$\lambda_{wd}$ controls the degree of penalization, balancing the tradeoff
between increased bias and decreased variance. The network optimizer
introduced in this section uses weight decay with the constant parameter
$\lambda_{wd}=0.001$ to mitigate the network’s overfitting risk.
Experimentally, this value delivered the optimal balance for the bias-variance
tradeoff. Although increasing the weight decay penalty could further reduce
overfitting risk, this was too restrictive for the networks.

It is important to note that weight decay reduces, but does not eliminate, the
risk of overfitting. Dropout is another explicit regularization technique
almost universally used in deep neural networks. Dropout forces the network to
learn multiple independent data representations, resulting in a more robust
model. When training networks on noisy financial data, dropout effectively
ensures the network ignores the noise. Similarly to weight decay, the dropout
rate is a tradeoff. There is no established heuristic for choosing the dropout
rate; instead, it is usually chosen through experimentation. In this case, a
dropout rate of $0.2$ provided a suitable regularizing effect where the model
generalized well and produced accurate predictions. Dropout is used between
all hidden layers in the networks.

Although explicit regularizers such as weight decay and dropout reduce
overfitting risk, it remains tricky to determine the optimal training
duration. This challenge is addressed with early stopping, which functions as
an implicit regularizer. The networks are trained in an early stopping scheme,
with testing on the validation set every $10$th epoch. As reinforcement
learning involves random exploration, the models are tested slightly less
frequently than conventional to prevent premature stopping.

##  Part III Experiments

###  8 Experiment and Results

Experiments play a vital role in science and provide the basis for scientific
knowledge. This chapter presents the experiments and results where the methods
presented in part II are tested on historical market data using the
backtesting framework described in section 2.9. The backtest requires
simplifying market assumptions, specified in chapter 5. Section 8.1 details
the experiment setting. The results of the experiment are presented and
discussed in sections 8.2 and 8.3. Finally, the overall approach is discussed
in section 8.4. The experiment aims to answer the research questions posed at
the start of this thesis.

  1. 1.

Can the risk of algorithmic trading agents operating in volatile markets be
controlled?

  2. 2.

What reinforcement learning algorithms are suitable for optimizing an
algorithmic training agent in an online, continuous time setting?

  3. 3.

What deep learning architectures are suitable for modeling noisy, non-
stationary financial data?

####  8.1 Materials and Methods

Chapter 6 described two reinforcement learning algorithms to solve the
commodity trading problem; the direct policy gradient (PG) and the
deterministic actor-critic (AC). Chapter 7 described two sequential
information layers, one based on CNN architecture and the other LSTM-based.
Both the actor and critic in the actor-critic algorithm are modeled using the
same architecture. In total, that leaves four combinations which are specified
below with their respective abbreviations

  * •

PG-CNN: Direct policy gradient algorithm where the policy network is modeled
using the CNN-based sequential information layer.

  * •

PG-LSTM: Direct policy gradient algorithm where the policy network is modeled
using the LSTM-based sequential information layer.

  * •

AC-CNN: Deterministic actor-critic algorithm where the policy and Q-network
are modeled using the CNN-based sequential information layer.

  * •

AC-LSTM: Deterministic actor-critic algorithm where the policy and Q-network
are modeled using the LSTM-based sequential information layer.

#####  8.1.1 Baselines

Defining a baseline can be helpful when evaluating the performance of the
methods presented in part II. A challenge with testing algorithmic trading
agents is the lack of established baselines. However, the by far most common
alternative is the buy-and-hold baseline [MWLS98][ZZR20][ZZW+20]. The buy-and-
hold baseline consists of buying and holding an instrument throughout the
experiment, i.e., $a_{t}=1,\forall t$. Compared to a naive buy-and-hold
baseline, an intelligent agent actively trading a market should be able to
extract excess value and reduce risk.

#####  8.1.2 Hyperparameters

Table 1 shows the hyperparameters used in this experiment. The learning rates
for the policy network and Q-network are denoted $\alpha_{actor}$ and
$\alpha_{critic}$, respectively, and were tuned experimentally.
$|\mathcal{B}|$ is the batch size, and $|\mathcal{D}|$ is the replay memory
size. Large batch sizes are necessary to obtain reliable gradient estimates.
However, large batch sizes also result in less frequent updates to the agent
and updates that may contain outdated information. As a result of this
tradeoff, the batch and replay memory sizes used in this experiment were
selected as appropriate values. The transaction cost fraction $\lambda_{c}$ is
set to a reasonable value that reflects actual market conditions. The initial
exploration rate is denoted $\epsilon$, with decay rate $\lambda_{\epsilon}$
and minimum $\epsilon_{min}$. The number of stacked past observations is given
by $n$, considered a reasonable value for the agent to use for short-term
market prediction.

Model |  $\mathbf{\alpha_{actor}}$ |  $\mathbf{\alpha_{critic}}$ |  $\mathbf{|\mathcal{B}|}$ |  $\mathbf{|\mathcal{D}|}$ |  $\mathbf{\lambda_{c}}$ |  $\mathbf{\epsilon}$ |  $\mathbf{\lambda_{\epsilon}}$ |  $\mathbf{\epsilon_{min}}$ |  $n$
---|---|---|---|---|---|---|---|---|---
PG |  $0.0001$ |  - |  128 |  - |  $0.0002$ |  $1$ |  $0.9$ |  $0.01$ |  $20$
AC |  $0.0001$ |  $0.001$ |  128 |  $1000$ |  $0.0002$ |  $1$ |  $0.9$ |  $0.01$ |  $20$

Table 1: Hyperparameters

#####  8.1.3 Training scheme

The dataset is split into three parts; a training set, a validation set, and a
test set, in fractions of $1/4$, $1/4$, and $1/2$, respectively. The RL agents
train on the training set, the first $1/4$ of the dataset, and then validate
on the validation set, the next $1/4$. Early stopping is used, with testing
every $10$th epoch. The early stopping frequency is low because the RL agents
exhibit randomness and stochasticity, especially in the early epochs. Setting
a high early stopping frequency can cause premature convergence. The weight
initialization scheme, described in section 7.6, causes the initial action
distribution of the policy to be centered around $0$ with a small standard
deviation. However, the agents learn faster when exploring the edge values of
the state space in the early stages. Exploration of the action space is
controlled, for both RL algorithms, by the three exploration parameters
$\epsilon,\lambda_{\epsilon},\epsilon_{min}$. During training, the exploration
rate starts at $\epsilon=1$ and decays at $\lambda_{\epsilon}=0.9$ per episode
to a minimum exploration rate of $\epsilon_{min}=0.01$.

When the agent has finished training, it tests once out-of-sample on the test
set, the last $1/2$ of the dataset. Leaving half of the dataset for final
testing ensures the test set is sufficiently large to evaluate the trading
agents. Exploring the action space is no longer necessary after initial
training. Therefore, $\epsilon=0$ for the out-of-sample test. According to the
optimization strategies specified in their respective pseudocodes 2 and 3, the
RL agents continuously refit themselves as they observe transitions. The
results section (8.2) presents results from these backtests.

#####  8.1.4 Performance metrics

The objective of the algorithmic trading agent is described by modern
portfolio theory (2.3) of maximizing risk-adjusted returns, usually
represented by the Sharpe ratio. Thus, the Sharpe ratio defined in equation
2.1 will be the primary performance metric for the backtest. The reward
function defined in equation 5.10 is not a comparable performance measure to
related work. Instead, the standard method for assessing performance by linear
returns net of transaction costs is adopted. In a backtest, the agent
interacts with the environment and generates a sequence of actions
$\\{a_{0},a_{1},...,a_{T-1},a_{T}\\}$. The linear net return after
$T\in\mathbb{N}_{+}$ trades is defined as

$R_{T}=\prod_{t=1}^{T}\left(y_{t}\cdot
a_{t-1}\right)+1-\lambda_{c}||a^{\prime}_{t-1}-a_{t-1}||$ (8.1)

where $y_{t},a_{t},a^{\prime}_{t},\lambda_{c}$ are defined in chapter 5. The
return $R_{T}$ is used to calculate the Sharpe ratio. As there is randomness
in the models, either through stochastic action selection or random mini-batch
sampling, along with random weight initialization, performance is averaged
over $10$ runs. In addition to the Sharpe ratio, additional performance
metrics can help paint a clearer picture of the performance of an algorithmic
trading agent. This thesis adopts some of the performance metrics most
frequently found in related work [JXL17, ZZW+20, ZZR20]. The performance
metrics used in this thesis are defined as

  1. 1.

$\mathbb{E}[R]$: the annualized expected rate of linear trade returns.

  2. 2.

$Std(R)$: the annualized standard deviation of linear trade returns.

  3. 3.

Sharpe: a measure of risk-adjusted returns defined in equation 2.1. The risk-
free rate is assumed to be zero, and the annualized Sharpe ratio is thus
$\mathbb{E}[R]/Std(R)$.

  4. 4.

MDD: Maximum Drawdown (MDD), the maximum observed loss from any peak.

  5. 5.

Hit: the rate of positive trade returns.

#####  8.1.5 Dataset

The dataset consists of the front-month TTF Natural Gas Futures contracts from
2011 to 2022. The observations are sampled according to the transacted euro
volume on the exchange, defined in section 5.2. Larger sample sizes are
desirable to ensure statistical significance, especially for highly
overparameterized approximators such as neural networks. In addition,
predictability is generally higher over shorter horizons [Isi21]. However, as
sampling frequency (and therefore trading frequency) increases, simplifying
assumptions, such as no impact and perfect liquidity, become increasingly
consequential. Thus, an appropriate target number of samples per day is
$tgt=5$, which provides a little over $20\;000$ total samples. The data
processing is limited to what is described in section 7.1.

The first quarter of the dataset consisting of trades from 01/01/2011 to
01/01/2014 makes up the training set. The validation set is the second quarter
of the dataset from 01/01/2014 to 01/01/2017. Finally, the test set is the
second half of the dataset from 01/01/2017 to 01/01/2023. Figure 8.1
illustrates the training-validation-test split.

Figure 8.1: The training-validation-test split

####  8.2 Results

This section presents the results of the experiments described in the previous
section. The models are tested using four different values of the risk-
sensitivity term $\lambda_{\sigma}$ ($0$, $0.01$, $0.1$, and $0.2$), and the
results of all four values are presented. The results are visualized using
three tools; a table and two types of plots, and they are briefly described
below

  * •

The table consists of the performance metrics (described in section 8.1.4) of
each model (described in section 8.1) from the backtests.

  * •

A standard line plot illustrates the performance of the models against the
baseline indexed in time of the cumulative product of logarithmic trade
returns, where the trade returns are defined in equation 8.1.

  * •

A boxplot illustrates the distribution of the monthly logarithmic
returns121212Again, trade returns are defined in equation 8.1 and resampled to
produce monthly values. The logarithmic monthly returns are then calculated
based on these values. of each model and the baseline. Boxplots summarize a
distribution by its sampled median, the first quantile ($Q_{1}$), and the
third quantile ($Q_{3}$), represented by the box. The upper whisker extends to
the largest observed value within $Q_{3}+\frac{3}{2}IQR$, and the lower
whisker extends to the smallest observed value within $Q_{1}-\frac{3}{2}IQR$,
where the interquartile range (IQR) is $Q_{3}-Q_{1}$. Dots represent all
values outside of the whiskers (outliers).

The plots display the performance of all models and the baseline and are
grouped by risk-sensitivity terms.

Table 2 below shows the results of the backtests averaged over $10$ runs. The
variation between runs was small enough to warrant the level of precision of
the results given in the table.

|  $\mathbb{E}[R]$ |  $Std(R)$ |  Sharpe |  MDD |  Hit
---|---|---|---|---|---
Buy & Hold |  $0.271$ |  $0.721$ |  $0.376$ |  $0.877$ |  $0.524$
$\lambda_{\sigma}=0$
PG-CNN |  $\mathbf{0.403}$ |  $0.558$ |  $\mathbf{0.722}$ |  $0.753$ |  $0.529$
PG-LSTM |  $0.297$ |  $\mathbf{0.502}$ |  $0.591$ |  $0.726$ |  $0.527$
AC-CNN |  $0.302$ |  $0.610$ |  $0.495$ |  $0.724$ |  $0.538$
AC-LSTM |  $0.226$ |  $0.694$ |  $0.325$ |  $\mathbf{0.637}$ |  $\mathbf{0.541}$
Average |  $0.307$ |  $0.591$ |  $0.533$ |  $0.710$ |  $0.534$
$\lambda_{\sigma}=0.01$
PG-CNN |  $\mathbf{0.401}$ |  $0.437$ |  $\mathbf{0.918}$ |  $0.665$ |  $0.537$
PG-LSTM |  $0.258$ |  $0.326$ |  $0.791$ |  $0.540$ |  $0.526$
AC-CNN |  $0.346$ |  $0.471$ |  $0.735$ |  $0.601$ |  $\mathbf{0.545}$
AC-LSTM |  $0.251$ |  $\mathbf{0.300}$ |  $0.837$ |  $\mathbf{0.443}$ |  $0.535$
Average |  $0.314$ |  $0.383$ |  $0.820$ |  $0.562$ |  $0.536$
$\lambda_{\sigma}=0.1$
PG-CNN |  $\mathbf{0.371}$ |  $0.356$ |  $\mathbf{1.042}$ |  $0.591$ |  $0.537$
PG-LSTM |  $0.235$ |  $0.264$ |  $0.890$ |  $0.373$ |  $0.524$
AC-CNN |  $0.091$ |  $0.239$ |  $0.380$ |  $0.392$ |  $\mathbf{0.539}$
AC-LSTM |  $0.110$ |  $\mathbf{0.190}$ |  $0.579$ |  $\mathbf{0.261}$ |  $0.525$
Average |  $0.202$ |  $0.262$ |  $0.723$ |  $0.404$ |  $0.531$
$\lambda_{\sigma}=0.2$
PG-CNN |  $\mathbf{0.243}$ |  $0.298$ |  $\mathbf{0.815}$ |  $0.410$ |  $0.533$
PG-LSTM |  $0.179$ |  $0.247$ |  $0.725$ |  $0.373$ |  $\mathbf{0.537}$
AC-CNN |  $0.136$ |  $\mathbf{0.198}$ |  $0.687$ |  $0.454$ |  $0.531$
AC-LSTM |  $0.114$ |  $0.229$ |  $0.498$ |  $\mathbf{0.341}$ |  $0.522$
Average |  $0.168$ |  $0.243$ |  $0.681$ |  $0.394$ |  $0.531$

Table 2: Backtest results

A pair of plots (line plot and boxplot) are grouped by risk-term values ($0$,
$0.01$, $0.1$, and $0.2$, respectively).

Figure 8.2: Cumulative logarithmic trade returns for $\lambda_{\sigma}=0$
Figure 8.3: Boxplot of monthly logarithmic trade returns for
$\lambda_{\sigma}=0$ Figure 8.4: Cumulative logarithmic trade returns for
$\lambda_{\sigma}=0.01$ Figure 8.5: Boxplot of monthly logarithmic trade
returns for $\lambda_{\sigma}=0.01$ Figure 8.6: Cumulative logarithmic trade
returns for $\lambda_{\sigma}=0.1$ Figure 8.7: Boxplot of monthly logarithmic
trade returns for $\lambda_{\sigma}=0.1$ Figure 8.8: Cumulative logarithmic
trade returns for $\lambda_{\sigma}=0.2$ Figure 8.9: Boxplot of monthly
logarithmic trade returns for $\lambda_{\sigma}=0.2$

####  8.3 Discussion of results

This section will discuss the experiment results and how they relate to the
three research questions posed at the start of this thesis.

#####  8.3.1 Risk/reward

The first research question posed at the start of this thesis was:

> Can the risk of algorithmic trading agents operating in volatile markets be
> controlled?

Note some general observations about the buy-and-hold baseline against which
the models are compared. The baseline mirrors the direction of the natural gas
market during the out-of-sample test. Due to the volatility and upwards price
pressure stemming from the energy crisis in 2021-2022, the buy-and-hold
baseline has a high annualized expected return but also a high annualized
standard deviation of returns. As a result, the Sharpe ratio, the primary
performance metric in this experiment, is relatively low. The primary goal of
the deep reinforcement learning models is to increase the Sharpe ratio.
Increasing the Sharpe ratio is achieved by increasing the expected returns or
decreasing the standard deviation. The reward function 5.10 defines this
trade-off, where the risk-sensitivity term $\lambda_{\sigma}$ functions as a
trade-off hyperparameter. Low values of $\lambda_{\sigma}$ make the agent more
risk-neutral, i.e., more concerned with increasing expected returns and less
concerned with decreasing the standard deviation of returns. Conversely, high
values of $\lambda_{\sigma}$ make the agent more risk-averse, i.e., more
concerned with decreasing the standard deviation of returns and less concerned
with increasing the expected returns. The experiments in this thesis use four
risk-sensitivity terms; $0$, $0.01$, $0.1$, and $0.2$. For
$\lambda_{\sigma}=0$, the agent is risk-neutral and only concerned with
maximizing the expected return. For values exceeding $0.1$, the agent becomes
so risk-averse that it hardly participates in the market. This trade-off is
evident in the results where the annualized expected return and the standard
deviation are on average $83\%$ and $143\%$ higher, respectively, for
$\lambda_{\sigma}=0$ compared to $\lambda_{\sigma}=0.2$. The boxplots (figures
8.3, 8.5, 8.7, 8.9) illustrate how the monthly logarithmic returns are more
concentrated as $\lambda_{\sigma}$ increase. This phenomenon is also observed
in the line plots (figures 8.2, 8.4, 8.6, 8.8), where it is apparent that the
variability of returns decreases as $\lambda_{\sigma}$ increases. The agent’s
action, i.e., the position size (and direction), is its only means to affect
the reward. By definition, a risk-averse agent will choose outcomes with low
uncertainty, so a preference for smaller positions is a natural consequence of
increasing risk sensitivity. Consequently, the risk-averse deep RL agents have
a significantly lower maximum drawdown than the baseline, but they do not
fully capitalize on the increasing prices from mid-2020 to 2023.

Averaged over all the deep RL models for all risk-sensitivity terms, they
produce $83\%$ higher Sharpe than the baseline. This gain comes from reducing
the standard deviation of returns, which is reduced by $49\%$, whereas the
return is only slightly reduced by $8\%$. Looking at the various risk-
sensitivity terms, the risk-neutral agents (i.e., those where
$\lambda_{\sigma}=0$), on average, increase the returns by $13\%$ compared to
the baseline. Although they have no risk punishment, they also decrease the
standard deviation of returns by $18\%$. This last point is surprising but
could be a byproduct of an intelligent agent trying to maximize returns. For
$\lambda_{\sigma}=0.01$, the deep RL agents, on average, produce $16\%$
increased returns and $47\%$ reduced standard deviation of returns compared to
the baseline. This combination results in a $118\%$ higher Sharpe ratio. For
$\lambda_{\sigma}=0.1$, the agents on average produce $25\%$ lower returns;
however, the standard deviation of returns is reduced more by $64\%$. Thus,
the Sharpe is increased by $92\%$ compared to the baseline. The most risk-
averse agents (i.e., those where $\lambda_{\sigma}=0.2$) on average produce
$38\%$ lower returns with $66\%$ lower standard deviation of returns, yielding
an $83\%$ increase in Sharpe compared to the baseline. The risk-sensitivity
term $\lambda_{\sigma}=0.01$ produces the highest Sharpe ratio on average.
Thus, the backtests suggest that of the four risk-sensitivity options tested
in this thesis, $\lambda_{\sigma}=0.01$ strikes the best risk/reward balance.

#####  8.3.2 RL models

The second research question posed at the start of this thesis was:

> What reinforcement learning algorithms are suitable for optimizing an
> algorithmic training agent in an online, continuous time setting?

A curious result from the experiment is that, for three out of four risk-
sensitivity terms131313$\lambda_{\sigma}=0$, $0.01$, and $0.1$, the model with
the highest hit-rate has the lowest Sharpe. In other words, the model making
the highest rate of profitable trades also produces the lowest risk-adjusted
returns. This result illustrates the complexity of trading financial markets
and justifies the methods chosen in this thesis. Firstly, there is no
guarantee that a higher percentage of correct directional calls will result in
higher returns. Therefore, a forecast-based supervised learning approach
optimized for making correct directional calls may not align with the
stakeholder’s ultimate goal of maximizing risk-adjusted returns. For an
algorithmic trading agent to achieve the desired results, e.g., making trades
that maximize the Sharpe ratio, it should be optimized directly for that
objective. However, doing so in a supervised learning setting is not
straightforward. Reinforcement learning, on the other hand, provides a
convenient framework for learning optimal sequential behavior under
uncertainty. Furthermore, discrete position sizing, a drawback of value-based
reinforcement learning, can expose the agent to high risks. However, the agent
can size positions based on confidence through the continuous action space
offered by policy gradient methods, allowing for more effective risk
management. Section 2.6 presented research arguing that, for algorithmic
trading, reinforcement learning is superior to supervised learning and policy
gradient methods are superior to value-based methods, and this result supports
those arguments.

Although previous research supported policy gradient methods, there was no
consensus on which one was superior in this context. Chapter 6 presented two
policy gradient methods: one based on an actor-only framework and the other
based on an actor-critic framework, and discussed their respective advantages
and disadvantages. The previous section (8.2) presented results from the
backtests where both algorithms were tested out-of-sample. For all risk-
sensitivity terms, the direct policy gradient algorithm, on average, produces
a $49\%$ higher Sharpe ratio than the deterministic actor-critic algorithm.
Comparing the two algorithms using the same network architecture and risk
sensitivity term reveals that the actor-based algorithm outperforms the actor-
critic-based algorithm in 7 out of 8 combinations. The only case where the
actor-critic-based algorithm performs better141414PG-LSTM vs. AC-LSTM for
$\lambda_{\sigma}=0.01$ is the case with the smallest performance gap.
Furthermore, the actor-only direct policy gradient method strictly increases
the Sharpe ratio for both network architectures as the risk-sensitivity
parameter $\lambda_{\sigma}$ increases to a maximum at $\lambda_{\sigma}=0.1$.
The actor-critic method does not follow this pattern, suggesting it fails to
achieve its optimization objective.

The performance gap between the actor-based and actor-critic-based algorithms
is significant enough to warrant a discussion. An explanation for the
performance gap could be that the actor-critic-based algorithm optimizes the
policy using a biased Q-network reward estimate instead of the observed
unbiased reward. As a data-generating process, the commodity market is complex
and non-stationary. If the Q-network closely models the data-generating
distribution, using reward estimates from sampled experience from a replay
memory is an efficient method for optimizing the policy. On the other hand, it
is also clear that a policy that is optimized using Q-network reward estimates
that are inaccurate will adversely affect performance. The direct policy
gradient algorithm optimizes the policy using the observed unbiased reward and
avoids this problem altogether. Given that the reward function is exactly
expressed, optimizing it directly, as the direct policy gradient method does,
is the most efficient approach. Many typical RL tasks work well with the
actor-critic framework, but the backtests indicate that financial trading is
not one of them.

#####  8.3.3 Networks

The third and final research question posed at the start of this thesis was:

> What deep learning architectures are suitable for modeling noisy, non-
> stationary financial data?

In the research presented in section 2.5, two types of deep learning
architectures stood out; the long short-term memory and the convolutional
neural network. Chapter 7 presented two types of parametric function
approximators based on the CNN- and LSTM-architecture, respectively. The
previous section (8.2) presented results from the backtests where both these
function approximators are tested out-of-sample. On average, the CNN-based
models produce over $5\%$ higher Sharpe than those based on the LSTM, which is
surprising, as LSTMs are generally viewed as superior in sequence modeling
and, due to their memory, are the preferred option when modeling POMDPs. In
contrast to the CNN, the LSTM can handle long-term dependencies, but it seems
the lookback window provides enough historical information for the CNN to make
trade decisions. However, the performance gap is not big enough to say
anything conclusive, and the LSTM outperforms the CNN in some tests, so it is
unclear which is most suitable.

One interesting observation is that the CNN-based models produce higher
returns and standard deviation of returns compared to the LSTM. On average,
the CNN-based models produce $37\%$ higher returns and $15\%$ higher standard
deviation of returns. From the line plots in figures 8.2, 8.4, 8.6, and 8.8,
it looks like a possible explanation for this is that the LSTM-based models
prefer smaller position sizes compared to the CNN-based models. One potential
reason for this phenomenon involves the difference in how the CNN and LSTM are
optimized. Generally speaking, the CNN-based model is far easier and quicker
to optimize than the LSTM-based model, partly due to batch norm, which in its
conventional form is incompatible with RNNs. Another reason is that when the
LSTM is trained for long sequences, the problem of vanishing gradients makes
back-propagating the error difficult and slow. Increasing the learning rate
leads to exploding gradients. The CNN-based model with batch norm quickly and
effectively adjusts its parameters to take full advantage of newly observed
information during out-of-sample tests. The LSTM-based model, on the other
hand, adjusts its parameters much slower. As a result, the actions it selects
often end up someplace in the middle of the action space causing smaller
position sizes, lower returns, and lower standard deviation of returns. For
that reason, the author of this thesis theorizes that the performance gap
between the CNN-based and LSTM-based models would increase with time.

####  8.4 Discussion of model

Following the discussion of the results, it is interesting to take a step back
and have a more general discussion of the model. This includes discussing the
strengths and weaknesses of the model, as well as potential applications and
limitations.

#####  8.4.1 Environment

Solving complex real-world problems with reinforcement learning generally
requires creating a simplified version of the problem that lends itself to
analytical tractability. Usually, this involves removing some of the frictions
and constraints of the real-world problem. In the context of financial
trading, the environment described in chapter 5 makes several simplifying
assumptions about the environment, including no market impact, no slippage,
the ability to purchase or sell any number of contracts at the exact quoted
price, no additional costs or restrictions on short-selling, and fractional
trading. It is imperative to note that these assumptions do not necessarily
reflect real-world conditions. As such, it is crucial to know the problem
formulation’s limitations and how it will negatively affect the model’s
generalizability to the real-world problem. Poorly designed environments,
where agents learn to exploit design flaws rather than the actual problem, are
a frequent problem in reinforcement learning [SB18]. At the same time, these
simplifying assumptions allow for a clean theoretical analysis of the problem.
Furthermore, the environment introduces some friction through transaction
costs, an improvement over many existing models.

Lookahead bias in the input data is avoided by using the price series alone as
input, as described in section 7.1. The price series of a financial instrument
is generally the most reliable predictor of future prices. However, price
series only provide a limited view of the market and do not consider the
broader economic context and the potential impact of external factors. As a
result, an agent relying solely on price series may miss out on meaningful
predictive signals. Furthermore, since the model learns online, an effective
data governance strategy is required to ensure the quality and integrity of
the real-time input data stream, as data quality issues can harm the model’s
performance. The dollar bars sampling scheme described in section 5.2 has
solid theoretical foundations for improving the statistical properties of the
sub-sampled price series compared to traditional time-based sampling. When
using this sampling scheme, however, the agent cannot be certain of the
prediction horizon, which complicates forecasting.

Commodity trading firms often conduct asset-backed trading in addition to
paper trading, which incurs additional costs for booking pipeline capacity,
storage capacity, or LNG tankers. The model does not currently include these
costs, but the environment could be adjusted to include them.

#####  8.4.2 Optimization

Statistical learning relies on an underlying joint feature-target distribution
$F(x,y)$, with non-vanishing mutual information. The algorithmic trading agent
approximates this function by learning the distribution through historical
data. As financial markets are nonstationary, statistical distributions
constantly change over time, partly because market participants learn the
market dynamics and adjust their trading accordingly. In order to remain
relevant for the near future, the model must be continuously refitted using
only data from the immediate past, at the expense of statistical significance.
On the other hand, training a complex model using only a relatively small set
of recent data is challenging in a high-noise setting such as financial
forecasting, often resulting in poor predictive power. This tradeoff between
timeliness and statistical significance is known as the timeliness-
significance tradeoff [Isi21]. The timeliness-significance tradeoff highlights
a central challenge in optimizing algorithmic trading models.

This thesis investigates the use of reinforcement learning in algorithmic
trading, a field traditionally dominated by supervised learning-based
approaches. Supervised learning is a straightforward method for easily labeled
tasks, such as forecasting financial prices. Reinforcement learning, on the
other hand, is better suited to complex problems, such as sizing positions,
managing risk and transaction costs. In fact, with only minor modifications,
the model outlined in this thesis can optimize an agent trading a portfolio of
arbitrary size. For this reason, reinforcement learning was chosen as the
algorithmic trading agents’ optimization framework. Temporal credit assignment
is one of the main strengths of reinforcement learning, making it ideal for
game playing and robotics, involving long-term planning and delayed reward. In
this problem, however, temporal credit assignment does not apply since trading
is an immediate reward process. Furthermore, the complexity of reinforcement
learning compared to supervised learning comes at a price. As well as
requiring a model of the environment in which the agent interacts and learning
to associate context with reward-maximizing actions, reinforcement learning
introduces added complexity by introducing the exploration-exploitation
tradeoff. With more moving parts, reinforcement learning can be significantly
more challenging to implement and tune and is generally less sample efficient
than supervised learning. The learning process is more convoluted as the agent
learns through reinforcement signals generated by interaction with the
environment and involves stochasticity. Consequently, the model can display
unstable behavior where the policy diverges, or the agent overfits to noise.
E.g., if the market experiences a sustained downward trend, the agent can be
deceived into believing that the market will continue to decline indefinitely.
As a result, the agent may adjust its policy to always short the market, which
will have disastrous effects once the market reverses. The phenomenon is
caused by the temporal correlation of sequential interactions between RL
agents and the market, and that reinforcement learning is sample inefficient,
making it difficult to obtain good gradient estimates. Replay memory can
ensure that gradient estimates are derived from a wide variety of market
conditions. However, replay memory introduces biased gradient estimates,
which, according to backtests, is a poor tradeoff. The timeliness-significance
tradeoff further complicates this problem of obtaining suitable gradient
estimates. A supervised learning framework is more straightforward and avoids
much of the complexity associated with reinforcement learning. Thus, it is
unclear whether reinforcement learning or supervised learning is the most
appropriate optimization framework for algorithmic trading.

#####  8.4.3 Interpretability and trust

Interpretability is the ability to determine the cause and effect of a model.
Due to their over-parameterized nature, deep neural networks possess a
remarkable representation capacity, enabling them to solve a wide range of
complex machine learning problems, but at the cost of being difficult to
interpret. Neural networks are black boxes, acceptable for low-risk commercial
applications such as movie recommendations and advertisements. Interpreting
these predictions is of little concern as long as the models have sufficient
predictive power. However, deep learning’s opaque nature prevents its adoption
in critical applications, as the failure of a commodity trading model could
result in substantial financial losses and threaten global energy security.
Setting aside people generally being risk-averse and unwilling to bet large
sums of money on a black box, is the aversion to applying deep learning in
critical domains such as commodity trading reasonable? Does understanding the
model even matter as long as it delivers satisfactory backtest performance?

This question can be answered by reviewing statistical learning theory.
Generally, machine learning models are tested under the assumption that
observations are drawn from the same underlying distribution, the data-
generating distribution, and that observations are IID. In this setting, the
test error serves as a proxy for the generalization error, i.e., the expected
error on new observations. However, the dynamics of the financial markets are
constantly changing. Fierce competition in financial markets creates a cycle
in which market participants attempt to understand the underlying price
dynamics. As market participants better understand market dynamics, they
adjust their trading strategies to exploit that knowledge, further changing
market dynamics. Due to the constantly changing dynamics of the market, models
that worked in the past may no longer work in the future as inefficiencies are
arbitraged away151515Volatility, for example, used to be a reliable indicator
of future returns [Isi21].. Therefore, it is important to be cautious when
interpreting backtest errors as generalization errors, as it is unlikely that
observations sampled at different points in time are drawn from the same
probability distribution. Even if one disregards all the flaws of a
backtest161616E.g., not accounting for market impact and lookahead bias., the
backtest, at best, only reflects performance on historical data. In no way is
this intended to discourage backtesting. However, naively interpreting
backtest performance as an assurance of future performance is dangerous.
Referring back to section 2.9; a backtest is a historical simulation of how
the model would have performed should it have been run over a past period.
Even exceptional results from the most flawlessly executed backtest can never
guarantee that the model generalizes to the current market. Furthermore, the
results should be interpreted cautiously if no ex-ante logical foundation
exists to explain them. Deep neural networks are highly susceptible to
overfitting to random noise when trained on noisy financial time series. It
is, however, difficult to determine if the agent has detected a legitimate
signal if the model is not interpretable. Even if the model detects a
legitimate signal in the backtests, other market participants may discover the
same signal and render the model obsolete. Again, determining this is
difficult without knowing what inefficiencies the model exploits, and
deploying it until it displays a sustained period of poor performance will be
costly.

In response to the question of whether or not understanding a model matters if
it performs well on a backtest, the answer is an emphatic yes. Blindly taking
backtest performance of a black box as an assurance of future performance in a
noisy and constantly changing environment can prove costly. Thus, the aversion
to adopting deep learning in algorithmic trading is reasonable. Ensuring that
the trading decisions are explainable and the models are interpretable is
essential for commercial and regulatory acceptance. To address this challenge,
models should be created with a certain degree of interpretability. This way,
stakeholders can get insight into which inefficiencies the model exploits,
evaluate its generalizability, and identify its obsolescence before incurring
significant losses. The use of deep learning in algorithmic trading can still
be viable with techniques such as explainable AI and model monitoring.

###  9 Future work

The methods presented in this thesis leave room for improvement in further
work. More investigation should be done to evaluate the effectiveness of
existing methods in different contexts. Further investigation will enable a
deeper understanding of the model and its generalizability and provide an
opportunity to identify potential areas for improvement. Considering the lack
of real-world market data, one option is to use generative adversarial
networks (GANs) to generate synthetic markets [YJVdS19]. GANs can generate
unlimited data, which can be used to train and test the model and its
generalizability. Additionally, the lack of established baselines could be
improved upon. While the buy-and-hold baseline is well understood and trusted,
it is unrealistic in this context, as futures contracts expire. Although it
presents its own challenges, developing a baseline more appropriate for
futures trading would improve the current model. Furthermore, a greater level
of interpretability is required to achieve real-world adoption. Therefore,
combining algorithmic trading research with explainable AI is imperative to
improve existing methods’ interpretability.

Incorporating non-traditional data sources, such as social media sentiment or
satellite images, may prove beneficial when forecasting market returns.
Alternative data can provide a more comprehensive and holistic view of market
trends and dynamics, allowing for more accurate predictions. By leveraging
alternative data, algorithmic trading agents can gain an edge over their
competitors and make better-informed decisions. Using deep learning techniques
such as natural language processing and computer vision to analyze text or
image data in an algorithmic trading context is promising. Neural networks are
generally effective in problems with complex structures and high signal-to-
noise ratios. Thus, it may be more appropriate to use deep learning to extract
features from images or text rather than analyzing price series.

Lastly, the methods presented in this thesis are limited to trading a single
instrument. They are, however, compatible with portfolio optimization with
minimal modifications. Further research in this area would be interesting, as
it better utilizes the potential of the reinforcement learning framework and
the scalability of data-driven decision-making.

###  10 Conclusion

This thesis investigates the effectiveness of deep reinforcement learning
methods in commodities trading. Previous research in algorithmic trading,
state-of-the-art reinforcement learning, and deep learning algorithms was
examined, and the most promising methods were implemented and tested. This
chapter summarizes the thesis’ most important contributions, results, and
conclusions.

This thesis formalizes the commodities trading problem as a continuing
discrete-time stochastic dynamical system. The system employs a novel time-
discretization scheme that is reactive and adaptive to market volatility,
providing better statistical properties of the sub-sampled financial time
series. Two policy gradient algorithms, an actor-based and an actor-critic-
based, are proposed to optimize a transaction-cost- and risk-sensitive agent.
Reinforcement learning agents parameterized using deep neural networks,
specifically CNNs and LSTMs, are used to map observations of historical prices
to market positions.

The models are backtested on the front month TTF Natural Gas futures contracts
from 01-01-2017 to 01-01-2023. The backtest results indicate the viability of
deep reinforcement learning agents in commodities trading. On average, the
deep reinforcement learning agents produce an $83\%$ higher Sharpe ratio out-
of-sample than the buy-and-hold baseline. The backtests suggest that deep RL
models can adapt to the unprecedented volatility caused by the energy crisis
during 2021-2022. Introducing a risk-sensitivity term functioning as a trade-
off hyperparameter between risk and reward produces satisfactory results,
where the agents reduce risk as the risk-sensitivity term increases. The risk-
sensitivity term allows the stakeholder to control the risk of an algorithmic
trading agent in volatile markets. The direct policy gradient algorithm
produces significantly higher Sharpe ($49\%$ on average) than the
deterministic actor-critic algorithm, suggesting that an actor-based policy
gradient method is more suited to algorithmic trading in an online, continuous
time setting. The parametric function approximator based on the CNN
architecture performs slightly better ($5\%$ higher Sharpe on average) than
the LSTM, possibly due to the problem of vanishing gradients for the LSTM.

The algorithmic trading problem is made analytically tractable by simplifying
assumptions that remove market frictions. Performance may be inflated due to
these assumptions and should be viewed with a high degree of caution.

### Acronyms

AC

    Actor-Critic
AMH

    Adaptive Market Hypothesis
ANN

    Artificial Neural Network
CNN

    Convolutional Neural Network
CV

    Cross-Validation
DL

    Deep Learning
DNN

    Deep Neural Network
DQN

    Deep Q-Network
DRQN

    Deep Recurrent Q-Network
EMH

    Efficient Market Hypothesis
FFN

    Feedforward Network
IID

    Independent and Identically Distributed
LSTM

    Long Short-Term Memory
MDD

    Maximum Drawdown
MDP

    Markov Decision Process
ML

    Machine Learning
MPT

    Modern Portfolio Theory
MSE

    Mean Squared Error
PG

    Policy Gradient
POMDP

    Partially Observable Markov Decision Process
ReLU

    Rectified Linear Unit
RL

    Reinforcement Learning
RNN

    Recurrent Neural Network
SGD

    Stochastic Gradient Descent
SL

    Supervised Learning

### References

  * [AAC+19] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al.  Solving rubik’s cube with a robot hand.  arXiv preprint arXiv:1910.07113, 2019.
  * [AG00] Thierry Ané and Hélyette Geman.  Order flow, transaction clock, and normality of asset returns.  The Journal of Finance, 55(5):2259–2284, 2000.
  * [AHM19] Rob Arnott, Campbell R Harvey, and Harry Markowitz.  A backtesting protocol in the era of machine learning.  The Journal of Financial Data Science, 1(1):64–74, 2019.
  * [ARS+20] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al.  What matters in on-policy reinforcement learning? a large-scale empirical study.  arXiv preprint arXiv:2006.05990, 2020.
  * [BHMM19] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias–variance trade-off.  Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019.
  * [BSF94] Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  IEEE transactions on neural networks, 5(2):157–166, 1994.
  * [CGLL19] Raymond H Chan, Yves ZY Guo, Spike T Lee, and Xun Li.  Financial Mathematics, Derivatives and Structured Products.  Springer, 2019.
  * [Cla73] Peter K Clark.  A subordinated stochastic process model with finite variance for speculative prices.  Econometrica: journal of the Econometric Society, pages 135–155, 1973.
  * [CLR+21] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  Advances in neural information processing systems, 34:15084–15097, 2021.
  * [Cyb89] George Cybenko.  Approximation by superpositions of a sigmoidal function.  Mathematics of control, signals and systems, 2(4):303–314, 1989\.
  * [DBK+16] Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai.  Deep direct reinforcement learning for financial signal representation and trading.  IEEE transactions on neural networks and learning systems, 28(3):653–664, 2016.
  * [DHS11] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of machine learning research, 12(7), 2011.
  * [DP18] Marcos Lopez De Prado.  Advances in financial machine learning.  John Wiley & Sons, 2018.
  * [eur] A european green deal.  https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/european-green-deal_en.
  * [Fam70] Eugene F Fama.  Efficient capital markets: A review of theory and empirical work.  The journal of Finance, 25(2):383–417, 1970.
  * [FFLa] Danfei Xu Fei-Fei Li, Ranjay Krishna.  Lecture 10: Recurrent neural networks.  http://cs231n.stanford.edu/slides/2020/lecture_10.pdf.
  * [FFLb] Danfei Xu Fei-Fei Li, Ranjay Krishna.  Lecture 4: Neural networks and backpropagation.  http://cs231n.stanford.edu/slides/2020/lecture_4.pdf.
  * [FFLc] Danfei Xu Fei-Fei Li, Ranjay Krishna.  Lecture 5: Convolutional neural networks.  http://cs231n.stanford.edu/slides/2020/lecture_5.pdf.
  * [Fis18] Thomas G Fischer.  Reinforcement learning in financial markets-a survey.  Technical report, FAU Discussion Papers in Economics, 2018.
  * [GBC16] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [GD34] Benjamin Graham and David Dodd.  Security analysis.  McGraw-Hill Book Co., 1934.
  * [HA18] Rob J Hyndman and George Athanasopoulos.  Forecasting: principles and practice.  OTexts, 2018.
  * [Has07] Joel Hasbrouck.  Empirical market microstructure: The institutions, economics, and econometrics of securities trading.  Oxford University Press, 2007.
  * [HGMS18] Ma Hiransha, E Ab Gopalakrishnan, Vijay Krishna Menon, and KP Soman.  Nse stock market prediction using deep-learning models.  Procedia computer science, 132:1351–1362, 2018.
  * [Hor91] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural networks, 4(2):251–257, 1991.
  * [HS97] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [HS15] Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  In 2015 aaai fall symposium series, 2015.
  * [HTFF09] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  The elements of statistical learning: data mining, inference, and prediction, volume 2.  Springer, 2009.
  * [Hua18] Chien Yi Huang.  Financial trading as a game: A deep reinforcement learning approach.  arXiv preprint arXiv:1807.02787, 2018.
  * [HZRS15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [Int20] Mordor Intelligence.  Algorithmic trading market—growth, trends, covid-19 impact, and forecasts (2021-2026), 2020.
  * [IS15] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [Isi21] M Isichenko.  Quantitative portfolio management: The art and science of statistical arbitrage, 2021.
  * [JES16] Olivier Jin and Hamza El-Saawy.  Portfolio management using reinforcement learning.  Stanford University, 2016.
  * [JXL17] Zhengyao Jiang, Dixing Xu, and Jinjun Liang.  A deep reinforcement learning framework for the financial portfolio management problem.  arXiv preprint arXiv:1706.10059, 2017.
  * [KB14] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [LB14] Michael Lewis and Dylan Baker.  Flash boys.  Simon & Schuster Audio, 2014.
  * [LHP+15] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [Lo04] Andrew W Lo.  The adaptive markets hypothesis.  The Journal of Portfolio Management, 30(5):15–29, 2004.
  * [LPW+17] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang.  The expressive power of neural networks: A view from the width.  Advances in neural information processing systems, 30, 2017.
  * [LXZ+18] Xiao-Yang Liu, Zhuoran Xiong, Shan Zhong, Hongyang Yang, and Anwar Walid.  Practical deep reinforcement learning approach for stock trading.  arXiv preprint arXiv:1811.07522, 2018.
  * [Man97] Benoit B Mandelbrot.  The variation of certain speculative prices.  In Fractals and scaling in finance, pages 371–418. Springer, 1997\.
  * [Mar68] Harry M Markowitz.  Portfolio selection.  In Portfolio selection. Yale university press, 1968.
  * [MBM+16] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * [MKS+13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [MM97] Tom M Mitchell and Tom M Mitchell.  Machine learning, volume 1.  McGraw-hill New York, 1997.
  * [MRC18] Sean McNally, Jason Roche, and Simon Caton.  Predicting the price of bitcoin using machine learning.  In 2018 26th euromicro international conference on parallel, distributed and network-based processing (PDP), pages 339–343. IEEE, 2018.
  * [MS01] John Moody and Matthew Saffell.  Learning to trade via direct reinforcement.  IEEE transactions on neural Networks, 12(4):875–889, 2001.
  * [MSA22] S Makridakis, E Spiliotis, and V Assimakopoulos.  The m5 accuracy competition: Results, findings and conclusions. 2020.  URL: https://www. researchgate. net/publication/344487258_The_M5_ Accuracy_competition_Results_findings_and_conclusions, 2022.
  * [MT67] Benoit Mandelbrot and Howard M Taylor.  On the distribution of stock price differences.  Operations research, 15(6):1057–1062, 1967.
  * [MW97] John Moody and Lizhong Wu.  Optimization of trading systems and portfolios.  In Proceedings of the IEEE/IAFE 1997 computational intelligence for financial engineering (CIFEr), pages 300–307. IEEE, 1997.
  * [MWLS98] John Moody, Lizhong Wu, Yuansong Liao, and Matthew Saffell.  Performance functions and reinforcement learning for trading systems and portfolios.  Journal of Forecasting, 17(5-6):441–470, 1998.
  * [Nar13] Rishi K Narang.  Inside the black box: A simple guide to quantitative and high frequency trading, volume 846.  John Wiley & Sons, 2013.
  * [Pet22] Martin Peterson.  The St. Petersburg Paradox.  In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2022 edition, 2022.
  * [Pla22] Aske Plaat.  Deep Reinforcement Learning.  Springer, 2022.
  * [PMR+17] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review.  International Journal of Automation and Computing, 14(5):503–519, 2017.
  * [RHW85] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation.  Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  * [SB18] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [SGO20] Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu.  Financial time series forecasting with deep learning: A systematic literature review: 2005–2019.  Applied soft computing, 90:106181, 2020.
  * [Sha98] William F Sharpe.  The sharpe ratio.  Streetwise–the Best of the Journal of Portfolio Management, pages 169–185, 1998.
  * [SHK+14] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [SHM+16] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [SHS+17] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  Mastering chess and shogi by self-play with a general reinforcement learning algorithm.  arXiv preprint arXiv:1712.01815, 2017.
  * [SLH+14] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In International conference on machine learning, pages 387–395. PMLR, 2014.
  * [SNN18] Sima Siami-Namini and Akbar Siami Namin.  Forecasting economics and financial time series: Arima vs. lstm.  arXiv preprint arXiv:1803.06386, 2018.
  * [SRH20] Simon R Sinsel, Rhea L Riemke, and Volker H Hoffmann.  Challenges and solution technologies for the integration of variable renewable energy sources—a review.  renewable energy, 145:2271–2285, 2020.
  * [SSS00] Robert H Shumway, David S Stoffer, and David S Stoffer.  Time series analysis and its applications, volume 3.  Springer, 2000.
  * [SSS+17] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [SVL14] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  Advances in neural information processing systems, 27, 2014.
  * [SWD+17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [Tal97] Nassim Nicholas Taleb.  Dynamic hedging: managing vanilla and exotic options, volume 64.  John Wiley & Sons, 1997.
  * [TOdSMJZ20] Danilo Tedesco-Oliveira, Rouverson Pereira da Silva, Walter Maldonado Jr, and Cristiano Zerbato.  Convolutional neural networks in predicting cotton yield from images of commercial fields.  Computers and Electronics in Agriculture, 171:105307, 2020.
  * [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [XNS15] Ruoxuan Xiong, Eric P Nichols, and Yuan Shen.  Deep learning stock volatility with google domestic trends.  arXiv preprint arXiv:1512.04916, 2015.
  * [YJVdS19] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar.  Time-series generative adversarial networks.  Advances in neural information processing systems, 32, 2019.
  * [YLZW20] Hongyang Yang, Xiao-Yang Liu, Shan Zhong, and Anwar Walid.  Deep reinforcement learning for automated stock trading: An ensemble strategy.  In Proceedings of the first ACM international conference on AI in finance, pages 1–8, 2020.
  * [ZBH+21] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.
  * [ZS10] Wenbin Zhang and Steven Skiena.  Trading strategies to exploit blog and news sentiment.  In Proceedings of the international AAAI conference on web and social media, volume 4, pages 375–378, 2010.
  * [Zuc19] Gregory Zuckerman.  The man who solved the market: How Jim Simons launched the quant revolution.  Penguin, 2019.
  * [ZZR20] Zihao Zhang, Stefan Zohren, and Stephen Roberts.  Deep reinforcement learning for trading.  The Journal of Financial Data Science, 2(2):25–40, 2020.
  * [ZZW+20] Yifan Zhang, Peilin Zhao, Qingyao Wu, Bin Li, Junzhou Huang, and Mingkui Tan.  Cost-sensitive portfolio selection via deep reinforcement learning.  IEEE Transactions on Knowledge and Data Engineering, 34(1):236–248, 2020.