# HyperRouter: Towards Efficient Training and Inference of
Sparse Mixture of Experts

Giang Do†, Khiem Le‡, Quang Pham${}^{\star,\textrm{{\char 0\relax}}}$,
TrungTin Nguyen⟂, Thanh-Nam Doan△,
Binh T. Nguyen♠, Chenghao Liu♣, Savitha Ramasamy⋆, Xiaoli Li⋆, Steven Hoi♢
⋆ Institute for Infocomm Research (I2R), A∗STAR, Singapore,
† University of Tennessee at Chattanooga, ‡ VinUniversity, △ Independent
Researcher,
⟂ Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble,
France,
♠ AISIA Lab, University of Science, Vietnam National University Ho Chi Minh
City,
♣ Salesforce Research Asia, ♢ Singapore Management University
${}^{\textrm{{\char 0\relax}}}$ Correspondence to<EMAIL_ADDRESS>
###### Abstract

By routing input tokens to only a few split experts, Sparse Mixture-of-Experts
has enabled efficient training of large language models. Recent findings
suggest that fixing the routers can achieve competitive performance by
alleviating the collapsing problem, where all experts eventually learn similar
representations. However, this strategy has two key limitations: (i) the
policy derived from random routers might be sub-optimal, and (ii) it requires
extensive resources during training and evaluation, leading to limited
efficiency gains. This work introduces HyperRouter, which dynamically
generates the router’s parameters through a fixed hypernetwork and trainable
embeddings to achieve a balance between training the routers and freezing them
to learn an improved routing policy. Extensive experiments across a wide range
of tasks demonstrate the superior performance and efficiency gains of
HyperRouter compared to existing routing methods. Our implementation is
publicly available at {https://github.com/giangdip2410/HyperRouter}.

##  1 Introduction

Recent years have witnessed tremendous successes of the Transformer model
(Vaswani et al., 2017) and its variants across a wide range of tasks, ranging
from natural language and speech processing Bao et al. (2022b); Gulati et al.
(2020), computer vision Dosovitskiy et al. (2021); Ruiz et al. (2021); Bao et
al. (2022a), reinforcement learning Chow et al. (2023), to life sciences Rives
et al. (2021). Since then, scaling up to larger models has become the
prevailing approach for advancing the state-of-the-art in pre-training and
finetuning tasks. However, training such large models comes with a high
computational cost Lin et al. (2022); therefore, there is a growing need to
develop efficient strategies that facilitate the training of large language
models (LLMs) Fedus et al. (2022a). One of the most effective strategies thus
far is the Sparse Mixture-of-Experts (SMoE) Shazeer et al. (2017); Fedus et
al. (2022b), which utilizes routers to direct each input token to a small
subset of network parameters (experts). SMoE improves both efficiency and
performance compared to dense training approaches Lewis et al. (2021); Artetxe
et al. (2022); Zhang et al. (2022); Du et al. (2022).

Figure 1: Perplexity (log-scaled) on the WikiText-103 dataset with varying
numbers of experts used for inference. All methods have the same FLOPs.

Despite the encouraging results, SMoE has been found to encounter the issue of
_representation collapse_ , where all experts converge to similar
representations Chi et al. (2022); Chen et al. (2022). This problem arises due
to the learning process of the router encouraging experts to cluster around a
centroid Chi et al. (2022). Therefore, significant efforts have been dedicated
to addressing the representation collapse issue while preserving the
simplicity and efficiency of SMoE training. To this end, one of the most
effective strategies is freezing the router, as demonstrated by SMoE-Dropout
Chen et al. (2023), where a randomly initialized router remains fixed
throughout the training process. Additionally, Chen et al. (2023) found that
progressively increasing the number of selected experts can be beneficial.
However, we argue that such a naive strategy exhibits two key limitations.
Firstly, the random routing policy may be sub-optimal, which hinders the
overall training process. Secondly, we will show in Section 3.3 that a fixed
router will restrict the model’s representation capabilities, necessitating a
progressive increase in the number of chosen experts to achieve satisfactory
performance. Therefore, SMoE-Dropout inherently suffers from limited
representation and does not offer efficiency gains during inference.

This work introduces a novel approach called HyperRouter to address the trade-
off between fixed and trainable routers in SMoE training. Instead of employing
a random but fixed router, HyperRouter utilizes a random but fixed
hypernetwork Ha et al. (2017) to generate the router’s parameters based on a
trainable router embedding. By doing so, HyperRouter can improve the routing
policy throughout training while mitigating the representation collapse
issues. To demonstrate its effectiveness, we conduct extensive evaluations on
various NLP tasks, comparing HyperRouter to several state-of-the-art SMoE
routing strategies. Moreover, HyperRouter achieves the same performance
threshold with fewer experts (compute) during inference, thereby significantly
enhancing the efficiency of deploying LLMs in real-world applications. Fig. 1
shows that HyperRouter consistently outperforms other competitors when the
same number of experts is used during inference.

##  2 Related Work

Existing Routing Mechanism for SMoE.  One of the most important components for
training SMoE is the expert routing strategy, which specifies experts to
process input tokens. There are two common classes of token-expert assignment
algorithms for SMoE training: (1) letting tokens select the top-$k$ experts
and (2) letting experts select the top-$k$ tokens. For the first approach,
(Fedus et al., 2022b, Switch Transformer), (Lepikhin et al., 2021, GShard),
(Zuo et al., 2022, THOR), (Lewis et al., 2021, BASE), (Clark et al., 2022,
S-BASE), and SMoE-Dropout (Chen et al., 2023) are representative methods.
Meanwhile, Zhou et al. (2022) introduced expert choice that enables selecting
different experts for each token and demonstrates the potential of the second
approach.

On the Representation Collapse of SMoE.  A major research focus is how to
improve the token-expert assignment to avoid the representation collapse issue
where: (i) all the inputs are routed to the same expert Zuo et al. (2022) or
(ii) all experts converge to similar representations Chi et al. (2022); Chen
et al. (2022). Such issues result in poor specialization among experts,
parameter redundancy, and limited performance gains. Existing works address
the issue (i) by employing various ad hoc-heuristics, e.g., adding Gaussian
noise Shazeer et al. (2017), limiting the maximum number of inputs that can be
routed to an expert Lepikhin et al. (2021), imposing a load balancing loss
Fedus et al. (2022b), using linear assignment Lewis et al. (2021), or
eliminating the necessity for router networks (Zuo et al. (2022) employed a
consistent regularized loss for stochastic expert assignment, Roller et al.
(2021) incorporated deterministic hashing assignments). To resolve the issue
(ii), Chi et al. (2022) proposed an X-MOE layer to improve the routing
algorithm via dimension reduction, $\ell_{2}$ normalization, and gating
temperature. Furthermore, Dai et al. (2022) proposed StableMoE with two
training stages to reduce the router’s fluctuations. The work related to ours
the most is SMoE-Dropout (Chen et al., 2023), which is considered a randomly
initialized and fixed router to route tokens. Thus, the routing policy is
stable during training and deterministic during inference. Our work improves
upon SMoE-Dropout by extending the fixed router to a hypernetwork to further
improve the routing policy during training while alleviating the collapsing
issues.

##  3 Methodology

This section describes SMoE training and details of the proposed HyperRouter
method.

###  3.1 SMoE Training

We first describe SMoE training of LLMs, which consists of a router
${\mathcal{R}}(\cdot)$ with parameter $W_{r}$ and $N$ expert networks
$\\{{\mathcal{E}}_{i}(\cdot)\\}_{i=1}^{N}$ with parameters $W_{e_{i}}$. We
follow the most common implementation Fedus et al. (2022b) to use a linear
network as the router and split the feedforward networks in LLMs into $N$
separated experts.

Switch Transformer.  Given an input token ${\bm{x}}$ with its representation
${\bm{h}}\in\mathbb{R}^{d}$, the SMoE output ${\bm{y}}$ is calculated by
routing ${\bm{h}}$ only to $k$-most suitable experts determined by the router,
i.e.,

${\bm{y}}={\sum_{j=1}^{N}{\mathcal{R}}({\bm{h}})_{j}\cdot{\mathcal{E}}_{j}({\bm{h}})}\quad\mathrm{and}$
(1) ${\mathcal{R}}({\bm{h}})=\mathrm{TopK}(\sigma(W_{r}\times{\bm{h}}),k),$

where the $\mathrm{TopK}(\cdot,k)$ function keeps the largest $k$ values of a
given vector while setting the remaining values to zero; $\sigma(\cdot)$ is
the standard Softmax function. In practice, only a few experts are activated
for the current token ($k\ll N$) for efficient training.

SMoE-Dropout.  SMoE-Dropout (Chen et al., 2023) is a state-of-the-art strategy
that addresses the representation collapse issues (Sec. 2) by simply fixing a
randomly initialized router to improve the token routing consistency.
Furthermore, SMoE-Dropout gradually increases the number of experts chosen
throughout training, starting with $k=2$ and ending at $k=N$. During the
evaluation, SMoE-Dropout proposes to use _half or all experts_ to balance its
efficiency and performance.

###  3.2 HyperRouter

Figure 2: An illustration of our HyperRouter that dynamically generates the
router’s parameters from a fixed hypernetwork. Yellow modules including the
router embedding and experts are trainable while the gray module, the
hypernetwork, is frozen. Best viewed in colors.

We describe our proposed HyperRouter strategy that balances between a random
but fixed router and a router optimized during training. HyperRouter employs a
fixed hypernetwork Ha et al. (2017) ${\mathcal{H}}(\cdot)$ to dynamically
generate the router’s parameters conditioned on a trainable router embedding
${\bm{e}}$. Specifically, HyperRouter obtains the output ${\bm{y}}$ as:

${\bm{y}}={\sum_{j=1}^{N}{\mathcal{R}}({\bm{h}})_{j}\cdot{\mathcal{E}}_{j}({\bm{h}})}\quad\mathrm{and}$
${\mathcal{R}}({\bm{h}})=\mathrm{TopK}(\sigma(W_{r}\times{\bm{h}}),k)\;\mathrm{and}\;W_{r}={\mathcal{H}}({\bm{e}}),$

where ${\bm{e}}$ is a low dimensional trainable vector associated with the
current layer. All parameters are jointly optimized to minimize the task loss.
Lastly, like SMoE-Dropout, HyperRouter also gradually increases the number of
activated experts throughout training, starting at $k=2$ and ending at $k=N$.
Fig. 2 visually illustrates our proposed HyperRouter method.

###  3.3 A Deeper Analysis of HyperRouter

We now investigate the representation capabilities of HyperRouter compared to
the naive SMoE and SMoE-Dropout. Due to space constraints, we only present the
main results here and provide the detailed calculations in Appendix B. First,
the naive SMoE jointly trains the router and expert parameters, leading to
entanglement and eventually the representation collapse issue Zuo et al.
(2022). SMoE-Dropout proposed to fix the router parameter $W_{r}$, which we
will show to result in a restricted representation. To do so, we need to
calculate the Jacobian with respect to ${\bm{h}}$, which characterizes how the
feature ${\bm{h}}$ is involved during training.

Jacobian of SMoE and SMoE-Dropout. Let
$S^{\text{SMOE}}_{j}({\bm{h}})=\sigma(W_{r}\times{\bm{h}})_{j}$,
$S^{\text{Hyper}}_{j}({\bm{h}})=\sigma({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{j}$,
$\mathbbm{1}_{ji}$ be the indicator function, and $W_{r,i}$ be the $i$-column
of $W_{r}$, the Jacobians for SMoE and SMoE-Dropout are calculated as:

$\displaystyle\nabla_{{\bm{h}}}{\mathcal{L}}=$
$\displaystyle{\bm{J}}_{1}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}+{\bm{J}}_{2}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}$
$\displaystyle=$
$\displaystyle{\bm{J}}_{1}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}+\sum_{i=1}^{k}c_{i}W_{r,i},\;\text{where
}$ (2) $\displaystyle c_{i}=$
$\displaystyle\sum_{j=1}^{k}S^{\text{SMOE}}_{j}({\bm{h}})\left(\mathbbm{1}_{ji}-S^{\text{SMOE}}_{i}({\bm{h}})\right)$
$\displaystyle\times$
$\displaystyle\left({\mathcal{E}}_{j}({\bm{h}})^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}\right).$

Here, the Jacobian $\nabla_{{\bm{h}}}{\mathcal{L}}$ is a combination of two
terms related to ${\bm{J}}_{1}$ and ${\bm{J}}_{2}$. The ${\bm{J}}_{1}$
component represents how ${\bm{h}}$ contributes to the output ${\bm{y}}$ and
is the same for all methods. The second component related to ${\bm{J}}_{2}$
characterizes learning better experts’ representation of ${\bm{h}}$ and is the
main difference. Since SMoE-Dropout fixes $W_{r}$, its
$\bar{{\bm{J}}_{2}}={\bm{J}}_{2}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}$ term is
expressed as a linear combination of columns in $W_{r}$, which lies in a lower
dimensional subspace because $W_{r}\in\mathbb{R}^{k\times d}\text{ with }k\ll
d$ \- number of experts is usually smaller than the feature dimension. Thus,
SMoE-Dropout alleviates the entanglement between the router and experts at the
cost of restricting the experts’ representation capabilities.

Jacobian of HyperRouter. By a similar calculation, the Jacobian of HyperRouter
is shown in equation 3.3. Given that ${\bm{e}}$ is trainable, HyperRouter’s
Jacobian is not expressed as a simple linear combination in a much lower-
dimensional subspace. In contrast, HyperRouter maps the feature to a subspace
whose dimension can be controlled by ${\bm{e}}$, which we can easily set to be
greater than $k$. Furthermore, by fixing the hypernetwork, the Jacobian does
not change freely as in SMoE, which helps alleviate the entanglement. Overall,
HyperRouter can alleviate the representation collapse while not sacrificing
the representation capabilities.

Jacobian of HyperRouter:

$\displaystyle\nabla_{{\bm{h}}}{\mathcal{L}}=$
$\displaystyle{\bm{J}}_{1}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}+{\bm{J}}_{2}^{\top}\nabla_{{\bm{y}}}{\mathcal{L}}$
$\displaystyle=$
$\displaystyle{\bm{J}}_{1}^{\top}\nabla_{{\bm{y}}}\mathcal{L}+\sum_{i=1}^{k}c_{i}\mathcal{H}_{i}(\mathbf{e}),~{}\text{where}$
(3) $\displaystyle{\bm{J}}_{1}=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\nabla_{{\bm{h}}}\left(\mathcal{E}_{j}({\bm{h}})\right),$
$\displaystyle c_{i}=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\left(\mathbbm{1}_{ji}-S^{\text{Hyper}}_{i}({\bm{h}})\right)$
$\displaystyle~{}~{}\times\left(\mathcal{E}_{j}({\bm{h}})^{\top}\nabla_{{\bm{y}}}\mathcal{L}\right).$

##  4 Experiments

We conduct experiments to investigate the following hypotheses. First,
HyperRouter improves efficiency compared to existing routing strategies,
requiring less computation to achieve similar performance thresholds during
evaluation. Second, with the same amount of experts used during inference,
HyperRouter can achieve better performances compared to other competitors.

###  4.1 Experiment Setting

Most of our experiments follow Chen et al. (2023) and consider a small
TransformerXL Dai et al. (2019) with four layers in our experiments due to the
limited resources. We also investigate the scalability by considering larger
variants of the TransformerXL with eight or twelve layers. We compare our
HyperRouter with several state-of-the-art routing strategies: (i) _SMoE_
(Fedus et al., 2022b) \- training with trainable routers; (ii) _THOR_ (Zhou et
al., 2022) \- replacing routers with a stochastic experts selection strategy
and a consistency regularizer; (iii) _SMoE-Dropout_ (Chen et al., 2023) \-
using a random router and gradually increasing the number of selected experts
during training. We also include two traditional baselines: (i) _Dense_ \- the
standard training of transformers where no routing mechanisms are implemented;
and (ii) _Dense+Dropout_ similar to _Dense_ but with Dropout Srivastava et al.
(2014) inserted in the fully connected layers. All baselines use the same
amount of trainable parameters, while our HyperRouter introduces a neglectable
1024 trainable parameters for the router embeddings.

We consider two training tasks. First, pre-training on the enwik8 Mahoney
(2011) and WikiText-103 Merity et al. (2017) datasets, where we follow the
same pre-training procedure as Chen et al. (2023) to train the small and
medium models for 400K iterations. We only train the large models for 100K
iterations due to resource constraints. Second, finetuning on the SST-2 Socher
et al. (2013), SST-5 Socher et al. (2013), IMDB Maas et al. (2011), and
BANKING77 Casanueva et al. (2020) datasets using the model pre-trained on
enwik8. Similar to Chen et al. (2023), we also perform dense finetuning
(always choosing all experts) for several epochs. In all experiments, we
report the evaluation metrics on the test set when using different numbers of
experts for routing methods. More implementation details and additional
results are provided in the Appendix.

###  4.2 Pre-training Result

Table 1: Bit-per-character and Perplexity on the enwik8 and WikiText-103 test
sets, respectively. Lower is better. $k$ denotes the number of experts chosen
during inference. The best results are in bold.  enwik8
---
$k$ | Dense | Dense+Dropout | SMoE | SMoE-Dropout | HyperRouter
1 | - | - | 7.20 | 3.02 | 1.48
2 | - | - | 3.30 | 1.61 | 1.25
4 | - | - | 1.68 | 1.29 | 1.19
8 | - | - | 1.32 | 1.19 | 1.17
16 | 1.25 | 1.16 | 1.27 | 1.16 | 1.16
WikiText-103
1 | - | - | 896.47 | 560.93 | 65.17
2 | - | - | 146.54 | 89.13 | 37.75
4 | - | - | 54.69 | 40.00 | 30.45
8 | - | - | 35.67 | 29.68 | 28.05
16 | 32.13 | 31.55 | 32.27 | 27.66 | 27.57

Tab. 1 reports the bit-per-character and perplexity on the enwik8 and
WikiText-103 datasets, respectively. First, we observe that SMoE-based
training offers improvements over Dense training. Furthermore, adding Dropout
can help with Dense training, which achieves comparable performances with
SMoE-Dropout, though its improvements diminish on the larger dataset of
WikiText-103. The benefit of SMoE training is the efficiency during inference
by using fewer experts. Our HyperRouter substantially outperforms both SMoE
and SMoE-Dropout on both datasets in this regime. Notably, HyperRouter
significantly outperforms SMoE-Dropout when using only one expert, reducing
BPC from 3.02 to 1.48 on enwik8, and perplexity from 560.93 to 65.17 on
WikiText-103. Overall, with eight experts or less, HyperRouter performs
competitively with SMoE-Dropout while only using half of the experts during
evaluation.

Table 2: Bit-per-character on the enwik8 test set using the TransformerXL medium and large models. Lower is better. $k$ denotes the number of experts chosen during inference, and SD denotes the SMoE-Dropout method. The best results are in bold.  | Medium TransformerXL | Large TransformerXL
---|---|---
k | SMOE | SD | HyperRouter | SMOE | SD | HyperRouter
1 | 2.170 | 3.741 | 1.457 | 1.370 | 1.766 | 1.215
2 | 1.395 | 1.642 | 1.201 | 1.216 | 1.298 | 1.181
4 | 1.260 | 1.257 | 1.156 | 1.189 | 1.225 | 1.174
8 | 1.230 | 1.167 | 1.137 | 1.182 | 1.197 | 1.171
16 | 1.226 | 1.141 | 1.132 | 1.181 | 1.183 | 1.170

Tab. 2 reports the BPC on the enwik8 dataset using the medium and large
TransformerXL. Notably, we only train the large model 100K iterations due to
resource constraints. With larger backbone networks, we observe the gap
between our HyperRouter and the baselines becomes more significant, indicating
our HyperRouter enjoys good scalability with the model complexity. Lastly, we
observe that SMoE-Dropout does not achieve satisfactory results with the large
backbone when only trained with 100K and is outperformed by the naive SMoE.
This result shows that SMoE-Dropout requires significant resources to achieve
good performance. On the other hand, our HyperRouter consistently outperforms
both baselines, regardless of the backbone size and number of experts
activated.

###  4.3 Finetuning Result

Table 3: Transfer performance (Accuracy) on the SST-2, SST-5, IMDB, and BANKING77 datasets. Higher is better. $k$ denotes the number of experts chosen during inference. The best results are in bold.  | SST-2 | SST-5 | IMDB | BANKING77
---|---|---|---|---
Dense | 82.18 | 39.72 | 88.30 | 84.54
Dense+Dropout | 81.94 | 39.95 | 88.37 | 84.34
SMoE-Dropout (k=8) | 81.37 | 41.17 | 87.78 | 87.40
HyperRouter (k=8) | 82.87 | 41.26 | 88.43 | 87.66
SMoE (k=16) | 79.98 | 39.45 | 88.16 | 86.52
SMoE-Dropout (k=16) | 81.83 | 41.30 | 87.81 | 87.89
HyperRouter (k=16) | 83.33 | 41.80 | 88.46 | 88.41

Tab. 3 reports the results of the finetuning experiment on the SST-2, SST-5,
IMDB, and BANKING77 datasets. Although we perform dense finetuning, we also
report the results of using only half the experts during evaluation. Overall,
we observe consistent accuracy gains from HyperRouter compared to other
baselines on all datasets. Notably, on SST-2 and IMDB datasets, HyperRouter
with only eight experts could substantially outperform other strategies when
using all experts.

###  4.4 Router Analysis

Table 4: Average entropy of the distribution of the routers on the enwik8 dataset. Lower is better. Method | Router 1 | Router 2 | Router 3 | Router 4
---|---|---|---|---
SMoE | 2.5164 | 2.2126 | 2.2424 | 2.2583
SMoE-Dropout | 2.5849 | 2.1624 | 2.2579 | 2.2690
HyperRouter | 1.4393 | 0.8894 | 1.1269 | 1.3477

We now investigate the distributional output from the routers (softmax of
router’s output) trained with different strategies. Such outputs determine how
tokens are assigned to experts. We hypothesize that high-entropy distributions
are not preferred since they are closer to being uniform, indicating the
router’s low confidence in choosing experts for the current token. In the
extreme, the router outputs a uniform distribution and assigns tokens to all
experts, eventually causing the collapse issue. Therefore, a router with a
lower entropy is preferred since it confidently assigns tokens to a few
experts, which can improve experts’ specialization. To this end, we report the
entropy of each router in the small TransformerXL trained on the enwik8
dataset in Tab. 4. The entropy is calculated on all samples in the test set
using the model obtained after training. The result clearly shows that
HyperRouter achieved much lower entropy than SMoE and SMoE-Dropout. Due to
space constraints, we will provide additional visualizations and discussions
in Appendix D.

##  5 Conclusion

In this work, we investigated the potentials and limitations of SMoE for
training LLMs and proposed HyperRouter to balance between two extremes of
SMoE: training and freezing the router. Consequently, HyperRouter can learn
better routing policies while alleviating the representation collapse of
conventional SMoE training. Experiments on both pre-training and fine-tuning
tasks demonstrated promising HyperRouter’s capabilities in facilitating
efficient, effective training and inference compared to state-of-the-art
strategies.

## Limitations

Our work focuses on the efficiency and efficacy of training LLMs using SMoE.
Despite the encouraging results, our experiments are conducted only on medium-
scale datasets with a small TransformerXL due to computation limitations.
Thus, further empirical evaluations are required to validate the scalability
of HyperRouter and other SMoE strategies on recent LLMs and larger datasets.

## Ethics Statement

Despite encouraging results, training large-scale LLMs is inevitably costly
and requires extensive computational resources, which need to be properly
managed. Moreover, our work used data collected on the web, which has been
known to suffer from gender and racial biases and requires additional efforts
to mitigate its negative impacts. Lastly, our study is a promising step
towards facilitating the development of new LLMs, which still requires careful
regularization to avoid potential misuses in harmful applications.

## Acknowledgement

This research/project is supported by the National Research Foundation,
Singapore, under its AI Singapore Programme (AISG Award No:
AISG2-RP-2021-027).

## References

  * Artetxe et al. (2022) Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. 2022.  Efficient Large Scale Language Modeling with Mixtures of Experts.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11699–11732, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bao et al. (2022a) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022a.  BEiT: BERT Pre-Training of Image Transformers.  In _International Conference on Learning Representations_.
  * Bao et al. (2022b) Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022b.  VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts.  In _Advances in Neural Information Processing Systems_.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45, Online. Association for Computational Linguistics.
  * Chang et al. (2020) Oscar Chang, Lampros Flokas, and Hod Lipson. 2020.  Principled Weight Initialization for Hypernetworks.  In _International Conference on Learning Representations_.
  * Chen et al. (2023) Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. 2023.  Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers.  In _The Eleventh International Conference on Learning Representations_.
  * Chen et al. (2022) Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022.  Towards Understanding the Mixture-of-Experts Layer in Deep Learning.  In _Advances in Neural Information Processing Systems_.
  * Chi et al. (2022) Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022.  On the Representation Collapse of Sparse Mixture of Experts.  In _Advances in Neural Information Processing Systems_.
  * Chow et al. (2023) Yinlam Chow, Azamat Tulepbergenov, Ofir Nachum, Dhawal Gupta, Moonkyung Ryu, Mohammad Ghavamzadeh, and Craig Boutilier. 2023.  A Mixture-of-Expert Approach to RL-based Dialogue Management.  In _The Eleventh International Conference on Learning Representations_.
  * Clark et al. (2022) Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc’Aurelio Ranzato, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. 2022.  Unified Scaling Laws for Routed Language Models.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 4057–4086. PMLR.
  * Dai et al. (2022) Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  StableMoE: Stable Routing Strategy for Mixture of Experts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7085–7095, Dublin, Ireland. Association for Computational Linguistics.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022.  GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 5547–5569. PMLR.
  * Fedus et al. (2022a) William Fedus, Jeff Dean, and Barret Zoph. 2022a.  A review of sparse expert models in deep learning.  _arXiv preprint arXiv:2209.01667_.
  * Fedus et al. (2022b) William Fedus, Barret Zoph, and Noam Shazeer. 2022b.  Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.  _Journal of Machine Learning Research_ , 23(120):1–39.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In _Proc. Interspeech 2020_ , pages 5036–5040.
  * Ha et al. (2017) David Ha, Andrew M. Dai, and Quoc V. Le. 2017.  HyperNetworks.  In _International Conference on Learning Representations_.
  * Lepikhin et al. (2021) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021.  GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.  In _International Conference on Learning Representations_.
  * Lewis et al. (2021) Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021.  BASE Layers: Simplifying Training of Large, Sparse Models.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 6265–6274. PMLR.
  * Lin et al. (2022) Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022.  A survey of transformers.  _AI Open_ , 3:111–132.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mahoney (2011) Matt Mahoney. 2011.  Large text compression benchmark.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017.  Pointer Sentinel Mixture Models.  In _International Conference on Learning Representations_.
  * Peng et al. (2015) Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015.  A comparative study on regularization strategies for embedding-based neural networks.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2106–2111.
  * Pham et al. (2022) Quang Pham, Chenghao Liu, Doyen Sahoo, and Steven CH Hoi. 2022.  Learning fast and slow for online time series forecasting.  _arXiv preprint arXiv:2202.11672_.
  * Rives et al. (2021) Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. 2021.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  _Proceedings of the National Academy of Sciences_ , 118.
  * Roller et al. (2021) Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E. Weston. 2021.  Hash Layers For Large Sparse Models.  In _Advances in Neural Information Processing Systems_.
  * Ruiz et al. (2021) Carlos Riquelme Ruiz, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021.  Scaling Vision with Sparse Mixture of Experts.  In _Advances in Neural Information Processing Systems_.
  * Shazeer et al. (2017) Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.  In _International Conference on Learning Representations_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Yin et al. (2021) Haiyan Yin, Ping Li, et al. 2021.  Mitigating forgetting in online continual learning with neuron calibration.  _Advances in Neural Information Processing Systems_ , 34:10260–10272.
  * Zhang et al. (2022) Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022.  MoEfication: Transformer Feed-forward Layers are Mixtures of Experts.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 877–890, Dublin, Ireland. Association for Computational Linguistics.
  * Zhou et al. (2022) Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. 2022.  Mixture-of-Experts with Expert Choice Routing.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 7103–7114. Curran Associates, Inc.
  * Zuo et al. (2022) Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022.  Taming Sparsely Activated Transformer with Stochastic Experts.  In _International Conference on Learning Representations_.

The supplementary document of our work is organized as follows. Appendix A
provides additional illustrations to compare our HyperRouter with other
strategies. Appendix B provides a detailed derivation of the Jacobians.
Appendix C provides the detailed experimental settings, additional results,
and analysis. Lastly, we conclude this work with a discussion on promising
future research venues in Appendix E.

##  Appendix A Additional Figures

(a) SMoE

(b) SMoE-Dropout

(c) HyperRouter

Figure 3: An illustrative comparison among SMoE, SMoE-Dropout, and our
HyperRouter . The input token is a representation vector (could be output from
the previous layer). Yellow modules are trainable. Gray modules are frozen,
indicated by a lock symbol (). Best viewed in colors.

Fig. 3 illustrates the conceptual difference between the traditional SMoE
(Fedus et al., 2022b), SMoE-Dropout (Chen et al., 2023), and our HyperRouter.
In summary, SMoE uses a trainable router to send tokens to experts. On the
other hand, SMoE-Dropout fixed a randomly initialized router and gradually
increased the number of experts chosen throughout training. Lastly,
HyperRouter improves upon SMoE-Dropout by replacing the router with a fixed
hypernetwork and trainable router embedding.

##  Appendix B Derivation of the Jacobian

This Section details the calculations of the Jacobians presented in Section.
3.3.

The Jacobian of SMoE and SMoE-Dropout is similar to the Jacobian of
HyperRouter. Therefore, only the later detailed calculation is shown. Recall
that HyperRouter obtains the output ${\bm{y}}$ as follows:

$\displaystyle{\bm{y}}=$
$\displaystyle\sum_{j=1}^{N}\mathcal{R}({\bm{h}})_{j}\cdot\mathcal{E}_{j}({\bm{h}}),$
$\displaystyle\mathcal{R}({\bm{h}})=$
$\displaystyle\mathrm{TopK}(\sigma(\mathcal{H}(\mathbf{e})\times{\bm{h}}),k).$

Here, $\sigma(\cdot)$ is the standard Softmax function given by:

$\displaystyle\sigma({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{j}=$
$\displaystyle\frac{\exp(({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{j})}{\sum_{i=1}^{N}\exp(({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{i})}$
$\displaystyle\equiv S^{\text{Hyper}}_{j}({\bm{h}}),~{}\forall j=1,\ldots,N.$

Without loss of generality and for simplicity, by rearranging the index of the
top $k$-experts, HyperRouter can be rewritten as follows:

$\displaystyle{\bm{y}}=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\cdot\mathcal{E}_{j}({\bm{h}}),~{}\text{where}$
$\displaystyle S^{\text{Hyper}}_{j}({\bm{h}})=$
$\displaystyle\frac{\exp(({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{j})}{\sum_{i=1}^{k}\exp(({\mathcal{H}}({\bm{e}})\times{\bm{h}})_{i})},$
$\displaystyle\forall$ $\displaystyle j=1,\ldots,k.$

The Jacobian matrix ${\bm{J}}$ of the output ${\bm{y}}$ with respect to
${\bm{h}}$ can be decomposed into two terms as follows:

$\displaystyle\nabla_{{\bm{h}}}{\bm{y}}=$
$\displaystyle\nabla_{{\bm{h}}}\left(\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\cdot\mathcal{E}_{j}({\bm{h}})\right)$
$\displaystyle=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\nabla_{{\bm{h}}}\left(\mathcal{E}_{j}({\bm{h}})\right)$
$\displaystyle~{}+\sum_{j=1}^{k}\nabla_{{\bm{h}}}\left(S^{\text{Hyper}}_{j}({\bm{h}})\right)\mathcal{E}_{j}({\bm{h}})$
$\displaystyle=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\nabla_{{\bm{h}}}\left(\mathcal{E}_{j}({\bm{h}})\right)$
$\displaystyle~{}+\sum_{j=1}^{k}\sum_{i=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\left(\mathbbm{1}_{ji}-S^{\text{Hyper}}_{i}({\bm{h}})\right)$
$\displaystyle~{}~{}\times\mathcal{E}_{j}({\bm{h}})\mathcal{H}_{i}(\mathbf{e})^{\top}\equiv{\bm{J}}_{1}+{\bm{J}}_{2}$
(4) (since $\mathbf{e}$ is independent of ${\bm{h}}$).

Note that the last equation in (4) is obtained by using the chain rule, the
logarithmic derivative, and the inner product as follows:

$\displaystyle\frac{\partial s_{j}}{\partial z_{i}}$
$\displaystyle=s_{j}\cdot\frac{\partial}{\partial
z_{i}}\log(s_{j})=s_{j}\cdot(\mathbbm{1}_{ji}-s_{i}),\text{ where}$
$\displaystyle s_{j}=$
$\displaystyle\frac{\exp(z_{j})}{\sum_{i=1}^{k}\exp(z_{i})},~{}\forall
j=1,\ldots,k.$

It is worth mentioning that the first term ${\bm{J}}_{1}$ means to produce a
better token representation given the current activation router
$S^{\text{Hyper}}_{j}({\bm{h}})$, while the second term ${\bm{J}}_{2}$
represents learning a better gating function for the appropriate activation
score router $S^{\text{Hyper}}_{j}({\bm{h}})$. After the back-propagation, the
gradient of the loss function $\mathcal{L}$ is obtained from the two paths
mentioned above and is written as follows

$\displaystyle\nabla_{{\bm{h}}}\mathcal{L}=$
$\displaystyle\nabla_{{\bm{y}}}\mathcal{L}\times\nabla_{{\bm{h}}}{\bm{y}}=\nabla_{{\bm{h}}}{\bm{y}}^{\top}\times\nabla_{{\bm{y}}}\mathcal{L}$
$\displaystyle=$
$\displaystyle({\bm{J}}_{1}+{\bm{J}}_{2})^{\top}\times\nabla_{{\bm{y}}}\mathcal{L}\quad\text{(using
(\ref{eq_nabla_y_h}))}$ $\displaystyle=$
$\displaystyle{\bm{J}}_{1}^{\top}\nabla_{{\bm{y}}}\mathcal{L}+{\bm{J}}_{2}^{\top}\nabla_{{\bm{y}}}\mathcal{L}.$
(5)

Finally, by expanding the second term as follows, we obtain the desired
result:

$\displaystyle{\bm{J}}_{2}^{\top}\nabla_{{\bm{y}}}\mathcal{L}=$
$\displaystyle\sum_{i=1}^{k}\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\left(\mathbbm{1}_{ji}-S^{\text{Hyper}}_{i}({\bm{h}})\right)$
$\displaystyle~{}~{}\times\left(\mathcal{E}_{j}({\bm{h}})^{\top}\nabla_{{\bm{y}}}\mathcal{L}\right)\mathcal{H}_{i}(\mathbf{e})$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{k}c_{i}\mathcal{H}_{i}(\mathbf{e}),~{}\text{where}$
(6) $\displaystyle c_{i}=$
$\displaystyle\sum_{j=1}^{k}S^{\text{Hyper}}_{j}({\bm{h}})\left(\mathbbm{1}_{ji}-S^{\text{Hyper}}_{i}({\bm{h}})\right)$
$\displaystyle~{}~{}\times\left(\mathcal{E}_{j}({\bm{h}})^{\top}\nabla_{{\bm{y}}}\mathcal{L}\right).$

##  Appendix C Additional Experiments

This section provides the implementation details of our experiments in Sec. 4.

###  C.1 General Setting

Our experiments are conducted based on the publicly available SMoE-Dropout
(Chen et al., 2023) implementation111https://github.com/VITA-Group/Random-MoE-
as-Dropout. Moreover, the pre-training experiments were conducted on a single
A40 or A100 GPU, while the finetuning experiments were conducted on a single
GeForce RTX 3090 GPU. We also emphasize that parallel training on multiple
GPUs might yield different results.

Model architecture.  The small TransformerXL variant (Chen et al., 2023)
consists of 4 Transformer decoder layers with an input dimension of 256. Each
layer consists of a self-attention layer with 8 attention heads, followed by a
feedforward network with an inner dimension of 512. The dropout ratio is kept
at 0.1. We split the feedforward network into 16 experts with the same
dimension. For the HyperRouter , we initialize the embeddings with size 256
and employ a 2-layer perceptron with an inner dimension of 256 as the
hypernetwork. The ReLU function is used as the activation function for the
hypernetwork. Totally our HyperRouter introduces an additional 1024 trainable
parameters in the TransformerXL model. It is worth noting that the parameter
overhead is fixed as we scale up to larger transformer models. For the medium
and large variants, we scale the model to eight and twelves layers,
respectively.

###  C.2 Pre-training Experiments

Tab. A1 provides the implementation details for pre-training our TransformerXL
small and medium on enwik8 and WikiText-103. The TransformerXL large network
was trained in the same maner, but only for 100K iterations.

Table A1: Implementation details for pre-training experimentson enwik8 and WikiText-103 datasets.  Dataset | Input length | Batch size | Optimizer | Lr | # Iterations
---|---|---|---|---|---
enwik8 | 512 | 22 | Adam | 2.5e-4 | 400000
WikiText-103 | 512 | 22 | Adam | 2.5e-4 | 400000

###  C.3 Finetuning Experiments

To conduct finetuning experiments, we use the same model architecture as in
pre-training. Tab. A2 provides the implementation details used for finetuning
experiments on four different datasets.

Table A2: Implementation details for finetuning experiments on four different datasets.  Dataset | Input length | Batch size | Optimizer | Lr | # Epochs
---|---|---|---|---|---
SST-2 | 512 | 16 | Adam | 1e-4 | 3
SST-5 | 512 | 16 | Adam | 1e-4 | 3
IMDB | 512 | 4 | Adam | 1e-4 | 3
BANKING77 | 512 | 16 | Adam | 1e-4 | 15

###  C.4 Inference Efficiency

We discuss an implementation detail that makes our HyperRouter have the same
efficiency during inference as SMoE and SMoE-Dropout. Particularly, before
performing inference on any input samples, we use the hypernetwork to generate
the router for each layer and then store them. During inference, HyperRouter
does not need to re-generate routers. Therefore, HyperRouter enjoys the same
inference complexity as SMoE and SMoE-Dropout because their routers have the
same dimensionality.

Tab. A3 reports the number of FLOPs of different methods during inference on
the enwik8 dataset according to a different number of experts chosen ($k$).
Note that the FLOPs do not scale linearly with $k$ since SMoE training is only
applied to the fully connected layers, while other layers (multi-head self-
attention, layer normalization, etc.) remain the same across baselines. In the
extreme, using only one expert only incurs 45% of the FLOPs compared to using
all 16 experts.

Table A3: Inference FLOPS ($10^{10}$) on the enwik8 dataset, $k$ denotes the number of experts used during inference.  k | Dense | Dense+Dropout | THOR | SMoE | SMoE-Dropout | HyperRouter
---|---|---|---|---|---|---
1 | - | - | - | 3.47 | 3.47 | 3.47
2 | - | - | - | 4.00 | 4.00 | 4.00
4 | - | - | - | 4.54 | 4.54 | 4.54
8 | - | - | - | 5.61 | 5.61 | 5.61
16 | 7.76 | 7.76 | 7.66 | 7.66 | 7.66 | 7.66

###  C.5 Parameter Comparison

Table A4: Number of parameters in different components of SMoE-Dropout and HyperRouter during training. Blue parameters are trainable, red parameters are frozen, and underline parameter are dynamically generated in each iteration.  Method | Router Emb. | Router | HyperNetwork | Transformer
---|---|---|---|---
SMoE-Dropout | $-$ | $16.448$ | $-$ | $19,505.350$
HyperRouter | $1.024$ | $16.448$ | $1,112.576$ | $19,505.350$

Tab. A4 provides the number of parameters in different components of SMoE-
Dropout and HyperRouter . There are three categories: (i) trainable parameters
which are the transformer backbone and the router embeddings; (ii) frozen
parameters which are the routers SMoE-Dropout and hypernetworks in HyperRouter
; (iii) dynamic parameters that are dynamically generated in each iteration,
such as the routers in HyperRouter . Overall, the additional trainable
parameters in HyperRouter is neglectable. Moreover, the number of frozen
parameters (hypernetworks) is quite small compared to the transformer backbone
($5.7\%$). Investigating into sharing the hypernetworks across layers or
generating the routers coordinately (Pham et al., 2022) can further reduce
this cost while improving the performance, which we will leave for the future
work.

##  Appendix D Routing Visualization

Table A5: Average entropy of the distribution of the routers’ output on the enwik8 dataset. Lower is better. Method | Router 1 | Router 2 | Router 3 | Router 4
---|---|---|---|---
SMoE | 2.5164 | 2.2126 | 2.2424 | 2.2583
|  $\pm$ 0.17 |  $\pm$ 0.31 |  $\pm$ 0.30 |  $\pm$ 0.25
SMoE-Dropout | 2.5849 | 2.1624 | 2.2579 | 2.2690
|  $\pm$ 0.11 |  $\pm$ 0.37 |  $\pm$ 0.28 |  $\pm$ 0.27
HyperRouter | 1.4393 | 0.8894 | 1.1269 | 1.3477
|  $\pm$ 0.42 |  $\pm$ 0.46 |  $\pm$ 0.46 |  $\pm$ 0.50

This Section provides the full details of the routers’ entropy and visualizes
its distributional output. This is supplementary to Section 4.4. Tab. A5
reports the mean and standard deviations of the entropy at each router. This
table is the full version of Tab. A5. We can see that all methods have rather
low standard deviation, indicating that the differences are significant.

We also provide an illustrative example of the routers’ outputs using a
randomly picked sample on the test set in Fig. 4. Here we can clearly see that
the policies from SMoE and SMoE-Dropout are much closer to uniform while
HyperRouter ’s policies are much sharper and have lower entropy. We emphasize
that this example is not cherry-picked since we already calculated the
averaged entropy on all samples in Tab. A5. Overall, this result shows
insights into how HyperRouter can perform better than other state-of-the-art
SMoE strategies.

---
(a) SMoE
(b) SMoE-Dropout
(c) HyperRouter
Figure 4: Visualization of the distribution of the routers’ output.

##  Appendix E Future Work

Our HyperRouter opens several promising venues for future research.
Particularly, we believe that investigating two HyperRouter components: (i)
fixed hypernetwork, and (ii) trainable embedding can yield further performance
and efficiency gains. Potentials directions include incorporating
regularization such as $\ell_{2}$-penalty or dropout (Peng et al., 2015) or
using better hypernetwork initialization techniques (Chang et al., 2020).
Furthermore, the current implementation uses a hypernetwork for each
transformer layer, and it generates all parameters of the router. Sharing
hypernetworks among layers or generating the router coordinate-wise can offer
knowledge sharing (Yin et al., 2021; Pham et al., 2022), which can further
improve the result.


# Ivancevic Option Pricing Model modulational instability through the
variational approach

Christopher Gaafele<EMAIL_ADDRESS>Department of
Physics and Astronomy, Botswana International University of Science and
Technology, Private Mail Bag 16 Palapye, Botswana

(June 2024)

###### Abstract

The instability of the Ivancevic option pricing model is studied through the
variational method. We have analytically derived the dispersion relation of
the IOPM for both constant volatility and Landau coefficient model and time-
dependent volatility and Landau coefficient model. Also the IOPM was studies
numerically using the 4th order Runge-Kutta method.

##  I Introduction

Since the discovery of the Nonlinear Schrodinger equation (NLS) like option
pricing model in 2010, it has drawn much interest theoretically in various
soliton phenomena in the financial world. Variuos techniques such as the
generalized Kudryashov method (GKM) demiray2022new , adomian decomposition
method (ADM) and spectral boundry approximation (SBA) method adama2022exact ,
improved physics-informed neural network (IPINN) bai2022application , the
method of Lie symmetries obaidullah2023analytical , the Elsaki-Adomian
decomposition method gonz2015solving , projected differential transform method
(PDTM) edeki2017analytical , He’s frecuency amplitude formulation
gonzalez2017solving , $q-homotopyanalysistransformmethod$ ($q$-HATM)
veeresha2022analysis  and modulational instability (MI) analysis
chen2022soliton  gaafele2024modulational  among others raheel2023research
wroblewski2023quantum .

Variational approach (VA) has also been a versatile method for nonlinear like
models for different branches of physics including optics
anderson1983variational  malomed1997pulse  malomed2002variational  and in
Bose-Einstein condesates sabari2013variational  wamba2008variational
wamba2014dynamical . In most studies this analysis has the same results as MI
and other method that are used for the NLS like models, in terms of soliton
dynamics and stability. Unlike MI, VA is suitable in discribing soliton pulse
propagation. VA can either use $Gaussian$ ansatz nicolin2008nonlinear  or
$sech$ ansatz malomed2002variational  among others since they are localize and
assumes the shape of a soliton. In this project we consider modulated wave
ansatz, which is the same wave ansatz used in modulational instability as
shown by wamba2008variational  sabari2013variational  wamba2014dynamical
ndzana2007modulational . The same can be done for the IOPM, and several
quantities such as the effective Hamiltonian and effective potantial can be
derived for the stock/asset market system, which is as important as MI in the
financial world chen2022soliton  gaafele2024modulational .

The IOPM is the NLS like model used in asset derivative markets alternative
approach to the traditional Black-Scholes (BS) model for pricing options.
Developed by Vladimir Ivancevic, this model incorporates elements of both
classical mechanics and financial mathematics ivancevic2010adaptive . It uses
a Hamiltonian formalism to describe the dynamics of the underlying asset price
and employs a different stochastic process than the standard geometric
Brownian motion used in the Black-Scholes model. The model incorporates
nonlinear dynamics, which can better capture the real-world behavior of
financial markets, including features such as volatility clustering and heavy
tails in the distribution of returns. Ivancevic’s approach sometimes draws
analogies from quantum mechanics, using path integrals to represent the
probabilities of different price paths kleinert2009path . This can provide a
more flexible and accurate representation of price movements over time. The
Black-Scholes equation is a linear partial differential equation, while the
Ivancevic model often results in nonlinear equations due to its more complex
stochastic processes. The IOPM can easily incorporate quantum neural
computations ivancevic2009quantum .

The remaining section are as follows: Section II will focus on the Ivancevic
option pricing model. Section III will focus on the variational calculation of
the model. Section IV shows direct simulation of the ansatz for different
conditions and the paper is concluded by Section V.

##  II The Model

The classic Black-Scholes (BS) model is a significant breakthrough in
financial mathematics, capturing the dynamic behavior of market prices for
financial assets, such as stock options, over time black1973pricing . This
model operates under certain assumptions, where variables such as the asset
price$s$, drift parameter $\mu$, and volatility $\sigma$ are considered
constants. Additionally, it presumes the existence of frictionless, arbitrage-
free, and efficient markets gonzalez2016nonlinear ,edeki2016modified .
However, to address the limitations of these assumptions, various
modifications have been suggested, including stochastic interest rate models,
jump-diffusion models, stochastic volatility models, and models incorporating
transaction costs. As a result, the Ivancevic option pricing model (IOPM) has
gained increasing attention in recent years for its effectiveness in
estimating option values, serving as an alternative to the Black-Scholes
equationchen2022soliton ,ali2023physical ,elmandouh2022integrability
,chen2021dark ,gonzalez2017solving . The price function $s=s(t)$ for $0\leq
t\leq T$ and follows the geometric brownian motion (BM) $ds=s(\mu dt+\sigma
dW_{t})$, where $\mu$ is the drift rate, $W_{t}\equiv W(t)$ is the Weiner
process. The BS model is given by

$\frac{\partial\psi}{\partial t}+rs\frac{\partial\psi}{\partial
s}+\frac{1}{2}\sigma^{2}s^{2}\frac{\partial^{2}\psi}{\partial s^{2}}-r\psi=0.$
(1)

Although there are versions of the Black-Scholes (BS) model tailored for
American stock regulations, the aforementioned BS model applies to European
stock regulations, where the option is exercised at maturity $T$. Recently,
Vukovic vukovic2015interconnectedness  established a connection between the
Schrödinger equation (SE) and the Black-Scholes model equation using
principles from quantum physics, particularly the Hamiltonian operator. It was
noted that the BS equation could be derived from the SE using tools from
quantum mechanics contreras2010quantum . While the SE is a complex-valued
equation, the BS model is a real-valued equation governing the price function.
The BS model equation, Eq.(1), can be extended to one-dimensional option
models involving $\psi$ and $s$. As discussed in voit2003statistical , one can
determine the probability density function (PDF) by solving the Fokker–Planck
equation using classical Kolmogorov probability methods, rather than relying
on the option value obtained from the BS equation.

In the contemporary era, exploring the intricate aspects of economic and
financial issues through mathematical modeling has become a highly researched
domain due to its extensive applications in nonlinear science. These
mathematical models often take the form of nonlinear partial differential
equations (NPDEs). Among the widely studied NPDEs is the Ivancevic option
pricing model. Ivancevic ivancevic2010adaptive  employed quantum-probabilistic
principles to derive a probability density function (PDF) equivalent for the
value of a stock option by constructing a complex-valued function. This
approach resulted in the proposal of a nonlinear model cont2001empirical ,
which was subsequently renamed the Ivancevic Option Pricing Model (IOPM), as
outlined below

$i\frac{\partial\psi}{\partial
t}+\frac{1}{2}\sigma^{2}\frac{\partial^{2}\psi}{\partial
s^{2}}+\beta|\psi|^{2}\psi=0.$ (2)

In this context, $\psi(s,t)$ represents the option price function at time $t$,
and $\sigma$ is the volatility, which remains constant in this model. The
adaptive market potential, represented by $\beta$, is the Landau coefficient
indicating the adaptive market potential. In its simplest form, it is
equivalent to the interest rate $r$; however, in this case, it depends on a
set of adjustable parameters $\\{{w_{i}}\\}$, known as the synaptic weight
vector components. This adaptive market potential is related to market
temperature, which follows the Maxwell-Boltzmann distribution kleinert2009path
. Additional relevant studies have been referenced to enhance understanding of
the current analysis. Edeki et al. edeki2017analytical  and Gonzalez-Gaxiola
and Ruiz de Chavez gonz2015solving  investigated the Ivancevic option pricing
model using analytical approaches, specifically the projected differential
transformation method and a hybrid technique combining the Adomian
decomposition method with the Elsaki transform, respectively. Chen et al.
chen2019new  introduced a novel operator splitting method to solve the
fractional Black-Scholes model for American options.

##  III Variational approximation

We perform a variational approach on the IOPM given by Eq.(2). The lagrangian
density of the IOPM is given by

$\mathcal{L}=\frac{i}{2}\left(\psi^{*}\frac{\partial\psi}{\partial
t}-\psi\frac{\partial\psi^{*}}{\psi}\right)-\frac{1}{2}\sigma^{2}\left|\frac{\partial\psi}{\partial
s}\right|^{2}+\frac{1}{2}\beta|\psi|^{4}.$ (3)

we assume the modulational instability motivated wave ansatz given by

$\psi(s,t)=[\psi_{0}+a_{1}(t)e^{i(b_{1}(t)+qx)}+a_{2}(t)e^{-i(b_{2}(t)-qx)}]e^{i(kx-\omega
t)},$ (4)

we substitute the ansatz into the lagrangian density to get

$L=\pi[-2(a_{1}^{2}\frac{db_{1}}{dt}+a_{2}^{2}\frac{db_{2}}{dt})\\\
+2(\frac{1}{2}\sigma^{2}k^{2}-\beta\psi_{0}^{2})(\psi_{0}^{2}+a_{1}^{2}+a_{2}^{2})\\\
+\beta(\psi_{0}^{4}+a_{1}^{4}+a_{2}^{4}+4\psi_{0}^{2}(a_{1}^{2}+a_{2}^{2})+4a_{1}^{2}a_{2}^{2}+4a_{1}a_{2}\psi_{0}^{2}cos(b_{1}+b_{2}))\\\
-\sigma^{2}(k^{2}\psi_{0}^{2}+k^{2}(a_{1}^{2}+a_{2}^{2})+(q^{2}+2kq)(a_{1}^{2}-a_{2}^{2}))],$
(5)

Where $L=\int_{0}^{2\pi}\mathcal{L}ds$. From Eq.(5) $b_{1}$,$b_{2}$ are
generalized coordinates of the system and $2a_{1}^{2}$, $2a_{2}^{2}$ are
generalized momenta and are canonically conjugate with respect to the
effective Hamiltonian where $A_{1}(t)=2a_{1}^{2}(t)$ and
$A_{2}(t)=2a_{2}^{2}(t)$. The effective Hamiltonian is given by

$H_{eff}=(\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q^{2})(A_{1}+A_{2})-kq\sigma^{2}(A_{1}-A_{2})\\\
+2\beta\psi_{0}^{2}\sqrt{A_{1}A_{2}}cos(b_{1}+b_{2})+\frac{1}{4}\beta(A_{1}^{2}+A_{2}^{2}+4A_{1}A_{2}).$
(6)

We use the Euler-Lagrange equation of motion based on the Lagrangian Eq.(5),
which are given by

$\frac{d}{dt}\frac{\partial L}{\partial\dot{\xi}_{i}}-\frac{\partial
L}{\partial\xi_{i}}=0$ (7)

where $\xi_{i}$ is the generalized coordinates. The evolution of the
variational equations $a_{1},b_{1},a_{2},b_{2}$ are given by

$\displaystyle\frac{db_{1}}{dt}$
$\displaystyle=\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q(q+2k),$ (8)
$\displaystyle\frac{db_{2}}{dt}$
$\displaystyle=\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q(q+2k),$ (9)
$\displaystyle\frac{da_{1}}{dt}$
$\displaystyle=a_{2}\psi_{0}^{2}sin(b_{1}+b_{2}),$ (10)
$\displaystyle\frac{da_{2}}{dt}$
$\displaystyle=a_{1}\psi_{0}^{2}sin(b_{1}+b_{2}).$ (11)

Assuming that $b=b_{1}+b_{2}$, we get

$\displaystyle a_{1}$ $\displaystyle=a_{2},$ (12)
$\displaystyle\frac{da_{1}}{dt}$ $\displaystyle=\beta\psi^{2}a_{1}sin(b),$
(13) $\displaystyle\frac{db}{dt}$
$\displaystyle=2\beta\psi_{0}^{2}-2q^{2}+2\beta\psi_{0}^{2}cos(b).$ (14)

The solution to Eq.(14) is given by

$b=2arctan\left[\frac{\sqrt{(2\beta\psi_{0}^{2}-\frac{1}{2}q^{2}\sigma^{2})(-\frac{1}{2}q^{2}\sigma^{2})}}{-q^{2}P}tan\sqrt{(2\beta\psi_{0}^{2}-\frac{1}{2}q^{2}\sigma^{2})(-\frac{1}{2}q^{2}\sigma^{2})t}\right].$
(15)

One can notice that Eq.(15) is used to solve Eq.(13). And also if
$q^{2}-4\frac{\psi_{0}^{2}}{\sigma^{2}}\beta<0$ the solution of Eq.(13) is
given by

$a=a_{0}\sqrt{1-4\frac{\beta\psi_{0}^{2}}{q^{2}\sigma^{2}}sin^{2}\left(\sqrt{\left(\frac{1}{2}q^{2}\sigma^{2}-2\beta\psi_{0}^{2}\right)\left(-\frac{1}{2}q^{2}\sigma^{2}\right)}t\right)},$
(16)

while if $q^{2}-4\frac{\psi_{0}^{2}}{\sigma^{2}}\beta>0$ we get

$a=a_{0}\sqrt{1+4\frac{\beta\psi_{0}^{2}}{q^{2}\sigma^{2}}sinh^{2}\left(\sqrt{\left(2\beta\psi_{0}^{2}-\frac{1}{2}q^{2}\sigma^{2}\right)\left(-\frac{1}{2}q^{2}\sigma^{2}\right)}t\right)}.$
(17)

The $2$ equations given by Eq.(16) and Eq.(17) are sinusoidal and exponential
respectively signifying stability and instability of the model.

Figure 1: The panel shows the time evolution of $a$ as a function of $t$ where
$\beta=1$, $\sigma=1.5$, $q=\sqrt{3}$, $\psi_{0}=1$ for Eq.(16)

Figure 2: The panel shows the time evolution of $a$ as a function of $t$ where
$\beta=1$, $\sigma=1.5$, $q=\sqrt{3}$, $\psi_{0}=1$ for Eq.(16)

We can reduce Eqs.(8)-(11) to the realm of one degree of freedom with an
effective potential energy landscape whose variation gives a more vivid
understanding of the instability. We also assume that $A_{1}(t=0)=0$ in Eq.(5)
and $A_{1}(t)=A_{2}(t)$, we have

$\dot{H}=\pi\left[2\left(\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q^{2}\right)+2\beta\psi_{0}^{2}Acos(b)+\frac{3}{2}\beta
A^{2}\right].$ (18)

Eliminating $b$ for $A(t=0)=0$ we get the mechanical energy equation for $A$

$\frac{\dot{A}^{2}}{2}+V_{eff}=0,$ (19)

where the effective potential $V_{eff}(A)$ is given by

$V_{eff}=-2A^{2}\left(\beta\psi_{0}^{2}\right)^{2}-\left(2\left(\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q^{2}\right)+\frac{3}{2}\beta
A\right)^{2}.$ (20)

We can examine the stability of the effective potential by evaluating its
curvature at $A=0$, we get
$V_{eff}^{"}=2\sigma^{2}q^{2}\left(\frac{1}{2}\sigma^{2}q^{2}-2\beta\psi_{0}^{2}\right)$,
which is convex for $\sigma^{2}q^{2}>4\beta\psi_{0}^{2}$, thus describing
stable dynamics, and $V_{eff}$ is concave for
$\sigma^{2}q^{2}<4\beta\psi_{0}^{2}$, thus describing unstable dynamics.

Figure 3: The panel shows the effective potentential given by Eq.(20) for
different values of $q$

###  III.1 Variational Approach for time depended volatility and adaptive
market heat potential

In this section we study the variational approach for IOPM Eq.(2) with time
dependent volatility $\sigma\equiv\sigma(t)$ and the adaptive market heat
potential $\beta\equiv\beta(t)$, given by

$i\frac{\partial\psi}{\partial
t}+\frac{1}{2}\sigma(t)^{2}\frac{\partial^{2}\psi}{\partial
s^{2}}+\beta(t)|\psi|^{2}\psi=0.$ (21)

This approach is important since it assumes the volatility is non-constant
which is the limitation of the BS and IOP models, since volatility is not
constant in the real world. Also in this case the adaptive market heat
potential is assumed to be non-constant.

We consider

$\psi=\psi_{1}[1+\eta(t)cos(qs)],$ (22)

where
$\psi_{1}=e^{i(-k^{2}\int_{0}^{t}-\frac{1}{2}\sigma(t^{\prime})^{2}dt^{\prime}+\int_{0}^{t}\beta(t^{\prime})dt^{\prime}+ks)}$,
without loss of generality, we assume that $a_{1}=a_{2}$, $b_{1}=b_{2}$ and
$\psi_{0}=1$. Following the same procedure that we used above, we get

$\displaystyle\dot{\eta}_{r}$
$\displaystyle=\frac{1}{2}q(t)^{2}\sigma^{2}\eta_{i}-\frac{3}{4}\beta(t)(\eta_{i}^{3}+\eta_{i}\eta_{r}^{2}),$
(23) $\displaystyle\dot{\eta}_{i}$
$\displaystyle=-(\frac{1}{2}q(t)^{2}\sigma^{2}-2\beta(t))\eta_{r}+\frac{3}{4}\beta(t)(\eta_{r}^{3}+\eta_{r}\eta_{i}^{2}),$
(24)

for a stability equation $w(t)=w_{r}(t)+iw_{i}(t)$, hence, at the linear
level, we get

$\ddot{\eta}=\frac{\dot{\sigma}(t)}{\sigma(t)}\dot{\eta}_{r}-\frac{1}{2}\sigma(t)^{2}q^{2}\left[\frac{1}{2}\sigma(t)^{2}q^{2}-2\beta(t)\right]\eta_{r}.$
(25)

We can determine the modulational stability by determining the criterion of
stability of the ordinary differential equation(25). In the case of constant
$\sigma(t)$ and periodic $\beta(t)$ Eq.(25) becomes Hill’s equation, which is
well studied ndzana2007modulational . Eq.(25) becomes the Mathieu equation for
$\beta(t)=2\alpha cos(2\omega t)$ where
$\alpha=\beta\psi_{0}^{2}-\frac{1}{2}\sigma^{2}q^{2}-\sigma^{2}qk$ which is
also well studied. In this case Eq.(25) becomes

$\ddot{\eta}+\frac{1}{2}\sigma^{2}q^{2}[\frac{1}{2}\sigma^{2}q^{2}-4\alpha
cos(2\omega t)]\eta_{r}=0,$ (26)

after some approximation Eq.(27) becomes

$\ddot{\eta}=Q^{2}[1+\alpha_{1}^{\prime}cos(2\omega t)]\eta_{r}=0,$ (27)

where $Q^{2}=\frac{1}{4}\sigma^{4}q^{4}$,
$\alpha_{1}^{\prime}=-\frac{8\alpha_{1}}{\sigma^{2}q^{2}}$. The non perturbed
version of Eq.(27) is given by

$\ddot{\eta}+Q^{2}\eta_{r}=0.$ (28)

The solurion of Eq.(28) is given by

$\eta_{r}=\eta_{0}e^{iQt}+c.c,$ (29)

where $\frac{Q}{2\pi}$ is the fundamental frequency of the system. Therefore
the perturbed solution of Eq.(27) is given by

$\eta(t)=e^{(\mu+i\omega)t+ib}+e^{(\mu-i\omega)t-ib},$ (30)

of which when substituted into Eq.(27) gives us

$\frac{1}{2}Q^{2}\alpha_{1}^{\prime}e^{-ib}+[(\mu+i\omega)^{2}+Q^{2}]e^{ib}=0.$
(31)

The solution of the above equation is such that

$X^{2}+2Q^{2}X+C=0,$ (32)

where $X=\mu^{2}+\omega^{2}$ and
$C=-4\omega^{2}Q^{2}+\frac{Q^{4}}{4}(4-\alpha_{1}^{\prime 2})$. This quadratic
equation, yields the solutions

$\displaystyle X_{1}$ $\displaystyle=-Q^{2}-2\omega
Q-\frac{Q^{3}}{16\omega}\alpha_{1}^{\prime 2},$ (33) $\displaystyle X_{2}$
$\displaystyle=-Q^{2}-2\omega Q+\frac{Q^{3}}{16\omega}\alpha_{1}^{\prime 2}.$
(34)

We can use the above equation to get

$\mu^{2}=-Q^{2}-\omega^{2}+2\omega Q+\frac{Q^{3}}{16\omega}\alpha_{1}^{\prime
2}.$ (35)

And $Q=\frac{1}{2}m\Gamma$ for $m=1,2,3,\dots$ are defined as the parametric
resonances, where $\Gamma=2\omega$. We get

$Q^{2}+\omega^{2}-2\omega
Q=\left(\frac{1}{2}\Gamma+\epsilon_{1}\right)^{2}+\frac{\Gamma}{4}-\Gamma\left(\frac{1}{2}\Gamma+\epsilon_{1}\right)=\epsilon_{1}^{2},$
(36)

by assuming $U=-Q^{2}-\omega^{2}+2\omega Q$. For
$Q=\frac{1}{2}\Gamma+\epsilon_{1}$ we get

$\lambda_{1}^{2}=\frac{1}{16}Q^{2}\alpha_{1}^{\prime 2}-\epsilon_{1}^{2},$
(37)

for $m=1$, $\mu=\lambda_{1}$ for Eq.(36) and Eq.(37) is called the value of
the instability increment for the first resonance. We can clearly observe that
the point of maximum instability coincides with the width of the resonance

$\displaystyle\lambda_{1max}^{2}$
$\displaystyle=\frac{1}{64}\sigma^{4}q^{4}\alpha_{1}^{\prime 2},$ (38)
$\displaystyle\epsilon_{1}^{2}$
$\displaystyle=\frac{1}{64}\sigma^{4}q^{4}\alpha_{1}^{\prime 2},.$ (39)

The second parametric resonance appears for m =2,

$Q=\Gamma+\epsilon_{2}.$ (40)

We get the increment of instability $\lambda_{2}$ for the second domain is
given by

$(4\lambda_{2})^{2}=\left(\frac{\alpha_{1}^{\prime
2}}{2}\right)+\alpha_{2}^{\prime
2}-\left(\frac{\alpha_{1}^{2}}{3}-4\epsilon_{2}\right)^{2},$ (41)

which is maximum at $\epsilon_{2}=0$. The edges of this instability region are
determined by the condition $\lambda_{2}=0$ as

$\displaystyle\epsilon_{2}^{+}$ $\displaystyle=\frac{1}{24}\alpha_{1}^{\prime
2},$ (42) $\displaystyle\epsilon_{2}^{-}$
$\displaystyle=\frac{-5}{24}\alpha_{1}^{\prime 2}.$ (43)

It is clear from Eq.(42) and Eq.(43) that the instability region is not
symmetrical around the center frequency.

It can be concluded that a slight periodic variation in the volatility of the
stock price generates new regions of modulation instability.

##  IV Numerical Simulations

In this section we study the IOPM Eq.(2) by direct numerical algoritms. We use
the improved Euler’s method refered to as the 4th order Runge-Kutta method. In
this method time is discrete and increases in time steps $\Delta
t=\frac{1}{4000}$. And also the initial solution is given by
$\psi(s,0)=\sqrt{n}+\epsilon cos(ks)$. In the algorithm we use $n=10$,
$\epsilon=0.01$ and $k=0.5$.

Figure 4: The Figure shows the temporal development of the CW wave of the
option price $\psi$ for $\sigma=0.2$, $\beta=2$.  Figure 5: The Figure shows
the temporal development of the CW wave of option price $\psi$ for
$\sigma=0.4$, $\beta=2$. Figure 6: The Figure shows the temporal development
of the CW wave of option price $\psi$ for $\sigma=0.2$, $\beta=4$. Figure 7:
The Figure shows the temporal development of the CW wave of option price
$\psi$ for $\sigma=0.4$, $\beta=4$.

The direct numerical simulations as shown by figure(4)-(7) shows that an
increase in both the volatility and the adaptive market heat potential leads
to the instability of the CW wave starting sooner as we can see on figure(4)
that the solitons begin to emerge at time close to $20$ when $\sigma=0.2$ and
$\beta=2$ which is smaller compared to on figure(7) where $\sigma=0.4$ and
$\beta=4$, the solitons emerge at $t=10$. Which agrees with MI results from
optics dauxois2006physics .

##  V Conclusion

We have studied the modulational instability of the IOPM using the variational
method. this method also derives a dispersion relation, also this same method
also studies the stability criterion of the IOPM with non-constant volatility
and the adaptive market heat potential, which gives new results different from
other studies given by previous results on the IOPM whose concern’s were of
constant volatility and adaptive market heat potential. Also we have used
direct numerical analysis on the IOPM.

## References

  * (1) Ş. T. Demiray and E. Ceren, “New solutions for iopm and (3+ 1)-dimensional nlwe in liquid with gas bubbles,” _Journal of the Institute of Science and Technology_ , vol. 12, no. 4, pp. 2425–2436, 2022.
  * (2) K. Adama, P. Daouda, M. L. Justin, and Y. Pare, “Exact analytical solution of ivancevic options pricing model (iopm) or schrodinger’s equation via adm and sba methods,” _Asian Research Journal of Mathematics_ , vol. 18, no. 3, pp. 50–61, 2022.
  * (3) Y. Bai, T. Chaolu, and S. Bilige, “The application of improved physics-informed neural network (ipinn) method in finance,” _Nonlinear Dynamics_ , vol. 107, no. 4, pp. 3655–3667, 2022.
  * (4) U. Obaidullah and S. Jamal, “Analytical solutions of the nonlinear ivancevic options pricing model,” _International Journal of Nonlinear Analysis and Applications_ , vol. 14, no. 3, pp. 245–259, 2023.
  * (5) O. Gonz _et al._ , “Solving the ivancevic option pricing model using the elsaki-adomian decomposition method,” _International Journal of Applied Mathematics_ , vol. 28, no. 5, pp. 515–525, 2015.
  * (6) S. Edeki, O. Ugbebor, and O. González-Gaxiola, “Analytical solutions of the ivancevic option pricing model with a nonzero adaptive market potential,” _International Journal of Pure and Applied Mathematics_ , vol. 115, no. 1, pp. 187–198, 2017.
  * (7) O. González-Gaxiola, S. Edeki, O. Ugbebor, and J. R. de Ch’avez, “Solving the ivancevic pricing model using the he’s frecuency amplitude formulation,” _European Journal of Pure and Applied Mathematics_ , vol. 10, no. 4, pp. 631–637, 2017.
  * (8) P. Veeresha and D. Kumar, “Analysis and dynamics of the ivancevic option pricing model with a novel fractional calculus approach,” _Waves in Random and Complex Media_ , pp. 1–18, 2022.
  * (9) Q. Chen, H. M. Baskonus, W. Gao, and E. Ilhan, “Soliton theory and modulation instability analysis: The ivancevic option pricing model in economy,” _Alexandria Engineering Journal_ , vol. 61, no. 10, pp. 7843–7851, 2022.
  * (10) C. Gaafele, E. B. Madimabe, K. Ndebele, P. Otlaadisa, B. Mozola, T. Matabana, K. Seamolo, and P. Pilane, “Modulational instability of the coupled nonlinear volatility and option price model,” _arXiv preprint arXiv:2405.19887_ , 2024.
  * (11) M. Raheel, K. K. Ali, A. Zafar, A. Bekir, O. A. Arqub, and M. Abukhaled, “Research article exploring the analytical solutions to the economical model via three different methods,” 2023.
  * (12) M. Wróblewski and A. Myślinski, “Quantum dynamics approach for non-linear black-scholes option pricing,” _Authorea Preprints_ , 2023.
  * (13) D. Anderson, “Variational approach to nonlinear pulse propagation in optical fibers,” _Physical review A_ , vol. 27, no. 6, p. 3135, 1983.
  * (14) B. A. Malomed, “Pulse propagation in a nonlinear optical fiber with periodically modulated dispersion: variational approach,” _Optics communications_ , vol. 136, no. 3-4, pp. 313–319, 1997.
  * (15) ——, “Variational methods in nonlinear fiber optics and related fields,” in _Progress in optics_. Elsevier, 2002, vol. 43, pp. 71–193.
  * (16) S. Sabari, E. Wamba, K. Porsezian, A. Mohamadou, and T. Kofané, “A variational approach to the modulational-oscillatory instability of bose–einstein condensates in an optical potential,” _Physics Letters A_ , vol. 377, no. 37, pp. 2408–2415, 2013.
  * (17) E. Wamba, A. Mohamadou, and T. C. Kofané, “A variational approach to the modulational instability of a bose–einstein condensate in a parabolic trap,” _Journal of Physics B: Atomic, Molecular and Optical Physics_ , vol. 41, no. 22, p. 225403, 2008.
  * (18) E. Wamba, S. Sabari, K. Porsezian, A. Mohamadou, and T. Kofané, “Dynamical instability of a bose-einstein condensate with higher-order interactions in an optical potential through a variational approach,” _Physical Review E_ , vol. 89, no. 5, p. 052917, 2014.
  * (19) A. I. Nicolin and R. Carretero-González, “Nonlinear dynamics of bose-condensed gases by means of a q-gaussian variational approach,” _Physica A: Statistical Mechanics and its Applications_ , vol. 387, no. 24, pp. 6032–6044, 2008.
  * (20) E. Wamba, A. Mohamadou, and T. C. Kofané, “A variational approach to the modulational instability of a bose–einstein condensate in a parabolic trap,” _Journal of Physics B: Atomic, Molecular and Optical Physics_ , vol. 41, no. 22, p. 225403, 2008.
  * (21) S. Sabari, E. Wamba, K. Porsezian, A. Mohamadou, and T. Kofané, “A variational approach to the modulational-oscillatory instability of bose–einstein condensates in an optical potential,” _Physics Letters A_ , vol. 377, no. 37, pp. 2408–2415, 2013.
  * (22) E. Wamba, S. Sabari, K. Porsezian, A. Mohamadou, and T. Kofané, “Dynamical instability of a bose-einstein condensate with higher-order interactions in an optical potential through a variational approach,” _Physical Review E_ , vol. 89, no. 5, p. 052917, 2014.
  * (23) F. I. Ndzana, A. Mohamadou, and T. C. Kofané, “Modulational instability in the cubic–quintic nonlinear schrödinger equation through the variational approach,” _Optics communications_ , vol. 275, no. 2, pp. 421–428, 2007.
  * (24) V. G. Ivancevic, “Adaptive-wave alternative for the black-scholes option pricing model,” _Cognitive Computation_ , vol. 2, no. 1, pp. 17–30, 2010.
  * (25) H. Kleinert, _Path integrals in quantum mechanics, statistics, polymer physics, and financial markets_. World scientific, 2009.
  * (26) V. G. Ivancevic, “Quantum neural computation for option price modelling,” _arXiv preprint arXiv:0903.0680_ , 2009.
  * (27) T. Dauxois and M. Peyrard, _Physics of solitons_. Cambridge University Press, 2006.
  * (28) F. Black and M. Scholes, “The pricing of options and corporate liabilities,” _Journal of political economy_ , vol. 81, no. 3, pp. 637–654, 1973.
  * (29) O. González-Gaxiola, J. Ruíz de Chávez, and J. A. Santiago, “A nonlinear option pricing model through the adomian decomposition method,” _International Journal of Applied and Computational Mathematics_ , vol. 2, pp. 453–467, 2016.
  * (30) S. Edeki, E. Owoloko, and O. Ugbebor, “The modified black-scholes model via constant elasticity of variance for stock options valuation,” in _AIP Conference proceedings_ , vol. 1705, no. 1. AIP Publishing LLC, 2016, p. 020041.
  * (31) Q. Chen, H. M. Baskonus, W. Gao, and E. Ilhan, “Soliton theory and modulation instability analysis: The ivancevic option pricing model in economy,” _Alexandria Engineering Journal_ , vol. 61, no. 10, pp. 7843–7851, 2022.
  * (32) K. K. Ali, S. Tarla, M. R. Ali, A. Yusuf, and R. Yilmazer, “Physical wave propagation and dynamics of the ivancevic option pricing model,” _Results in Physics_ , vol. 52, p. 106751, 2023.
  * (33) A. Elmandouh, M. Elbrolosy _et al._ , “Integrability, variational principle, bifurcation, and new wave solutions for the ivancevic option pricing model,” _Journal of Mathematics_ , vol. 2022, 2022.
  * (34) Y.-Q. Chen, Y.-H. Tang, J. Manafian, H. Rezazadeh, and M. Osman, “Dark wave, rogue wave and perturbation solutions of ivancevic option pricing model,” _Nonlinear Dynamics_ , vol. 105, pp. 2539–2548, 2021.
  * (35) O. González-Gaxiola, S. Edeki, O. Ugbebor, and J. R. de Ch’avez, “Solving the ivancevic pricing model using the he’s frecuency amplitude formulation,” _European Journal of Pure and Applied Mathematics_ , vol. 10, no. 4, pp. 631–637, 2017.
  * (36) O. Vukovic _et al._ , “On the interconnectedness of schrodinger and black-scholes equation,” _Journal of Applied Mathematics and Physics_ , vol. 3, no. 9, pp. 1108–1113, 2015.
  * (37) M. Contreras, R. Pellicer, M. Villena, and A. Ruiz, “A quantum model of option pricing: When black–scholes meets schrödinger and its semi-classical limit,” _Physica A: Statistical Mechanics and its Applications_ , vol. 389, no. 23, pp. 5447–5459, 2010.
  * (38) J. Voit, _The statistical mechanics of financial markets_. Springer, 2003.
  * (39) V. G. Ivancevic, “Adaptive-wave alternative for the black-scholes option pricing model,” _Cognitive Computation_ , vol. 2, no. 1, pp. 17–30, 2010.
  * (40) R. Cont, “Empirical properties of asset returns: stylized facts and statistical issues,” _Quantitative finance_ , vol. 1, no. 2, p. 223, 2001.
  * (41) H. Kleinert, _Path integrals in quantum mechanics, statistics, polymer physics, and financial markets_. World scientific, 2009.
  * (42) S. Edeki, O. Ugbebor, and O. González-Gaxiola, “Analytical solutions of the ivancevic option pricing model with a nonzero adaptive market potential,” _International Journal of Pure and Applied Mathematics_ , vol. 115, no. 1, pp. 187–198, 2017.
  * (43) O. Gonz _et al._ , “Solving the ivancevic option pricing model using the elsaki-adomian decomposition method,” _International Journal of Applied Mathematics_ , vol. 28, no. 5, pp. 515–525, 2015.
  * (44) C. Chen, Z. Wang, and Y. Yang, “A new operator splitting method for american options under fractional black–scholes models,” _Computers & Mathematics with Applications_, vol. 77, no. 8, pp. 2130–2144, 2019. @bookdauxois2006physics, title=Physics of solitons, author=Dauxois, Thierry and Peyrard, Michel, year=2006, publisher=Cambridge University Press