# KronA: Parameter Efficient Tuning with Kronecker Adapter

Ali Edalati
McGill University
<EMAIL_ADDRESS>&Marzieh Tahaei
Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>&Ivan Kobyzev
Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>Vahid Partovi Nia
Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>&James J. Clark
McGill University
<EMAIL_ADDRESS>&Mehdi Rezagholizadeh
Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>
###### Abstract

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task
has been a well-known paradigm in Natural Language Processing. However, with
the ever-growing size of PLMs, training the entire model on several downstream
tasks becomes very expensive and resource-hungry. Recently, different
Parameter Efficient Tuning (PET) techniques are proposed to improve the
efficiency of fine-tuning PLMs. One popular category of PET methods is the
low-rank adaptation methods which insert learnable truncated SVD modules into
the original model either sequentially or in parallel. However, low-rank
decomposition suffers from limited representation power. In this work, we
address this problem using the Kronecker product instead of the low-rank
representation. We introduce KronA, a Kronecker product-based adapter module
for efficient fine-tuning of Transformer-based PLMs. We apply the proposed
methods for fine-tuning T5 on the GLUE benchmark to show that incorporating
the Kronecker-based modules can outperform state-of-the-art PET methods.

##  1 Introduction

Large PLMs are used as a backbone model in a variety of NLP tasks to achieve
state-of-the-art results Devlin et al. (2019); Radford et al. (2019). These
large pre-trained models are adapted to the downstream applications either via
in-context learning or fine-tuning of the model parameters. In-context
learning imposes substantial memory and computational overhead during
inference as all the training examples have to be processed for each sample
Liu et al. (2022). On the other hand, full Fine-Tuning (FT) the entire model
provides both less inference latency and improved accuracy. However, as these
models become larger, full fine-tuning of their parameters becomes more
challenging. Additionally, one has to store an entire model checkpoint for
each downstream application, which makes deployment and switching between
different tasks extremely inefficient.

To address these challenges, several works have proposed to insert a small
number of trainable parameters while freezing most (or even all) of the pre-
trained model parameters. This significantly reduces the memory and
computation requirements for fine-tuning. Furthermore, instead of storing one
copy of the entire model, a small set of tuned parameters can be stored for
each task. We refer to these methods as PET methods.

Among the PET methods, soft prompts Li and Liang (2021); Lester et al. (2021a)
prepend trainable parameters to the input of the layers. The increase in
length of the embedding layers leads to a significant computation overhead
during the inference.

In another category of the PET methods, adapter modules are inserted Houlsby
et al. (2019); Karimi Mahabadi et al. (2021); He et al. (2022) into the
Transformer. Adapters are low-rank modules that are composed of an up
projection followed by a down projection. One limitation of these approaches
is that they increase the computational overhead and the latency during the
inference which makes them inefficient for latency-critical scenarios.

Therefore, Low Rank Adaption (LoRA) Hu et al. (2021) was developed, which also
uses extra low-rank modules as the trainable parameters. However, once fine-
tuned, the task-specific parameters can be merged with the original pre-
trained model weights, making the latency and energy requirements for
inference, intact. Despite of fast inference, usually LoRA suffers from an
accuracy drop compared to full fine-tuning. This is because of the strong
assumption imposed by its low-rank structure for task-specific updates.

Kronecker product decomposition is another factorization method that does not
rely on the low-rank assumption. This powerful decomposition method, when used
for model compression, has proven to outperform low-rank factorization methods
Thakker et al. (2019); Hameed et al. (2021). It has also been successfully
used for the compression of Transformer-based language models Tahaei et al.
(2021); Edalati et al. (2021).

Inspired by the ubiquitous success of Kronecker decomposition, in this work we
replace low-rank decomposition in LoRA with Kronecker product decomposition to
develop the Kronecker Adapter (KronA). We show that this simple modification
can improve the accuracy without increasing the inference latency. Also for
applications where an increase in latency is tolerable, we propose to use
$\text{KronA}^{\text{B}}$. This module is a version of KronA developed to be
utilized in parallel to Feed-Forward Network (FFN) modules and achieves
significant improvements over full fine-tuning on the General Language
Understanding Evaluation (GLUE) benchmark Wang et al. (2018). In addition,
when a proposed learnable residual connection is added to
$\text{KronA}^{\text{B}}$, $\text{KronA}^{\text{B}}_{\text{res}}$ is developed
to achieve even better results.

We evaluated our methods on the GLUE benchmark Wang et al. (2018) to study the
impact of Kronecker product on the performance. To summarize, our
contributions are:

  * •

Proposing KronA, a Kronecker Adapter module that can be inserted in parallel
to the weight matrices and is suitable for latency-critical scenarios.

  * •

Using the KronA module in parallel to the FFN module
($\text{KronA}^{\text{B}}$) along with a learnable residual connection
($\text{KronA}^{\text{B}}_{\text{res}}$) to further improve the accuracy at
the cost of increased inference latency.

  * •

Providing evaluation of the methods in comparison to the state-of-the-art in
terms of the GLUE score, training time and inference latency.

Figure 1: This figure shows the structure of the proposed Kronecker-based
modules and their low-rank counterparts. For simplicity, the scaling factor at
output of the modules is not depicted. Figure d shows
$\text{KronA}^{\text{B}}_{\text{res}}$. The residual connection is depicted by
a dotted-line to remind that this connection can simply be removed to have
$\text{KronA}^{\text{B}}$.

##  2 Related Works and Baselines

(Zaken et al., 2021) proposed freezing the weights and tuning only biases or a
subset of biases in the PLM to fine-tune it on the downstream tasks. This
technique, called BitFit, is parameter-efficient and fast, but it usually
cannot achieve good performance compared to the state-of-the-art methods.

Houlsby et al. (2019) introduced Adapters as a PET method. In this method, the
entire parameters of a PLM are frozen and some trainable modules, called
Adapters, are inserted after the FFN or attention blocks, sequentially. Every
Adapter module has a down projection, a non-linear function and an up
projection in addition to a residual connection. He et al. (2022) developed
Parallel-Adapter (PA) which outperforms Adapter. PA is inserted in parallel to
the original PLM module and has a scaling factor, but the residual connection
inside the module is removed since the PLM module has a residual connection
(Figure 1.c). He et al. (2022) also introduced a unified view over PET methods
and combined some of the techniques like Prefix tuning Li and Liang (2021) and
PA.

In Karimi Mahabadi et al. (2021), a modified version of the sequential adapter
is used for PET, named Compacter. In Compacter, Kronecker product of multiple
Kronecker factors are added to reconstruct the module’s weight matrix. Each
Kronecker factor itself is a result of matrix multiplication of two sub-
factors. Compacter achieves good results on GLUE, but it is notably slow in
both training and inference phases.

LoRA Hu et al. (2021) inserts trainable modules which are made from a down
projection and an up projection in different parts of a PLM. It is worth
mentioning that the authors recommended to insert the LoRA modules in parallel
to the query and value matrices. During the training, the PLM weights are
frozen and only the LoRA modules are tuned. During the inference, the LoRA
weights are merged with the original weight matrices of the PLM. Therefore, in
contrast to the Adapter and Compacter, LoRA does not increase the inference
time.

Since LoRA, PA and Compacter have outperformed most of the existing baselines
such as Pfeiffer-Adapter Pfeiffer et al. (2020), AdapterDrop Rücklé et al.
(2021), VGLM-Adapter Lin et al. (2020) Prompt tuning (Lester et al., 2021b),
and Prefix tuning Li and Liang (2021), we considered these works as baseline
for comparison in this work.

##  3 Methodology

###  3.1 Kronecker Product

Kronecker product is an operation on two input matrices
($\mathbf{A}\in\mathbb{R}^{a_{1}\times a_{2}}$ and
$\mathbf{B}\in\mathbb{R}^{b_{1}\times b_{2}}$) that results a block matrix
($\mathbf{W}\in\mathbb{R}^{w_{1}\times w_{2}}$). In $\mathbf{W}$, each block
$(i,j)$ is equal to the multiplication of the element $a_{i,j}$ and the matrix
$\mathbf{B}$. Also, the shape of the resulting matrix is $(w_{1},w_{2})$,
where $w_{1}=a_{1}\times b_{1}$ and $w_{2}=a_{2}\times b_{2}$. Equation 1
shows how the Kronecker product works. for more details see Henderson et al.
(1983).

$\mathbf{W}=\mathbf{A}\otimes\mathbf{B}=\begin{bmatrix}a_{11}\mathbf{B}&\cdots&a_{1n}\mathbf{B}\\\
\vdots&\ddots&\vdots\\\ a_{m1}\mathbf{B}&\cdots&a_{mn}\mathbf{B}\end{bmatrix}$
(1)

The Kronecker product has some interesting features that makes it suitable for
PET. First, it is not rank deficient. In other words, unlike the low-rank
down-projection used in LoRA and Adapter, Kronecker product decomposition
maintains the rank of the input matrix. Second, to speed up and reduce the
required number of FLOPS, one can avoid the reconstruction of $\mathbf{W}$ and
use Equation 2 to calculate the output of a Kronecker module. More precisely,
multiplying an input vector $\mathbf{x}\in\mathbb{R}^{d_{h}}$, where $d_{h}$
is the embedding dimension, by the Kronecker product of $\mathbf{A}$ and
$\mathbf{B}$ can be performed using the following equation:

$(\mathbf{A}\otimes\mathbf{B})\mathbf{x}=\gamma(\mathbf{B}\eta_{b_{2}\times
a_{2}}(\mathbf{x})\mathbf{A^{\top}})$ (2)

where $\mathbf{A}^{\top}$ is $\mathbf{A}$ transpose. $\eta_{m\times
n}(\mathbf{y})$ is a mathematical operation that reshapes a vector
$\mathbf{y}\in\mathbb{R}^{mn}$ into a matrix of size $m\times n$.
$\gamma(\mathbf{Y})$ is another operation that converts a matrix
$\mathbf{Y}\in\mathbb{R}^{m\times n}$ into a vector by stacking its columns.

The following two sub-sections describe our proposed Kronecker modules. See
Appendix B for the ablation experiments justifying our design choices. These
include studying the effect of initialization, the use of scale factor and
non-linearity, as well as the position of the Kronecker module (parallel vs
sequential).

###  3.2 KronA

Figure 1.a shows the structure of a LoRA module where $\mathbf{A}$ is the down
projection and $\mathbf{B}$ is the up projection. To modify this module into
KronA, the normal product is replaced by the Kronecker product. Also, LoRA
projections are replaced with the Kronecker factors (See Figure 1.b and
Appendix A). Equation 3 shows how the output is generated when KronA is
applied. $\mathbf{A}_{k}$ and $\mathbf{B}_{k}$ are the Kronecker factors that
replaced the LoRA projections. Similar to LoRA, our KronA has a fixed scale
factor, $s$, which is a hyperparameter.

$\mathbf{Y}=\mathbf{X}\mathbf{W}+s\mathbf{X}[\mathbf{A}_{k}\otimes\mathbf{B}_{k}]$
(3)

KronA modules are applied in parallel to the weight matrices in a PLM during
the tuning phase. Once fine-tuned, the Kronecker factors are multiplied, then
scaled and merged to the original PLM weight matrix (Equation 4). Therefore,
similar to LoRA, KronA does not increase the inference time.

$\mathbf{W}_{\text{tuned}}=\mathbf{W}+s[\mathbf{A}_{k}\otimes\mathbf{B}_{k}]$
(4)

For initialization of the Kronecker factors, we observed that initializing one
of the factors with zero significantly improves the results compared to
initializing both factors using the Normal distribution. See Appendix B for
more details.

###  3.3 $\text{KronA}^{\text{B}}$ and $\text{KronA}^{\text{B}}_{\text{res}}$

Inspired by the promising performance of the PA method, we also investigate
KronA module when used in parallel to the FNN block and call it
KronA${}^{\text{B}}$. The B superscript in the name means that this module is
applied to the PLM blocks, opposed to KronA which is applied to the PLM weight
matrices. Note that similar to the PA method, the non-linearity in the FFN
block does not allow our KronA module to be merged into the pre-trained model
after fine-tuning. This imposes an increase in the inference time and
computations. Equation 5 shows how KronA${}^{\text{B}}$ works in parallel to
the FFN block.

$\mathbf{Y}=\text{FFN}(\mathbf{X})+s\mathbf{X}[\mathbf{A}_{k}\otimes\mathbf{B}_{k}]$
(5)

To further improve the representation power, we incorporate a scaled residual
connection inside the $\text{KronA}^{\text{B}}$ module to develop
$\text{KronA}^{\text{B}}_{\text{res}}$. Scale of the residual connection
($s_{res}$) is initialized with one and tuned during the fine-tuning. Equation
6 shows how the $\text{KronA}^{\text{B}}_{\text{res}}$ works in parallel to
the FFN. Also, Figure 1.c and 1.d show the structure of the PA module and our
$\text{KronA}^{\text{B}}_{\text{res}}$, respectively.

$\mathbf{Y}=\text{FFN}(\mathbf{X})+s\mathbf{X}[\mathbf{A}_{K}\otimes\mathbf{B}_{K}]+s_{res}\mathbf{X}$
(6)

Method | Params | CoLA | RTE | MRPC | SST-2 | STS-B | MNLI | QNLI | QQP | Avg
---|---|---|---|---|---|---|---|---|---|---
FT | 100 | 63.37 | 74.82 | 92.73 | 93.58 | 90.07 | 86.16 | 92.77 | 91.74 | 85.65
BitFit | 0.12 | 58.19 | 68.34 | 92.58 | 94.61 | 90.69 | 85.73 | 92.91 | 90.33 | 84.17
Adapter | 0.07 | 64.66 | 71.94 | 91.27 | 94.84 | 90.49 | 85.91 | 92.97 | 90.35 | 85.30
LoRA | 0.07 | 64.76 | 74.10 | 92.10 | 93.92 | 91.21 | 86.08 | 92.97 | 90.68 | 85.73
Compacter | 0.07 | 64.42 | 76.26 | 91.52 | 93.92 | 91.04 | 86.14 | 92.93 | 90.36 | 85.82
PA | 0.06 | 64.80 | 74.10 | 93.20 | 94.04 | 91.10 | 86.24 | 93.12 | 90.30 | 85.86
KronA | 0.07 | 63.27 | 77.70 | 92.52 | 94.26 | 91.30 | 86.34 | 93.15 | 90.57 | 86.14
$\text{KronA}^{\text{B}}$ | $0.07^{*}$ | 65.74 | 75.54 | 92.78 | 94.72 | 91.41 | 86.22 | 93.19 | 90.68 | 86.28
$\text{KronA}^{\text{B}}_{\text{res}}$ | $0.07^{*}$ | 66.73 | 76.98 | 93.15 | 94.38 | 91.35 | 86.20 | 93.21 | 90.57 | 86.57

Table 1: This table shows performance of the methods on the GLUE. $*$ shows
that number of parameters might be a slightly different depending on the
choice of one or two bias vectors in our KronA module.

Method | FT | LoRA | KronA | BitFit | Adapter | PA | Compacter | $\text{KronA}^{\text{B}}$ | $\text{KronA}^{\text{B}}_{\text{res}}$
---|---|---|---|---|---|---|---|---|---
Inference Latency($\%$) | 100 | 100 | 100 | 100 | 146 | 113 | 181 | 127 | 136
Training Time($\%$) | 100 | 72 | 75 | 64 | 73 | 71 | 79 | 74 | 81

Table 2: In this table, the first row shows the normalized latency of the
methods in the inference phase, and the second row lists the average
normalized training time of the methods on the GLUE tasks.

##  4 Results and Discussion

###  4.1 GLUE Results

Table 1 shows111Details regarding the hyperparameters and experimental setups
are mentioned in Appendix D. the GLUE score of our proposed methods compared
to other baselines when applied to the T5 model Raffel et al. (2019). As the
results show, KronA and KronA${}^{\text{B}}$ outperform LoRA and PA as their
low-rank counterparts, respectively. Also, all of our proposed modules
outperform other state-of-the-art baselines on average and most of the GLUE
tasks. In addition, $\text{KronA}^{\text{B}}_{\text{res}}$ which benefits from
an extra learnable residual connection achieves remarkably better results.

###  4.2 Inference and Training Time

Table 2 shows the normalized inference delay for the discussed methods. KronA,
LoRA, FT, and Bitfit do not increase the inference latency since these
techniques do not add any extra parameters or computations to the model during
the inference stage. Although $\text{KronA}^{\text{B}}$ is significantly
faster than Compacter and Adapter, it is slower than PA which is as we
expected; calculation of the Kronecker product is generally slower than normal
matrix multiplication. Also, adding the learnable residual connection
increases the latency.

Normalized training time averaged over the GLUE tasks for each technique is
shown in Table 2. Based on these results, the significant improvement in the
accuracy of the proposed KronA module and its variants is at the expense of a
slight increase in the training time compared to the low-rank counterparts
like LoRA, PA, and Adapter. However, the training time increase is not
remarkable and KronA modules are still significantly faster than FT.

##  5 Conclusion and Future Directions

In this work, we developed Kronecker-based adapters by replacing the low-rank
projections of the state-of-the-art PET methods with the Kronecker product. In
addition to a comparison of the training and inference time, we evaluated our
proposed adapter for fine-tuning T5 on the GLUE benchmark to show its
superiority over the state-of-the-art baselines.

## References

  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Edalati et al. (2021) Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2021.  Kronecker decomposition for gpt compression.  _arXiv preprint arXiv:2110.08152_.
  * Elfwing et al. (2018) Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural networks : the official journal of the International Neural Network Society_ , 107:3–11.
  * Hameed et al. (2021) Marawan Gamal Abdel Hameed, Marzieh S Tahaei, Ali Mosleh, and Vahid Partovi Nia. 2021.  Convolutional neural network compression through generalized kronecker product decomposition.  _arXiv preprint arXiv:2109.14710_.
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_.
  * Henderson et al. (1983) Harold V. Henderson, Friedrich Pukelsheim, and Shayle R. Searle. 1983.  On the history of the kronecker product.  _Linear & Multilinear Algebra_, 14:113–120.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035.
  * Lester et al. (2021a) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021a.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_.
  * Lester et al. (2021b) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021b.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020.  Exploring versatile generative language model via parameter-efficient transfer learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 441–459, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _arXiv preprint arXiv:2205.05638_.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterfusion: Non-destructive task composition for transfer learning.  _arXiv preprint arXiv:2005.00247_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rücklé et al. (2021) Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021.  AdapterDrop: On the efficiency of adapters in transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7930–7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tahaei et al. (2021) Marzieh S Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021.  Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation.  _arXiv preprint arXiv:2109.06243_.
  * Thakker et al. (2019) Urmish Thakker, Igor Fedorov, Jesse Beu, Dibakar Gope, Chu Zhou, Ganesh Dasika, and Matthew Mattina. 2019.  Pushing the limits of rnn compression.  In _2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)_ , pages 18–21. IEEE.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zaken et al. (2021) Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  _arXiv preprint arXiv:2106.10199_.
  * Zhang et al. (2022) Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. 2022.  Hyperpelt: Unified parameter-efficient language model tuning for both language and vision-and-language tasks.  _arXiv preprint arXiv:2203.03878_.

##  Appendix A Kronecker Factors vs Down and Up Projections

Table 3 shows details about the Kronecker factors that replaced the
projections in the LoRA module. Note that $r<\frac{d_{h}}{2}$ to reduce the
number of trainable parameters compared to the original PLM matrix.

Module Name | Factor Name | Symbol | Shape | Parameters | Module Parameters | Constraint
---|---|---|---|---|---|---
KronA | Kronecker Factor | $\mathbf{A}_{k}$ | $a_{1}\times a_{2}$ | $a_{1}a_{2}$ | $a_{1}a_{2}+b_{1}b_{2}$ | $a_{1}b_{1}=a_{2}b_{2}=d_{h}$
Kronecker Factor | $\mathbf{B}_{k}$ | $b_{1}\times b_{2}$ | $b_{1}b_{2}$
LoRA | Down Projection | $\mathbf{A}$ | $d_{h}\times r$ | $d_{h}r$ | $2d_{h}r$ | $r<\frac{d_{h}}{2}$
Up Projection | $\mathbf{B}$ | $r\times d_{h}$ | $d_{h}r$

Table 3: This table compares some details of Kronecker factors with LoRA projections. Init Method | CoLA | RTE | MRPC | SST2 | STSB | MNLI | QNLI | QQP | Avg
---|---|---|---|---|---|---|---|---|---
$\mathbf{A}_{K}$,$\mathbf{B}_{K}$ $\thicksim$ Normal | 63.36 | 66.91 | 91.69 | 91.97 | 90.46 | 86.03 | 92.33 | 90.19 | 84.12
$\mathbf{A}_{K}$ $\thicksim$ KU ,$\mathbf{B}_{K}$=0 | 63.27 | 77.70 | 92.52 | 94.04 | 91.26 | 86.03 | 93.13 | 90.57 | 86.06
Table 4: This table shows performance of KronA on GLUE using different
initialization options for the Kronecker factors.

##  Appendix B Ablation Study

###  B.1 KronA Initialization

Our empirical results show that initialization of the Kronecker factors
affects the performance of the KronA module. Table 4 shows the performance of
two investigated strategies for KronA initialization. We observe that by
initializing one of the Kronecker factors from the Kaiming-uniform
($a=\sqrt{5}$) distribution and the other one with zero, KronA module performs
significantly better than initializing both of the factors from the Normal
($\mu=0,\sigma=\frac{1}{\sqrt{d_{h}}}$, where $d_{h}$ is the embedding
dimension) distribution.

Modification | CoLA | RTE | MRPC | SST-2 | STS-B | MNLI | QNLI | QQP | Avg
---|---|---|---|---|---|---|---|---|---
Sequential | 15.26 | 53.28 | 86.39 | 87.38 | 83.78 | 74.70 | 84.29 | 86.63 | 71.46
Parallel | 58.17 | 69.78 | 91.58 | 93.81 | 90.86 | 85.68 | 93.35 | 90.14 | 84.17
Parallel+Scale (PS) | 62.27 | 70.50 | 91.58 | 94.04 | 91.01 | 86.16 | 93.39 | 90.61 | 84.94
PS+SiLU | 62.74 | 69.78 | 91.89 | 94.15 | 90.97 | 85.98 | 93.30 | 90.15 | 84.87
PS only on FFN | 63.74 | 72.66 | 92.20 | 94.72 | 90.98 | 85.98 | 93.12 | 90.68 | 85.51
Table 5: This table shows performance of $\text{KronA}^{\text{B}}$ after implementing step by step modifications on GLUE. nonlinear function | mish | relu | gelu | $\text{gelu}_{\text{new}}$ | silu
---|---|---|---|---|---
QNLI Performance | 93.21 | 93.28 | 93.13 | 93.26 | 93.30
Table 6: This table shows performance of $\text{KronA}^{\text{B}}$ on QNLI
using different non-linear functions.

###  B.2 Step by Step Improvement of $\text{KronA}^{\text{B}}$

At first, $\text{KronA}^{\text{B}}$ was initialized similar to a normal
adapter. It was sequentially inserted after both FFN and attention blocks and
it did not have scaling. Based on He et al. (2022), we made some modifications
to our module to improve its performance.

Table 5 shows the results of our experiments. We observed that inserting
$\text{KronA}^{\text{B}}$ modules in parallel to the PLM modules instead of
inserting them sequentially, improves performance significantly. Also, adding
a scaling factor to our module increases the GLUE score further. Moreover,
adding two modules to each FFN instead of adding to both FFN and attention
blocks resulted in a higher score.

In addition, motivated by the presence of a non-linear function in PA and
Adapter, we tested different non-linear functions between the two
multiplications (by $\mathbf{A}_{k}^{\text{T}}$ and $\mathbf{B}_{k}$) in
Equation 2. As Table 6 shows, SiLU Elfwing et al. (2018) is the best option
among others, but based on Table 5, adding SiLU decreases GLUE score of the
$\text{KronA}^{\text{B}}$. Therefore, we removed the non-linearity from our
module.

###  B.3 Learnable Residual Connection

In the $\text{KronA}^{\text{B}}_{\text{res}}$ module, a residual connection
that is multiplied by a learnable scale is added to the output of
$\text{KronA}^{\text{B}}$. Also, we studied another scenario in which the
residual is multiplied by Sigmoid(learnable scale). This module is called
$\text{KronA}^{\text{B}}_{\text{sigres}}$. We wanted to answer the question
"Is it better to limit the residual scale between 0 and 1?". Our empirical
results (Table 7) show that by adding a Sigmoid function, the performance of
the module drops and the latency increases. Therefore, the Sigmoid function
was removed from our module.

Method | Avg Score | Latency
---|---|---
$\text{KronA}^{\text{B}}_{\text{res}}$ | 86.57 | 1
$\text{KronA}^{\text{B}}_{\text{sigres}}$ | 86.42 | 1.18
Table 7: This table shows the effect of adding a Sigmoid function to the
$\text{KronA}^{\text{B}}_{\text{res}}$ module. Avg Score is the averaged score
on the GLUE tasks and Latency represents the relative training time.

##  Appendix C Details of Measuring Training and Inference Time

To measure the inference latency, a random dummy input with the batch size
equal to one and the sequence length equal to ten is generated. Then, the
dummy input is given to the model for 150 iterations to warm-up the GPU.
Finally, the dummy input is fed to the model for 200 iterations and the
required time to generate the output is measured, averaged, and recorded. This
experiment is repeated three times and the average latency is reported.
Finally, the reported latencies are normalized and shown in Table 2.

Table 8 shows the normalized training time for each technique on the GLUE
tasks. All the experiments are done with the same number of epochs, batch
size, number of GPUs, and gradient accumulation step.

Method | CoLA | RTE | MRPC | SST-2 | STS-B | MNLI | QNLI | QQP | Avg
---|---|---|---|---|---|---|---|---|---
FT | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
BitFit | 0.58 | 0.64 | 0.65 | 0.66 | 0.66 | 0.65 | 0.64 | 0.62 | 0.64
Adapter | 0.82 | 0.71 | 0.72 | 0.78 | 0.76 | 0.72 | 0.69 | 0.69 | 0.73
LoRA | 0.79 | 0.69 | 0.7 | 0.76 | 0.72 | 0.7 | 0.68 | 0.68 | 0.72
KronA | 0.8 | 0.72 | 0.75 | 0.81 | 0.77 | 0.74 | 0.73 | 0.73 | 0.75
Compacter | 0.88 | 0.74 | 0.78 | 0.86 | 0.81 | 0.75 | 0.74 | 0.76 | 0.79
PA | 0.7 | 0.78 | 0.81 | 0.73 | 0.7 | 0.67 | 0.66 | 0.65 | 0.71
$\text{KronA}^{\text{B}}$ | 0.84 | 0.7 | 0.72 | 0.79 | 0.75 | 0.72 | 0.7 | 0.71 | 0.74
$\text{KronA}^{\text{B}}_{\text{res}}$ | 0.91 | 0.85 | 0.81 | 0.91 | 0.76 | 0.78 | 0.73 | 0.74 | 0.81
Table 8: This table shows the normalized training time of methods on the GLUE tasks. Shape of $\mathbf{A}_{k}$ | MNLI (Accuracy)
---|---
(48, 16) | 86.50
(32, 24) | 86.31
(3, 256) | 86.16
(24, 32) | 86.40
(2, 384) | 86.63
(192, 4) | 86.46
(12, 64) | 86.56
Table 9: This table shows the performance of the tested options for
$\mathbf{A}_{k}$ in KronA on MNLI. Note that for each option, the shape of the
corresponding $\mathbf{B}_{k}$ is in the reversed order of $\mathbf{A}_{k}$.

Method | FT | BitFit | Adapter | Compacter | LoRA | PA | KronA | $\text{KronA}^{\text{B}}$ | $\text{KronA}^{\text{B}}_{\text{res}}$
---|---|---|---|---|---|---|---|---|---
Block | Entire Model | Biases | FFN & Attention | FFN & Attention | Query & Value | FFN | Query & Value | FFN | FFN

Table 10: This table shows the PLM block each method has been applied to.

##  Appendix D Experimental Setups and Hyperparameters

###  D.1 Datasets

We used the GLUE Wang et al. (2018) benchmark to evaluate our methods compared
to the baselines. This benchmark covers a variety of tasks including natural
language inference (MNLI, RTE, QNLI), linguistic acceptability (CoLA),
similarity and paraphrasing (MRPC, QQP), and sentiment classification (SST-2).
The original test set of GLUE is not published, so similar to Karimi Mahabadi
et al. (2021); Zhang et al. (2022), we generated our test sets from the
evaluation and the training data. For the small datasets (CoLA, RTE, MRPC, and
STSB), we use half of the task dev set for evaluation and the other half as
the test set. For the rest of the GLUE tasks with larger datasets, we take 1K
samples out of the train set and use it as our test set. The reported
evaluation metric for CoLA, MRPC, STS-B, is Matthew correlation coefficient,
F1, and the average of Pearson/Spearman correlations, respectively. Accuracy
is used for the other tasks.

###  D.2 Experimental Setup

All experiments were performed on one NVIDIA Tesla V100. We used PyTorch and
Hugging Face Transformers library Wolf et al. (2019) for our experiments. To
re-implement LoRA and PA, we used their publicly available code. For the
experiments on the Compacter, BitFit, FT and Adpater, we used the Compacter’s
official code. The Backbone model for this work is $\text{T5}_{\text{base}}$
Raffel et al. (2019).

The size of the trainable parameters for all of the methods is set roughly
equal to have a fair comparison. However, for the BitFit tuning we could not
match the trainable parameters since all of the biases are trainable.

Given the number of trainable parameters, we have several choices for the
shapes of Kronecker factors. For KronA, We tested some of the options and
selected the option with the best results.

Table 10 shows the specific PLM block on which each technique is applied.

$\text{KronA}^{\text{B}}$ and $\text{KronA}^{\text{B}}_{\text{res}}$ modules
can have one or two biases. We selected the number of biases that maximized
the score on each task.

###  D.3 Hyperparameters

Since we wanted to ignore the effect of the scaling factor when comparing LoRA
and KronA, the scaling factor for these two modules is set to one in all of
the experiments.

For FT, BitFit, Compacter, and Adapter experiments, we used the
hyperparameters that are mentioned in Karimi Mahabadi et al. (2021). However,
we changed the learning rate and the rank of modules to match the desired
number of trainable parameters in the Adapter experiments.

The rank of LoRA and PA is set to one and two, respectively. For the KronA
modules, the shape of the Kronecker factors are selected based on the best dev
results among different options for the shapes. Due to time and resource
limitations, we did not tune the shape of the Kronecker factors for
$\text{KronA}^{\text{B}}$ and $\text{KronA}^{\text{B}}_{\text{res}}$.

All of the other hyperparameters are set based on Karimi Mahabadi et al.
(2021), expect for the learning rate and scaling factor which are tuned based
on the best dev results. All of the methods are trained for 20 epochs and the
checkpoint that achieves the best performance on dev set is reported as the
final model. Tables 11, 12, 13, 14, 15, 16, 17, 18 and 19 show the tuned
hyperparameters for each method on the GLUE tasks.

FT hyperparameters
---
Task | learning rate | batch size | warmup steps | source sentence length | epoch
GLUE | 3e-4 | 100 | 500 | 128 | 20
Table 11: This table shows the hyperparameters used for FT experiments on the
GLUE tasks. BitFit hyperparameters
---
Task | learning rate | batch size | warmup steps | source sentence length | epoch
GLUE | 3e-4 | 100 | 500 | 128 | 20
Table 12: This table shows the hyperparameters used for BitFit experiments on
the GLUE tasks. Adapter hyperparameters
---
Task | learning rate | batch size | task reduction factor | epoch
GLUE | 3e-3 | 100 | 32 | 20
Table 13: This table shows the hyperparameters used for Adapter experiments on
the GLUE tasks.

Compacter hyperparameters
---
Task | learning rate | batch size | hypercomplex division | task reduction factor | epoch
GLUE | 3e-3 | 100 | 4 | 32 | 20

Table 14: This table shows the hyperparameters used for Compacter experiments
on the GLUE tasks. LoRA hyperparameters
---
Task | learning rate | batch size | rank | $s$ | epoch
GLUE | 1e-3 | 100 | 1 | 1 | 20
Table 15: This table shows the hyperparameters used for LoRA experiments on
the GLUE tasks. PA hyperparameters
---
Task | learning rate | batch size | rank | $s$ | epoch
CoLA | 3e-3 | 100 | 2 | 16 | 20
RTE | 5e-3 | 100 | 2 | 16 | 20
MRPC | 5e-3 | 100 | 2 | 16 | 20
SST-2 | 1e-3 | 100 | 2 | 16 | 20
SSTS-B | 1e-3 | 100 | 2 | 16 | 20
MNLI | 1e-3 | 100 | 2 | 16 | 20
QNLI | 1e-3 | 100 | 2 | 16 | 20
QQP | 1e-3 | 100 | 2 | 16 | 20
Table 16: This table shows the hyperparameters used for PA experiments on the
GLUE tasks. KronA hyperparameters
---
Task | learning rate | batch size | $\mathbf{A}_{k}$ | $\mathbf{B}_{k}$ | $s$ | epoch
CoLA | 1e-3 | 100 | (32,24) | (24,32) | 1 | 20
RTE | 2e-3 | 100 | (32,24) | (24,32) | 1 | 20
MRPC | 1e-3 | 100 | (32,24) | (24,32) | 1 | 20
SST-2 | 1e-3 | 100 | (24,32) | (32,24) | 1 | 20
SSTS-B | 1e-3 | 100 | (2,384) | (384,2) | 1 | 20
MNLI | 1e-3 | 100 | (2,384) | (384,2) | 1 | 20
QNLI | 1e-3 | 100 | (3,256) | (256,3) | 1 | 20
QQP | 1e-3 | 100 | (24,32) | (32,24) | 1 | 20
Table 17: This table shows the hyperparameters used for KronA experiments on
the GLUE tasks. $\text{KronA}^{\text{B}}$ hyperparameters
---
Task | learning rate | batch size | $\mathbf{A}_{k}$ | $\mathbf{B}_{k}$ | $s$ | module bias | epoch
CoLA | 1e-3 | 100 | (32,24) | (24,32) | 16 | 2 | 20
RTE | 5e-3 | 100 | (32,24) | (24,32) | 16 | 1 | 20
MRPC | 5e-3 | 100 | (32,24) | (24,32) | 16 | 1 | 20
SST-2 | 1e-3 | 100 | (32,24) | (24,32) | 4 | 1 | 20
SSTS-B | 1e-3 | 100 | (32,24) | (32,24) | 16 | 1 | 20
MNLI | 1e-3 | 100 | (32,24) | (24,32) | 4 | 2 | 20
QNLI | 1e-3 | 100 | (32,24) | (24,32) | 4 | 1 | 20
QQP | 1e-3 | 100 | (32,24) | (24,32) | 4 | 1 | 20
Table 18: This table shows the hyperparameters used for
$\text{KronA}^{\text{B}}$ experiments on the GLUE tasks.
$\text{KronA}^{\text{B}}_{\text{res}}$ hyperparameters
---
Task | learning rate | batch size | $\mathbf{A}_{k}$ | $\mathbf{B}_{k}$ | $s$ | module bias | epoch
CoLA | 1e-3 | 100 | (32,24) | (24,32) | 16 | 2 | 20
RTE | 5e-3 | 100 | (32,24) | (24,32) | 16 | 2 | 20
MRPC | 5e-3 | 100 | (32,24) | (24,32) | 16 | 2 | 20
SST-2 | 1e-3 | 100 | (32,24) | (24,32) | 16 | 1 | 20
SSTS-B | 9e-4 | 100 | (32,24) | (32,24) | 16 | 1 | 20
MNLI | 1e-3 | 100 | (32,24) | (24,32) | 16 | 1 | 20
QNLI | 1e-3 | 100 | (32,24) | (24,32) | 4 | 2 | 20
QQP | 1e-3 | 100 | (32,24) | (24,32) | 16 | 1 | 20
Table 19: This table shows the hyperparameters used for
$\text{KronA}^{\text{B}}_{\text{res}}$ experiments on the GLUE tasks.


# CONMOD: Controllable Neural Frame-based Modulation Effects

###### Abstract

Deep learning models have seen widespread use in modelling LFO-driven audio
effects, such as phaser and flanger. Although existing neural architectures
exhibit high-quality emulation of individual effects, they do not possess the
capability to manipulate the output via control parameters. To address this
issue, we introduce Controllable Neural Frame-based Modulation Effects
(CONMOD), a single black-box model which emulates various LFO-driven effects
in a frame-wise manner, offering control over LFO frequency and feedback
parameters. Additionally, the model is capable of learning the continuous
embedding space of two distinct phaser effects, enabling us to steer between
effects and achieve creative outputs. Our model outperforms previous work
while possessing both controllability and universality, presenting
opportunities to enhance creativity in modern LFO-driven audio effects.
Additional demo of our model is available in the accompanying
website.111https://lgbin81.notion.site/Audio-Demo-for-
DAFx-2024-Paper-70705a992d5c4730a1f70fbaf951243b?pvs=4

##  1 Introduction

The study of virtual analog (VA) modelling via deep neural networks have
become a well-established topic nowadays. Neural networks are strongly capable
of modelling the behavior of the system in a black-box manner, and is being
applied in various VA modelling tasks [1, 2, 3, 4]. Furthermore, the
strongpoint of deep neural networks as a regression model anticipates an
intuitive and creative effect by adjusting a representative embedding vector
of the trained model [5, 6].

The field of VA is largely divided into white-box, grey-box and black-box
approaches. The white-box approach discretizes the acoustic system of the
analog circuit structure to obtain a digital model [7, 8]. The black-box
approach incorporates parametric models which emulate the input-output
behavior of the analog circuits without knowing the physical structure [9, 10,
11]. Grey-box approach leverages some knowledge of the acoustic system to
better emulate the parametric model [12, 13]. Among them, the black-box
approach has recently drawn a great attention as it can behave more robust and
general in the case of deep neural networks when the model is well-structured
and trained with a sufficient amount of data [14, 11, 15, 16, 2, 3].

Phaser and flanger effects modulate the input audio using a periodic signal
typically within the human preferred range (0.1-10Hz), commonly referred to as
LFO-driven effects. Core aspect of modelling these effects is maintaining
precise periodicity regardless of the input audio duration. Therefore, early
neural architectures integrated an LFO signal in the form of a rectified sine
wave [17, 18] or a sinusoidal wave [4] to enable the model to focus on
generating the spectral structure. Early works related to these approach [17,
18] extracts the LFO signal manually, which is a clear disadvantage for
training. [19] further proposed a CNN-based LFO extraction model, achieving
plausible results for unseen audio and effects. Unlike existing black-box
approaches that learn the direct mapping from dry input audio sets to
corresponding wet output audio sets, Carson et al. simplifies the role of
model into emulating solely the effect of a phaser pedal using a frame-based
processing [4]. Nevertheless, the paper does not explore the model’s
controllability with respect to control parameters(e.g. LFO frequency,
feedback, etc), even though directly manipulating the sinusoidal wave may
alter the LFO frequency. And even so, the model is controllable with only a
single control parameter.

In this paper, we build upon the model architecture proposed by Carson et al.
[4] and introduce a novel controllable neural effects designed for two types
of modulation effects, phaser and flanger. Specifically, our neural
architecture enables adjustment of the output with user-specified rate and
feedback parameters for the target effect. Rather than directly optimizing the
parameters related to the physics of the phaser effect, we optimize a neural
architecture consisting of long-term short-memory (LSTM), multi-layer
perceptron (MLP), and conditional feature wise linear modulation (FiLM)
blocks[20]. To enhance the model’s controllability on the LFO frequency, we
introduce a training procedure which optimizes multiple LFO signals within a
single neural network. We compare our work with a baseline model [4] and show
that our model shows superior accuracy in trained control parameter settings,
while also exhibiting impressive performance with unseen control parameters.
Inspired by [5, 6], we further explored our model’s capability to train
multiple modulation effects simultaneously. Through training on two unique
phaser effects concurrently, and by manipulating each effect’s learned
embedding, we show that our model can produce intriguing mixed results between
the two distinct phaser effects. The main contributions of our work can be
summarized as follows:

  * •

We introduce a controllable neural frame-based modulation effect pedal
targeting phaser and flanger, which outperforms previous model in emulation
capability.

  * •

Our model possesses robust controllability on multiple control parameters,
capable of modelling effects on unseen control parameter settings, a feat
unattainable by previous models.

  * •

For the first time, we train a single model on two distinct phaser effects and
demonstrate the model’s steerability by manipulating the learned
representative embeddings.

##  2 Target Modulation Effects

In this section, we briefly explain the characteristics of our target
modulation effects, and the specific effect pedals that were trained on our
model. For each of the effects, we trained our model with digital and analog
devices.

###  2.1 Phaser effects and its controls

The phaser pedal modulates the input signal by utilizing low frequency
oscillating notch filters or low-order all-pass filters. When cascaded, these
filters periodically attenuate certain frequency levels of the signal, thereby
introducing soft, moving sounds. In this paper, we focus on learning to
control the two main parameters: LFO frequency and feedback, following [4].
LFO frequency, also called Speed or Rate, governs the frequency of the LFO
signal, while feedback adjusts the amount of filtered output signal that is
fed back to the input. The adjustment of feedback leads to an overall change
in the shape of the frequency response, notably bringing resonance peaks in
the unattenuated area [21]. Feedback is often termed Color due to its ability
to alter the frequency response shape.

####  2.1.1 Studio One Phaser2

Studio One Phaser2 is a built-in digital phaser included in Studio One,
featuring various controllable parameters such as Speed, Feedback, Center, and
Range [22]. In Phaser2 the Speed knob, which controls the LFO frequency,
allows for continuous adjustment within a range of 0 Hz to 10 Hz. The Feedback
knob also offers continuous range from 0 to 95%. For the remaining controls,
we maintain the default values.

####  2.1.2 Moore Liquid Phaser

Moore Liquid Phaser is a modern analog phaser offering selection among 5
distinct Phaser Types and featuring controls among 3 different Waveform
Types(Round, Triangular, or Square), Speed, and Color. We set the Waveform
Type to Round, and the Phaser Type to feedback mode, which corresponds to the
12 o’ clock direction of the knob (’iii@’).

###  2.2 Flanger effects and its controls

The flanger effect has a similar effect compared to the phaser effect,
generating spectral notches. The main difference is that flanger creates
notches by introducing a variable-length feedforward delayline which leads to
a comb filter. Therefore the notches, in general, are regularly distributed
across the frequency spectrum, extending infinitely. In this work, we adopt a
similar approach as in the phaser case, concentrating on the LFO frequency and
feedback parameters, which function on the same principle as in the phaser
effect.

####  2.2.1 Studio One Flanger

Studio One Flanger is a built-in digital flanger with controllable parameters
similar to those found in a phaser, such as Speed, Feedback, Delay, and LFO
amount. The Speed control allows for adjustments within a continuous value
from 0.01Hz to 10Hz. The Feedback knob provides a range from -90% to 90%,
where negative values lead to inverted feedback [22].

####  2.2.2 Flamma FC15 Analog Flanger

The Flamma FC 15 analog flanger is a classic device carrying two effect modes,
namely filter and normal, along with controllable knobs for Color, Range, and
Rate [23]. We maintain the Modes setting at normal, and set the Range
parameter to the 12 o’ clock position, while training our model on variable
settings for Color and Rate.

##  3 Methodology

We introduce a novel black-box methodology controllable via various effect
parameters, which we refer to as CONMOD(Controllable Neural Frame-based
Modulation Effects). Our model efficiently models the audio effect frame-wise
with less reliance on prior domain knowledge.

###  3.1 Model

Our model predicts the frame-dependent transfer function of the target effect
using a sinusoidal LFO signal with trainable frequency and initial phase as
input, following [4]. This approach approximates the effects, which are
analogous to an IIR filter, by representing them as a truncated impulse
response, where the truncation length is equal to the window size of each
frame. As the frame-based process remains differentiable throughout the
system, we can train the model to predict solely the modulation effect, with
the training objective being the target wet audio waveform. The input for the
training phase is a chirp signal, and the target pair is the wet chirp signal.

The main difference with [4] lies in the neural network structure responsible
for predicting the transfer function. We begin by employing LSTM on the LFO
signal to enhance the model’s robustness under varying LFO frequency
conditions. The features are then passed through fully-connected (FC) layers
to shape the frame-dependent transfer function. FiLM layers are inserted
between these FC layers to incorporate the feedback condition and effect type
embeddings. The final output comprises predicted transfer functions for each
input frame, resulting in a complex matrix equivalent in size to the STFT
matrix of the dry signal. After element-wise multiplication of these matrices,
an inverse STFT is conducted to derive the predicted wet signal. The
comprehensive model architecture is depicted in Figure 1.

Figure 1: Overall model architecture. Blocks that have both dotted lines and
solid lines as input indicate that the input data is distinctive for the
training phase and inference phase. Grey-colored blocks indicate they have no
trainable parameters and their processes are differentiable, thus enabling
backpropagation. $\mathbf{z_{a}}$, $\mathbf{z_{b}}$, and $\mathbf{c_{emb}}$
are trainable parameters.

Our model employs both time domain loss and frequency domain loss for the
training objective. For the time domain loss, we utilize the error-to-signal
ratio (ESR) defined as:

$\mathcal{L}_{ESR}(y,\hat{y})=\frac{\sum_{l=0}^{L-1}(y[l]-\hat{y}[l])^{2}}{\sum_{l=0}^{L-1}y[l]^{2}}$
(1)

where $L$ is the length of the training data in samples. For the frequency
domain loss, we adopt the multi-resolution spectral loss (MRSL) defined as:

$\mathcal{L}_{MRSL}=\sum_{i}||S_{i}-\hat{S_{i}}||_{1}+||\log
S_{i}-\log\hat{S_{i}}||_{1}$ (2)

where $i$ is the index of each FFT size, which is set as (512, 1024, 2048).
The neighboring frames during the STFT are overlapped by 75%. Consequently,
the final training objective of our model becomes:

$\mathcal{L}(y,\hat{y})=\lambda\mathcal{L}_{ESR}(y,\hat{y})+\mathcal{L}_{MRSL}$
(3)

where the weight $\lambda$ is set to 2 throughout all experiments.

###  3.2 Conditioning the control parameters

We employed the FiLM layer to apply the feedback condition to the model [20].
This is mathematically formalized as the following:

$\begin{split}&\text{MLP}([\mathbf{c_{fb}},\mathbf{c_{emb}}])=[\gamma,\beta]\\\
&\text{FiLM:\ \ }\mathbf{y_{fc}}=\gamma\cdot\mathbf{x_{fc}}+\beta\end{split}$
(4)

where $\mathbf{c_{fb}}$ denotes the feedback, $\mathbf{c_{emb}}$ is the
embedding of effect type, $[\cdot,\cdot]$ denotes simple concatenation of two
vectors, $\gamma$, $\beta$ are the output factors of FiLM, and
$\mathbf{x_{fc}}$, $\mathbf{y_{fc}}$ are the intermediate input feature and
its affine transformed output respectively. Effect type embedding
$\mathbf{c_{emb}}$ is only used when the model is trained on multiple phaser
effects. (refer to Section 5.4).

In [4], the time-varying behavior of the target wet audio is modeled as a
sinusoidal LFO signal and is fitted through gradient descent during the
training phase [24]. The sinusoidal LFO signal is generated from input
parameters directly related to the frequency $\mathbf{z_{a}}$ and initial
phase $\mathbf{z_{b}}$, which is then fed into the LSTM block. In our
approach, we follow the LFO fitting methodology in [4] and additionally train
the model on target wet audio modulated with multiple LFO frequencies to
achieve robust controllability over frequency. The parameters being optimized
are thus $(z_{a}^{1},z_{b}^{1}),(z_{a}^{2},z_{b}^{2})$, …,
$(z_{a}^{N},z_{b}^{N})$ where each of the $N$ couple of parameters is assigned
to train different LFO signals. When an audio pair with LFO frequency
corresponding to the $i$-th signal is input to the model, other parameters are
not updated, utilizing the stop gradient operator [25], as illustrated in
Figure 2.

Figure 2: Proposed model training technique on multiple LFO frequency
controls. For a random input-output pair with specified LFO frequency
settings, only the corresponding $z_{a}$, $z_{b}$ parameters are optimized.
Figure depicts the case when $z_{a}^{2}$, $z_{b}^{2}$ are optimized. Other
parameters are not updated by employing stop gradient operation.

##  4 Experiments

###  4.1 Dataset Preparation

We utilize a chirp-train signal with impulse period 40ms, passed through
cascaded 64 all-pass filters with a coefficient of $p=0.9$, following [4].
Training data consists of 6.67 seconds of chirp signal, while direct-input
(DI) recordings provided in [4]222https://github.com/a-carson/ddsp-
phaser/tree/main/audio_data are utilized for inference. For digital phaser and
flanger effects, we vary the LFO frequency(Hz) among $0.23,0.73,1.13$ and the
feedback (%) parameters among $0,25,50,75$. The number of control parameter
values are selected heuristically, resulting in a total of 12 audio samples
used for training the model. All other effect parameters except for LFO
frequency and feedback parameter remain fixed. Overall parameter settings are
organized in Table 1 Notably, 23, 73, and 113 are all prime numbers, ensuring
that the LFO generator provides various and non-overlapping LFO phase to the
LSTM block.

Table 1: Control parameter experiment settings for the digital effect. Studio
One Digital Phaser (Phaser2)
---
Modulation | Stages | 6
Speed (Hz) | 0.23, 0.73, 1.13
Color | Center | 1300
Range | 80%
Feedback (%) | 0, 25, 50, 75
Global | Stereo Spread | 2.0%
Mix | 70.0%
Studio One Digital Flanger
Color | LFO-Amount | 80%
Speed (Hz) | 0.23, 0.73, 1.13
Modulation | Feedback (%) | 0, 25, 50, 75
Delay | 1.50ms
Global | Mix | 70.0%

The LFO frequency of the analog effect was adjusted to closely match the LFO
settings of the digital effects. For the analog phaser, the LFO frequency knob
was set to 8 o’clock, halfway between 8 o’clock and 9 o’clock, and 9 o’clock,
resulting in corresponding LFO frequencies (in Hz) of ${0.3549,\allowbreak
0.7519,\allowbreak 1.430}$. Similarly, for the analog flanger, the LFO
frequency knob was set to 12 o’clock, 3 o’clock, and halfway between 3 o’clock
and 4 o’clock, resulting in corresponding LFO frequencies of
${0.2438,0.6816,1.004}$. Regarding the feedback parameter, we determined the
maximum feedback value that produced an audible sound (3 o’clock for the
analog phaser and 12 o’clock for the analog flanger). Subsequently, we set
four feedback values that evenly divided the range between no feedback and the
maximum audible feedback. We designated each feedback such that a clockwise
rotation of the knob by ${45^{\circ}}$ corresponds to a 12.5 increment in
feedback. Consequently, the resulting feedback labels are
${0.0,12.5,37.5,62.5}$ for the analog phaser and ${0.0,12.5,25,37.5}$ for the
analog flanger. As with the digital effects, all other effect parameters
except for the LFO frequency and feedback parameters, are held constant.
Specific parameter settings are presented in Table 2.

Table 2: Control parameter experiment settings for the analog effect. Moore
Liquid Phaser
---
Phaser Type | 12 o’ clock
LFO shape | Round
Speed (Hz) | 0.3549, 0.7519, 1.430
Color | 0.0, 12.5, 37.5, 62.5
Flamma FC15 Analog Flanger
Mode | Normal
Range | 12 o’clock
Rate (Hz) | 0.2438, 0.6816, 1.004
Color | 0.0, 12.5, 25, 37.5

###  4.2 Baselines

For model comparison, we choose the previous grey-box phaser model[4] as our
baseline. This model has a frame-based approach similar to our proposed model
but is limited in learning the phaser effect due to its confined structure.
Consequently, it cannot be trained for other time-varying filters or delay-
based audio effects such as flanger. Additionally, although the baseline model
may have the potential for controllability by manipulating the LFO signal,
this capability is not explicitly explored in the paper. Therefore, multiple
baseline models are trained individually for each LFO frequency and feedback
condition. We conducted the comparison for both digital and analog phasers.

###  4.3 Training and Inference Setup

To ensure that all LFO-related parameters are learned independently, all
models are trained with a batch size of 1. We train audio with sample rate of
44.1kHz and adopt an FFT size of 4096, hop size of 441 and a fixed frame size
of 1764 throughout the experiments. Adam optimizer with exponential learning
rate decay ($\gamma=0.9997$) and an initial learning rate of 0.001 is
employed. The models are trained for 10,000 epochs while each epoch is
composed of 12 chirp signals, resulting in a total training duration of 80.04
seconds. The hidden size of LSTM layer and MLP layers are 32 and 512
respectively, where a single MLP layer predicts the real and imaginary part of
the STFT matrix. With these settings, the model has total 2.4M learnable
parameters. For the digital effect, the initial LFO phase is identical for
each recording. Consequently, only three LFO generator parameters,
corresponding to the variations in LFO frequency, need to be trained. In the
case of analog effect, we could not designate a desired initial LFO phase
during the recording of the 12 wet targets. Therefore the model had to predict
12 distinct LFO signals for each of the 12 wet chirp signals, leading to
unstable convergence. Since we cannot obtain the groundtruth initial phase and
LFO frequency values for the analog effect, we conducted pilot experiments by
training the 12 signals individually and provide the model the trained
$\\{z_{a}^{i}\\}_{i=1}^{12}$, $\\{z_{b}^{i}\\}_{i=1}^{12}$ as initial values
to stabilize the training.

For quantitative evaluation, we utilized the ESR metric on 10s direct-
input(DI) guitar recording.

##  5 Results

In this section, we aim to evaluate the performance of our model in modelling
digital and analog phaser/flanger effects compared to the baseline model. We
also highlight our model’s controllability across various control parameter
settings. Ablation studies on long-term audio inference capability and the
effect of training on multiple LFO frequencies are conducted. Lastly, we
showcase a model trained on two distinct phasers simultaneously, which implies
the potential of our approach in universal modelling.

###  5.1 Main Results

####  5.1.1 Digital Phaser

Table 3 presents the performance of our model targeting digital phaser under
seen control parameter settings. Results show a consistent trend of increased
ESR with higher feedback. As shown in the table, our model significantly
outperforms the baseline model.

Table 3: ESR(%) results for model trained on the digital phaser (Studio One Phaser2) under seen control parameter conditions. Training separate models for each control parameter configuration obtained results for the baseline model[4]. LFO freq.(Hz) | Feedback(%) | ESR(%)
---|---|---
CONMOD |  Baseline[4]
0.23 | 0 | 0.30 | 18.3
25 | 0.21 | 19.6
50 | 0.86 | 22.6
75 | 4.34 | 32.2
0.73 | 0 | 0.08 | 19.1
25 | 0.09 | 21.9
50 | 0.21 | 23.7
75 | 2.20 | 34.3
1.13 | 0 | 0.07 | 38.6
25 | 0.08 | 22.4
50 | 0.17 | 22.8
75 | 1.96 | 49.8

Figure 3 shows the model inference score for both seen and unseen LFO
frequencies and feedback during the training. Evaluations with unseen
parameter configurations have been included to assess the model’s
controllability. We found that the model demonstrates robust inference
capabilities for unseen LFO frequencies, even outside the range of the
training data. This indicates that the LSTM in our model is capable of
predicting the spectral behavior for the varying LFO frequencies. We also
found that our model shows consistent performance for unseen feedback
conditions. It is evident that the FiLM layer in our model demonstrates strong
performance in facilitating continuous control inferences.

Figure 3: ESR(%) results for model trained on the digital phaser under seen
and unseen LFO frequencies and Feedback parameters. Seen cases are labeled
black, and orange for unseen cases.

####  5.1.2 Analog Phaser

Figure 4 outlines the ESR results for the analog phaser model under seen
control parameter settings. Similar to digital phaser, increased feedback
leads to reduced accuracy. Notably, the baseline model failed to converge. We
attribute this to the baseline structure learning model parameters assuming
that the phaser is comprised of cascaded first-order all-pass filters.

Figure 4: ESR(%) results for model trained on the analog phaser under seen LFO
frequencies and Color parameters.

Figure 5 presents the model performance on unseen control parameter settings.
The model shows effectiveness in predicting unseen feedback cases. However, it
encounters challenges when inferring unseen LFO frequencies, especially those
significantly higher than those observed within the seen frequency range. This
difficulty may arise from complexities in modelling the nonlinear
characteristics of analog circuits.

Figure 5: ESR(%) results for model trained on the analog phaser under seen and
unseen LFO frequencies and Color parameters. Seen cases are labeled black, and
orange for unseen cases. Color is controlled over a fixed LFO frequency (Top).
LFO frequency is controlled over a fixed Color rate (Bottom).

####  5.1.3 Digital Flanger

The flanger effect demonstrates higher spectral complexity compared to the
phaser, introducing greater number of notches in the spectrogram of transfer
function[17]. As a result, the flanger exhibits an overall higher ESR than the
phaser.

Figure 6 displays the model’s inference capability across both seen and unseen
control parameters. The model exhibits strong performance for both seen and
unseen feedback values. However, the model encounters difficulties in
precisely predicting very high LFO frequencies, such as 10Hz. We anticipate
that implementing adversarial loss [26] could mitigate these issues for the
extreme control parameter settings, and leave this as a consideration for
future work.

Figure 6: ESR(%) results for model trained on digital flanger for under seen
and unseen LFO frequencies and Feedback parameters. Seen cases are labeled
black, and orange for unseen cases.

####  5.1.4 Analog Flanger

Figure 7 presents the ESR outcomes for the analog flanger model. Similar to
previous experiments, the ESR decreases as feedback increases. Although the
ESR is higher compared to the phaser scenario, it maintains an acceptable
value overall.

Figure 8 illustrates the model’s inference performance on unseen LFO frequency
and feedback parameters. We observed that the model demonstrates robust
interpolation for unseen feedback. Analogously, the model exhibited
limitations in accurately inferring higher LFO frequencies.

###  5.2 Robustness on long-term audio inference

Robust long-term audio generation is essential for effect pedals, considering
their use cases. Therefore, we analyzed our model’s performance in long-term
generation, with the results illustrated in Figure 10. In the case of the
analog flanger, a minor increase in ESR was observed as the sequence length
extended. However, in general, our model demonstrated the ability to generate
stable sequences of up to 60 seconds. Due to the periodic nature of the LFO,
the LSTM was able to perform inference effectively even as the sequence length
increased.

###  5.3 Effect of training the model on multiple LFO frequencies

To investigate the effects of training the model with multiple LFO
frequencies, we compared the unseen LFO frequency inference capabilities of
model trained with single LFO frequency and multiple LFO frequencies. As shown
in Figure 11, the spectrogram of the audio output from the model trained with
a single LFO frequency (left) exhibits an inaccurate rightward-shifted notch.
In contrast, our proposed architecture leveraging multiple LFO frequencies
(center) displays a notch shape identical to the target audio (right). This
outcome aligns with expectations, due to the restricted diversity of LFO
phases encountered when the model is trained using a single oscillator.

Figure 7: ESR(%) results for model trained on the analog flanger under seen
LFO frequencies and Color parameters.

Figure 8: ESR(%) results for model trained on the analog flanger under varying
LFO frequencies and Color parameters. Seen cases are labeled black, and orange
for unseen cases. Color is controlled over a fixed LFO frequency (Top). LFO
frequency is controlled over a fixed Color rate (Bottom). Figure 9: Analysis
of the multiple-phaser effects trained results, depicted in spectrograms. For
better visualization, we used chirp signal as input. The model is
simultaneously trained with two digital phaser effects: Studio One Phaser2 and
Pedalboard phaser. Each spectrogram illustrates the model’s output with the
Pedalboard phaser embedding(leftmost), the model’s output with the Studio One
Phaser2 embedding(left), a naive linear sum of the outputs from each
embedding(right), and model’s output with the interpolated
embedding(rightmost).

Figure 10: ESR(%) for various input sequence length.

Figure 11: Analysis on the effect of training the model on multiple LFO
frequencies. Each spectrogram illustrates the output of single LFO frequency
model(left), multiple LFO frequency model(center), and target audio(right).

###  5.4 Training on multiple phaser effects

To investigate the capability of our black-box modelling approach, we
additionally trained our model concurrently on two distinct digital phasers:
Studio One Phaser2 and the Pedalboard
Phaser333https://github.com/spotify/pedalboard. The detailed control parameter
settings for the training are described in Table 4. The 2-dimensional
embedding vector $c_{emb}$ was concatenated with $c_{fb}$ to be an input
condition of the FiLM block. After training the model, we derived a unique
embedding vector interpolated between the two vectors each representing
individual effects and analyzed the model’s output upon this condition. As
illustrated in Figure 9, the model demonstrates consistently robust
performance when conditioned with the learned embeddings for both phaser
effects (leftmost and left). Notably, when the interpolated embedding is
employed as the model condition (rightmost), the output exhibits a distinctive
characteristic that lies between the two effects. For comparative analysis, we
also present a result of a naive approach: a spectrogram of linearly summed
audio from the two phaser effects (right).

Table 4: Control parameter experiment settings for training multiple phaser
effects. Studio One Digital Phaser (Phaser2)
---
Stages | 6
Speed (Hz) | 0.37, 1.13
Center | 1300
Range | 80%
Feedback (%) | 0, 25, 50, 75
Stereo Spread | 2.0%
Mix | 70.0%
Pedalboard Phaser
Rate (Hz) | 0.37, 1.13
Centre Frequency (Hz) | 1300
Depth | 0.5
Feedback | 0.0, 0.25, 0.5, 0.75
Mix | 0.5

##  6 Conclusion

In this study, we presented a black-box model for LFO-driven effects targeting
phaser and flanger. Our model offers controllability over LFO frequency and
feedback parameters, by employing a combination of LSTM, MLP, and FiLM blocks,
along with a unique training procedure optimizing multiple LFO frequencies.
Comparative analysis with a baseline model revealed our model’s superiority in
both seen and unseen control parameters when modelling the phaser effect.
Additionally, we found that our model can effectively generate audible flanger
effects with reasonable controllability. Also, our model is capable of
generating long sequences consistently with negligible amount of decrease in
accuracy. Finally, we demonstrated our model’s capability to train on two
distinct phaser effects concurrently, resulting in creative mixed-up outputs
between the effects.
However, our model faced challenges in inferring LFO frequencies that were
outside the training data range, particularly evident in analog effect
modelling. The performance was relatively lower for the flanger effect, due to
its higher spectral complexity. Addressing these challenges may involve
implementing methods capable of modelling complex distributions, such as
adversarial loss. We aim to explore these solutions in future work.

## References

  * [1] Tara Vanhatalo, Pierrick Legrand, Myriam Desainte-Catherine, Pierre Hanna, Antoine Brusco, Guillaume Pille, and Yann Bayle,  “A review of neural network-based emulation of guitar amplifiers,”  Applied Sciences, vol. 12, no. 12, 2022.
  * [2] Riccardo Simionato and Stefano Fasciani,  “Fully conditioned and low-latency black-box modeling of analog compression,”  in Proceedings of the International Conference on Digital Audio Effects. DAFx Board, 2023.
  * [3] Marco Comunità, Christian J Steinmetz, Huy Phan, and Joshua D Reiss,  “Modelling black-box audio effects with time-varying feature modulation,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [4] Alistair Carson, Cassia Valentini-Botinhao, Simon King, and Stefan Bilbao,  “Differentiable grey-box modelling of phaser effects using frame-based spectral processing,”  in International Conference on Digital Audio Effects (DAFx), 2023.
  * [5] Jong Wook Kim, Rachel Bittner, Aparna Kumar, and Juan Pablo Bello,  “Neural music synthesis for flexible timbre control,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 176–180.
  * [6] Christian J Steinmetz and Joshua D Reiss,  “Steerable discovery of neural audio effects,”  in 5th Workshop on Creativity and Design at NeurIPS, 2021.
  * [7] Felix Eichas, Marco Fink, Martin Holters, and Udo Zölzer,  “Physical modeling of the mxr phase 90 guitar effect pedal.,”  in DAFx, 2014, pp. 153–158.
  * [8] Jaromír Mačák,  “Simulation of analog flanger effect using bbd circuit,”  in Proceedings of the 19th International Conference on Digital Audio Effects (DAFx-16), Brno, Czech Republic, 2016, pp. 5–9.
  * [9] Thomas Hélie,  “Volterra series and state transformation for real-time simulations of audio circuits including saturations: Application to the moog ladder filter,”  IEEE transactions on audio, speech, and language processing, vol. 18, no. 4, pp. 747–759, 2009.
  * [10] Felix Eichas and Udo Zölzer,  “Black-box modeling of distortion circuits with block-oriented models,”  in Proceedings of the International Conference on Digital Audio Effects (DAFx), Brno, Czech Republic, 2016, pp. 5–9.
  * [11] Marco A Martínez Ramírez, Emmanouil Benetos, and Joshua D Reiss,  “A general-purpose deep learning approach to model time-varying audio effects,”  in International Conference on Digital Audio Effects (DAFx), 2019.
  * [12] Roope Kiiski, Fabián Esqueda, and Vesa Välimäki,  “Time-variant gray-box modeling of a phaser pedal,”  in Proc. Int. Conf. Digital Audio Effects (DAFx-16), Brno, Czech Republic, 2016, pp. 31–38.
  * [13] Joseph T Colonel, Marco Comunità, and Joshua Reiss,  “Reverse engineering memoryless distortion effects with differentiable waveshapers,”  in Audio Engineering Society Convention 153. Audio Engineering Society, 2022.
  * [14] M Martinez Ramirez, Joshua Reiss, et al.,  “End-to-end equalization with convolutional neural networks,” 2018.
  * [15] Marco A Martínez Ramírez, Emmanouil Benetos, and Joshua D Reiss,  “Deep learning for black-box modeling of audio effects,”  Applied Sciences, vol. 10, no. 2, pp. 638, 2020.
  * [16] Christian J Steinmetz and Joshua D Reiss,  “Efficient neural networks for real-time modeling of analog dynamic range compression,”  in 152nd Convention of the Audio Engineering Society (AES), 2022.
  * [17] Alec Wright and Vesa Välimäki,  “Neural modelling of periodically modulated time varying effects,”  in International Conference on Digital Audio Effects. DAFx, 2020, pp. 281–288.
  * [18] Alec Wright and Vesa Välimäki,  “Neural modeling of phaser and flanging effects,”  Journal of the Audio Engineering Society, vol. 69, pp. 517–529, 07 2021.
  * [19] Christopher Mitcheltree, Christian J. Steinmetz, Marco Comunità, and Joshua D. Reiss,  “Modulation extraction for lfo-driven audio effects,”  in International Conference on Digital Audio Effects (DAFx), 2023.
  * [20] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville,  “Film: Visual reasoning with a general conditioning layer,”  in Proceedings of the AAAI conference on artificial intelligence, 2018, vol. 32.
  * [21] Michael L Beigel,  “A digital phase shifter for musical applications, using the bell labs (alles-fischer) digital filter module,”  Journal of the Audio Engineering Society, vol. 27, no. 9, pp. 673–676, 1979.
  * [22] “Studio one manual,” https://s1manual.presonus.com/en/StudioOneReferenceManual.htm,  Accessed: 2024-03-23.
  * [23] “Flamma fc15 analog flanger pedal manual,” https://flammainnovation.com/file/pdf?url=group1/M00/00/07/wKgA0GNFRIeAX4hnAAsXY4lMquE867.pdf&fileName=FC15_Manual_EN_V01_2022.10.11.pdf,  Accessed: 2024-03-23.
  * [24] Ben Hayes, Charalampos Saitis, and György Fazekas,  “Sinusoidal frequency estimation by gradient descent,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [25] Aaron Van Den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  Advances in neural information processing systems, vol. 30, 2017.
  * [26] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  Advances in neural information processing systems, vol. 33, pp. 17022–17033, 2020.