# Study of Encoder-Decoder Architectures for Code-Mix Search Query Translation

Mandar Kulkarni, Soumya Chennabasavaraj, Nikesh Garera  Flipkart Data
ScienceIndia (mandar.kulkarni, soumya.cb<EMAIL_ADDRESS>
###### Abstract.

With the broad reach of the internet and smartphones, e-commerce platforms
have an increasingly diversified user base. Since native language users are
not conversant in English, their preferred browsing mode is their regional
language or a combination of their regional language and English. From our
recent study on the query data, we noticed that many of the queries we receive
are code-mix, specifically Hinglish i.e. queries with one or more Hindi words
written in English (Latin) script. We propose a transformer-based approach for
code-mix query translation to enable users to search with these queries. We
demonstrate the effectiveness of pre-trained encoder-decoder models trained on
a large corpus of the unlabeled English text for this task. Using generic
domain translation models, we created a pseudo-labelled dataset for training
the model on the search queries and verified the effectiveness of various data
augmentation techniques. Further, to reduce the latency of the model, we use
knowledge distillation and weight quantization. Effectiveness of the proposed
method has been validated through experimental evaluations and A/B testing.
The model is currently live on Flipkart app and website, serving millions of
queries.

transformers, query translation

##  1\. Introduction

With the wide reach of the internet and smartphones, e-commerce platforms have
an increasingly diversified user base. Due to the language barrier, native
language speakers feel comfortable interacting with the platform in the
language of their choice. Specifically in India, due to the relatively low
percentage of the English-speaking population, the vernacular platform
capability is essential to provide a reliable and convenient shopping
experience to such users.

The analysis of the in-house query data showed that, apart from English, the
major portion of the queries (around 21%) seen on the platform are code-mixed
Hinglish i.e. queries where one or more Hindi words are written in Latin
script. This could be because of the large Hindi-speaking population in India.
To utilize traditional search workflows for English with Hinglish queries, it
was important to translate Hinglish queries to English.

Code-mix translation has its unique challenges. One of the major challenges is
the unavailability of a parallel training corpus. A well-known approach for
improving the translation model with large target-side monolingual data is
back-translation (Sennrich et al., 2016). However, such an approach is not
feasible with code-mix data since there is no publicly available model which
can reliably translate from English to Hinglish. On top of it, for search
query translation, factors such as spell errors in the input queries,
ambiguity in language identification, non-standard query format, and free-
flowing text pose additional challenges. For some queries, we also encounter
articulation gap issues i.e. the user is not able to correctly articulate the
query to search the product e.g. user inputting a query ’juta bina dori wala’
where the intended product is ’shoe without lace’. In such cases, literal
translation may not provide good search results. Table 1 shows examples and
issues associated with Hinglish queries.

Hinglish Query | Issue
---|---
semsung phone 8 hajar vala | Brand mispelling
redami 5a ki battery | Brand mispelling
laptop ke barrte | Spell errors
makhamal wala seutar | Spell errors
juta bina dori wala | Articulation Gap
baal hawa machine | Articulation Gap
teen char sau wali saree sadi | Free flowing text
kam keemat mein four g mobile | Free flowing text
Table 1. Various issues associated with Hinglish queries.

In this paper, we propose a transformer-based approach for code-mix query
translation. We propose to use the pre-trained encoder-decoder models such as
T5, and BART as the base model and further fine-tune it on the in-domain query
corpus. As opposed to other works on code-mix data such as (Jawahar et al.,
2021), which proposes to use multi-lingual pre-trained models, we experiment
with models trained only on English text. A parallel corpus for the training
is created with monolingual Hindi and Hinglish queries. To detect Hinglish
queries, we developed a ML-based approach. We use translation and
transliteration APIs for quick and in-expensive labeling with unlabeled
queries. We observe that translations on the search queries are noisy,
possibly due to domain mismatch. To make the translation robust against noise
and issues such as spell errors, we experiment with different data
augmentation schemes, and the model is trained using a combination of
supervised and data augmentation objectives. To mitigate issues with the
articulation gap, we use a small set of manually tagged Hinglish queries. To
reduce the latency of the model in production, we use Knowledge Distillation
(KD) and Weight Quantization (WQ). Experimental results under various settings
demonstrate that our approach can provide reliable translations for the
queries. Post successful results with AB testing, the model is currently live
in production, serving millions of queries. The main contributions of the
paper are as follows.

  * •

Validating the effectiveness of English pre-trained models and data
augmentations for code-mix search query translation which achieves 25 BLEU
points improvement over generic in-house models and third-party APIs

  * •

Developing a Knowledge Distillation (KD) approach for faster inference which
achieves more than 4x reduction in latency with a small drop in the accuracy

  * •

Developing Hinglish language detection approach to identify code-mix queries

##  2\. Related work

Transformers (Vaswani et al., 2017) are widely used for seq2seq applications
such as translation. Warm starting the transformer training with pre-trained
checkpoints proved to be an effective method for improving the translation
model. Rothe et.al (Rothe et al., 2020) extensively studied the effectiveness
of such a method for a wide variety of Natural Language Processing (NLP)
tasks. It was observed that, for encoder-decoder models, good results are
observed when the encoder and the decoder share the vocabulary.

In a recent work on code-mix translation, Jawahar et al. (Jawahar et al.,
2021) explored the use of multilingual pre-trained encoder-decoder models such
as mBART and mT5 for translation from English to Hinglish. On the contrary, in
our work, we explore the use of pre-trained models trained only on English
text.

Due to the lack of parallel corpus for training, few of the earlier works
attempted to use existing MT systems for code-mix translation. Dhar et al.
(Dhar et al., 2018) explored augmenting code-mix data for use with existing MT
systems such as Google Translate. Srivastava et al. (Srivastava and Singh,
2020) also explored the use of existing MT systems such as Google Translate
and Bing for translating code-mix text. They also propose a partial
translation approach based on token language labels along with MT systems.

##  3\. Exploring pre-trained checkpoints for query translation

Setting | BART-base | T5-base | Bert2Bert
---|---|---|---
Baseline | 31.88 | 32.8 | 19.88
+AutoEncoder | +1.37 | +1.13 | +7.13
+DropChar | +1.32 | +1.14 | +1.8
+Masking | +1.14 | +1.43 | +5.02
+WordOrderPermute | -3.1 | -7.4 | +5.6
Table 2. Comparison result for models and data augmentations. Baseline result
indicates BLEU scores with pre-trained models without using data augmentation.
Rows 2-5 indicate a change in the Baseline BLEU score when a particular data
augmentation is used during the training.

Pre-trained transformer models are proved to be effective in a wide variety of
Natural Language Processing (NLP) applications. For translating code-mix
Hinglish queries to English, we experiment with different pre-trained encoder-
decoder models and further fine-tune these models on the in-domain query data.
The model is fine-tuned with the combination of two loss components as shown
in Eq. 1. $Loss_{s}$ corresponds to the cross-entropy loss for the labeled set
of queries while the $Loss_{d}$ corresponds to the cross-entropy loss for the
data augmentation.

(1) $\displaystyle L=(1-\lambda)\hskip 2.84544ptLoss_{s}+\lambda\hskip
2.84544ptLoss_{d}$

where $\lambda$ indicates the weighting factor for the supervised and the data
augmentation loss.

In the following sections, we describe the details of the training data,
models, and data augmentations techniques experimented with for the training.

###  3.1. Data preparation

As noted earlier, one of the major challenges with code-mix translation is the
unavailability of parallel corpus for training. To identify the pre-trained
model best suited for the code-mix translation task, for the initial set of
experiments, we created a parallel training set of 200k queries. We used
monolingual Hindi query data and transliterated it to English (to get
Hinglish) and translated it to English (to get the English target label). For
translation and transliteration of the monoloingual data, we use generic
domain third-party APIs and also observe comparable results of the same with
generic in-house models in 8.2.

###  3.2. Test data preparation

For evaluation, we created a parallel test set of 15k manually tagged Hinglish
queries. The test set is formed based on the query impressions (repetitions)
i.e. the test set consists of approximately equal proportions of Head, Torso,
and Tail queries. The same test dataset is used for all the experimental
evaluations in the paper.

###  3.3. Pre-trained models

For query translation use-case, we explored three pre-trained encoder-decoder
models.

####  3.3.1. T5

Text-To-Text Transfer Transformer (T5) (Raffel et al., 2020) is the
transformer encoder-decoder model trained in a self-supervised manner on a
large corpus of unlabeled English text. We use the T5-base model which has
222M trainable parameters and 12 layers in the encoder and decoder. For our
application, since T5 is fine-tuned on a single task, we do not add any pre-
text to the input queries during training/inference. The model is fine-tuned
for 3 epochs using AdamW optimizer with a learning rate of 1e-5 and batch size
of 32.

####  3.3.2. BART

BART (Lewis et al., 2019) is a transformer model trained using de-noising
autoencoder objective. We use the Bart-base model which has 139M trainable
parameters and 6 layers in the encoder and decoder. The model is fine-tuned
for 3 epochs using AdamW optimizer with a learning rate of 5e-6 and batch size
of 64.

####  3.3.3. Bert2Bert

Rothe et al. (Rothe et al., 2020) demonstrated the effectiveness of pre-
trained checkpoints for sequence generation. We use pre-trained bert-base-
uncased (Devlin et al., 2018) model as the encoder and the decoder since it
showed good performance for translation task (Rothe et al., 2020). Since the
BERT is inherently an encoder model while using it as the decoder, additional
cross attention weights are introduced and these are initialized randomly.
Therefore, the encoder and the decoder both have 12 layers. The model is fine-
tuned for 3 epochs using AdamW optimizer with a learning rate of 5e-6 and
batch size of 32.

For all the models, we use label smoothing during the training since it is
shown to provide robustness to the label noise (Vaswani et al., 2017). The
label smoothing parameter is set to 0.1 for all the experiments. During the
inference, we use beam search decoding where the beam size is set to 3. All
the experiments were carried out on A100 GPU.

###  3.4. Data Augmentation

We have experimented with the following data augmentation techniques.

####  3.4.1. Masking

A word/token masking has proved be an effective self-supervised training
strategy (Lewis et al., 2019) (Raffel et al., 2020). Since search queries are
short in length, we randomly mask a single word from the target (English) and
train the model to reconstruct the entire target. For the BART and BERT2BERT
models, a randomly selected word is replaced with the ’[MASK]’ token while for
T5, it is replaced with ’<extra_id_0>’ token.

####  3.4.2. AutoEncoder

We experiment with Autoencoder (AE) as the self-supervised data augmentation
objective. For this, we use target data as the input and train the model to
reconstruct it. The intuition behind it is that we want to teach the model to
reproduce English words in the query as it is. AE techniques have been
experimented with NMT (Cheng et al., 2016), however, authors have used an
additional target-to-source model to auto-encode the source sentence. In our
work, we only use a single source-to-target model for the AE objective.

####  3.4.3. DropChar

Since non-English speakers attempt to spell the words based on the phoneme
sound of it, we observe that typically the first and last character of the
word is spelled correctly while the majority of the spelling mistakes are
present in the middle of the word. To render translation robust to such spell
mistakes, we randomly drop a character from 30-50% of the words in the query
and train the model with the corresponding translation as the target.

####  3.4.4. WordOrderPermute

As shown in (Lewis et al., 2019), permuting the order of words in the sentence
has proved to be an effective noise technique. In our case, we randomly
permute the order of the words in the target and train the model to de-noise
it.

For all the data augmentations, for each batch of supervised labeled data, we
randomly sample a batch of queries from the dataset and apply a data
augmentation to it. The model is then updated with a linear combination of
supervised and data augmentation loss. For all the experiments, we set
$\lambda$ to 0.5 (Eq. 1).

###  3.5. Results

To judge the effectiveness of different data augmentation techniques, we
performed an experiment where we first train a Baseline model without data
augmentation. Then, we add each data augmentation, one at a time, and observe
its effect. This experiment is carried out for all the models. Table 2 shows
the BLEU score result of the experiment. All the BLEU scores are computed
using SacreBLEU (Post, 2018).

From the results, we draw a few observations. AutoEncoder, CharDrop, and
Masking provide an improvement over the baseline for all the models.
Surprisingly, a simple data augmentation such as Auto-encoder provides
improvement to the accuracy. We further analyze the effect of AE augmentation
with additional experiments and details are provided in Appendix section 8.3.
WordOrderPermute has a negative effect over the baseline in the case of BART
and T5. The reason could be the lack of grammar in the search queries and
hence, the model is not able to identify the relative positions of the words.
BART and T5 provide comparable accuracies while Bert2Bert gave inferior
results. This indicates that the selection of the pre-trained model has an
effect on the accuracy and some of the pre-trained embeddings are more
favorable than others.

##  4\. Hinglish query detection

To detect Hinglish code-mix queries, we developed a ML based approach. We
chose 80k search queries randomly and got them manually tagged for each word
in the sequence. Each word is tagged as English, Hinglish or Other languages.
From this, we set aside 20k search queries as a test set.

For query language detection, we explored text classification and sequence
labeling approaches. For text classification, the entire query is tagged as
English if all the words are English, it is tagged as Hinglish if any of the
words is Hinglish and it is tagged as Other if all the words are neither
English nor Hinglish. We tried following classification models. 1. fasttext
classifier, where the word embeddings are averaged and passed through a
softmax classifier. 2. 1-D CNN model with softmax classifier where filter size
is set to three. 3. Single layer LSTM model with softmax classifier.

Model | Precision | Recall | F1 score
---|---|---|---
Fasttext | 0.90 | 0.61 | 0.73
CNN | 0.80 | 0.65 | 0.72
LSTM | 0.82 | 0.63 | 0.71
CRF | 0.84 | 0.72 | 0.76
Table 3. Comparison of different approaches for Hinglish query detection

Apart from this, we also tried the sequence labeling approach - where we
utilize word-level labels in the query. We tried Conditional Random Field
(CRF) with a context window of 3 - current, previous, and next word. For CRF,
the features used were character n-grams (uni, bi, tri, quad), length of the
word, whether the word contains digits, and whether the word contains special
characters. While inference, a query is termed as Hinglish if any of the words
is classified as Hinglish.

Table 3 shows the comparison results for the different approaches. CRF model
provides the best results compared to other approaches. On the test set, a
simple keyword-based approach provides precision and recall of 77% and 28%
respectively while the CRF-based approach provides recall of 72%, thus
improving the detection recall by more than 2x.

##  5\. Training corpus preparation

For capturing more word variations, we prepared a training corpus derived from
the incoming traffic of unlabeled Hindi and Hinglish queries. We used generic
domain third-party APIs for translation and transliteration and also
experimented with training in-house generic translation and transliteration
models for data tagging, and the results are given in section 8.2. The
parallel query training dataset is created as follows:

  * •

Unlabeled Hindi queries: Translating and transliterating them to English

  * •

Unlabeled Hinglish queries: First transliterating to Hindi followed by
Translation to English

Hindi (Devanagari) queries are filtered based on the unicode ranges for the
script. We also observe a very small percentage of Marathi queries since it
shares a script with Hindi. We did not specifically filter out Marathi
queries. For Hinglish language detection, we use the approach mentioned in
section 4.

We created a parallel corpus of 5M queries for training the model. We observe
that translations with are noisy possibly due to domain mismatch. Therefore,
for testing and fine-tuning with cleaner labels, we also created a dataset of
70k Hinglish queries tagged in-house. Note that the in-house tagged data is
less than 1.5% of the total labeled set.

For model training, we follow a two-stage process. In the first stage, we use
5M noisy query annotations for fine-tuning the model where we warm start the
training with the existing pre-trained checkpoints. We use the loss function
described in Eq. 1 for training the model. We use DropChar, AutoEncoder, and
Word Masking as the data augmentation techniques. For each batch of supervised
data, a data augmentation method is chosen randomly and applied to the
randomly sampled query batch to compute $Loss_{d}$. The model is trained for 5
epochs with the learning rate 5e-6 and batch size of 64. In the second stage,
we further fine-tune the model on the in-house labeled set of 55k queries
where 10% of queries are used for validation and the remaining 15k queries are
used as the test set. During the second stage of fine-tuning, we only use
DropChar, and AutoEncoder data augmentations and the model is fine-tuned until
the validation loss seizes to improve. We use label smoothing for both stages
where the label smoothing parameter is set to 0.1. During inference, we use
beam search decoding with a beam size of 3.

Model | BLEU
---|---
Azure Translate | 25.5
Google Translate | 35.8
RandomInit BART | 49.9
BART w/o DataAug | 50.3
BART with DataAug | 51.3
Table 4. Results on Hinglish test set Model | # layers | KD loss | BLEU | p95
---|---|---|---|---
BART teacher | 6 | - | 51.3 |  $\sim$ 200 ms
Student (Q.) | 1 | CE | 49.7 |  $\sim$ 47 ms
Student (Q.) | 1 | JS | 50.8 |  $\sim$ 47 ms
Table 5. Results for Knowledge Distillation. (Q.) indicates the Weight
Quantization in the student.

In an earlier experiment, since BART and T5 gave comparable results, we
experimented with both the models and computed the BLEU scores on the test
set. BART-base and T5-base gave BLEU scores of 51.3 and 51.33 respectively.
Interestingly, though the BART-base model is less deep, it provides comparable
results T5-base which has twice the number of layers. This indicates that for
the query translation, deeper models may not necessarily provide an advantage
possibly due to the lack of grammar in the queries. We choose BART for further
experiments because it gave comparable results to T5.

Further, to validate the effectiveness of warm starting the training with pre-
trained weights, we did an additional experiment. We randomly initialize the
BART model with weights as $\mathcal{N}(0,0.02)$ and follow the two-stage
training procedure as explained earlier. Table 4 shows the comparison result.
The result indicates that starting the training with pre-trained weights
provides better BLEU scores. Hence, pre-trained weights as well data
augmentations are essential for good performance.

We also compared our result with general domain translations from Azure and
Google translate. Hinglish queries are first transliterated to Hindi, and then
a Hindi-to-English translation is used. The first and second row in Table 4
indicates the BLEU score result. The generic domain translators have
significantly lower accuracies than the proposed model, possibly due to domain
mismatch.

Hinglish Query | English Translation | Issue
---|---|---
semsung phone 8 hajar vala | samsung phone for 8 thousand rupees | Brand mispelling
redami 5a ki baitiry | redmi 5a battery | Brand mispelling
goldan kalar ka aie linar | golden colour eye liner | Spell errors
lenovo k6 ka kabar | lenovo k6 cover | Spell errors
makhamal wala seutar | velvet sweater | Spell errors
resmi 7a kwr | redmi 7a cover | Spell errors
250 tak ki biluthuth | bluetooth upto 250 | Spell errors
baal hawa machine | hair dryer machine | Articulation Gap
juta bina dori wala | shoe without lace | Articulation Gap
teen char sau wali saree sadi | saree for 300-400 rupees | Free flowing text
kam keemat mein four g mobile | 4g mobile in low price | Free flowing text
Table 6. Example results for Hinglish to English query translations.

##  6\. Knowledge Distillation (KD)

For a good user experience, a quick turnaround with search queries is
essential. Hence, it is important to have a low latency translation model. Due
to a large number of parameters, self-attention computation across all layers,
and auto-regressive generation, translation models typically have a higher
latency during inference. To reduce the latency of the model, we use Knowledge
Distillation (KD) where we train a smaller student model with the larger BART
model as the teacher. In particular, we follow an approach proposed in (Kim
and Rush, 2016) for training the student model. We use a single-layer BART
model as the student. We use the following loss function to train the student
model.

(2) $\displaystyle L_{student}=(1-\lambda)\hskip
2.84544pt(Loss_{s}+Loss_{d})+\lambda\hskip 2.84544ptLoss_{KD}$

The training loss consists of three loss components: supervised cross-entropy
loss, data augmentation cross-entropy loss, and Knowledge Distillation (KD)
loss. $\lambda$ is set to 0.5 for all experiments. For the KD loss component,
we experimented with Cross-Entropy (CE) and Jensen-Shannon (JS) loss. In
contrast to Kullback–Leibler (KL) divergence or Cross-Entropy (CE), JS loss is
symmetric, bounded, does not require absolute continuity (Englesson and
Azizpour, 2021). The JS loss is defined as follows.

(3) $\displaystyle JS=D_{KL}(T_{p}||m)+D_{KL}(S_{p}||m)$

Where $T_{p}$ and $S_{p}$ indicate output probability distributions of the
teacher and the student respectively. The distribution $m$ is obtained as
follows.

(4) $\displaystyle m=0.5*T_{p}+0.5*S_{p}$

While training the student model, $Loss_{s}$ and $Loss_{d}$ are computed on
the batch of manually labeled data. For each batch of manually tagged data, we
randomly sample a batch of Hinglish queries from the 5M corpus. We obtain the
pseudo translation labels from the teacher on this Hinglish query batch. Using
input Hinglish queries and corresponding pseudo labels and with teacher
forcing strategy, we obtain probability distributions for the teacher
($T_{p}$) and the student ($S_{p}$). Note that the teacher forcing term here
is generic to transformers and not related to the teacher model in KD.
$Loss_{KD}$ is then computed with either CE or JS. We use Weight Quantization
to further speed up the inference on the CPU devices.

|
---|---
Query: kale jute 300 tak |  Query: mbile phone ki battry
|
Query: baal hawa machine |  Query: peele klr ka gol chasma
Figure 1.  Hinglish query search results on Flipkart website.

Table 5 shows the accuracy result for different KD loss types. JS loss gives a
BLEU score improvement of 1.1 over CE and facilitates much better knowledge
transfer from teacher to student. Our findings are in line with (Englesson and
Azizpour, 2021). We calculate latency for all the models on the test set. The
best student model provides more than 4x speed up for the inference with a 0.5
drop in BLEU score. We also further optimised the latencies via C libraries
for production deployment.

##  7\. Production Search Results

The code-mix translation model is currently live on the Flipkart app and
website. Fig. 1 shows the search results for different Hinglish queries. Note
that we get good search results even for the queries with spell errors
(queries: mbile phone ki battry, peele klr ka chasma), for the queries with
articulation gap (query: baal hawa machine), and for the queries where
attributes such as color and price are specified (query: kale jute 300 tak).

##  8\. Additional experiments

###  8.1. Comparison with multilingual pre-trained models

We compared the results of BART model with multilingual pre-trained models.
Specifically, we consider 2 pre-trained models, mT5-small and mBART-large-50.
We follow a similar training procedure for multilingual models where we first
fine-tune the model with the 5M corpus for 3 epochs using a learning rate of
5e-6 and then further fine-tune with a manually labeled corpus using a
learning rate of 1e-5. Table 7 shows the comparison accuracy. The result
indicates that models pre-trained with English text provide better
initialization than multilingual models for query translation.

Model | BLEU
---|---
BART | 51.3
mT5 | 48.3
mBART | 46.6
Table 7. Multilingual models on Hinglish test set

###  8.2. Comparison with in-house generic models for dataset tagging

We also experimented with building generic translation and transliteration
models in-house for dataset tagging and observed comparable results with the
generic third-party APIs. Model details below:

####  8.2.1. Translation model

We train a generic translation model for English to Hindi translation. For
training the model, we use the following publicly available translation
datasets: Opensubtitiles, Pib, PMI, WikiMatrix, ALT, FLORES-101, indian-
parallel-corpora, Man ki bat,WikiMedia,Tatoeba, Global voices, GNOME, KDE,
Ubuntu. The total size of the training corpus is $\sim$1.4M. We train a
transformer model with 6 layers in encoder and 6 layers in decoder layers. The
model has 12 attention heads, 512 hidden dimension, and $\sim$96M trainable
parameters. Using this model, we get translations on a large corpus
($\sim$157M) of Hindi monolingual text where text is translated from Hindi to
English. The monolingual Hindi text is obtained from datasets such as CC-100,
OSCAR, news-crawl, Wikipedia. Using this Hindi-English parallel corpus, we
then train an English to Hindi translation model. The model architecture is
the same as mentioned earlier in this section.

####  8.2.2. Transliteration model

For building a transliteration model, we use a dataset of size 96.4k of a
word-level manual taggings of Hindi-English transliterations. We train a 2
layer transformer model that uses character level tokenization on this
dataset. The model has 4 attention heads and a hidden dimension size of 128.
For transliterating of an input text, a hybrid approach is followed. If a word
from input text is present in the manual mappings, the corresponding
transliteration is used. If the word is not present, then a transformer model
is used with greedy decoding to get the word transliteration.

To create a large parallel Hinglish-English query corpus, we first translate
$\sim$18M unlabeled English search queries to Hindi using the translation
model. We then use the transliteration model to transliterate Hindi text to
English.

The BART model is then pre-trained on this dataset and finetuned on the
manually tagged Hinglish queries as before. Table 8 shows the BLEU score
comparison result on the test set. With the dataset created with generic in-
house models, we get comparable results as with third-party APIs.

Setting | BLEU
---|---
Dataset created with generic domain third-party APIs | 51.3
Dataset created with in-house generic models | 51.1
Table 8.  Third-party APIs vs in-house generic models

###  8.3. Analysis of Auto-Encoder (AE) objective

Language Model (LM) regularization is known to help in the translation tasks
(Gulcehre et al., 2015). We hypothesize that Auto-Encoder (AE) augmentation
may have a similar effect as adding a LM regularization. To validate this, we
design the following experiment. During the training, we analyze the cross
attention matrices in different layers in the decoder. This is because the
decoder in the conditional generative seq2seq models differs from the decoder
in language models (such as GPT) in terms of additional cross attention
layers. While generating the output tokens, a conditional decoder also pays
attention to encoder states through cross attention layers. Intuitively, the
conditional decoder with identity cross attention matrices would be equivalent
to a language model. To verify if ’any’ of the cross attention matrices in the
decoder are becoming similar to the identity matrix with AE data augmentation
loss, we calculate error using the following equation.

(5) $\displaystyle e=\min_{h}||C-I||_{F}$

where $C$ indicates the cross attention matrix and $I$ indicates the identity
matrix. We calculate the Frobenius norm of the difference between the cross
attention matrix and the identity matrix and the error ($e$) is calculated by
taking the minimum across the attention heads. Note that, since the same
tokenizer is used for the encoder and the decoder, cross attention matrices
are square matrices. The error ($e$) is calculated across all layers in the
decoder. We use the same 200k query dataset as mentioned in section 3.1 for
the experiment. Out of 200k samples, 190k are used for training and 10k are
used for validation. The model is trained with the loss as specified in Eq. 1
where $loss_{d}$ is only calculated with AE augmentation. The model is trained
for 5 epochs and at the end of each epoch, we calculate the average error $e$
on the validation set where target-side data is used as the input and the
output as mentioned in section 3.4.2.

Figure 2 shows the error values for different layers across 5 epochs as the
training progresses. Interestingly, for the initial layers (layer: 2,3 4), the
error goes on decreasing indicating that at least one of the cross attention
matrices is gradually becoming similar to an identity matrix. For the last
(6th) layer though, we do not observe a similar trend indicating that the last
layer features are more task-specific.

---
Figure 2.  Analysis of cross attention matrices for Auto-Encoder data
augmentation.

##  9\. Conclusion

In the paper, we describe a transfer learning-based approach for code-mix
query translation. We found the pre-trained encoder-decoder model and data
augmentation technique best suited for the task through experimentation. A
larger training corpus is created with translation of the unlabeled query data
using generic domain models. To reduce the inference time latency of the
model, we used Knowledge Distillation and Weight Quantization. The
effectiveness of the proposed method has been validated through experimental
evaluations and AB testing. The model is currently live in production,
enabling search with Hinglish language for millions of queries.

##  10\. Acknowledgements

We would like to thank Amey Patil and Gaurav Gupta for their help in getting
results with in-house translation and transliteration models.

## References

  * (1)
  * Cheng et al. (2016) Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016\.  Semi-Supervised Learning for Neural Machine Translation.  arXiv:1606.04596 [cs.CL]
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dhar et al. (2018) Mrinal Dhar, Vaibhav Kumar, and Manish Shrivastava. 2018\.  Enabling Code-Mixed Translation: Parallel Corpus Creation and MT Augmentation Approach. In _Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing_. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 131–140.  https://aclanthology.org/W18-3817
  * Englesson and Azizpour (2021) Erik Englesson and Hossein Azizpour. 2021.  Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels.  arXiv:2105.04522 [cs.LG]
  * Gulcehre et al. (2015) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015.  On Using Monolingual Corpora in Neural Machine Translation.  arXiv:1503.03535 [cs.CL]
  * Jawahar et al. (2021) Ganesh Jawahar, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, and Laks V. S. Lakshmanan. 2021\.  Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing.  arXiv:2105.08807 [cs.CL]
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-Level Knowledge Distillation.  arXiv:1606.07947 [cs.CL]
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  arXiv:1910.13461 [cs.CL]
  * Post (2018) Matt Post. 2018\.  A Call for Clarity in Reporting BLEU Scores.  arXiv:1804.08771 [cs.CL]
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  arXiv:1910.10683 [cs.LG]
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020\.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 264–280.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016\.  Improving Neural Machine Translation Models with Monolingual Data.  arXiv:1511.06709 [cs.CL]
  * Srivastava and Singh (2020) Vivek Srivastava and Mayank Singh. 2020.  PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation. In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_. Association for Computational Linguistics, Online, 41–49.  https://doi.org/10.18653/v1/2020.wnut-1.7
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).


# Optically excited spin dynamics of thermally metastable skyrmions in
Fe0.75Co0.25Si

J. Kalin∗<EMAIL_ADDRESS>Physikalisch-Technische Bundesanstalt, 38116
Braunschweig, Germany  S. Sievers  Physikalisch-Technische Bundesanstalt,
38116 Braunschweig, Germany  F. García-Sánchez  Departamento de Física
Aplicada, Universidad de Salamanca, 37008 Salamanca, Spain  A. Bauer  Zentrum
für QuantumEngineering (ZQE), Technische Universität München, 85748 Garching,
Germany  H. Füser  Physikalisch-Technische Bundesanstalt, 38116 Braunschweig,
Germany  H. W. Schumacher  Physikalisch-Technische Bundesanstalt, 38116
Braunschweig, Germany  C. Pfleiderer  Zentrum für QuantumEngineering (ZQE),
Technische Universität München, 85748 Garching, Germany  Munich Center for
Quantum Science and Technology (MCQST), Technische Universität München, 85748
Garching, Germany  M. Bieler  Physikalisch-Technische Bundesanstalt, 38116
Braunschweig, Germany

###### Abstract

We investigate the microwave spin excitations of the cubic chiral magnet
Fe0.75Co0.25Si as driven by the thermal modulation of magnetic interactions
via laser heating and probed by time-resolved measurements of the magneto-
optical Kerr effect. Focusing on the topologically nontrivial skyrmion lattice
state, the dynamic properties in thermodynamic equilibrium are compared with
those of a metastable state prepared by means of rapid field cooling. In both
cases, we find precessional and exponential contributions to the dynamic
response, characteristic of a breathing mode and energy dissipation,
respectively. When taking into account the universal scaling as a function of
temperature, the precession frequencies in the equilibrium and metastable
skyrmion state are in excellent quantitative agreement. This finding
highlights that skyrmion states far from thermal equilibrium promise great
flexibility, for instance with respect to temperature and field scales, both
for possible microwave applications and the study of fundamental properties.

Magnetic skyrmions are nanoscale spin whirls that show inherent robustness
against external perturbation due to their special topology and have
nontrivial magnetoelectrical properties, such as their intriguing collective
spin dynamics in the GHz-frequency region [1, 2, 3, 4]. Due to that skyrmion
hosting materials are considered a potential candidate for the development of
future microwave- and magnonics-related applications [5, 6, 7, 8]. This has
triggered intensive research of collective skyrmion dynamics [1, 2, 4, 3, 9,
10, 11, 12]. In the pioneering theoretical work by Mochizuki the dominating
skyrmion eigenmodes (i.e., breathing and gyration modes) and their selection
rules were proposed and soon after experimentally verified for several
skyrmion hosting materials [3, 2, 9, 13]. Despite these efforts, one important
obstacle for the transition towards successful skyrmion devices still remains:
The skyrmions exist only in a small temperature and magnetic field region,
difficult to reach in applications. The observation of a thermodynamically
metastable skyrmion state in cubic chiral magnets [14, 15, 16, 17, 18, 19, 20,
21, 22, 23] over large parts of the magnetic phase diagram with long lifetime
[19] paves a new pathway for the realization of microwave- and magnonics-
related applications. However, so far, little is known on how the skyrmion
dynamics develop outside the parameter regime of the equilibrium state in the
non-thermal limit and how they compare to the equilibrium dynamics.

In this letter, we explore the skyrmion dynamics in the chiral B20 magnet
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si in thermodynamic equilibrium and in the
metastable state using time-resolved magneto-optical Kerr effect (TR-MOKE)
measurements. In contrast to the microwave magnetic field [3, 1, 2] and
magneto-optical excitation [9] applied so far to study the dynamics of B20
magnets, we use a thermal excitation mechanism by laser heating with linearly
polarized light [24]. The spin dynamics are triggered by the thermal
modulation of the effective field due to the optically induced temperature
change [25]. As the dynamics are driven by an indirect coupling between
photons and the spin system [26], this technique enables the time-resolved
characterization not only of precession modes, but also of energy dissipation
processes after laser excitation. We study and compare the magnetic field and
temperature dependence of these different dynamical contributions in the
thermodynamic equilibrium and metastable skyrmion state of
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si for the first time. While our study
shows a higher potential of the metastable skyrmion state for possible
microwave applications as compared to the equilibrium state, it also
demonstrates that the metastable skyrmion state is well suited to investigate
generic properties of skyrmion dynamics.

The sample used in this study is a ${2\times 2\times 0.5\,\text{mm}^{3}}$
large Fe0.75Co0.25Si crystal with all edges parallel to the magnetically easy
axes $<100>$ and a magnetic ordering temperature of
${T_{\text{c}}=39\,\text{K}}$. Details about the magnetic properties and
crystal growth are provided in Ref. [15]. In thermal equilibrium,
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si shows the generic magnetic phase
diagram of B20 magnets [27, 28, 14, 29, 30], schematically depicted in Fig.
1(a). At high temperatures Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si is in a
paramagnetic state. Below $T_{\text{c}}$ the helical phase is observed for
fields smaller than the critical field $H_{\text{c1}}$. This phase is
characterized by long-wavelength helices, which are aligned with the
magnetically easy axes $<100>$ given by the cubic anisotropy. At the phase
boundary the spins tilt towards the magnetic field direction and form the
conical phase. Above the critical field $H_{\text{c2}}$ the spins align
collinearly in the field-aligned phase. Note here, that the helical phase of
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si is only observed for zero-field cooling
[15], characteristic for doped compounds [16]. After a cooldown under magnetic
fields, e.g., high-field cooling with $H>H_{\text{c2}}$, the conical phase
extends to zero magnetic fields [15]. Most prominent in the magnetic phase
diagram is the small area just below $T_{\text{c}}$ for intermediate fields
where the periodic magnetic skyrmion lattice (SkL) with hexagonal symmetry
[31], the so called skyrmion pocket, is observed. Beyond that, skyrmions in
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si can also be generated outside the
parameter regime of their equilibrium state by supercooling [14, 15, 16, 19,
20, 21, 22, 23]. In this case the phase transition between the skyrmion pocket
and equilibrium conical phase is suppressed. As a consequence the skyrmion
lattice evolves as thermodynamically metastable state (MSkL), which extends
over large parts of the magnetic phase diagram (see Fig. 1(b)). To that end
the material is cooled down rapidly from temperatures well above
$T_{\text{c}}$ to the desired temperature, while applying a magnetic field
which allows to cross the phase pocket of the equilibrium SkL [16].

Figure 1:  Sketch of the magnetic phase diagram of
Fe${}_{\text{1-x}}$Co${}_{\text{x}}$Si in thermal equilibrium (a) and under
supercooling for rapid cooldowns through the skyrmion pocket (b). The dashed
line in (a) indicates that the helical phase is only present for zero-field
cooling. (c) Schematic optical setup for excitation and detection of the spin
dynamics by TR-MOKE. (d) Typical TR-MOKE signal for the field-aligned (green),
conical (gray) and helical (blue) phase at $15\,$K. The black arrow in (a)
shows the change of the magnetic field during these measurements.

The experimental pump-probe setup for TR-MOKE is depicted in Fig. 1(c). Laser
pulses of about $150\,$fs duration with photon wavelength of $800\,$nm and a
repetition rate of $76\,$MHz were generated by a Ti:Sapphire oscillator. The
linearly polarized laser light was then split into a pump and probe beam. The
pump beam thermally excites the spin ensemble and triggers the magnetization
dynamics. They are detected by analyzing the polarization state of the probe
beam, which rotates by a Kerr angle $\phi_{\text{k}}$ upon reflection from the
sample surface. For this purpose, we use a polarization bridge consisting of a
half-wave plate ($\lambda/2$), Wollaston prism (WP) and a pair of balanced
photodiodes. To enhance the detection sensitivity, the pump beam is
mechanically amplitude modulated at $3\,$kHz. Accordingly, the TR-MOKE signal
$\Delta\phi_{\text{k}}$ is extracted from the balanced photodiode voltage by
subsequent lock-in demodulation. Time-resolved measurements are realized by
adjusting the delay between pump and probe beam, so that the change in
magnetization can be measured as a function of time after excitation with sub-
picosecond resolution. The two laser beams hit the sample at almost normal
incidence and consequently the Kerr rotation is mostly sensitive to the
variation of the out-of-plane magnetization component [32]. In the experiment,
the applied magnetic field and sample temperature were controlled using a
cryostat with superconducting magnet, where the applied magnetic field is
always oriented out of the sample plane.

First, we characterize the equilibrium states at low temperatures. Typical TR-
MOKE signals for the field-aligned, conical and helical phase at $15\,$K are
shown in Fig. 1(d). The measurements are performed subsequent to zero-field
cooling in a field scan going from zero to high magnetic fields, which is
shown schematically by the black arrow in Fig. 1(a). In the field-aligned and
conical phase the TR-MOKE signal rises after the thermal excitation by the
optical pump beam at zero time and reaches its maximum after approximately
$20\,$ps. In the helical phase, the maximum occurs at about $15\,$ps. We
attribute the initial rise of the TR-MOKE signal to the change of the Kerr
rotation induced by the demagnetization of the sample by laser heating. It is
much slower than what is observed for 3d ferromagnets [26]. After the maximum
signal is reached the TR-MOKE signal decreases corresponding to a
magnetization recovery. In the field-aligned phase we observe a monotonous
decay of the TR-MOKE signal, which does not reach its initial state (at $t<0$)
within ${800\,\text{ps}}$. Hereby, the decay does not change significantly
with the applied magnetic field (not shown). In comparison, in the conical
phase the TR-MOKE signal decays on a much faster timescale with a small
overshoot to negative values. This dynamic gets faster with the applied
magnetic field and the overshoot rises for smaller magnetic field values (not
shown). In the helical phase the rapid rise of the TR-MOKE signal is followed
by a highly damped GHz oscillation. This behavior is a clear signature of the
excitation of collective precessional out-of-plane spin dynamics and thus of
helical eigenmodes, the so called helimagnons as already observed for
Fe0.8Co0.2Si [33]. In our measurements the field-aligned and the conical
phases do not reveal an oscillatory signal as typical fingerprint of
collective precessional dynamics. Note however, that the presence of such
collective dynamics cannot be ruled out since they might dominantly only occur
in the sample plane to which our setup is insensitive. Despite of this, the
equilibrium states can be unambiguously identified using characteristic
signatures within the TR-MOKE signal.

Figure 2: (a) TR-MOKE measurements subsequent to rapid field cooling (black
curves), high-field cooling (green curves) and field cooling (gray curve) for
$80\,$mT (upper panel), $55\,$mT (center panel) and $30\,$mT (lower panel).
(b) The solid arrows in the sketched magnetic phase diagram illustrate the
change of temperature during the rapid field cooling (black), high-field
cooling (green) and field cooling (gray), respectively. The dashed green arrow
indicates the change of magnetic field after the high-field cooling.
Temperature evolution of a sensor in sample proximity during rapid (c) and
moderate (d) cooldown. The sample is cooled down from T = $70\,$K (red line) >
$T_{\text{c}}$ (orange line) to $15\,$K (blue line).

We now explore the magnetization dynamics of Fe0.75Co0.25Si after rapid field
cooling to ${15\,\text{K}}$. Taking the phase transition fields from ac
susceptibility measurements [15], we apply three characteristic magnetic field
values: ${H=30\,\text{mT}}$ (below SkL), ${H=55\,\text{mT}}$ (center SkL), and
${H=80\,\text{mT}}$ (above SkL), see Fig. 2(a-b). The sample is cooled via the
helium bath provided by the cryostat from ${70\text{ K}>T_{\text{c}}}$ to
$15\,$K and the cooldown is realized by a sudden increase of the helium flow
in the sample chamber. The temperature evolution of a sensor placed in
proximity to the sample in the cryostat for rapid cooldowns is shown in Fig.
2(c). At zero time the cooling is initialized. The temperature sensor shows an
abrupt cooldown within $30\,$s and settles soon after to the temperature
setpoint of $15\,$K. We take this as an estimation of the time-dependent
sample temperature. In the upper panel in Fig. 2(a) we show the TR-MOKE
measurement subsequent to rapid cooldowns at magnetic fields above the field
span of the equilibrium SkL exemplary for $80\,$mT. The TR-MOKE signal shows
the characteristic of the conical phase measured after high-field cooling
(green curves and also compare with Fig. 2(b)). The same we observe for rapid
cooldowns at magnetic fields below the field span of the equilibrium SkL,
shown for $30\,$mT in the lower panel in Fig. 2(a). Recall here that the
helical phase, which is present for zero-field cooling in this field range, is
not recovered under field cooling. When we cross the equilibrium SkL during
the rapid field cooling (see center panel of Fig. 2(a)), we observe different
dynamics than for the equilibrium states at the same magnetic field and
temperature. The TR-MOKE signal shows a damped GHz oscillation, which settles
to the initial value ($t<0$) after approximately $600\,\text{ps}$. We assign
this behavior to the excitation of collective spin dynamics in the MSkL, which
we stabilized via the rapid field cooling. With that, we demonstrate here the
first experimental observation of collective skyrmion dynamics far from
thermal equilibrium in Fe0.75Co0.25Si. It is worth noticing that even though
we heat the sample locally by laser excitation during the experiment, the MSkL
remains stable on timescales relevant for the characterization. This shows
that the thermal activation cannot trigger the decay of the MSkL,
demonstrating its robustness [18, 19]. In contrast to previous works [14, 15],
moderate cooldowns through the skyrmion pocket (see Fig. 2(d)) are not
sufficient to stabilize the MSkL in our experiment. We observe again the
conical state, shown as the gray curve in Fig. 2(a). Possibly, temperature
gradients across the Fe0.75Co0.25Si sample promote the unwinding of the
equilibrium skyrmion lattice (skyrmion collapse) in our experiments, requiring
higher cooling rates for the stabilization of the MSkL.

Figure 3:  (a) TR-MOKE measurements at $15\,$K for different magnetic fields
after rapid field cooling at $55\,$mT. The measurements were collected in FC+
(vertical arrow upwards) and FC- scans (vertical arrow downwards). (b)
Comparison of the skyrmion dynamics in the MSkL at $15\,$K (red circle) and
the equilibrium SkL at $35\,\text{K}$ (turquoise square) at $55\,\text{mT}$.
The solid lines show the fit with the phenomenological model and the dashed
lines represent the exponential and oscillatory component.

Having established the existence of the MSkL in our experiment, we further
investigate its dynamics versus magnetic field at $15\,$K, see Fig. 3(a). The
sample is rapidly cooled down at $55\,$mT from ${T>T_{\text{c}}}$ to $15\,$K.
The TR-MOKE measurements are recorded in field sweeps for increasing and
decreasing fields, denoted FC+ and FC- scans. Before each scan the cooldown
procedure is repeated. We observe that for magnetic fields ranging from $0$ to
$120\,$mT, the precessional dynamics associated with the MSkL persist. This is
true even outside the magnetic field range where the equilibrium SkL exists.
For fields larger than $120\,$mT we observe again the TR-MOKE characteristic
of the conical phase. This can be explained by the unwinding of the skyrmions
for a critical magnetic field value and the transition to the equilibrium
state. Above H${}_{\text{c2}}$ we enter the field-aligned phase, as indicated
by the slow remagnetization, compare with Fig. 1(d).

Next, we address the question how the skyrmion dynamics in the MSkL compare to
the one in the equilibrium SkL. To this end we plot in Fig. 3(b) the TR-MOKE
signal obtained from the MSkL (${15\,\text{K},55\,\text{mT}}$) and from the
equilibrium SkL (${35\,\text{K},55\,\text{mT}}$). As in the MSkL we also
observe in the SkL an oscillation of the TR-MOKE signal resulting from a
precessional mode, but with smaller amplitude and lower frequency. For a
quantitative comparison of the magnetization dynamics, we use a
phenomenological model which separates oscillatory and exponential components.
Within this framework the TR-MOKE signal $\Delta\phi_{\text{k}}$ is given by

$\Delta\phi_{\text{k}}=\left(e^{-t/\tau_{\text{k}}}(A+B\sin(2\pi
f_{\text{p}}t+\delta))+C\right)\left(1-e^{-t/\tau_{\text{rise}}}\right)\Theta(t)$
(1)

The energy dissipation after laser excitation is modeled by the exponential
term with amplitude $A$. The rise-time $\tau_{\text{rise}}$ is attributed to
the demagnetization and the decay time $\tau_{\text{k}}$ to the magnetization
recovery. The precessional spin dynamics are described by the oscillatory
component with frequency $f_{\text{p}}$ and amplitude $B$. We note here that
phase offsets $\delta$ are required to fit the early dynamics of the TR-MOKE
data properly. The offset term $C$ is needed to fit the dynamics for times
$t>1\,\text{ns}$. The term $\Theta(t)$ is an error function and accounts for
the finite width of the probe pulse and the start of excitation at $t=0$.
Applying this model to fit the measurements, we obtain an excellent agreement,
see Fig. 3(b). In the following we use the model to study the amplitude
($A,B$) and frequency ($f_{\text{p}}$) dependence versus magnetic field and
temperature.

Figure 4: (a) and (b) Amplitudes of the exponential ($A$, solid markers) and
precessional ($B$, open markers) dynamics as a function of magnetic field and
temperature for the MSkL (red markers) and equilibrium SkL (turquoise
markers). Fig. (c) and (d) present the magnetic field and temperature
dependence of the precession frequency in the MSkL (red dots) and SkL
(turquoise square). The solid lines in (c) are linear approximations of the
data to extract $\Delta f/\Delta B$ and $f_{0}$. In (d) the critical field
$H_{\text{c2}}$ as a function of temperature, extracted from Ref. [15], is
plotted for comparison and the black line presents a fit of the data to the
function ${f_{\text{p}}=f_{0}(1-(T/T_{\text{c}})^{\alpha})^{0.5}}$. In a small
temperature gap neither the MSkL, nor the SKL is stable and we observe the
conical phase (gray area).

We first compare the amplitudes of the precessional and exponential dynamics
in the MSkL and equilibrium SkL. We therefore plot the fit parameter $A$
(exponential) and $B$ (precessional) in Fig. 4(a) and (b) as function of
magnetic field and temperature, respectively. In the MSkL, we observe a
strongly non-linear magnetic field dependence of $A$ and $B$. For fields
smaller than ${\mu_{0}H=110\,\text{mT}}$ both increase with magnetic field,
with the precessional part dominating the spin dynamics. However, for higher
fields the amplitude of the precessional mode tends to zero when reaching the
phase transition and the exponential dynamics become the leading process.
Regarding the equilibrium SkL, the precessional mode strongly dominates the
dynamics (${B>>A}$) and scales linearly with the applied magnetic field. In
the limited parameter range, the exponential part ($A$) does not show a clear
field dependence in the SkL. Overall, the TR-MOKE signal (${~{}A+B}$) in the
MSkL is much larger compared to the SkL, which can be explained by the
increased magnetization at low temperatures. This is supported by the
temperature evolution of the amplitudes (see Fig. 4(b)), which shows a
decrease of the signal with temperature. Here, primarily the precessional
dynamics change and the exponential term is rather unaffected. Note, that in a
small gap from approximately $31\,\text{K}$ to $34\,\text{K}$ neither the SkL
nor the MSkL is observed in our experiment, but the exponential dynamics
associated with the conical phase are present.

This amplitude study demonstrates that the MSkL offers an extended view on the
physical processes contributing to skyrmion dynamics, i.e., collective spin
excitation and energy dissipation, compared to the SkL, beneficial for the
investigation of generic skyrmion properties. Furthermore, it shows the higher
potential of the MSkL for applications in comparison to its equilibrium state,
since larger precessional amplitudes promise larger signal-to-noise ratios. In
addition, the increased tunability with the applied magnetic field of the
precessional amplitude $B$, but also of the amplitude ratio ($B/A$) in the
MSkL are important features for future applications.

Finally, we focus on the precessional dynamics and compare the frequency
$f_{\text{p}}$ of the MSkL and equilibrium SkL. For this purpose the magnetic
field and temperature dependence of $f_{\text{p}}$ are presented in Fig. 4(c)
and 4(d), respectively. We observe a frequency decrease for higher magnetic
fields in both states, which is characteristic for the breathing mode of
skyrmions [1, 3]. Our measurement technique enables the selective detection of
the breathing mode, as the other dominating skyrmion eigenmodes, i.e.,
gyration modes, do not cause a modulation of the average magnetization
component out of the sample plane [13]. Our data reveal a linear dependence of
the breathing mode frequency on the magnetic field. The predicted sublinear
dependence [3] approaching the critical field to the conical phase is not
observed. By a linear approximation we determine the frequency change with the
applied magnetic field $\Delta f/\Delta B$ and the zero-field frequency
$f_{0}$. For the MSkL we obtain ${\Delta f_{\text{MSkL}}/\Delta B=(-14.5\pm
0.4)\,\text{GHz/T}}$ and ${f_{\text{MSkL,0}}=(6.19\pm 0.03)\,\text{GHz}}$ and
for the equilibrium SkL ${\Delta f_{\text{SkL}}/\Delta B=(-13\pm
4)\,\text{GHz/T}}$ and ${f_{\text{SkL,0}}=(3.6\pm 0.2)\,\text{GHz}}$,
respectively. The frequency change with the applied magnetic field is
remarkably similar for the MSkL and equilibrium SkL. This indicates that the
characteristics of the dominating skyrmion eigenmodes are preserved in non-
thermal equilibrium. As the MSkL is stable in an extended magnetic field
range, however, it offers a higher frequency tunability as the equilibrium
SkL, beneficial for future applications.

The zero-field frequency $f_{\text{0}}$ of the equilibrium SkL is much smaller
compared to the one in the MSkL, indicating slower spin dynamics in the
equilibrium state. In order to investigate this further, we show in Fig. 4(d)
the precession frequency $f_{\text{p}}$ extracted from the fit of Eq. 1 to the
experimental data for various temperatures. The precession frequency of the
MSkL and equilibrium SkL decreases for higher temperatures and can be fitted
using the phenomenological model
${f_{\text{p}}=f_{0}(1-(T/T_{\text{c}})^{\alpha})^{0.5}}$ with $\alpha=2.2$
and ${f_{0}=5.9\,\text{GHz}}$ (black line in Fig. 4(d)). The precession
frequency change with temperature tracts the $H_{\text{c2}}$ line (see blue
markers taken from Ref. [15] in Fig. 4(d)), which describes the temperature
dependence of the critical field $H_{\text{c2}}$. This universal scaling of
the precession frequency, i.e., the proportionality to the temperature
dependence of magnetic order parameters, demonstrates that the observed
differences of the skyrmion eigenmodes of MSkL and equilibrium SkL is solely
an effect of temperature. Thus, the MSkL is well suited to investigate generic
properties of skyrmion dynamics.

In conclusion, this study explores the spin dynamics of the chiral magnet
Fe0.75Co0.25Si in the equilibrium and metastable skyrmion state using time-
resolved magneto-optical Kerr effect measurements. The skyrmion dynamics are
well modeled by a phenomenological approach, which separates the spin
precession signal, characteristic for the skyrmion breathing mode, from the
exponential part, which denotes the energy dissipation. We observe an
excellent quantitative agreement of the precession frequency in the
equilibrium and metastable skyrmion state considering the universal scaling
with temperature. Our results strongly indicate that the characteristics of
skyrmion eigenmodes are preserved far from thermal equilibrium. Besides this
fundamental significance, the larger amplitude and enhanced tunability of the
MSkL precessional mode point out the higher potential for possible microwave-
and magnonics-related applications compared to the equilibrium SkL.

###### Acknowledgements.

Funding: This work was supported by the European Metrology Research Programme
(EMRP) and EMRP participating countries under the European Metrology Programme
for Innovation and Research (EMPIR) Project No. 17FUN08-TOPS Metrology for
topological spin structures. In part, this study has been funded by the
Deutsche Forschungsgemeinschaft (DFG, German ResearchFoundation) under TRR80
(From Electronic Correlations to Functionality, Project No. 107745057, Project
E1), SPP2137 (Skyrmionics, Project No. 403191981, Grant PF393/19 and Grant
SCHU 2250/8-1), and the excellence cluster MCQST under Germany’s Excellence
Strategy EXC-2111 (ProjectNo. 390814868). Financial support by the European
Research Council (ERC) through Advanced Grants No. 291079 (TOPFIT) and No.
788031 (ExQuiSid) is gratefully acknowledged. Author contributions: MB and HWS
initiated the study. AB and CP grew the crystal. JK performed the TR-MOKE
study under the supervision of MB and HF. The data analysis was performed by
JK. MB, HF, SS, HWS, JK, FGS, AB and CP discussed the results. JK wrote the
paper. Competing interests: The authors declare that they have no competing
interests. Data availability: All data needed to evaluate the conclusions of
the paper are present in the paper.

## References

  * Mochizuki [2012] M. Mochizuki, Spin-wave modes and their intense excitation effects in skyrmion crystals, Physical Review Letters 108, 017601 (2012).
  * Onose  _et al._ [2012] Y. Onose, Y. Okamura, S. Seki, S. Ishiwata, and Y. Tokura, Observation of magnetic excitations of skyrmion crystal in a helimagnetic insulator Cu2OSeO3, Physical Review Letters 109, 037603 (2012).
  * Schwarze  _et al._ [2015] T. Schwarze, J. Waizner, M. Garst, A. Bauer, I. Stasinopoulos, H. Berger, C. Pfleiderer, and D. Grundler, Universal helimagnon and skyrmion excitations in metallic, semiconducting and insulating chiral magnets, Nature Materials 14, 478 (2015).
  * Okamura  _et al._ [2013] Y. Okamura, F. Kagawa, M. Mochizuki, M. Kubota, S. Seki, S. Ishiwata, M. Kawasaki, Y. Onose, and Y. Tokura, Microwave magnetoelectric effect via skyrmion resonance modes in a helimagnetic multiferroic, Nature Communications 4, 1 (2013).
  * Garst  _et al._ [2017] M. Garst, J. Waizner, and D. Grundler, Collective spin excitations of helices and magnetic skyrmions: review and perspectives of magnonics in non-centrosymmetric magnets, Journal of Physics D: Applied Physics 50, 293002 (2017).
  * Mochizuki and Seki [2013] M. Mochizuki and S. Seki, Magnetoelectric resonances and predicted microwave diode effect of the skyrmion crystal in a multiferroic chiral-lattice magnet, Physical Review B 87, 134403 (2013).
  * Finocchio  _et al._ [2015] G. Finocchio, M. Ricci, R. Tomasello, A. Giordano, M. Lanuzza, V. Puliafito, P. Burrascano, B. Azzerboni, and M. Carpentieri, Skyrmion based microwave detectors and harvesting, Applied Physics Letters 107, 262401 (2015).
  * Xing  _et al._ [2020] X. Xing, Y. Zhou, and H. Braun, Magnetic skyrmion tubes as nonplanar magnonic waveguides, Physical Review Applied 13, 034051 (2020).
  * Ogawa  _et al._ [2015] N. Ogawa, S. Seki, and Y. Tokura, Ultrafast optical excitation of magnetic skyrmions, Scientific Reports 5, 1 (2015).
  * Seki  _et al._ [2020] S. Seki, M. Garst, J. Waizner, R. Takagi, N. Khanh, Y. Okamura, K. Kondou, F. Kagawa, Y. Otani, and Y. Tokura, Propagation dynamics of spin excitations along skyrmion strings, Nature Communications 11, 1 (2020).
  * Pöllath  _et al._ [2019] S. Pöllath, A. Aqeel, A. Bauer, C. Luo, H. Ryll, F. Radu, C. Pfleiderer, G. Woltersdorf, and C. H. Back, Ferromagnetic resonance with magnetic phase selectivity by means of resonant elastic x-ray scattering on a chiral magnet, Physical Review Letters 123, 167201 (2019).
  * Aqeel  _et al._ [2021] A. Aqeel, J. Sahliger, T. Taniguchi, S. Mändl, D. Mettus, H. Berger, A. Bauer, M. Garst, C. Pfleiderer, and C. H. Back, Microwave spectroscopy of the low-temperature skyrmion state in Cu2OSeO3, Physical Review Letters 126, 017202 (2021).
  * Padmanabhan  _et al._ [2019] P. Padmanabhan, F. Sekiguchi, R. B. Versteeg, E. Slivina, V. Tsurkan, S. Bordács, I. Kézsmárki, and P. Van Loosdrecht, Optically driven collective spin excitations and magnetization dynamics in the Néel-type skyrmion host GaV4S8, Physical Review Letters 122, 107203 (2019).
  * Münzer  _et al._ [2010] W. Münzer, A. Neubauer, T. Adams, S. Mühlbauer, C. Franz, F. Jonietz, R. Georgii, P. Böni, B. Pedersen, M. Schmidt, _et al._ , Skyrmion lattice in the doped semiconductor Fe${}_{\text{1- x}}$Co${}_{\text{x}}$Si, Physical Review B 81, 041203 (2010).
  * Bauer  _et al._ [2016] A. Bauer, M. Garst, and C. Pfleiderer, History dependence of the magnetic properties of single-crystal Fe${}_{\text{1- x}}$Co${}_{\text{x}}$Si, Physical Review B 93, 235144 (2016).
  * Bauer  _et al._ [2018] A. Bauer, A. Chacon, M. Halder, and C. Pfleiderer, Skyrmion lattices far from equilibrium, in  _Topology in Magnetism_ (Springer, 2018) pp. 151–176.
  * Oike  _et al._ [2016] H. Oike, A. Kikkawa, N. Kanazawa, Y. Taguchi, M. Kawasaki, Y. Tokura, and F. Kagawa, Interplay between topological and thermodynamic stability in a metastable magnetic skyrmion lattice, Nature Physics 12, 62 (2016).
  * Berruto  _et al._ [2018] G. Berruto, I. Madan, Y. Murooka, G. Vanacore, E. Pomarico, J. Rajeswari, R. Lamb, P. Huang, A. Kruchkov, Y. Togawa, _et al._ , Laser-induced skyrmion writing and erasing in an ultrafast cryo-Lorentz transmission electron microscope, Physical Review Letters 120, 117201 (2018).
  * Wild  _et al._ [2017] J. Wild, T. N. Meier, S. Pöllath, M. Kronseder, A. Bauer, A. Chacon, M. Halder, M. Schowalter, A. Rosenauer, J. Zweck, _et al._ , Entropy-limited topological protection of skyrmions, Science Advances 3, e1701704 (2017).
  * Ritz  _et al._ [2013] R. Ritz, M. Halder, C. Franz, A. Bauer, M. Wagner, R. Bamler, A. Rosch, and C. Pfleiderer, Giant generic topological Hall resistivity of MnSi under pressure, Physical Review B 87, 134424 (2013).
  * Okamura  _et al._ [2016] Y. Okamura, F. Kagawa, S. Seki, and Y. Tokura, Transition to and from the skyrmion lattice phase by electric fields in a magnetoelectric compound, Nature Communications 7, 1 (2016).
  * Yu  _et al._ [2018] X. Yu, D. Morikawa, T. Yokouchi, K. Shibata, N. Kanazawa, F. Kagawa, T.-h. Arima, and Y. Tokura, Aggregation and collapse dynamics of skyrmions in a non-equilibrium state, Nature Physics 14, 832 (2018).
  * Bannenberg  _et al._ [2016] L. Bannenberg, K. Kakurai, F. Qian, E. Lelièvre-Berna, C. Dewhurst, Y. Onose, Y. Endoh, Y. Tokura, and C. Pappas, Extended skyrmion lattice scattering and long-time memory in the chiral magnet Fe${}_{\text{1- x}}$Co${}_{\text{x}}$Si, Physical Review B 94, 104406 (2016).
  * [24] For completeness we note that using the thermal excitation technique the spin eigenmodes of a Néel type skyrmion hosting material [13] were characterized in a previous study.
  * Van Kampen  _et al._ [2002] M. Van Kampen, C. Jozsa, J. Kohlhepp, P. LeClair, L. Lagae, W. De Jonge, and B. Koopmans, All-optical probe of coherent spin waves, Physical Review Letters 88, 227201 (2002).
  * Beaurepaire  _et al._ [1996] E. Beaurepaire, J.-C. Merle, A. Daunois, and J.-Y. Bigot, Ultrafast spin dynamics in ferromagnetic nickel, Physical Review Letters 76, 4250 (1996).
  * Bauer and Pfleiderer [2016] A. Bauer and C. Pfleiderer, Generic aspects of skyrmion lattices in chiral magnets, in  _Topological Structures in Ferroic Materials_ (Springer, 2016) pp. 1–28.
  * Mühlbauer  _et al._ [2009] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion lattice in a chiral magnet, Science 323, 915 (2009).
  * Seki  _et al._ [2012] S. Seki, X. Yu, S. Ishiwata, and Y. Tokura, Observation of skyrmions in a multiferroic material, Science 336, 198 (2012).
  * Adams  _et al._ [2012] T. Adams, A. Chacon, M. Wagner, A. Bauer, G. Brandl, B. Pedersen, H. Berger, P. Lemmens, and C. Pfleiderer, Long-wavelength helimagnetic order and skyrmion lattice phase in Cu2OSeO3, Physical Review Letters 108, 237204 (2012).
  * Yu  _et al._ [2010] X. Yu, Y. Onose, N. Kanazawa, J. H. Park, J. Han, Y. Matsui, N. Nagaosa, and Y. Tokura, Real-space observation of a two-dimensional skyrmion crystal, Nature 465, 901 (2010).
  * You and Shin [1998] C.-Y. You and S.-C. Shin, Generalized analytic formulae for magneto-optical kerr effects, Journal of Applied Physics 84, 541 (1998).
  * Koralek  _et al._ [2012] J. Koralek, D. Meier, J. Hinton, A. Bauer, S. Parameswaran, A. Vishwanath, R. Ramesh, R. Schoenlein, C. Pfleiderer, and J. Orenstein, Observation of coherent helimagnons and Gilbert damping in an itinerant magnet, Physical Review Letters 109, 247204 (2012).
  * Nakajima  _et al._ [2017] T. Nakajima, H. Oike, A. Kikkawa, E. P. Gilbert, N. Booth, K. Kakurai, Y. Taguchi, Y. Tokura, F. Kagawa, and T.-h. Arima, Skyrmion lattice structural transition in MnSi, Science Advances 3, e1602562 (2017).
  * Kimel  _et al._ [2004] A. Kimel, A. Kirilyuk, A. Tsvetkov, R. Pisarev, and T. Rasing, Laser-induced ultrafast spin reorientation in the antiferromagnet TmFeO3, Nature 429, 850 (2004).
  * Bigot  _et al._ [2005] J.-Y. Bigot, M. Vomir, L. Andrade, and E. Beaurepaire, Ultrafast magnetization dynamics in ferromagnetic cobalt: The role of the anisotropy, Chemical Physics 318, 137 (2005).
  * Hashimoto  _et al._ [2008] Y. Hashimoto, S. Kobayashi, and H. Munekata, Photoinduced precession of magnetization in ferromagnetic (Ga,Mn)As, Physical Review Letters 100, 067202 (2008).
  * Shibata  _et al._ [2018] H. Shibata, M. Okano, and S. Watanabe, Ultrafast control of coherent spin precession in ferromagnetic thin films via thermal spin excitation processes induced by two-pulse laser excitation, Physical Review B 97, 014438 (2018).
  * Garanin  _et al._ [2020] D. A. Garanin, R. Jaafar, and E. M. Chudnovsky, Breathing mode of a skyrmion on a lattice, Physical Review B 101, 014418 (2020).
  * Burn  _et al._ [2020] D. M. Burn, S. Wang, W. Wang, G. van der Laan, S. Zhang, H. Du, and T. Hesjedal, Field and temperature dependence of the skyrmion lattice phase in chiral magnet membranes, Physical Review B 101, 014446 (2020).
  * Milde  _et al._ [2013] P. Milde, D. Köhler, J. Seidel, L. Eng, A. Bauer, A. Chacon, J. Kindervater, S. Mühlbauer, C. Pfleiderer, S. Buhrandt, _et al._ , Unwinding of a skyrmion lattice by magnetic monopoles, Science 340, 1076 (2013).
  * Bauer  _et al._ [2010] A. Bauer, A. Neubauer, C. Franz, W. Münzer, M. Garst, and C. Pfleiderer, Quantum phase transitions in single-crystal Mn${}_{\text{1- x}}$Fe${}_{\text{x}}$Si and Mn${}_{\text{1- x}}$Co${}_{\text{x}}$Si: Crystal growth, magnetization, ac susceptibility, and specific heat, Physical Review B 82, 064404 (2010).