# What Happens When Small Is Made Smaller? Exploring the Impact of Compression
on Small Data Pretrained Language Models

Busayo Awobade, Mardiyyah Oduwole11footnotemark: 1, Steven
Kolawole11footnotemark: 1
ML Collective
{busayo.awobade, mardiyyah.oduwole<EMAIL_ADDRESS>
These authors contributed equally to this work.

###### Abstract

Compression techniques have been crucial in advancing machine learning by
enabling efficient training and deployment of large-scale language models.
However, these techniques have received limited attention in the context of
low-resource language models, which are trained on even smaller amounts of
data and under computational constraints, a scenario known as the "low-
resource double-bind." This paper investigates the effectiveness of pruning,
knowledge distillation, and quantization on an exclusively low-resourced,
small-data language model, AfriBERTa. Through a battery of experiments, we
assess the effects of compression on performance across several metrics beyond
accuracy. Our study provides evidence that compression techniques
significantly improve the efficiency and effectiveness of small-data language
models, confirming that the prevailing beliefs regarding the effects of
compression on large, heavily parameterized models hold true for less-
parameterized, small-data models.

##  1 Introduction

One of the most challenging aspects of working with large language models
(LLMs) is their computational complexity (Zhang et al., 2021). With 340M
parameters, even the BERT-large model is impractical for deployment on low-end
devices with inadequate computational power (Treviso et al., 2022). Several
architectural changes to make the BERT model more efficient have been made
(Jiao et al., 2019; Sanh et al., 2019; Lan et al., 2019). However, to achieve
adequate performance on downstream tasks, LLMs require huge training corpora
(billions of tokens), which are unavailable for most African languages (Nekoto
et al., 2020). The omission of African languages from the pre-training phase
of LLMs results in low performance in these languages, making NLP tasks
participatory difficult (Kreutzer et al., 2022). Ahia et al. (2021) termed
this situation the "low-resource double-bind" to describe the coexistence of
data and computation limitations on resources. This is a popular NLP setting
for low-resource languages, although the performance trade-offs are
understudied.

One of the most promising attempts to mitigate the sparse presence of low-
resource African languages in model training is the creation of AfriBERTa, the
first multilingual language model trained purely and from scratch on African
languages with $<1$GB of data. AfriBERTa (Ogueji et al., 2021) beats
competitive models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al.,
2020) on text categorization and NER tasks. Instead of depending on high-
resource languages for transfer learning, AfriBERTa takes advantage of
linguistic similarities between languages from low-resource environments to
yield promising results, which is critical in determining the sustainability
of language models trained on small datasets. However, with 126M parameters,
the AfriBERTa-large model is still impractical for deployment on low-end
devices with inadequate computational power. Moreover, little is known about
the ability of a small-data, low-resource-focused model like AfriBERTa to
generalize to unseen-before languages, considering its small size, and its
ability to get "smaller" for efficient usage by resource-constrained users.

This research attempts to bridge the gap between a low-resource, small-data,
high-performance multilingual language model and an ultra-efficient,
deployable model for double-bind users. Our experimental results demonstrate
that pruning achieves $\approx 60\%$ reduction in model size with a minimal
performance drop. Furthermore, generalization tests reveal varied outcomes,
with some languages surpassing dense models even with extreme pruning.
Distillation achieves compression rates between 22% and 33% with comparable
performances. Additionally, quantization reduces the model size by 64.08%,
inference time by 52.3%, and even outperforms the baseline model in the F1
score for certain languages. Our contributions address the following
questions:

  1. 1.

How tiny can we construct a small-data model using the knowledge distillation
framework?

  2. 2.

What are the efficiency and generalization limits of pruning on a small-data
model?

  3. 3.

What are the optimal reductions we can achieve in size and latency utilizing
quantization?

##### Related work

First introduced by Hinton et al. (2015), Sanh et al. (2019) demonstrated
similar performances on downstream tasks with smaller language models pre-
trained using distillation, which is faster at inference and suitable for edge
devices. Jiao et al. (2019) proposed a transformer-specific distillation
method, employing a two-stage learning framework with general and task-
specific distillation using BERT. Han et al. (2015) reintroduced modern
pruning as "network pruning." and the spotlight intensified with Frankle &
Carbin (2019) suggesting the existence of subnetworks within a dense neural
network that matches or surpass the dense model’s performance—termed "winning
tickets." Yu et al. (2019) and Renda et al. (2020) also found winning tickets
early in training for Transformers and LSTMs. Chen et al. (2020) and Prasanna
et al. (2020) also explored trainable subnetworks in pre-trained BERT models,
locating matching subnetworks at 40% to 90% sparsity across various
applications. Li et al. (2020) demonstrated that heavily compressing large
models resulted in higher accuracy than lightly compressing small models. Bai
et al. (2022) also introduced post-training quantization for language models,
minimizing training time, memory, and data consumption, while Wang et al.
(2022) achieved 16$\times$ compression by quantizing transformer backbones to
4-bit and applying 50% fine-grained structural sparsity. Additionally, Xiao et
al. (2022) enabled 8-bit weight, 8-bit activation quantization for large
language models, addressing activation outliers, and Dettmers et al. (2022)
used LLM.int8() on transformers with 16 or 32-bit weights for immediate
inference using vector-wise quantization and mixed-precision decomposition. As
far as we know, our work is the first to explore these techniques’ efficacy in
a low-resource double-bind setting.

##  2 Approach

This section details our experimental settings for the compression techniques
we used to evaluate efficiency in our small-data, pre-trained model of choice.
Further details on our training setup can be found in Appendix A, and all
settings stay consistent for all our experiments.

##### Data

We use the AfriBERTa corpus for distillation; it comprises 11 African
languages. We use the MasakhaNER dataset (Adelani et al., 2021), a Named
Entity Recognition dataset that spans 10 African languages, for task-specific
compression evaluation.

##### Model Training

This study uses the AfriBERTa models, _Base_ and _Large_ , which are based on
the XLM-R architecture. For the different compression strategies we explore,
we use either or both the large variant and base variant for our study. The
final results were averaged over three training runs with different training
seeds for all reported results. We notice an insignificant standard deviation
and distribution between the results of the different seeds.

##### Tasks Evaluation

The evaluation of the compressed models in this study focuses on the NER task
due to its significant relevance in downstream applications such as question
answering and information extraction (Tjong Kim Sang & De Meulder, 2003). We
adopt the F1 score as our primary evaluation metric, and the evaluation
dataset is the MasakhaNER dataset (Adelani et al., 2021).

###  2.1 Efficiency Evaluation

##### Distillation

We apply both task-agnostic and task-specific distillation approaches on the
base and large variants of the AfriBERTa model. We distil knowledge from the
pre-trained AfriBERTa model into relatively smaller models for task-agnostic
distillation and evaluate them for the NER downstream task. For the task-
specific distillation, we distil knowledge from a model fine-tuned for the NER
downstream task into smaller models.

##### Pruning

Our pruning experiments at various sparsity levels range from $10\%$ to
$95\%$. We prune the model before, after, and during fine-tuning to examine
the impact of pruning at each phase of the training process. We also examine
the computational efficiency of the pruned models by measuring their inference
time on the test data at all sparsity levels. Furthermore, we evaluate the
generalization capabilities of the pruned models by fine-tuning and testing
them on out-of-distribution (OOD) data. We compare the performance of the
pruned models to the original dense model to assess any influence of pruning
on cross-lingual knowledge transfer and the model’s level of generalizability,
using MasakhaNER 2.0 (Adelani et al., 2022) and MSRA NER (Feng et al., 2006)
dataset.

##### Quantization

We examine the effects of quantization on the large AfriBERTa model, which has
been fine-tuned for Named Entity Recognition (NER) tasks. Our study utilizes
two quantization approaches. The first is the LLM.int8() method, which
uniformly converts the model’s weights, activations, and attention mechanisms
to an 8-bit integer (int8) format. The second approach is dynamic
quantization, which dynamically converts the weights of linear layers from
floating-point to integer data types at runtime and quantizes the activation
layers during CPU-based inference.

Table 1: Average results for the distilled models on NER task across 10
languages. The best student variant for each teacher is highlighted, and the
best variant for each strategy is underlined.

Distillation strategy Teacher #Layers #Att. Heads #Params amh hau ibo kin lug
luo pcm swa wol yor avg Task-agnostic AfriBERTa-base 4 4 83M 64.96 87.28 83.58
68.15 74.57 63.02 78.92 83.89 55.46 73.62 73.35 4 6 83M 64.23 87.34 83.84
67.59 74.60 60.00 79.40 84.00 57.21 73.38 73.16 6 4 97M 65.96 87.60 85.55
70.16 75.90 64.61 81.65 85.48 57.70 74.82 74.94 6 6 97M 66.92 87.91 85.28
69.81 77.19 68.40 81.72 85.08 60.28 75.47 75.81 AfriBERTa-large 4 4 83M 65.28
87.28 84.15 68.83 73.82 63.79 79.80 84.13 56.30 73.43 73.68 4 6 83M 65.25
87.62 84.28 68.82 74.66 62.60 78.88 84.07 55.11 73.85 73.51 6 4 97M 69.38
88.25 85.08 69.49 75.44 63.87 82.71 85.87 56.42 73.89 75.04 6 6 97M 71.98
88.72 85.76 71.76 78.30 68.10 84.24 87.07 61.45 77.61 77.50 Task-specific
AfriBERTa-base 4 4 83M 65.04 87.12 83.13 67.62 75.01 62.90 78.19 83.96 54.04
69.22 72.62 4 6 83M 65.52 87.27 83.93 68.18 75.56 63.78 79.03 83.70 57.46
69.98 73.44 6 4 97M 67.28 87.27 85.72 71.68 77.18 66.58 81.85 84.92 60.20
74.78 75.75 6 6 97M 69.45 88.23 85.47 69.88 74.90 65.79 82.27 85.36 59.12
76.25 75.67 AfriBERTa-large 4 4 83M 65.66 87.60 83.42 67.38 74.28 62.37 79.62
83.78 55.72 72.89 73.27 4 6 83M 71.20 88.27 84.66 70.70 77.11 65.58 82.09
86.06 58.00 76.21 75.99 6 4 97M 69.38 88.25 85.08 69.49 75.44 63.87 82.71
85.87 56.42 73.89 75.04 6 6 97M 72.58 88.33 86.05 71.16 78.56 69.87 84.03
86.32 61.49 76.66 77.51

Table 2: Comparison of NER results between the teachers and the best students.
The underlined scores are instances where the distilled model outperformed any
of the teacher models.

Language AfriBERTa-base AfriBERTA-large Distiled AfriBERTa-base Distiled
AfriBERTa-base Distiled AfriBERTa-large Distiled AfriBERTa-large (Teacher)
(Teacher) (Task agnostic) (Task specific) (Task agnostic) (Task Specific)
<111M <126M> <97M> <97M> <97M> <97M> amh 71.8 73.28 69.45 66.92 71.98 72.58
hau 90.01 90.17 88.23 87.91 88.72 88.33 ibo 86.63 87.38 85.47 85.28 85.76
86.05 kin 69.91 73.78 69.88 69.81 71.76 71.16 lug 76.44 78.85 74.9 77.19 78.3
78.56 luo 67.31 70.23 65.79 68.4 68.1 69.87 pcm 82.92 85.7 82.27 81.72 84.24
84.03 swa 85.68 87.96 85.36 85.08 87.07 86.32 wol 60.1 61.81 59.12 60.28 61.45
61.49 yor 76.08 81.32 76.25 75.47 77.61 76.66 avg 76.688 79.048 75.672 75.806
77.499 77.505

##  3 Results and Discussion

###  3.1 How small can we make these language models?

Using our distillation methods, we achieve up to 31% compression while
maintaining competitive results, with only a 7% performance drop for our
least-performing model and only a 1.9% decline compared to the best-performing
AfriBERTa model at 22% compression, as shown in table 2. We notice only
marginal differences between the teachers’ and students’ performances in some
languages. We also see the student model trained by the large teacher
outperform the base teacher in specific languages. Additionally, our task-
agnostic models outperform the task-specific models in terms of F1 score, but
with relatively minimal differences.

###  3.2 Which is the best teacher: Base vs Large?

Although there is a performance decline of roughly 1.9% from the AfriBERTa-
large baseline, we discover that the AfriBERTa-large model produced the
student with the best grade. However, the best-performing student by the base
model only showed a performance decline of 1.3% between the original scores of
the base model and the student scores (see Table 2). According to the results,
the base model is comparatively better at imparting most of its knowledge to
its students, even though the larger model creates the best overall student.
Additionally, as the attention head and layer ratio reduce, the students being
taught using the base model catch up to those taught by the larger model with
no discernible difference in performance, as seen in Table 1. Our results
suggest that the selected instructor model significantly influences the
performance of student models.

(a) Pruning before fine-tuning

(b) Pruning after fine-tuning

Figure 1: Pruning before vs after fine-tuning: F1 score of sparsified models,
averaged across languages.

###  3.3 How does pruning before and after fine-tuning affect model
performance?

As seen in Figure 1, our results show that pruning before fine-tuning produces
fairly consistent performance with the dense model up to a sparsity level of
60%. However, above this threshold, the model’s performance gradually
declines. Still, it remains competitive even at 80% sparsity, with an average
F1 score of over 70% (detailed tables and charts in Appendix D.1). When fine-
tuning is performed before pruning, however, our results demonstrate that the
model’s performance stays firmly on par with, and even exceeds, the
performance of the dense model up to 50% sparsity. However, when the sparsity
level increases, especially from the 70% sparsity, we find a dramatic
deterioration in performance. It is worth mentioning that both methods have
advantages and disadvantages. Pruning before fine-tuning results in more
stable and predictable performance at greater sparsity levels, making it a
feasible alternative for applications requiring high sparsity levels. On the
other hand, pruning after fine-tuning can greatly improve the model’s
performance at lower sparsity levels, making it a better technique for
applications that emphasize high accuracy.

###  3.4 Exploring the limits of pruning for small-data pre-trained Models

In our findings (see Appendix D.1), we notice that despite the limitations of
AfriBERTa’s training data and architecture, we find constant performance up to
50% and 60% sparsity. Notably, certain languages maintain a moderate degree of
performance even at 95% sparsity, suggesting that the model might have a
certain level of robustness to pruning. Nonetheless, we notice dramatic
reductions in performance for several languages, such as Yoruba and Luganda,
at this level. This might be attributable to these languages’ particular
characteristics, such as their high inflectional complexity and the sparse
nature of their datasets, which may make them more prone to pruning-induced
degeneration. To confirm the robustness of our results, we also explored
cross-lingual transfer on AfriBERTa leveraging pruning, as detailed in
Appendix D.2. While our results show that aggressive pruning is possible for
small-data pre-trained models, it is also critical to take into account the
unique qualities of each language and dataset when calculating the ideal
sparsity level for pruning.

###  3.5 How does Pruning affect Out-of-Domain Generalization?

Our findings (Figure 7) reveal that pruning can positively impact OOD
generalization for some languages, while for others, the benefits are limited.
Surprisingly, for many languages, including Swahili, the performance of the
pruned models remains consistent with or surpasses that of the original dense
model up to a sparsity level of 60%, with $\approx 90\%$ for Swahili. However,
for languages such as Yoruba, which exhibit a higher level of linguistic
complexity, the performance is lower even for the dense model, with an F1
score of around 60%, highlighting the challenge of compressing models with
complex linguistic structures.

###  3.6 Effectiveness of Quantization on Model Efficiency

Table 3: F1 Scores by Language and Quantization Methods. Language | Baseline | Dynamic | LLM.int8()
---|---|---|---
Amh | 73.36 | 68.02 | 73.28
Hau | 89.93 | 85.35 | 89.95
Ibo | 86.96 | 82.21 | 86.88
Kin | 73.98 | 61.58 | 73.91
Lug | 79.78 | 68.94 | 79.83
Luo | 70.04 | 42.40 | 69.77
Pcm | 85.23 | 74.37 | 85.18
Swa | 87.89 | 84.58 | 87.93
Wol | 61.73 | 47.36 | 61.71
Yor | 80.76 | 65.10 | 80.74

Table 4: Inference Time Comparison (ms) for the quantization Methods and the baseline model. Language | Baseline | Dynamic | LLM.int8()
---|---|---|---
amh | 26.01 | 12.78 | 13.27
hau | 31.08 | 19.99 | 13.31
ibo | 31.84 | 21.67 | 15.03
kin | 27.19 | 20.95 | 16.85
lug | 21.10 | 12.35 | 10.62
luo | 22.40 | 5.53 | 5.47
pcm | 41.70 | 17.96 | 16.34
swa | 35.50 | 20.14 | 17.37
wol | 25.38 | 20.95 | 14.78
yor | 34.45 | 23.14 | 18.36

Our results, as shown in Table 4, show that the LLM.int8() quantization method
generally outperformed the dynamic quantization method across all languages,
with an average decrease in the F1-score of just 4.7%. Moreover, our findings
suggest that for some languages, such as Swahili, Luganda, and Hausa,
LLM.int8() may be preferable to the original dense model.

Model size reduced varyingly across languages, with dynamic quantization
resulting in a 42.44% reduction and LLM.int8() resulting in a 64.08%
reduction. There is no one-size-fits-all solution when it comes to
quantization. The performance of quantized models depends on various factors,
such as the language, the type of data being processed, and the adapted
quantization technique.

Table 4 shows that quantization can significantly reduce inference time for
all languages. For example, in the case of Amharic, quantization lead to a 50%
reduction in inference time compared to the baseline model. Similarly, for
Hausa and Swahili, quantization resulted in a 35% and 40% reduction in
inference time, respectively. An average reduction of 40.9% for dynamic
quantization and 52.3% for LLM.int8() was observed. These findings suggest
that quantization effectively optimises small data-pre-trained models for
deployment on devices with limited resources.

##  4 Conclusion and Future Work

This study investigates the effectiveness of pruning, knowledge distillation,
and quantization on a small-data language model, AfriBERTa, trained on low-
resource languages. Our findings indicate that compression techniques can
significantly improve the efficiency and effectiveness of small-data language
models. Also, we identify the importance of balancing the attention head and
hidden layers when using knowledge distillation to compress small-data
language models. Additionally, further experiments with different variations
of quantization strategies yield results comparable to the original models.
Our study balances compressed small-data language models’ efficiency-accuracy
tradeoff and generalization capabilities.

However, our work’s novelty lies in applying existing compression techniques
to a low-resource setting. We do not introduce new techniques or approaches
but adapt and evaluate existing methods. Moreover, while NER is a crucial NLP
task, a focus for future work is to explore the applicability of our findings
to other NLP tasks.

## Acknowledgements

We thank the ML Collective community for the generous computational support,
as well as helpful discussions, ideas, and feedback on experiments.

## References

  * Adelani et al. [2021] D. Adelani, Jade Abbott, Graham Neubig, Daniel D’Souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, S. Muhammad, Chris C. Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, J. Alabi, Seid Muhie Yimam, Tajuddeen R. Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, V. Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin P. Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, C. Chukwuneke, N. Odu, Eric Peter Wairagala, S. Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane Mboup, D. Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima Diop, A. Diallo, Adewale Akinfaderin, T. Marengereke, and Salomey Osei.  Masakhaner: Named entity recognition for african languages.  _ArXiv_ , abs/2103.11811, 2021.
  * Adelani et al. [2022] David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O Alabi, Shamsuddeen H Muhammad, Peter Nabende, et al.  Masakhaner 2.0: Africa-centric transfer learning for named entity recognition.  _arXiv preprint arXiv:2210.12391_ , 2022.
  * Ahia et al. [2021] Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker.  The low-resource double bind: An empirical study of pruning for low-resource machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 3316–3333, 2021.
  * Alabi et al. [2022] Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow.  Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pp. 4336–4349, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.382.
  * Bai et al. [2022] Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu.  Towards efficient post-training quantization of pre-trained language models.  _Advances in Neural Information Processing Systems_ , 35:1405–1418, 2022.
  * Chen et al. [2020] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin.  The lottery ticket hypothesis for pre-trained bert networks.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 15834–15846, 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/b6af2c9703f203a2794be03d443af2e3-Abstract.html.
  * Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, 2020.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_ , 2022.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Eberhard et al. [2019] David M Eberhard, Gary F Simons, and Charles D Fennig.  Ethnologue: languages of the world. dallas, texas: Sil international.  _Online version: http://www. ethnologue. com_ , 22, 2019.
  * Feng et al. [2006] Yuanyong Feng, Le Sun, and Yuanhua Lv.  Chinese word segmentation and named entity recognition based on conditional random fields models.  In _Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_ , pp. 181–184, 2006.
  * Frankle & Carbin [2019] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _International Conference on Learning Representations_ , 2019.
  * Han et al. [2015] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  _arXiv: Computer Vision and Pattern Recognition_ , 2015.
  * Hinton et al. [2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  _ArXiv_ , abs/1503.02531, 2015.
  * Jiao et al. [2019] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding.  In _Findings_ , 2019.
  * Kingma & Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * Kreutzer et al. [2022] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, January 2022.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00447.  URL https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00447/109285/Quality-at-a-Glance-An-Audit-of-Web-Crawled.
  * Lan et al. [2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  April 2019.  URL https://openreview.net/forum?id=H1eA7AEtvS&utm_campaign=The%20Batch&utm_source=hs_email&utm_medium=email&_hsenc=p2ANqtz-_QwAkpWYd5cbmMTX5gb9_GYEBsWkI_vi0WyIti1i3vzXI7Qw0zTGiLe6VfcuW-v15PRAlZ.
  * Li et al. [2020] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez.  Train big, then compress: Rethinking model size for efficient training and inference of transformers.  In _International Conference on machine learning_ , pp. 5958–5968. PMLR, 2020.
  * Nekoto et al. [2020] Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, et al.  Participatory research for low-resourced machine translation: A case study in african languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 2144–2160, 2020.
  * Ogueji et al. [2021] Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin.  Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pp. 116–126, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  URL https://aclanthology.org/2021.mrl-1.11.
  * Prasanna et al. [2020] Sumanth Prasanna, Anna Rogers, and Anna Rumshisky.  When bert plays the lottery, all tickets are winning.  _arXiv preprint arXiv:2005.00561v2_ , May 2020.  URL https://arxiv.org/abs/2005.00561v2.
  * Renda et al. [2020] Alaa Renda, Jonathan Frankle, and Michael Carbin.  Comparing rewinding and fine-tuning in neural network pruning.  _arXiv preprint arXiv:2003.02389v1_ , 2020.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108, 2019.
  * Tjong Kim Sang & De Meulder [2003] Erik F Tjong Kim Sang and Fien De Meulder.  Introduction to the conll-2003 shared task: language-independent named entity recognition.  In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4_ , pp. 142–147, 2003.
  * Treviso et al. [2022] Marcelo Treviso, Tao Ji, Jung-uk Lee, Betty van Aken, Qingyun Cao, Marius R Ciosici, Meni Hassid, Kenneth Heafield, Simon Hooker, Pedro Henrique Martins, André FT Martins, Peter Milder, Colin Raffel, Edmund Simpson, Noam Slonim, Niranjan Balasubramanian, Leon Derczynski, and Roy Schwartz.  Efficient methods for natural language processing: A survey.  _arXiv preprint arXiv:2209.00099_ , 2022.  doi: 10.48550/arXiv.2209.00099.
  * Wang et al. [2022] Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia-Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang.  Deep compression of pre-trained transformer models.  _Advances in Neural Information Processing Systems_ , 35:14140–14154, 2022.
  * Xiao et al. [2022] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  _arXiv preprint arXiv:2211.10438_ , 2022.
  * Yu et al. [2019] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos.  Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp.  _arXiv preprint arXiv:1906.02768_ , 2019.
  * Zhang et al. [2021] Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman.  When Do You Need Billions of Words of Pretraining Data?  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1112–1125, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.90.  URL https://aclanthology.org/2021.acl-long.90.

##  Appendix A Training Setup

###  A.1 Data

The AfriBERTa corpus is a multilingual dataset comprising 11 African
languages. The data was primarily sourced from the BBC
news111https://www.bbc.co.uk/ws/languages (scraped up to January 17, 2021) and
the Common Crawl Corpus [Conneau et al., 2020]. Although relatively small at
0.91 GB, it was specifically engineered to present a first-of-its-kind attempt
to train a multilingual language model exclusively on low-resource languages.
Table 5 shows language-specific information and token details.

The MasakhaNER dataset [Adelani et al., 2021] is used for task-specific
compression and evaluation. It is a Named Entity ecognition dataset comprising
PER, ORG, LOC, and DATE entities annotated for the 10 African languages. It
was used to evaluate the original AfriBERTa model. The languages contained in
the dataset are Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Luo, Nigerian
Pidgin, Swahili, Wolof, and Yorùbá. Table 7 summarizes the dataset’s details,
including the languages included in the dataset.

###  A.2 Data Preprocessing

Table 5: AfriBERTa corpus [Ogueji et al., 2021]: Size of each language in the dataset covering numbers of sentences, tokens, and uncompressed disk size. Language | # Sent. | # Tok. | Size (GB)
---|---|---|---
Afaan Oromoo | 410,840 | 6,870,959 | 0.051
Amharic | 525,024 | 1,303,086 | 0.213
Gahuza | 131,952 | 3,669,538 | 0.026
Hausa | 1,282,996 | 27,889,299 | 0.150
Igbo | 337,081 | 6,853,500 | 0.042
Nigerian Pidgin | 161,842 | 8,709,498 | 0.048
Somali | 995,043 | 27,332,348 | 0.170
Swahili | 1,442,911 | 30,053,834 | 0.185
Tigrinya | 12,075 | 280,397 | 0.027
Yorùbá | 149,147 | 4,385,797 | 0.027
Total | 5,448,911 | 108,800,600 | 0.939

Similar to the original preprocessing step for the AfriBERTa model [Ogueji et
al., 2021], we use the AfriBERTa tokenizer. We also remove lines that are
empty or contain only punctuations to ensure that the dataset is clean and
contains meaningful text. We also enforce a minimum length restriction by
retaining sentences with more than 11 tokens. This step helps to filter out
concise sentences that may not provide enough context for the model to learn
effectively. Furthermore, we take preprocessing steps significant to the
performance of the AfriBERTa model and DistilBERT [Sanh et al., 2019]. We use
the entire pre-training corpus since it’s already a small amount of data of
1GB in size.

Table 6: Model architecture details for AfriBERTa-base and AfriBERTa-large. Model | #Params | #Layers | #Att. Heads
---|---|---|---
AfriBERTa-base | 111M | 8 | 6
AfriBERTa-large | 126M | 10 | 6

###  A.3 NER

Adapted from the AfriBERTa experiment setup

learning rate: 5e-5

max sequence length: 164

batch size: 16

num‘ epochs:50

Tokenizer: ‘afriberta original tokenizer‘ [Ogueji et al., 2021]

optimizer: Adam [Kingma & Ba, 2014]

training seeds: [1, 3, 5]

##  Appendix B Distillation

We experiment using task-agnostic and task-specific distillation approaches on
both the base and significant variants of the AfriBERTa model.

##### Task-Agnostic Distillation

Task-agnostic distillation involves distilling a large pre-trained language
model into a smaller model that is not optimized for any specific downstream
task. Knowledge from the teacher model was used to pretrain the student model
which is then further fine-tuned on a downstream task, in our case here, the
NER downstream task.

##### Task-Specific Distillation

The already fine-tuned teacher model was used to teach the already distilled
student model on a downstream task. Task-specific distillation involves fine-
tuning the pre-trained model on the target task and distilling knowledge from
the fine-tuned model into the already distilled pretrained student model.

####  B.0.1 Task agnostic

Adapted from HuggingFace distillation

temperature: [2, 3, 6]

####  B.0.2 Task specific

Adapted from TextBrewer.

temperature: 8

Table 7: Statistics of MasakhaNER datasets Adelani et al. [2021] including
their source, number of sentences in each split, number of annotators, and
number of entities of each label type, combined with information on language,
family, number of speakers [Eberhard et al., 2019], and African regions.
Adapted from [Adelani et al., 2021]

Language Family Speakers Region Data Source Train/Dev/Test # Anno PER ORG LOC
DATE Amharic Afro-Asiatic-Ethio-Semitic 33M East DW & BBC 1750/250/500 4 730
403 1,420 580 Hausa Afro-Asiatic-Chadic 63M West VOA Hausa 1903/272/545 3
1,490 766 2,779 922 Igbo Niger-Congo-Volta-Niger 27M West BBC Igbo
2233/319/638 6 1,603 1,292 1,677 690 Kinyarwanda Niger-Congo-Bantu 12M East
IGIHE news 2110/301/604 2 1,366 1,038 2,096 792 Luganda Niger-Congo-Bantu 7M
East BUKEDDE news 2003/200/401 3 1,868 838 943 574 Luo Nilo Saharan 4M East
Ramogi FM news 644/92/185 2 557 286 666 343 Nigerian-Pidgin English Creole 75M
West BBC Pidgin 2100/300/600 5 2,602 1,042 1,317 1,242 Swahili Niger-Congo-
Bantu 98M Central & East VOA Swahili 2104/300/602 6 1,702 960 2,842 940 Wolof
Niger-Congo-Senegambia 5M West & NW Lu Defu Waxu & Saabal 1871/267/536 2 731
245 836 206 Yorùbá Niger-Congo-Volta-Niger 42M West GV & VON news 2124/303/608
5 1,039 835 1,627 853

##  Appendix C Additional Results References for Distillation

See table 1 and figures 4(b) & 4 for references.

Figure 2: A chart of the F1 scores for the best-performing student models and
the teacher models across each language in the NER task. Figure 3: Performance
(F1 score) during task agnostic and task-specific distillation across
different languages in the NER task.

(a) Task agnostic

(b) Task specific

(c) Base teacher vs Students

(d) Large teacher vs Students

Figure 4: Performance comparison between students and teachers (distillation).

##  Appendix D Pruning

Unstructured magnitude pruning involves setting a binary mask, $M$, that
determines which weights in the network are pruned based on their magnitude
relative to a pruning threshold, $t$. Specifically, we define the mask $M$ as:

$M_{ij}=\begin{cases}1&\text{if }|W_{ij}|\geq t\\\
0&\text{otherwise}\end{cases}$

where $W_{i,j}$ is the weight at position $(i,j)$ in the weight matrix, and
$t$ is a threshold determined by the desired sparsity level.

###  D.1 Average performances before and after fine-tuning

Table 8: Pruning before Finetuning: Performance metrics of each language at all sparsity levels |  | loss | precision | recall | f1 | inference_time | pruned_params
---|---|---|---|---|---|---|---
prune_rate | lang |  |  |  |  |  |
0.10 | amh | 0.38 | 0.71 | 0.76 | 0.74 | 1.36 | 7078579
| hau | 0.15 | 0.88 | 0.93 | 0.90 | 1.83 | 7078579
| ibo | 0.20 | 0.86 | 0.88 | 0.87 | 1.85 | 7078579
| kin | 0.35 | 0.72 | 0.78 | 0.75 | 1.73 | 7078579
| lug | 0.29 | 0.78 | 0.82 | 0.80 | 1.24 | 7078579
| luo | 0.44 | 0.70 | 0.73 | 0.71 | 0.57 | 7078579
| pcm | 0.15 | 0.84 | 0.87 | 0.86 | 1.76 | 7078579
| swa | 0.20 | 0.86 | 0.90 | 0.88 | 1.76 | 7078579
| wol | 0.37 | 0.66 | 0.60 | 0.63 | 1.52 | 7078579
| yor | 0.24 | 0.78 | 0.83 | 0.81 | 2.01 | 7078579
0.20 | amh | 0.37 | 0.71 | 0.76 | 0.73 | 1.25 | 14157158
| hau | 0.16 | 0.87 | 0.93 | 0.90 | 1.56 | 14157158
| ibo | 0.20 | 0.85 | 0.89 | 0.87 | 1.80 | 14157158
| kin | 0.34 | 0.70 | 0.78 | 0.74 | 1.73 | 14157158
| lug | 0.29 | 0.78 | 0.82 | 0.80 | 1.12 | 14157158
| luo | 0.46 | 0.67 | 0.72 | 0.69 | 0.58 | 14157158
| pcm | 0.16 | 0.83 | 0.87 | 0.85 | 1.62 | 14157158
| swa | 0.21 | 0.86 | 0.90 | 0.88 | 1.61 | 14157158
| wol | 0.37 | 0.63 | 0.60 | 0.61 | 1.41 | 14157158
| yor | 0.26 | 0.78 | 0.83 | 0.81 | 1.79 | 14157158
0.30 | amh | 0.37 | 0.69 | 0.76 | 0.72 | 1.24 | 21235738
| hau | 0.16 | 0.88 | 0.93 | 0.90 | 1.57 | 21235738
| ibo | 0.19 | 0.86 | 0.89 | 0.87 | 1.68 | 21235738
| kin | 0.35 | 0.70 | 0.78 | 0.74 | 1.62 | 21235738
| lug | 0.29 | 0.77 | 0.82 | 0.79 | 1.14 | 21235738
| luo | 0.46 | 0.69 | 0.72 | 0.70 | 0.60 | 21235738
| pcm | 0.15 | 0.83 | 0.86 | 0.85 | 1.62 | 21235738
| swa | 0.21 | 0.86 | 0.90 | 0.88 | 1.80 | 21235738
| wol | 0.36 | 0.64 | 0.60 | 0.62 | 1.38 | 21235738
| yor | 0.25 | 0.79 | 0.83 | 0.81 | 1.80 | 21235738
0.40 | amh | 0.38 | 0.70 | 0.76 | 0.73 | 1.27 | 28314317
| hau | 0.16 | 0.86 | 0.93 | 0.90 | 1.69 | 28314317
| ibo | 0.19 | 0.86 | 0.89 | 0.87 | 1.66 | 28314317
| kin | 0.35 | 0.70 | 0.78 | 0.73 | 1.63 | 28314317
| lug | 0.28 | 0.77 | 0.82 | 0.79 | 1.12 | 28314317
| luo | 0.44 | 0.68 | 0.71 | 0.70 | 0.53 | 28314317
| pcm | 0.15 | 0.84 | 0.87 | 0.85 | 1.73 | 28314317
| swa | 0.21 | 0.85 | 0.89 | 0.87 | 1.61 | 28314317
| wol | 0.37 | 0.65 | 0.59 | 0.62 | 1.38 | 28314317
| yor | 0.26 | 0.78 | 0.82 | 0.80 | 1.80 | 28314317
0.50 | amh | 0.37 | 0.69 | 0.76 | 0.72 | 1.36 | 35392896
| hau | 0.17 | 0.87 | 0.93 | 0.90 | 1.56 | 35392896
| ibo | 0.21 | 0.85 | 0.88 | 0.87 | 1.67 | 35392896
| kin | 0.34 | 0.70 | 0.78 | 0.74 | 1.65 | 35392896
| lug | 0.29 | 0.76 | 0.81 | 0.78 | 1.30 | 35392896
| luo | 0.44 | 0.68 | 0.71 | 0.70 | 0.52 | 35392896
| pcm | 0.16 | 0.82 | 0.86 | 0.84 | 1.74 | 35392896
| swa | 0.22 | 0.84 | 0.89 | 0.86 | 1.63 | 35392896
| wol | 0.37 | 0.64 | 0.59 | 0.61 | 1.38 | 35392896
| yor | 0.26 | 0.78 | 0.82 | 0.80 | 1.79 | 35392896
prune_rate | lang | loss | precision | recall | f1 | inference_time | pruned_params
---|---|---|---|---|---|---|---
0.60 | amh | 0.39 | 0.68 | 0.75 | 0.71 | 1.25 | 42471475
| hau | 0.18 | 0.86 | 0.92 | 0.89 | 1.57 | 42471475
| ibo | 0.19 | 0.85 | 0.88 | 0.87 | 1.63 | 42471475
| kin | 0.36 | 0.69 | 0.78 | 0.73 | 1.60 | 42471475
| lug | 0.29 | 0.76 | 0.81 | 0.78 | 1.20 | 42471475
| luo | 0.43 | 0.68 | 0.70 | 0.69 | 0.58 | 42471475
| pcm | 0.16 | 0.82 | 0.85 | 0.84 | 1.75 | 42471475
| swa | 0.21 | 0.84 | 0.89 | 0.86 | 1.67 | 42471475
| wol | 0.37 | 0.62 | 0.59 | 0.61 | 1.41 | 42471475
| yor | 0.28 | 0.78 | 0.81 | 0.80 | 1.78 | 42471475
0.70 | amh | 0.40 | 0.66 | 0.73 | 0.69 | 1.30 | 49550054
| hau | 0.19 | 0.85 | 0.92 | 0.88 | 1.55 | 49550054
| ibo | 0.20 | 0.83 | 0.87 | 0.85 | 1.69 | 49550054
| kin | 0.37 | 0.68 | 0.76 | 0.71 | 1.59 | 49550054
| lug | 0.29 | 0.73 | 0.80 | 0.76 | 1.14 | 49550054
| luo | 0.44 | 0.65 | 0.69 | 0.67 | 0.52 | 49550054
| pcm | 0.17 | 0.80 | 0.85 | 0.83 | 1.76 | 49550054
| swa | 0.21 | 0.83 | 0.88 | 0.86 | 1.77 | 49550054
| wol | 0.38 | 0.57 | 0.56 | 0.56 | 1.43 | 49550054
| yor | 0.29 | 0.76 | 0.79 | 0.77 | 1.85 | 49550054
0.80 | amh | 0.43 | 0.62 | 0.70 | 0.66 | 1.25 | 56628634
| hau | 0.20 | 0.83 | 0.90 | 0.87 | 1.55 | 56628634
| ibo | 0.21 | 0.81 | 0.84 | 0.83 | 1.67 | 56628634
| kin | 0.41 | 0.62 | 0.71 | 0.66 | 1.63 | 56628634
| lug | 0.32 | 0.69 | 0.76 | 0.72 | 1.14 | 56628634
| luo | 0.47 | 0.58 | 0.63 | 0.60 | 0.59 | 56628634
| pcm | 0.19 | 0.76 | 0.82 | 0.79 | 1.60 | 56628634
| swa | 0.22 | 0.80 | 0.87 | 0.83 | 1.66 | 56628634
| wol | 0.39 | 0.53 | 0.53 | 0.53 | 1.38 | 56628634
| yor | 0.32 | 0.72 | 0.76 | 0.74 | 1.78 | 56628634
0.90 | amh | 0.50 | 0.55 | 0.64 | 0.59 | 1.24 | 63707213
| hau | 0.22 | 0.80 | 0.88 | 0.84 | 1.61 | 63707213
| ibo | 0.24 | 0.79 | 0.83 | 0.81 | 1.71 | 63707213
| kin | 0.45 | 0.57 | 0.68 | 0.62 | 1.61 | 63707213
| lug | 0.36 | 0.63 | 0.72 | 0.67 | 1.20 | 63707213
| luo | 0.53 | 0.50 | 0.58 | 0.53 | 0.52 | 63707213
| pcm | 0.24 | 0.69 | 0.77 | 0.73 | 1.76 | 63707213
| swa | 0.24 | 0.77 | 0.85 | 0.81 | 1.70 | 63707213
| wol | 0.44 | 0.43 | 0.46 | 0.45 | 1.50 | 63707213
| yor | 0.39 | 0.62 | 0.70 | 0.66 | 1.96 | 63707213
0.95 | amh | 0.51 | 0.50 | 0.61 | 0.55 | 1.24 | 67246502
| hau | 0.23 | 0.77 | 0.86 | 0.81 | 1.56 | 67246502
| ibo | 0.25 | 0.75 | 0.81 | 0.78 | 1.66 | 67246502
| kin | 0.44 | 0.53 | 0.65 | 0.59 | 1.62 | 67246502
| lug | 0.37 | 0.58 | 0.69 | 0.63 | 1.15 | 67246502
| luo | 0.54 | 0.42 | 0.52 | 0.47 | 0.53 | 67246502
| pcm | 0.27 | 0.64 | 0.74 | 0.69 | 1.66 | 67246502
| swa | 0.90 | 0.50 | 0.59 | 0.53 | 1.65 | 67246502
| wol | 1.05 | 0.28 | 0.32 | 0.29 | 1.42 | 67246502
| yor | 1.02 | 0.37 | 0.46 | 0.40 | 1.82 | 67246502
Table 9: Pruning after Finetuning: Performance metrics of each language at all sparsity levels |  | loss | precision | recall | f1 | inference_time | pruned_params
---|---|---|---|---|---|---|---
prune_rate | lang |  |  |  |  |  |
0.10 | amh | 0.37 | 0.72 | 0.76 | 0.74 | 1.88 | 7078579
| hau | 0.17 | 0.87 | 0.93 | 0.90 | 1.93 | 7078579
| ibo | 0.19 | 0.85 | 0.88 | 0.87 | 2.96 | 7078579
| kin | 0.34 | 0.70 | 0.79 | 0.74 | 4.01 | 7078579
| lug | 0.28 | 0.78 | 0.82 | 0.80 | 1.18 | 7078579
| luo | 0.45 | 0.69 | 0.71 | 0.70 | 0.69 | 7078579
| pcm | 0.15 | 0.84 | 0.87 | 0.85 | 1.76 | 7078579
| swa | 0.20 | 0.86 | 0.90 | 0.88 | 1.76 | 7078579
| wol | 0.36 | 0.63 | 0.60 | 0.61 | 1.64 | 7078579
| yor | 0.25 | 0.78 | 0.83 | 0.80 | 2.35 | 7078579
0.20 | amh | 0.35 | 0.71 | 0.76 | 0.74 | 1.21 | 14157158
| hau | 0.16 | 0.87 | 0.93 | 0.90 | 1.65 | 14157158
| ibo | 0.19 | 0.85 | 0.88 | 0.87 | 1.75 | 14157158
| kin | 0.32 | 0.70 | 0.79 | 0.74 | 1.71 | 14157158
| lug | 0.27 | 0.77 | 0.82 | 0.80 | 1.18 | 14157158
| luo | 0.43 | 0.68 | 0.71 | 0.69 | 0.55 | 14157158
| pcm | 0.14 | 0.83 | 0.86 | 0.85 | 1.73 | 14157158
| swa | 0.19 | 0.86 | 0.90 | 0.88 | 1.60 | 14157158
| wol | 0.34 | 0.63 | 0.60 | 0.61 | 1.45 | 14157158
| yor | 0.24 | 0.78 | 0.83 | 0.80 | 1.94 | 14157158
0.30 | amh | 0.32 | 0.72 | 0.76 | 0.73 | 1.21 | 21235738
| hau | 0.14 | 0.87 | 0.93 | 0.90 | 1.66 | 21235738
| ibo | 0.17 | 0.85 | 0.88 | 0.87 | 1.90 | 21235738
| kin | 0.30 | 0.70 | 0.79 | 0.74 | 1.58 | 21235738
| lug | 0.25 | 0.77 | 0.82 | 0.79 | 1.15 | 21235738
| luo | 0.40 | 0.68 | 0.70 | 0.69 | 0.56 | 21235738
| pcm | 0.13 | 0.82 | 0.86 | 0.84 | 1.58 | 21235738
| swa | 0.18 | 0.85 | 0.90 | 0.88 | 1.73 | 21235738
| wol | 0.32 | 0.63 | 0.59 | 0.61 | 1.51 | 21235738
| yor | 0.22 | 0.78 | 0.83 | 0.81 | 2.04 | 21235738
0.40 | amh | 0.29 | 0.72 | 0.75 | 0.74 | 1.21 | 28314317
| hau | 0.12 | 0.87 | 0.93 | 0.90 | 1.80 | 28314317
| ibo | 0.15 | 0.85 | 0.88 | 0.87 | 1.62 | 28314317
| kin | 0.26 | 0.69 | 0.78 | 0.74 | 1.57 | 28314317
| lug | 0.22 | 0.77 | 0.81 | 0.79 | 1.18 | 28314317
| luo | 0.36 | 0.69 | 0.69 | 0.69 | 0.51 | 28314317
| pcm | 0.12 | 0.82 | 0.85 | 0.84 | 1.77 | 28314317
| swa | 0.15 | 0.86 | 0.89 | 0.87 | 1.60 | 28314317
| wol | 0.29 | 0.64 | 0.58 | 0.61 | 1.50 | 28314317
| yor | 0.19 | 0.78 | 0.82 | 0.80 | 2.12 | 28314317
0.50 | amh | 0.23 | 0.71 | 0.74 | 0.72 | 1.32 | 35392896
| hau | 0.10 | 0.86 | 0.93 | 0.89 | 1.51 | 35392896
| ibo | 0.12 | 0.85 | 0.88 | 0.86 | 1.73 | 35392896
| kin | 0.21 | 0.68 | 0.77 | 0.72 | 1.58 | 35392896
| lug | 0.18 | 0.77 | 0.78 | 0.78 | 1.06 | 35392896
| luo | 0.31 | 0.67 | 0.67 | 0.67 | 0.55 | 35392896
| pcm | 0.11 | 0.81 | 0.83 | 0.82 | 1.68 | 35392896
| swa | 0.12 | 0.85 | 0.89 | 0.87 | 1.66 | 35392896
| wol | 0.25 | 0.64 | 0.55 | 0.59 | 1.49 | 35392896
| yor | 0.17 | 0.79 | 0.80 | 0.79 | 1.95 | 35392896
prune_rate | lang | loss | precision | recall | f1 | inference_time | pruned_params
---|---|---|---|---|---|---|---
0.60 | amh | 0.19 | 0.71 | 0.66 | 0.68 | 1.32 | 42471475
| hau | 0.09 | 0.83 | 0.88 | 0.86 | 1.70 | 42471475
| ibo | 0.10 | 0.84 | 0.84 | 0.84 | 1.75 | 42471475
| kin | 0.19 | 0.67 | 0.65 | 0.66 | 1.72 | 42471475
| lug | 0.21 | 0.72 | 0.60 | 0.66 | 1.08 | 42471475
| luo | 0.36 | 0.52 | 0.47 | 0.49 | 0.50 | 42471475
| pcm | 0.11 | 0.77 | 0.76 | 0.76 | 1.60 | 42471475
| swa | 0.10 | 0.84 | 0.85 | 0.84 | 1.63 | 42471475
| wol | 0.23 | 0.64 | 0.42 | 0.50 | 1.49 | 42471475
| yor | 0.18 | 0.76 | 0.66 | 0.71 | 1.94 | 42471475
0.70 | amh | 0.38 | 0.45 | 0.23 | 0.30 | 1.35 | 49550054
| hau | 0.30 | 0.66 | 0.58 | 0.62 | 1.71 | 49550054
| ibo | 0.31 | 0.66 | 0.50 | 0.57 | 1.73 | 49550054
| kin | 0.38 | 0.63 | 0.22 | 0.32 | 1.60 | 49550054
| lug | 0.47 | 0.48 | 0.11 | 0.18 | 1.13 | 49550054
| luo | 0.63 | 0.23 | 0.09 | 0.12 | 0.58 | 49550054
| pcm | 0.31 | 0.49 | 0.27 | 0.35 | 1.60 | 49550054
| swa | 0.32 | 0.64 | 0.50 | 0.56 | 1.77 | 49550054
| wol | 0.30 | 0.59 | 0.13 | 0.21 | 1.53 | 49550054
| yor | 0.42 | 0.57 | 0.20 | 0.29 | 1.89 | 49550054
0.80 | amh | 1.25 | 0.23 | 0.04 | 0.06 | 1.27 | 56628634
| hau | 1.39 | 0.48 | 0.33 | 0.35 | 1.54 | 56628634
| ibo | 1.29 | 0.64 | 0.18 | 0.28 | 1.78 | 56628634
| kin | 1.31 | 0.59 | 0.13 | 0.21 | 1.71 | 56628634
| lug | 1.25 | 0.35 | 0.02 | 0.04 | 1.20 | 56628634
| luo | 1.57 | 0.15 | 0.05 | 0.05 | 0.57 | 56628634
| pcm | 1.22 | 0.37 | 0.05 | 0.09 | 1.75 | 56628634
| swa | 1.35 | 0.54 | 0.25 | 0.31 | 1.72 | 56628634
| wol | 1.13 | 0.05 | 0.01 | 0.01 | 1.40 | 56628634
| yor | 1.09 | 0.59 | 0.04 | 0.06 | 1.77 | 56628634
0.90 | amh | 2.02 | 0.01 | 0.02 | 0.06 | 1.24 | 63707213
| hau | 2.02 | 0.03 | 0.05 | 0.18 | 1.75 | 63707213
| ibo | 2.01 | 0.03 | 0.04 | 0.12 | 1.81 | 63707213
| kin | 2.04 | 0.02 | 0.02 | 0.08 | 1.76 | 63707213
| lug | 2.05 | 0.01 | 0.02 | 0.08 | 1.23 | 63707213
| luo | 2.07 | 0.01 | 0.01 | 0.04 | 0.52 | 63707213
| pcm | 2.04 | 0.02 | 0.03 | 0.11 | 1.65 | 63707213
| swa | 2.02 | 0.03 | 0.04 | 0.14 | 1.82 | 63707213
| wol | 2.01 | 0.01 | 0.02 | 0.07 | 1.43 | 63707213
| yor | 1.96 | 0.02 | 0.02 | 0.06 | 1.99 | 63707213
0.95 | amh | 2.19 | 0.01 | 0.09 | 0.02 | 1.37 | 67246502
| hau | 2.19 | 0.01 | 0.12 | 0.02 | 1.62 | 67246502
| ibo | 2.19 | 0.01 | 0.13 | 0.03 | 2.02 | 67246502
| kin | 2.19 | 0.01 | 0.15 | 0.02 | 1.80 | 67246502
| lug | 2.20 | 0.01 | 0.13 | 0.02 | 1.13 | 67246502
| luo | 2.21 | 0.01 | 0.10 | 0.01 | 0.60 | 67246502
| pcm | 2.20 | 0.01 | 0.10 | 0.01 | 1.69 | 67246502
| swa | 2.18 | 0.01 | 0.13 | 0.02 | 1.88 | 67246502
| wol | 2.19 | 0.00 | 0.08 | 0.01 | 1.61 | 67246502
| yor | 2.19 | 0.01 | 0.10 | 0.01 | 1.90 | 67246502

This section details the performance scores of all language tasks on all
sparsity levels, before and after fine-tuning. See Tables 8 & 9 and Figures 5
& 6 for references.

(a) Pruning before fine-tuning

(b) Pruning after fine-tuning

Figure 5: Pruning before vs after fine-tuning: F1 scores averaged across the
performances of each language.

(a) Inference Time wrt Sparsity Level

(b) Inference Time wrt Language Groups

Figure 6: Comparison of how Sparsity Level and Language Groups affect
Inference Time.

###  D.2 The Impact of Pruning on Cross-Lingual Transfer

(a) Mean F1 scores of languages over sparsity levels

(b) Distribution of the languages F1-scores over sparsity levels

Figure 7: OOD Generalization: Performances stay consistent with dense models
up till 50% sparsity.

This section analyses the impact of pruning on the cross-lingual transfer
learning capabilities of the AfriBERTa model.

##### Few-shot learning

To evaluate the effectiveness of the AfriBERTa model in capturing linguistic
intricacies in "unknown" languages, we fine-tuned it on two low-resource
African languages - Fon and Bambara - from MasakhaNER 2.0. Results show that
the performances when pruning after fine-tuning were comparable to the
performances of the known languages. However, we observed that inference time
increased by 2 to 3 times the usual inference time, and performances
deteriorated rapidly from 70% sparsity downwards. These observations suggest
that the AfriBERTa model might have leveraged linguistic similarities and
relationships inherent in languages sharing geographical regions.

##### Knowledge transfer from low-resource to high-resource

Our results show that AfriBERTa leverages the linguistic similarities and
relationships inherent in languages that share geographical regions [Ogueji et
al., 2021]. To test this assumption, we performed downstream on a high-
resource language, Chinese, from the MSRA NER dataset [Feng et al., 2006].
Surprisingly, AfriBERTa performed very strongly, implying that cross-lingual
transfer is possible even between languages with no known affiliation.
However, inference time is about four times (5–7 sec) the usual time it takes
to perform inference on familiar language data.

##### Zero-shot learning

We further explored the zero-shot transferability of AfriBERTa on an unseen-
before language at different levels of sparsity. Our findings reveal that F1
scores ranging between 40% and 60% remain competitive with MasakhaNER 2.0’s
zero-shot experiments on Afro-XLMR[Alabi et al., 2022], even at 50% sparsity.
Predicting on a new language from models trained on languages from the same
geographical region seems to perform zero-shot more confidently, consistent
with findings in MasakhaNER 2.0’s experiments [Adelani et al., 2022]. However,
we observed a considerable drop in performance at 70% sparsity, indicating
that pruning might not be suitable for zero-shot learning in low-resource
settings beyond a certain threshold.

###  D.3 Impact of pruning on inference time

The effect of pruning on inference time is a significant component of our
research, and we discovered that when the sparsity level grows, inference time
decreases noticeably. However, our investigation in Figure 6 indicated a wide
range of inference time disparities, which might be related to variables other
than the pruning rate itself. Our findings indicate that language-specific
characteristics may have a considerable impact on inference time, as
performance indicators showed no true connection with inference time. Further
investigation is needed to study the language-specific aspects influencing
inference time and to establish the ideal sparsity values for each language,
taking both performance metrics and inference time into account.


# Magnetic neutron scattering from spherical nanoparticles with Néel surface
anisotropy: Atomistic simulations

Michael P. Adams<EMAIL_ADDRESS>Department of Physics and Materials
Science, University of Luxembourg, 162A avenue de la Faiencerie, L-1511
Luxembourg, Grand Duchy of Luxembourg  Andreas Michels<EMAIL_ADDRESS>Department of Physics and Materials Science, University of Luxembourg, 162A
avenue de la Faiencerie, L-1511 Luxembourg, Grand Duchy of Luxembourg  Hamid
Kachkachi<EMAIL_ADDRESS>Université de Perpignan via Domitia,
Laboratoire PROMES CNRS UPR8521, Rambla de la Thermodynamique, Tecnosud,
F-66100 Perpignan, France

###### Abstract

We consider a dilute ensemble of randomly-oriented noninteracting spherical
nanomagnets and investigate its magnetization structure and ensuing neutron-
scattering response by numerically solving the Landau-Lifshitz equation.
Taking into account the isotropic exchange interaction, an external magnetic
field, a uniaxial magnetic anisotropy for the particle core, and in particular
the Néel surface anisotropy, we compute the magnetic small-angle neutron
scattering cross section and pair-distance distribution function from the
obtained equilibrium spin structures. The numerical results are compared to
the well-known analytical expressions for uniformly magnetized particles and
provide guidance to the experimentalist. Moreover, the effect of a particle-
size distribution function is modeled.

##  I Introduction

Magnetic nanoparticles are the subject of intense worldwide research efforts
which are partly motivated by potential applications in e.g. medicine,
biology, and nanotechnology (see, e.g. Refs. [1, 2, 3, 4, 5, 6, 7] and
references therein). In the majority of studies, the internal spin structure
of the nanoparticles is neglected and assumed to be uniform (so-called macro-
or superspin model). While this is probably justified in many application-
oriented approaches in which an overall understanding is sufficient, it is of
interest, at least from the standpoint of fundamental science, to elucidate
the effect of a nonuniform spin structure on a certain physical property.

Scattering techniques, in particular employing x-rays and neutrons, have
proven to be very powerful in this endeavor, since they provide statistically-
averaged information on a large number of scattering particles. For instance,
using Monte Carlo simulations of a discrete atomistic spin model, Köhler et
al. [8] have numerically studied the influence of antiphase boundaries in iron
oxide nanoparticles on their spin structure. These authors used the Debye
scattering equation to relate the internal spin disorder to the broadening of
certain x-ray Bragg peaks. Vivas et al. [9] carried out micromagnetic
continuum calculations of the spin structure of defect-free iron nanoparticles
and related a vortex-type magnetization configuration to certain signatures in
the magnetic neutron scattering cross section and correlation function.

Magnetic small-angle neutron scattering (SANS) is a powerful technique for
investigating spin structures on the mesoscopic length scale ($\sim
1$$-$$100\,\mathrm{nm}$) and inside the volume of magnetic materials [10, 11].
Recent SANS studies of magnetic nanoparticles, in particular employing spin-
polarized neutrons, unanimously demonstrate that their spin textures are
highly complex and exhibit a variety of nonuniform, canted, or core-shell-type
configurations (see, e.g. Refs. [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24] and references therein). The magnetic SANS data analysis largely
relies on structural form-factor-models for the cross section, borrowed from
nuclear SANS, which do not properly account for the existing spin
inhomogeneity inside a magnetic nanoparticle. Progress in magnetic SANS theory
[25, 26, 27, 28, 29, 30, 31, 32, 33, 34] strongly suggests that for the
analysis of experimental magnetic SANS data, the spatial nanometer scale
variation of the orientation and magnitude of the magnetization vector field
must be taken into account, going beyond the macrospin-based models that
assume a uniform magnetization.

In this paper, we employ atomistic simulations using the Landau-Lifshitz
equation (LLE) to investigate the role of the Néel surface anisotropy in
magnetic nanoparticles and its effect on the magnetic SANS cross section and
correlation function. We take into account the isotropic exchange interaction,
an external magnetic field, a magnetocrystalline anisotropy for the core of
the nanoparticles and Néel’s anisotropy for spins on the surface. Moreover,
the influence of a particle-size distribution function on the magnetic SANS
cross section and pair-distance correlation function is studied. The numerical
results reveal marked differences with the superspin model and provide
guidance for the experimentalist to identify nonuniform spin structures inside
magnetic nanoparticles. We also refer to our analytical study of the problem
[35], which is restricted to a linear approximation in the magnetization
deviation.

The paper is organized as follows: In Section II, we provide information on
the atomistic simulations using the LLE. In Section III, we display the
expressions for the magnetic SANS cross section and for the pair-distance
distribution function. The results of the numerical calculations are discussed
in Section IV, with Section IV.1 focusing on the effect of the Néel surface
anisotropy and Section IV.2 discussing the influence of a lognormal particle-
size distribution on the SANS observables. Section V summarizes the main
findings of this study and provides an outlook on future challenges.

##  II Details of the atomistic SANS modeling using the Landau-Lifshitz
equation

Fig. 1 schematically depicts the adopted procedure to generate and calculate
the spin structure, and to obtain the ensuing magnetic SANS cross section and
correlation function. This flowchart-type representation will be discussed in
more detail in the following.

A spherical many-spin nanomagnet is viewed as a crystallite consisting of
$\mathcal{N}$ atomic magnetic moments $\bm{\mu}_{i}=\mu_{a}\mathbf{m}_{i}$,
where $\mu_{a}$ denotes the magnitude of the atomic magnetic moment and
$\mathbf{m}_{i}$ is a unit vector specifying its orientation. We assume the
spins to ‘sit’ on a simple cubic lattice, so that $\mu_{a}=M_{s}a^{3}$, where
$M_{s}$ is the saturation magnetization of the material and $a$ is the lattice
constant. The spherical shape of the nanomagnet is cut from a simple cubic
regular grid [Fig. 1(a)], and its radius $R$ is defined as $R=\frac{N-1}{2}a$,
where the integer $N$ is the number of atoms on the side of the cubic grid.
The magnetic state of the nanomagnet is investigated with the help of the
atomistic approach based on the following Hamiltonian [36, 37, 38, 39, 40, 41,
42, 43]:

$\displaystyle\mathcal{H}$
$\displaystyle=\mathcal{H}_{{\mathrm{EX}}}+\mathcal{H}_{\mathrm{Z}}+\mathcal{H}_{\mathrm{A}}$
(1)
$\displaystyle=-\frac{1}{2}J\sum_{i,j\in{\mathrm{n.n.}}}\mathbf{m}_{i}\cdot\mathbf{m}_{j}-\mu_{a}\mathbf{B}_{0}\cdot\sum_{i=1}^{\mathcal{N}}\mathbf{m}_{i}+\sum_{i=1}^{\mathcal{N}}\mathcal{H}_{\mathrm{A},i},$
(2)

where $\mathcal{H}_{{\mathrm{EX}}}$ is the nearest-neighbor (n.n.) exchange
energy, with $J>0$ the exchange parameter, $\mathcal{H}_{{\mathrm{Z}}}$
denotes the Zeeman energy, with $\mathbf{B}_{0}=\mu_{0}\mathbf{H}_{0}$ the
homogeneous externally applied magnetic field, and
$\mathcal{H}_{{\mathrm{A}}}$ represents the magnetic anisotropy energy. For
the core spins, we assume the anisotropy to be of uniaxial symmetry, while for
surface spins we adopt the model proposed by Néel [44].
$\mathcal{H}_{\mathrm{A},i}$ can then be expressed as follows:

$\mathcal{H}_{\mathrm{A},i}=\begin{cases}-K_{c}\left(\mathbf{m}_{i}\cdot\mathbf{e}_{\mathrm{A}}\right)^{2},&i\in{\mathrm{core}}\\\
\\\
+\frac{1}{2}K_{s}{\displaystyle\sum_{j\in{\mathrm{n.n.}}}}\left(\mathbf{m}_{i}\cdot\mathbf{u}_{ij}\right)^{2},&i\in\mathrm{surface},\end{cases}$
(3)

where $K_{c}>0$ and $K_{s}>0$ denote, respectively, the core and surface
anisotropy constants, $\mathbf{e}_{\mathrm{A}}$ is a unit vector along the
core anisotropy easy direction, and
$\mathbf{u}_{ij}=(\mathbf{r}_{i}-\mathbf{r}_{j})/\|\mathbf{r}_{i}-\mathbf{r}_{j}\|$
is a unit vector connecting the nearest-neighbor spins $i$ and $j$.

The magnetodipolar interaction has been ignored in our simulations. This is
motivated by the numerical complexity of this energy term, in particular for
atomistic simulations (here for a $10\,\mathrm{nm}$ diameter particle the
number of spins is $\mathcal{N}=11633$), and by the expectation that it is of
minor relevance for smaller-sized nanomagnets [8, 45].

Figure 1: Flow chart explaining the atomistic SANS simulation procedure. (a) A
spherical nanoparticle is cut from a simple cubic grid with $N\times N\times
N$ atoms. (b) Time evolution of the Cartesian magnetization components
obtained by solving the Landau-Lifshitz equation. (c) Computed equilibrium
spin structure of a spherical nanoparticle at remanence (cut through the
center of the particle). (d) Hysteresis loop of an ensemble of randomly-
oriented nanoparticles. (e) Computed Fourier transform and (f) two-dimensional
magnetic SANS cross section $d\Sigma_{M}/d\Omega$. (g) Azimuthally-averaged
magnetic SANS cross section $I(q)$ and (h) pair-distance distribution function
$p(r)$.

The dynamics of each individual magnetic moment $\mathbf{m}_{i}$ is described
by the Landau-Lifshitz equation (LLE) [46]:

$\frac{d{\mathbf{m}}_{i}}{dt}=-\gamma\,\mathbf{m}_{i}\times\mathbf{B}_{i}^{\mathrm{eff}}-\alpha\,\mathbf{m}_{i}\times(\mathbf{m}_{i}\times\mathbf{B}_{i}^{\mathrm{eff}}),$
(4)

where $\gamma$ is the gyromagnetic ratio, and $\alpha$ denotes the damping
constant. The deterministic effective magnetic field acting on the spin $i$ is
given by:

$\displaystyle\mathbf{B}_{i}^{\mathrm{eff}}$
$\displaystyle=-\frac{1}{\mu_{a}}\frac{\delta\mathcal{H}}{\delta\mathbf{m}_{i}}$
$\displaystyle=\mathbf{B}_{0}+\frac{J}{\mu_{a}}\sum_{j\in\mathrm{n.n.}}\mathbf{m}_{j}$
$\displaystyle-\frac{1}{\mu_{a}}\begin{cases}-2K_{c}(\mathbf{m}_{i}\cdot\mathbf{e}_{\mathrm{A}})\,\mathbf{e}_{\mathrm{A}},&\;\;\;i\in\text{core},\\\
K_{s}\sum_{j\in\mathrm{n.n.}}(\mathbf{m}_{i}\cdot\mathbf{u}_{ij})\,\mathbf{u}_{ij},&\;\;\;i\in\text{surface}.\end{cases}$
(5)

The LLE is numerically solved by using the explicit Euler-forward-projection
method [47], which consist of two steps. The first step, as seen from equation
(6) below, is the simple Euler forward scheme and the second step, as seen
from equation (7) is the projection (or normalization) onto the unit sphere to
enforce the constraint $\|\mathbf{m}_{i}\|=1$. Since we are interested in the
static equilibrium, this first-order method is fully appropriate. In equations
(6) and (7), $k$ is the time iteration index while $i$ refers to the $i$th
lattice site:

$\displaystyle\mathbf{m}_{i}^{\mathrm{Euler}}$
$\displaystyle=\mathbf{m}_{i}^{k}+h_{t}\,\frac{d{\mathbf{m}}_{i}^{k}}{dt},$
(6) $\displaystyle\mathbf{m}_{i}^{k+1}$
$\displaystyle=\frac{\mathbf{m}_{i}^{\mathrm{Euler}}}{\|\mathbf{m}_{i}^{\mathrm{Euler}}\|},$
(7)

where $h_{t}$ denotes the time step for the integration procedure. For the
termination of the energy minimization, we have employed the following
criterion:

$\frac{h_{t}}{\mathcal{N}}\sqrt{\sum_{i=1}^{\mathcal{N}}\left\|\frac{d{\mathbf{m}}_{i}^{k}}{dt}\right\|^{2}}<10^{-8}.$
(8)

The macroscopic state of the nanomagnet is then described by the following
super- or macrospin (representing the net magnetic moment):

$\overline{\mathbf{m}}=\frac{1}{\mathcal{N}}\sum_{i=1}^{\mathcal{N}}\mathbf{m}_{i}.$
(9)

As an example, we show in Fig. 1(b) the temporal evolution of the Cartesian
magnetization components of $\overline{\mathbf{m}}$ and in Fig. 1(c) the
numerically-computed equilibrium spin configuration for a spherical nanomagnet
at zero applied field, in a plane across its center. It is seen that the spins
in the center of the nanoparticle are directed along $\overline{\mathbf{m}}$,
while the surface spins exhibit significant misalignment, which is due to the
presence of the Néel surface anisotropy. Note that the $\mathbf{m}_{i}$ are
unit vectors, whereas generally $\|\overline{\mathbf{m}}\|\neq 1$.

In our simulations, we used the following parameters: atomic magnetic moment
$\mu_{a}=1.577\times 10^{-23}$ Am2 (corresponding to $1.7\,\mu_{\mathrm{B}}$
with $\mu_{\mathrm{B}}$ the Bohr magneton), lattice constant
$a=0.3554\,\mathrm{nm}$, $M_{s}=351\,\mathrm{kA/m}$, exchange constant
$J=8.7\times 10^{-22}\,\mathrm{J/atom}$, core anisotropy constant
$K_{c}=3\times 10^{-24}\,\mathrm{J/atom}$, damping constant $\alpha=3\times
10^{11}$ (Ts)-1, gyromagnetic constant $\gamma=1.76\times 10^{11}$ (Ts)-1, and
an integration time step of $h_{t}=5$ fs. The surface anisotropy constant
$K_{s}$ was used as an adjustable parameter.

For the calculation of the magnetic SANS cross section $d\Sigma_{M}/d\Omega$
[Fig. 1(d)], it is necessary to compute the discrete Fourier transform of all
the ${\mathbf{m}}_{i}$ belonging to the spherical nanomagnet [Fig. 1(e)]. In
Sec. III, the expressions for $d\Sigma_{M}/d\Omega$ are formulated for a
continuous magnetization distribution $\mathbf{M}(\mathbf{r})$ and of its
Fourier transform $\widetilde{\mathbf{M}}(\mathbf{q})$. These functions are
defined as follows:

$\displaystyle\mathbf{M}(\mathbf{r})$
$\displaystyle=\frac{1}{(2\pi)^{3/2}}\int\widetilde{\mathbf{M}}(\mathbf{q})\exp\left(\mathrm{i}\mathbf{q}\cdot\mathbf{r}\right)\;d^{3}q,$
(10) $\displaystyle\widetilde{\mathbf{M}}(\mathbf{q})$
$\displaystyle=\frac{1}{(2\pi)^{3/2}}\int\mathbf{M}(\mathbf{r})\exp\left(-\mathrm{i}\mathbf{q}\cdot\mathbf{r}\right)\;d^{3}r.$
(11)

Using $\bm{\mu}_{i}=\mu_{a}\mathbf{m}_{i}$, the discrete-space Fourier
transform is computed as:

$\displaystyle\widetilde{\mathbf{M}}(\mathbf{q})\cong\frac{\mu_{a}}{(2\pi)^{3/2}}\sum_{i=1}^{\mathcal{N}}\mathbf{m}_{i}\exp\left(-\mathrm{i}\mathbf{q}\cdot\mathbf{r}_{i}\right),$
(12)

where $\mathbf{r}_{i}$ is the location point of the $i$th spin and
$\mathbf{q}$ represents the wave vector (scattering vector). Equation (12)
establishes the relation between the outcome of the simulations,
$\mathbf{m}_{i}$, and the magnetic SANS cross section $d\Sigma_{M}/d\Omega$.
In the standard SANS geometry, the $\mathbf{q}$-space of interest is defined
by $\mathbf{q}=q[0,\,\sin\theta,\,\cos\theta]$, which corresponds to the two-
dimensional detector plane ($q_{x}=0$, see Fig. 2). The two- and one-
dimensional magnetic SANS cross section $d\Sigma_{M}/d\Omega$ [Fig. 1(f) and
(g)] is then computed according to equation (13). A further Fourier
transformation yields the pair-distance distribution function [Fig. 1(h)].

At each value of the external field, atomistic simulations of the spin
structure and of the ensuing magnetic SANS cross section were carried out for
$256$ random orientations of the core anisotropy axes
$\mathbf{e}_{\mathrm{A}}$ of the particle with respect to the field
$\mathbf{B}_{0}$. More specifically, once the lattice orientation has been
randomly selected, the easy-axis orientation of the particle’s core and the
distribution of the Néel anisotropy are fixed. The whole system (core plus
surface anisotropy) is then randomly rotated relative to $\mathbf{B}_{0}$. For
the generation of the random angles, we used the low-discrepancy Sobol
sequence [48]. Therefore, except Fig. 3, all the data shown in this paper
correspond to an ensemble of randomly-oriented particles. The simulations were
carried out by starting from a large positive (saturating) field of about
$10\,\mathrm{T}$ and then the field was reduced in steps of typically
$30\,\mathrm{mT}$.

##  III Magnetic SANS cross section and pair-distance distribution function

The quantity of interest in experimental SANS studies is the elastic magnetic
differential scattering cross section $d\Sigma_{M}/d\Omega$, which is usually
recorded on a two-dimensional position-sensitive detector. For the most
commonly used scattering geometry in magnetic SANS experiments, where the
applied magnetic field $\mathbf{B}_{0}\parallel\mathbf{e}_{z}$ is
perpendicular to the wave vector $\mathbf{k}_{0}\parallel\mathbf{e}_{x}$ of
the incident neutrons (see Fig. 2), $d\Sigma_{M}/d\Omega$ (for unpolarized
neutrons) can be written as [10]:

$\displaystyle\frac{d\Sigma_{M}}{d\Omega}(\mathbf{q})=\frac{8\pi^{3}}{V}b_{H}^{2}\left(|\widetilde{M}_{x}|^{2}+|\widetilde{M}_{y}|^{2}\cos^{2}\theta\right.$
$\displaystyle\left.+|\widetilde{M}_{z}|^{2}\sin^{2}\theta-(\widetilde{M}_{y}\widetilde{M}_{z}^{\ast}+\widetilde{M}_{y}^{\ast}\widetilde{M}_{z})\sin\theta\cos\theta\right),$
(13)

where $V$ is the scattering volume, $b_{H}=2.91\times
10^{8}\,\mathrm{A}^{-1}\mathrm{m}^{-1}$ is the magnetic scattering length in
the small-angle regime (the atomic magnetic form factor is approximated by $1$
since we are dealing with forward scattering),
$\widetilde{\mathbf{M}}(\mathbf{q})=[\widetilde{M}_{x}(\mathbf{q}),\widetilde{M}_{y}(\mathbf{q}),\widetilde{M}_{z}(\mathbf{q})]$
represents the Fourier transform of the magnetization vector field
$\mathbf{M}(\mathbf{r})=[M_{x}(\mathbf{r}),M_{y}(\mathbf{r}),M_{z}(\mathbf{r})]$,
$\theta$ denotes the angle between $\mathbf{q}$ and $\mathbf{B}_{0}$, and the
asterisk ‘$*$’ stands for the complex-conjugated quantity. Note that in the
perpendicular scattering geometry the Fourier components are evaluated in the
plane $q_{x}=0$ (see Fig. 2).

The numerically computed magnetic SANS cross sections that are displayed in
this paper correspond to the following average:

$\displaystyle\frac{d\Sigma_{M}}{d\Omega}=\left\langle\frac{d\Sigma_{M}}{d\Omega}\right\rangle_{\mathbf{e}_{\mathrm{A}}}=\frac{1}{\mathcal{K}}\sum_{k=1}^{\mathcal{K}}\frac{d\Sigma_{M,k}}{d\Omega},$
(14)

where $d\Sigma_{M,k}/d\Omega$ represents (for fixed $K_{s}$ and $B_{0}$) the
magnetic SANS cross section for a particular core easy-axis orientation
$\mathbf{e}_{\mathrm{A}}$ (referred to index ‘$k$’), and $\mathcal{K}$ denotes
the number of random configurations. Equation (14) implies the absence of
interparticle interactions.

For a uniformly magnetized spherical particle with its saturation direction
parallel to $\mathbf{e}_{z}$, i.e. $M_{x}=M_{y}=0$ and $M_{z}=M_{s}$, equation
(13) reduces to:

$\displaystyle\frac{d\Sigma_{M}}{d\Omega}(q,\theta)=V_{p}(\Delta\rho)_{\mathrm{mag}}^{2}\,9\left(\frac{j_{1}(qR)}{qR}\right)^{2}\sin^{2}\theta,$
(15)

where $V_{p}=\frac{4\pi}{3}R^{3}$ is the particle’s volume,
$(\Delta\rho)_{\mathrm{mag}}^{2}=b_{H}^{2}\left(\Delta
M\right)^{2}=b_{H}^{2}M_{s}^{2}$ is the magnetic scattering-length density
contrast, and $j_{1}(qR)$ is the first-order spherical Bessel function. The
well-known analytical result for the homogeneous sphere case, equation (15),
and its correlation function [see equation (18) below] serve as a reference
for comparison with the nonuniform case.

Figure 2: Sketch of the neutron scattering geometry. The applied magnetic
field $\mathbf{B}_{0}\parallel\mathbf{e}_{z}$ is perpendicular to the wave
vector $\mathbf{k}_{0}\parallel\mathbf{e}_{x}$ of the incident neutron beam
($\mathbf{B}_{0}\perp\mathbf{k}_{0}$). The momentum-transfer or scattering
vector $\mathbf{q}$ is defined as the difference between $\mathbf{k}_{0}$ and
$\mathbf{k}_{1}$, i.e. $\mathbf{q}=\mathbf{k}_{0}-\mathbf{k}_{1}$. SANS is
usually implemented as elastic scattering ($k_{0}=k_{1}=2\pi/\lambda$), and
the component of $\mathbf{q}$ along the incident neutron beam, here $q_{x}$,
is much smaller than the other two components, so that
$\mathbf{q}\cong[0,q_{y},q_{z}]=q[0,\sin\theta,\cos\theta]$. This demonstrates
that SANS probes predominantly correlations in the plane perpendicular to the
incident beam. For elastic scattering, the magnitude of $\mathbf{q}$ is given
by $q=(4\pi/\lambda)\sin(\psi)$, where $\lambda$ denotes the mean wavelength
of the neutrons and $2\psi$ is the scattering angle. The angle
$\theta=\angle(\mathbf{q},\mathbf{B}_{0})$ is used to describe the angular
anisotropy of the recorded scattering pattern on the two-dimensional position-
sensitive detector.

It is often convenient to average the two-dimensional SANS cross section
$\frac{d\Sigma_{M}}{d\Omega}(\mathbf{q})=\frac{d\Sigma_{M}}{d\Omega}(q_{y},q_{z})=\frac{d\Sigma_{M}}{d\Omega}(q,\theta)$
along certain directions in $\mathbf{q}$-space, e.g. parallel ($\theta=0$) or
perpendicular ($\theta=\pi/2$) to the applied magnetic field, or even over the
full angular $\theta$ range. In the following, we consider the
$2\pi$-azimuthally-averaged magnetic SANS cross section

$\displaystyle
I(q)\equiv\frac{1}{2\pi}\int_{0}^{2\pi}\frac{d\Sigma_{M}}{d\Omega}(q,\theta)\,d\theta,$
(16)

which is used to compute the pair-distance distribution function $p(r)$
according to:

$\displaystyle p(r)=r^{2}\int\limits_{0}^{\infty}I(q)j_{0}(qr)q^{2}dq,$ (17)

where $j_{0}(qr)=\sin(qr)/(qr)$ is the spherical Bessel function of zero
order; $p(r)$ corresponds to the distribution of real-space distances between
volume elements inside the particle weighted by the excess scattering-length
density distribution; see the reviews by Glatter [49] and by Svergun and Koch
[50] for detailed discussions of the properties of $p(r)$ and for information
on how to compute it by indirect Fourier transformation [51]. For our discrete
simulation data, the integrals in equations (16) and (17) were approximated by
the trapezoidal rule. Apart from constant prefactors, the $p(r)$ of the
azimuthally-averaged single-particle cross section [equation (15)],
corresponding to a uniform sphere magnetization, is given by (for $r\leq 2R$):

$\displaystyle p(r)=r^{2}\left(1-\frac{3r}{4R}+\frac{r^{3}}{16R^{3}}\right).$
(18)

We also display results for the correlation function $c(r)$, which is related
to $p(r)$ by

$\displaystyle c(r)=p(r)/r^{2}.$ (19)

As we will demonstrate in the following, when the particles’ spin structure is
inhomogeneous, the $d\Sigma_{M}/d\Omega$ and the corresponding $p(r)$ and
$c(r)$ differ significantly from the homogeneous case [equations (15) and
(18)], which serve as a reference. Due to the $r^{2}$ factor, features in
$p(r)$ at medium and large distances are more pronounced than in $c(r)$.

##  IV Results and Discussion

###  IV.1 Effect of the Néel surface anisotropy

Figure 3: Selected 3D equilibrium spin structures arising from the Néel
surface anisotropy (compare also to Figs. 2 and 3 in the analytical study
[35]). (a) $K_{s}=5.22\times 10^{-23}\,\mathrm{J/atom}$; (b) $K_{s}=52.2\times
10^{-23}\,\mathrm{J/atom}$. Further parameters are: core-anisotropy axis
$\mathbf{e}_{\mathrm{A}}=[0,0,1]$, core-anisotropy constant $K_{c}=3\times
10^{-24}\,\mathrm{J/atom}$, and external magnetic field
$\mathbf{B}_{0}=[0,0,150\,\mathrm{mT}]$. The color code depicts the spin
misalignment relative to the average magnetization vector, namely $\delta
m_{j}=\|\mathbf{m}_{j}-\overline{\mathbf{m}}/\|\overline{\mathbf{m}}\|\|$. At
the surface of the nanomagnet the spin deviations are larger compared to the
core.

Figure 4: Computed normalized magnetization $\overline{m}_{z}$ [compare
equation (9)] of an ensemble of randomly-oriented $10$-nm-sized spherical
nanomagnets for different values of the surface anisotropy constant $K_{s}$
(in units of $10^{-23}\,\mathrm{J/atom}$, see inset).

Fig. 3 displays as an example the spin structures of a $5$-nm-sized spherical
nanomagnet for the cases of a small and large surface-anisotropy constant
$K_{s}$, and Fig. 4 shows computed hysteresis curves for an ensemble of
randomly-oriented $10$-nm-sized nanomagnets. As expected, increasing $K_{s}$
results, for a given particle size, in a progressive surface spin disorder
which propagates into the bulk of the nanomagnet. The effect of an enhanced
$K_{s}$ also becomes visible in the magnetization curves via an increased
coercivity $H_{c}$ and remanence $m_{r}$. For $K_{s}=0$ and dominant exchange,
we recover the well-known results from the Stoner-Wohlfarth model [52], i.e.
we find a reduced remanence of $m_{r}=0.5$ and a coercivity of

$\displaystyle\mu_{0}H_{c}=0.48\frac{2K_{c}}{\mu_{a}}\frac{\mathcal{N}_{c}}{\mathcal{N}}=183\,\mathrm{mT},$
(20)

where $\mathcal{N}_{c}$ denotes the number of atoms belonging to the
particle’s core. Note that for the case of a strong surface anisotropy [Fig.
3(b)], the mean magnetization at remanence deviates strongly from the core
anisotropy axis, which is in contrast to the case of weak anisotropy [Fig.
3(a)]. This observation is in agreement with the analytical calculations by
Garanin and Kachkachi [53] who have predicted the emergence of an effective
anisotropy of cubic symmetry for dominant $K_{s}$. Therefore, with increasing
$K_{s}$, we observe in Fig. 4 an increase of the remanence.

Figure 5: Decryption of the two-dimensional magnetic SANS cross section
$d\Sigma_{M}/d\Omega$ in the remanent state ($B_{0}=0\,\mathrm{T}$) into the
individual magnetization Fourier components $|\widetilde{M}_{x}|^{2}$,
$|\widetilde{M}_{y}|^{2}$, $|\widetilde{M}_{z}|^{2}$, and
$CT=-(\widetilde{M}_{y}\widetilde{M}_{z}^{\ast}+\widetilde{M}_{y}^{\ast}\widetilde{M}_{z})$
(see insets) (logarithmic color scale). Note that the respective Fourier
components are multiplied with the constant $8\pi^{3}V^{-1}b_{H}^{2}$ (in
order to have the same unit as $d\Sigma_{M}/d\Omega$), but not with the
trigonometric functions in the expression for $d\Sigma_{M}/d\Omega$ [see
equation (13)]. The $\%$ values specify the fraction of the respective Fourier
component of the total $d\Sigma_{M}/d\Omega$ [see equation (21) and related
text in the main paper]. The $CT$ (and hence the corresponding
$\eta_{\alpha}$) can take on negative values, however, in this figure we show
(due to the chosen logarithmic color scale) the absolute value of the $CT$.
The data correspond to an ensemble of randomly-oriented
$10\,\mathrm{nm}$-sized nanomagnets. (upper row) $K_{s}=0$; (2nd row)
$K_{s}=5.22\times 10^{-23}\,\mathrm{J/atom}$; (3rd row) $K_{s}=26.1\times
10^{-23}\,\mathrm{J/atom}$; (lower row) $K_{s}=52.2\times
10^{-23}\,\mathrm{J/atom}$.

Figure 6: Same as Fig. 5, but for $B_{0}=10\,\mathrm{T}$.

Fig. 5 displays the two-dimensional magnetic SANS cross section
$d\Sigma_{M}/d\Omega$ of an ensemble of $10$-nm-sized nanomagnets in the
remanent magnetization state along with the individual Fourier components
$|\widetilde{M}_{x}|^{2}$, $|\widetilde{M}_{y}|^{2}$,
$|\widetilde{M}_{z}|^{2}$, and the cross term
$CT=-(\widetilde{M}_{y}\widetilde{M}_{z}^{\ast}+\widetilde{M}_{y}^{\ast}\widetilde{M}_{z})$
[see equation (13)]. Fig. 6 shows the corresponding plots at a (nearly)
saturating field of $B_{0}=10\,\mathrm{T}$. We emphasize that the depicted
scalar functions represent projections of the corresponding three-dimensional
quantities onto the $q_{y}$$-$$q_{z}$ detector plane at $q_{x}=0$ (see Fig.
2). The surface anisotropy constant $K_{s}$ increases from the top to the
bottom row in Figs. 5 and 6. It is seen that, generally, all the Fourier
components are contributing to $d\Sigma_{M}/d\Omega$.

Near saturation (Fig. 6), $d\Sigma_{M}/d\Omega$ is dominated for all values of
$K_{s}$ by the isotropic ($\theta$ independent) $|\widetilde{M}_{z}|^{2}$
Fourier component and exhibits the characteristic $\sin^{2}\theta$ anisotropy
with two maxima along the vertical direction [compare equation (13)].
Increasing $K_{s}$ enhances the contributions of both transversal Fourier
components $|\widetilde{M}_{x}|^{2}$ and $|\widetilde{M}_{y}|^{2}$ and of the
$CT$. Moreover, the latter contributions develop a pronounced angular
anisotropy with increasing $K_{s}$.

At remanence (Fig. 5), $d\Sigma_{M}/d\Omega$ and all the Fourier components
are isotropic for small values of $K_{s}$ and become progressively more
anisotropic with increasing $K_{s}$. For instance, $|\widetilde{M}_{z}|^{2}$
is initially isotropic and develops a pronounced angular anisotropy that is
elongated along the $q_{y}$ direction for larger $K_{s}$. The $CT$ also
develops an anisotropy with increasing $K_{s}$ with maxima roughly along the
detector diagonals. An anisotropic magnetic SANS cross section at zero applied
magnetic field of an ensemble of randomly-oriented nanoparticles has also been
found in the micromagnetic continuum simulations of Vivas et al. [9]. These
authors did not consider the Néel surface anisotropy but included the
magnetodipolar interaction.

To quantify the fraction of the individual Fourier components in equation (13)
to the total magnetic SANS cross section $d\Sigma_{M}/d\Omega$, we have
computed the following dimensionless quantity:

$\displaystyle\eta_{\alpha}=\frac{1}{\pi
q_{\mathrm{max}}^{2}}\int_{0}^{2\pi}\int_{0}^{q_{\mathrm{max}}}\frac{\alpha(q,\theta)}{d\Sigma_{M}/d\Omega}qdqd\theta,$
(21)

where $\alpha(q,\theta)$ is, respectively, given by
$K|\widetilde{M}_{x}|^{2}$, $K|\widetilde{M}_{y}|^{2}\cos^{2}\theta$,
$K|\widetilde{M}_{z}|^{2}\sin^{2}\theta$, and $K\,CT\sin\theta\cos\theta$ with
$K=8\pi^{3}b_{H}^{2}V^{-1}$; $q_{\mathrm{max}}$ has been taken as
$q_{\mathrm{max}}=10\,\mathrm{nm}^{-1}$. The sum of the $\eta_{\alpha}$ is
then equal to one. The corresponding numbers are specified in the insets of
Fig. 5, and we note that the contribution related to
$K\,CT\sin\theta\cos\theta$ can be negative, in contrast to the other three
contributions which are strictly positive. Moreover, using the inequality
$|\widetilde{M}_{y}\cos\theta-\widetilde{M}_{z}\sin\theta|^{2}\geq 0$, it can
easily be shown that the contribution $K\,CT\sin\theta\cos\theta$ is, however,
always smaller than the sum of the other terms (as it must be). We emphasize
that the color-coded plots in Fig. 5 show the respective Fourier components
without the trigonometric functions in equation (13), whereas the quantities
$\eta_{\alpha}$ do contain the trigonometric terms. For $K_{s}=0$ and zero
field, the contributions of $|\widetilde{M}_{x}|^{2}$,
$|\widetilde{M}_{y}|^{2}$, and $|\widetilde{M}_{z}|^{2}$ to
$d\Sigma_{M}/d\Omega$ are approximately equal (while $CT=0$). This can be
understood by noting the isotropy of these functions and by taking into
account the trigonometric terms $\cos^{2}\theta$ (for
$|\widetilde{M}_{y}|^{2}$) and $\sin^{2}\theta$ (for
$|\widetilde{M}_{z}|^{2}$), which yield a factor of $1/2$ on azimuthal
averaging [$\theta$ integration, compare equation (21)].

Figure 7: Effect of the surface anisotropy constant $K_{s}$ (in units of
$10^{-23}\,\mathrm{J/atom}$, see inset) on (a) the azimuthally-averaged
magnetic SANS cross section $I(q)=\frac{d\Sigma_{M}}{d\Omega}(q)$ (log-log
scale), (b) the value of the magnetic SANS cross section at the origin,
$I(q=0)$ versus $K_{s}$, (c) the pair-distance distribution function $p(r)$,
and (d) the correlation function $c(r)$. The data correspond to the remanent
state ($B_{0}=0\,\mathrm{T}$), and the nanomagnets’ diameter is
$10\,\mathrm{nm}$. The green dashed line in (c) displays the analytical pair-
distance distribution function for the case of a uniformly magnetized
spherical particle [proportional to equation (18)], where the magnitude is
normalized to the maximum value from the numerical simulation in the case
$K_{s}=0$.

Figure 8: Same as Fig. 7, but for $B_{0}=10\,\mathrm{T}$.

The effect of increasing $K_{s}$ on the $2\pi$ azimuthally-averaged
$I(q)=d\Sigma_{M}/d\Omega$ and on $p(r)$ and $c(r)$ is shown in Fig. 7 for the
remanent state and in Fig. 8 for $B_{0}=10\,\mathrm{T}$. With increasing spin
disorder (induced by an increasing $K_{s}$) we observe in Fig. 7(a) that (i)
the characteristic form-factor oscillations of $I(q)$ are progressively damped
and (ii) that the maxima in $I(q)$ shift to larger $q$ values. The smearing of
the form-factor oscillations at increased $K_{s}$ mimicks the effect of a
particle-size distribution function and/or of instrumental resolution.
Therefore, in experimental situations, when such a data is fitted to e.g. a
sphere form factor with a distribution of particle sizes, an erroneous value
for the particle size may result. At (quasi)saturation [Fig. 8(c)] and for
small $K_{s}$ at remanence [Fig. 7(c)], we recover the analytically-known
expressions for $I(q)$, $p(r)$, and $c(r)$ for uniformly magnetized spherical
particles [equations (15) and (18), compare Figs. 7(c) and 8(c)]. We have also
plotted in Figs. 7(b) and 8(b) the $K_{s}$ dependence of the $q=0$
extrapolated value of $I(q)$. The quantity $I(q=0)$ is directly proportional
to the static susceptibility $\chi(q=0)$ (as it can be measured with a
magnetometer), which itself is proportional to the mean-square fluctuation of
the magnetization per atom [54]. We see that, as expected, the increase in
$K_{s}$ has a large effect on $\chi(0)$, whereas the reduction is relatively
small at $10\,\mathrm{T}$.

###  IV.2 Effect of a particle-size distribution

In SANS experiments on nanoparticles one always has to deal with a
distribution of particle sizes and shapes. The size of a particle has an
important effect on its spin structure, e.g. smaller particles tend to be
uniformly magnetized, whereas larger particles may exhibit inhomogeneous spin
structures. It is therefore also of interest to study the influence of a
distribution of particle sizes on the magnetic SANS observables
[$d\Sigma_{\mathrm{M}}/d\Omega$, $p(r)$, $c(r)$]. This has been done using a
lognormal probability distribution function, which is defined as [55]:

$\displaystyle w(D)$ $\displaystyle=\frac{1}{\sqrt{2\pi
D^{2}\operatorname{ln}\left(1+\frac{\sigma^{2}}{\mu^{2}}\right)}}\exp\left(-\frac{\mathrm{ln}^{2}\left(\frac{D}{\mu}\sqrt{1+\frac{\sigma^{2}}{\mu^{2}}}\right)}{2\operatorname{ln}\left(1+\frac{\sigma^{2}}{\mu^{2}}\right)}\right),$
(22)

where $\mu$ denotes the expectation value and $\sigma^{2}$ is the variance,
such that:

$\displaystyle\mu$ $\displaystyle=\int_{0}^{\infty}w(D)D\;dD>0,$ (23)
$\displaystyle\sigma^{2}$
$\displaystyle=\int_{0}^{\infty}w(D)(D-\mu)^{2}\;dD,$ (24)

where the corresponding median $\mu^{\ast}$ is determined by the following
relation:

$\displaystyle\int_{0}^{\mu^{\ast}}w(D)\;dD$ $\displaystyle=\frac{1}{2},$
$\displaystyle\mu^{\ast}$
$\displaystyle=\frac{\mu^{2}}{\sqrt{\mu^{2}+\sigma^{2}}}.$ (25)

For given values of $\mu$ and $\sigma$, the averaged magnetic SANS cross
section $\langle...\rangle$ is computed as:

$\displaystyle\left\langle\frac{d\Sigma_{M}}{d\Omega}\right\rangle=\sum_{\ell=1}^{L}\frac{d\Sigma_{M,\ell}}{d\Omega}P_{\ell},$
(26)

where $P_{\ell}$ denotes the probability related to the particle-size class
$D_{\ell}=2R_{\ell}$ (diameter), which is computed as:

$\displaystyle P_{\ell}=\int_{D_{\ell}-\Delta D/2}^{D_{\ell}+\Delta
D/2}w(D)\;dD.$ (27)

Fig. 9 summarizes the results obtained for the magnetic SANS cross section and
correlation function. As expected, one observes a smearing of the SANS cross
section with increasing width $\sigma$ of the distribution, which becomes
particularly visible in the azimuthally-averaged $\left\langle
I(q)\right\rangle$ curves via the suppression of the form-factor oscillations.
The angular anisotropy of the SANS cross section in the remanent state, which
can be seen as a characteristic signature of the Néel anisotropy (compare also
the lower row in Fig. 5), becomes less pronounced for large $\sigma$. With
increasing field, the $\left\langle p(r)\right\rangle$ are reduced in
magnitude and the maximum is shifted to larger sizes (due to the suppression
of the internal spin disorder).

Figure 9: Effect of a lognormal particle-size distribution function on the
SANS observables ($K_{s}=52.2\times 10^{-23}\,\mathrm{J/atom}$). Shown are the
two-dimensional $\left\langle d\Sigma_{M}/d\Omega\right\rangle$, the
corresponding azimuthally-averaged $\left\langle I(q)\right\rangle$, the pair-
distance distribution functions $\left\langle p(r)\right\rangle$, and the
particle-size distributions $w(D)$ for $\sigma$ values of (a)
$\sigma=3\,\mathrm{nm}$, (b) $\sigma=2\,\mathrm{nm}$, and (c)
$\sigma=1\,\mathrm{nm}$. The nanoparticles’ mean diameter (expectation value)
was chosen as $\mu=10\,\mathrm{nm}$ in each case. The data correspond to the
remanent ($B_{0}=0\,\mathrm{T}$) and saturated ($B_{0}=10\,\mathrm{T}$)
magnetization state. The discrete particle-size classes are defined by the
particle diameters $D=3\dots 16\,\mathrm{nm}$ with an equidistant step size of
$\Delta D=1\,\mathrm{nm}$. The black dashed $\left\langle p(r)\right\rangle$
curves are the analytically known solution for uniformly magnetized spheres of
size $\mu=10\,\mathrm{nm}$ in the fully saturated state.

##  V Conclusions and Outlook

We have studied the spin structure and magnetic neutron scattering signal of
an ensemble of randomly-oriented spherical nanomagnets using the Landau-
Lifshitz equation with particular focus on the Néel surface anisotropy. Taking
into account the isotropic exchange interaction, an external magnetic field, a
uniaxial magnetic core anisotropy, and the Néel surface anisotropy, we compute
the magnetic small-angle neutron scattering cross section and the pair-
distance distribution function from the obtained equilibrium spin structures.
The numerical results are compared to the well-known analytical expressions
for uniformly magnetized particles. Upon increasing internal spin disorder
(increasing surface anisotropy $K_{s}$), the pair-distance distribution
function (at remanence) exhibits a systematic shift of its maximum to smaller
$r$ values, and the total magnetic SANS cross section develops a
characteristic anisotropic scattering pattern. The strength of the simulation
methodology is that the field evolution of the individual Fourier components
and their contribution to the magnetic SANS signal can be monitored. Atomistic
and micromagnetic continuum simulations have contributed and will continue to
contribute to the fundamental understanding of magnetic SANS. In our future
work, we will focus on the inclusion of both the intraparticle and the
interparticle dipole-dipole energy and the Dzyaloshinskii-Moriya interaction,
which will give rise to more complicated spin textures (e.g. vortex-type
structures), in particular for larger particle sizes. Moreover, it is of
interest to compare the Néel anisotropy to other phenomenological expressions
for the surface anisotropy, such as energy densities of the type
$\pm\frac{1}{2}K_{s}(\mathbf{m}\cdot\mathbf{n})^{2}$, where $\mathbf{n}$ is
the unit normal vector to the surface (instead of the $\mathbf{u}_{ij}$), or
to the case of a truly random surface anisotropy, where the $\mathbf{u}_{ij}$
are random vectors. In this regard, the present first atomistic simulations
may be considered as the starting point towards a more complete description of
magnetic SANS.

###### Acknowledgements.

Michael Adams and Andreas Michels thank the National Research Fund of
Luxembourg for financial support (AFR Grant No. 15639149).

## References

  * Lak  _et al._ [2021] A. Lak, S. Disch, and P. Bender, Adv. Sci. 8, 2002682 (2021).
  * Diebold and Calonge [2010] Y. Diebold and M. Calonge, Prog. Ret. Eye Res. 29, 596 (2010).
  * De  _et al._ [2008] M. De, P. S. Ghosh, and V. M. Rotello, Adv. Mater. 20, 4225 (2008).
  * Baetke  _et al._ [2015] S. C. Baetke, T. Lammers, and F. Kiessling, Br. J. Radiol. 88, 20150207 (2015).
  * Stark  _et al._ [2015] W. J. Stark, P. R. Stoessel, W. Wohlleben, and A. Hafner, Chem. Soc. Rev. 44, 5793 (2015).
  * Han  _et al._ [2019] X. Han, K. Xu, O. Taratula, and F. Khashayar, Nanoscale 11, 799 (2019).
  * Batlle  _et al._ [2022] X. Batlle, C. Moya, M. Escoda-Torroella, $\mathrm{\grave{O}}$. Iglesias, A. Fraile Rodríguez, and A. Labarta, J. Magn. Magn. Mater. 543, 168594 (2022).
  * Köhler  _et al._ [2021] T. Köhler, A. Feoktystov, O. Petracic, N. Nandakumaran, A. Cervellino, and T. Brückel, J. Appl. Cryst. 54, 1719 (2021).
  * Vivas  _et al._ [2020] L. G. Vivas, R. Yanes, D. Berkov, S. Erokhin, M. Bersweiler, D. Honecker, P. Bender, and A. Michels, Phys. Rev. Lett. 125, 117201 (2020).
  * Mühlbauer  _et al._ [2019] S. Mühlbauer, D. Honecker, E. A. Périgo, F. Bergner, S. Disch, A. Heinemann, S. Erokhin, D. Berkov, C. Leighton, M. R. Eskildsen, and A. Michels, Rev. Mod. Phys. 91, 015004 (2019).
  * Michels [2021] A. Michels,  _Magnetic Small-Angle Neutron Scattering: A Probe for Mesoscale Magnetism Analysis_ (Oxford University Press, Oxford, 2021).
  * Disch  _et al._ [2012] S. Disch, E. Wetterskog, R. P. Hermann, A. Wiedenmann, U. Vainio, G. Salazar-Alvarez, L. Bergström, and T. Brückel, New J. Phys. 14, 013025 (2012).
  * Krycka  _et al._ [2014] K. L. Krycka, J. A. Borchers, R. A. Booth, Y. Ijiri, K. Hasz, J. J. Rhyne, and S. A. Majetich, Phys. Rev. Lett. 113, 147203 (2014).
  * Hasz  _et al._ [2014] K. Hasz, Y. Ijiri, K. L. Krycka, J. A. Borchers, R. A. Booth, S. Oberdick, and S. A. Majetich, Phys. Rev. B 90, 180405(R) (2014).
  * Günther  _et al._ [2014] A. Günther, D. Honecker, J.-P. Bick, P. Szary, C. D. Dewhurst, U. Keiderling, A. V. Feoktystov, A. Tschöpe, R. Birringer, and A. Michels, J. Appl. Cryst. 47, 992 (2014).
  * Maurer  _et al._ [2014] T. Maurer, S. Gautrot, F. Ott, G. Chaboussant, F. Zighem, L. Cagnon, and O. Fruchart, Phys. Rev. B 89, 184423 (2014).
  * Dennis  _et al._ [2015] C. L. Dennis, K. L. Krycka, J. A. Borchers, R. D. Desautels, J. van Lierop, N. F. Huls, A. J. Jackson, C. Gruettner, and R. Ivkov, Adv. Funct. Mater. 25, 4300 (2015).
  * Grutter  _et al._ [2017] A. J. Grutter, K. L. Krycka, E. V. Tartakovskaya, J. A. Borchers, K. S. M. Reddy, E. Ortega, A. Ponce, and B. J. H. Stadler, ACS Nano 11, 8311 (2017).
  * Oberdick  _et al._ [2018] S. D. Oberdick, A. Abdelgawad, C. Moya, S. Mesbahi-Vasey, D. Kepaptsoglou, V. K. Lazarov, R. F. L. Evans, D. Meilak, E. Skoropata, J. van Lierop, I. Hunt-Isaak, H. Pan, Y. Ijiri, K. L. Krycka, J. A. Borchers, and S. A. Majetich, Sci. Rep. 8, 3425 (2018).
  * Ijiri  _et al._ [2019] Y. Ijiri, K. L. Krycka, I. Hunt-Isaak, H. Pan, J. Hsieh, J. A. Borchers, J. J. Rhyne, S. D. Oberdick, A. Abdelgawad, and S. A. Majetich, Phys. Rev. B 99, 094421 (2019).
  * Bender  _et al._ [2019] P. Bender, D. Honecker, and L. F. Barquín, Appl. Phys. Lett. 115, 132406 (2019).
  * Bersweiler  _et al._ [2019] M. Bersweiler, P. Bender, L. G. Vivas, M. Albino, M. Petrecca, S. Mühlbauer, S. Erokhin, D. Berkov, C. Sangregorio, and A. Michels, Phys. Rev. B 100, 144434 (2019).
  * Zákutná  _et al._ [2020] D. Zákutná, D. Ni$\mathrm{\check{z}}$$\mathrm{\check{n}}$anský, L. C. Barnsley, E. Babcock, Z. Salhi, A. Feoktystov, D. Honecker, and S. Disch, Phys. Rev. X 10, 031019 (2020).
  * Honecker  _et al._ [2022] D. Honecker, M. Bersweiler, S. Erokhin, D. Berkov, K. Chesnel, D. A. Venero, A. Qdemat, S. Disch, J. K. Jochum, A. Michels, and P. Bender, Nanoscale Adv. 4, 1026 (2022).
  * Honecker and Michels [2013] D. Honecker and A. Michels, Phys. Rev. B 87, 224426 (2013).
  * Michels  _et al._ [2014] A. Michels, S. Erokhin, D. Berkov, and N. Gorn, J. Magn. Magn. Mater. 350, 55 (2014).
  * Mettus and Michels [2015] D. Mettus and A. Michels, J. Appl. Cryst. 48, 1437 (2015).
  * Erokhin  _et al._ [2015] S. Erokhin, D. Berkov, and A. Michels, Phys. Rev. B 92, 014427 (2015).
  * Metlov and Michels [2015] K. L. Metlov and A. Michels, Phys. Rev. B 91, 054404 (2015).
  * Metlov and Michels [2016] K. L. Metlov and A. Michels, Sci. Rep. 6, 25055 (2016).
  * Michels  _et al._ [2016] A. Michels, D. Mettus, D. Honecker, and K. L. Metlov, Phys. Rev. B 94, 054424 (2016).
  * Michels  _et al._ [2019] A. Michels, D. Mettus, I. Titov, A. Malyeyev, M. Bersweiler, P. Bender, I. Peral, R. Birringer, Y. Quan, P. Hautle, J. Kohlbrecher, D. Honecker, J. R. Fernández, L. F. Barquín, and K. L. Metlov, Phys. Rev. B 99, 014416 (2019).
  * Mistonov  _et al._ [2019] A. A. Mistonov, I. S. Dubitskiy, I. S. Shishkin, N. A. Grigoryeva, A. Heinemann, N. A. Sapoletova, G. A. Valkovskiy, and S. V. Grigoriev, J. Magn. Magn. Mater. 477, 99 (2019).
  * Zaporozhets  _et al._ [2022] V. D. Zaporozhets, Y. Oba, A. Michels, and K. L. Metlov, J. Appl. Cryst. xy, in press (2022).
  * Adams  _et al._ [2022] M. P. Adams, A. Michels, and H. Kachkachi, J. Appl. Cryst. xy, abc (2022).
  * Dimitrov and Wysin [1994] D. A. Dimitrov and G. M. Wysin, Phys. Rev. B 50, 3077 (1994).
  * Kodama and Berkovitz [1999] R. H. Kodama and A. Berkovitz, Phys. Rev. B 59, 6321 (1999).
  * Kachkachi and Garanin [2001a] H. Kachkachi and D. A. Garanin, Physica A 300, 487 (2001a).
  * Kachkachi and Garanin [2001b] H. Kachkachi and D. A. Garanin, Eur. Phys. J. B 22, 291 (2001b).
  * Iglesias and Labarta [2001] O. Iglesias and A. Labarta, Phys. Rev. B 63, 184416 (2001).
  * Kachkachi and Dimian [2002] H. Kachkachi and M. Dimian, Phys. Rev. B 66, 174419 (2002).
  * H. Kachkachi and D. A. Garanin [2005] H. Kachkachi and D. A. Garanin, in  _Surface Effects in Magnetic Nanoparticles_ , edited by D. Fiorani (Springer, Berlin, 2005) p. 75.
  * Kazantseva  _et al._ [2008] N. Kazantseva, D. Hinzke, U. Nowak, R. W. Chantrell, U. Atxitia, and O. Chubykalo-Fesenko, Phys. Rev. B 77, 184428 (2008).
  * Néel [1954] L. Néel, J. Phys. Radium 15, 225 (1954).
  * Pathak and Hertel [2021] S. A. Pathak and R. Hertel, Phys. Rev. B 103, 104414 (2021).
  * Berkov [2007] D. V. Berkov, in  _Handbook of Magnetism and Advanced Magnetic Materials_ , edited by H. Kronmüller and S. Parkin (Wiley, Chichester, 2007) volume 2: Micromagnetism ed., pp. 795–823.
  * Baňas [2005] L. Baňas, in  _Numerical Analysis and Its Applications_ , edited by Z. Li, L. Vulkov, and J. Waśniewski (Springer, Berlin, 2005) pp. 158–165.
  * [48] https://www.mathworks.com/help/stats/sobolset.html [Accessed May 2022].
  * Glatter [1982] O. Glatter, in  _Small Angle X-ray Scattering_ , edited by O. Glatter and O. Kratky (Academic Press, London, 1982) pp. 167–196.
  * Svergun and Koch [2003] D. I. Svergun and M. H. J. Koch, Rep. Prog. Phys. 66, 1735 (2003).
  * Bender  _et al._ [2017] P. Bender, L. K. Bogart, O. Posth, W. Szczerba, S. E. Rogers, A. Castro, L. Nilsson, L. J. Zeng, A. Sugunan, J. Sommertune, A. Fornara, D. González-Alonso, L. Fernández Barquín, and C. Johansson, Sci. Rep. 7, 45990 (2017).
  * Usov and Peschany [1997] N. A. Usov and S. E. Peschany, J. Magn. Magn. Mater. 174, 247 (1997).
  * Garanin and Kachkachi [2003] D. A. Garanin and H. Kachkachi, Phys. Rev. Lett. 90, 065504 (2003).
  * Marshall and Lowde [1968] W. Marshall and R. D. Lowde, Rep. Prog. Phys. 31, 705 (1968).
  * Krill and Birringer [1998] C. E. Krill and R. Birringer, Philos. Mag. A 77, 621 (1998).