# Compressed Meta-Optical Encoder for Image Classification

Anna Wirth-Singh1,∗
&Jinlin Xiang2,∗
&Minho Choi2,∗
&Johannes E. Fröch1,2
&Luocheng Huang2
&Shane Colburn2
&Eli Shlizerman3,2
&Arka Majumdar1,2
&
1Department of Physics, University of Washington, Seattle, WA 98195, USA
2Department of Electrical and Computer Engineering, University of Washington,
Seattle, WA 98195, USA
3Department of Applied Mathematics, University of Washington, Seattle, WA
98195, USA

∗These authors contributed equally to this work

(April 2024)

###### Abstract

Optical and hybrid convolutional neural networks (CNNs) recently have become
of increasing interest to achieve low-latency, low-power image classification
and computer vision tasks. However, implementing optical nonlinearity is
challenging, and omitting the nonlinear layers in a standard CNN comes at a
significant reduction in accuracy. In this work, we use knowledge distillation
to compress modified AlexNet to a single linear convolutional layer and an
electronic backend (two fully connected layers). We obtain comparable
performance to a purely electronic CNN with five convolutional layers and
three fully connected layers. We implement the convolution optically via
engineering the point spread function of an inverse-designed meta-optic. Using
this hybrid approach, we estimate a reduction in multiply-accumulate
operations from 17M in a conventional electronic modified AlexNet to only 86K
in the hybrid compressed network enabled by the optical frontend. This
constitutes over two orders of magnitude reduction in latency and power
consumption. Furthermore, we experimentally demonstrate that the
classification accuracy of the system exceeds 93% on the MNIST dataset.

_Keywords_ neural networks, meta-optics, image classification, knowledge
distillation, optical computing

##  1 Introduction

Convolutional neural networks (CNNs) represent a significant milestone in
image classification, recognition, and tracking [1]. CNNs, for example,
AlexNet, are composed of several convolutional layers that adaptively learn
spatial representations from input images. While powerful, the convolution
operation is computationally expensive, leading to high latency and power
consumption. In fact, it has been estimated that about 80% of the total
runtime of CNNs is used in performing convolution operations [2]. Reducing
this latency and as a result power consumption has become an active area of
research, with multiple works proposing free-space optical systems as a
solution [3, 4, 5, 6]. Beyond latency reduction and power consumption, optical
information processing features qualities including high bandwidth, spatial
parallelism, and low-loss transmission which have led to a surge of interest
in the field [3].

For decades, it has been known that a $4f$ lens system can be used to perform
convolutions optically by placing an appropriate filter at the Fourier plane
of the lens [7, 8, 9, 4]. This was demonstrated in 2018 [4] using a
diffractive optical element as the filtering element and traditional
refractive lenses composing the $4f$ system. Spatial light modulators [6] and
digital micromirror devices [10] can also be used as the filtering element.
However, one drawback of the Fourier-based $4f$ approach is that it requires
three elements (two lenses and a spatial filter), resulting in a bulky optical
system with greater propensity for misalignments than single-element optical
systems. Such misalignments from each optical convolutional layer cannot be
ignored even when weights are trained with noisy inputs (more details in the
Supplementary material). In addition, the filtering optics must be contained
within a compact area at the focal plane in the $4f$ system, which limits
parallel processing ability unless creative measures are taken such as
utilizing naturally present diffraction orders [10] or lenslet arrays [5].

Advantageously, the convolution operation can also be performed using free-
space optics and requires only a single element. The resultant image produced
by any optics is the input convolved with the point spread function (PSF) of
the optics [11, 12, 13]. Therefore, by engineering optics to produce a
particular PSF, convolution can be performed optically simply via passing
light through the optics. Further, by passing the input through several of
these optics in parallel, multiple convolution operations can be performed
simultaneously at the speed of light [5, 14]. This approach leverages the
inherent parallelism of light enabling the passive processing of a vast amount
of data without increasing computation time [3, 10, 5]. This unique optical
capability circumvents scalability issues when handling high-resolution images
in traditional electronic-based CNN systems.

However, a challenge to all optically-implemented CNN approaches is that
nonlinear layers are interspersed with the linear layers. For example, AlexNet
consists of five convolutional layers followed by three fully connected layers
[15]. Specifically, each convolutional layer in the architecture utilizes the
rectified linear unit (ReLu) as its non-linear activation function, followed
by the local response normalization and Max Pooling layer, ensuring an
effective mechanism for spatial hierarchy extraction. Therefore, nonlinearity
is consistently applied across all layers, serving as a foundational element
of the network’s design to enhance its learning capability. Without the
particular nonlinear layers that are effective in CNNs (e.g., ReLu), the
classification accuracy of the CNN drops by about 20% [16]. The nonlinear
layers cannot be implemented using simple lens-like optics; to implement them
optically, some physical nonlinearity must be introduced, for instance by
using an atomic vapor cell [6, 17] or image intensifier [18]. Hybrid
approaches involving repeated transduction of the signal to perform linear
operations in optics and nonlinear operations in electronics provide little
benefit due to large latency and power consumption in signal transduction [5,
18, 9]. Implementing only one of many required convolution operations does not
provide much benefit in terms of speed and latency. Alternatively, there have
been recent breakthroughs in using end-to-end designs for physical and hybrid
networks designs which perform image classification or other tasks without
explicitly using convolution [19, 3, 20, 21]. Such an approach can effectively
implement multiple linear layers in one optical frontend. While novel, these
end-to-end neural networks are computationally expensive to train and are
applicable only to the physical system for which they were specifically
designed.

Figure 1: Schematic of convolutional neural networks for image classification
tasks. (a) All-electronic multi-layered CNN. (b) All-electronic compressed
CNN. (c) Hybrid CNN which combines an optical meta-optic front end and
electronic backend. (d) The number of multiply-accumulate (MAC) operations of
each network configuration, with convolutional MACs in green and fully-
connected (FC) MACs in brown.

In this work, we experimentally demonstrate a hybrid optical-electronic CNN
consisting of a single optical convolution layer with an electronic single
fully connected layer to achieve similar accuracy as AlexNet on hand-written
digit classification tasks. To overcome the limitation from the absence of
optical nonlinearity, we apply knowledge distillation to remove the nonlinear
layers, and compress multiple layers into a single linear layer [16].
Knowledge distillation (more details in Methods) circumvents the need for
nonlinearity without a significant reduction in the performance by
transferring knowledge from a larger, pre-trained network (the ‘teacher’
network) to a more compact network (the ‘student’ network). Here, we use a
modified AlexNet, denoted AlexNet-Mod, as the teacher network and a single
convolutional layer coupled with a single fully connected layer as the student
network. We use this architecture to demonstrate a hybrid meta-optical
platform, wherein an optical frontend based on a single meta-optic performs
the linear convolution operation, followed by an electronic backend which
contains a linear calibration layer and a fully connected layer. In such a
way, the most computationally expensive operation is performed optically to
leverage the benefits of optical computing, namely high spatial bandwidth and
low power consumption. The use of a single meta-optic layer drastically
simplifies the experimental setup and provides a compact geometry.

The optical frontend of this network is realized using inverse-designed meta-
optics. The meta-optics are arrays of sub-wavelength scatterers which act as
phase masks, imparting spatially-coded phase shifts to incident light. Here,
we design meta-optics to realize a phase mask which performs the desired
convolutional steps of the CNN by engineering the PSF. We fabricate and
experimentally validate the performance of the designed optics using
incoherent green light illumination from a light emitting diode, centered at
525 nm. Further, we experimentally demonstrate the classification accuracy of
the entire hybrid CNN on the MNIST dataset. The hybrid CNN is described in
Figure 1, where we compare the architectures of multi-layer electronic CNNs,
compressed electronic CNNs (a linear single layer CNN), and our hybrid system.
The number of multiply-accumulate (MAC) operations in the entire network is
reduced by over two orders of magnitude through compressing multiple
convolutional layers into a single layer and implementing them optically. The
classification accuracy of the compressed hybrid CNN is reduced by only 5%
from AlexNet-Mod (98% accuracy) to achieve 93% classification accuracy on the
MNIST dataset.

##  2 Results

In each convolutional layer of a CNN, an optimized kernel is convolved with
the input to generate a feature map which is then passed to the next layer.
Using the knowledge distillation approach, we optimize eight convolutional
kernels (each $6\times 6$ pixels in size) for the MNIST dataset of handwritten
digits. The selected number and size of the kernels are based on previous
experimental results (details shown in the Supplementary Information) [22]. As
described in the next section, we design the optics to implement these
optimized convolutional kernels and combine them with an electronic backend
for image classification.

###  2.1 Compressing Multiple Convolutional Layers using Knowledge
Distillation

To select the ideal network architecture for the linear optical-electronic
hybrid system, underfitting and overfitting issues must be avoided. Smaller
networks face underfitting concerns, particularly in optical settings that
limit the system to just 8 kernels and remove nonlinear functions [23]. This
is in stark contrast to AlexNet, which uses over 300 kernels [1]. However,
complex models are prone to overfitting, and practical issues such as
fabrication noise and misalignments introduce further challenges for optically
implementing a large number of kernels.

Therefore, to obtain a balanced network that could be implemented optically,
we use knowledge distillation to compress a AlexNet-Mod as a base model.
AlexNet-Mod consists of 5 convolutional layers and 3 fully connected layers (8
total layers) which we compress to the desired structure of one convolutional
layer and one fully connected layer (2 layers). The knowledge distillation
approach assumes that the teacher network is already trained and performs the
desired task with high accuracy; in this case, we use AlexNet-Mod as the
teacher network, which achieves 98.9%$\pm$ 0.33% on training and
98.4%$\pm$0.32% on testing classification accuracy MNIST datasets over
repeated trials. Additionally, AlexNet-Mod employs nonlinear activation
functions (ReLU) to optimize performance; these are circumvented by knowledge
distillation for a result that is compatible with our optical setting. In the
compressed network, we limit the number of kernels in the compressed
convolutional layer to 8, and each kernel is $6\times 6$ pixels in size. After
training, the compressed electronic network achieves an approximate
classification accuracy of 96% on both training and testing datasets.

Figure 2: Schematic of the optical system. (a) PSF measurement setup using a
monochromatic point light source (left) and optical convolution measurements
using a micro-LED display (right). (b) A photograph of the fabricated meta-
optics. The meta-optic contains 16 different sub-optics, spatially distributed
in a single layer, operating in parallel for classification tasks. (c) Phase
maps and SEM images of exemplary sub-optics corresponding to the positive and
negative parts of a particular convolutional kernel. (d) The positive and
negative parts of an example convolutional kernel (left) and the corresponding
PSF simulation (middle) and right (experiment). (e) The simulated electronic
output (left) and optical experiment (right) convolved output for the example
kernel, for the case of an input “7" from MNIST.

###  2.2 Optical Convolution using Meta-Optics

The optical component of this network is implemented using inverse-designed
meta-optics. We used the Gerchberg-Saxton (GS) phase retrieval method [24] to
design phase masks that correspond to the optimized convolutional kernels.
Specifically, each sub-optic is designed to have a PSF which resembles the
convolutional kernel. Since electronic convolutional kernels include both
positive and negative values, we separated each kernel into positive and
negative parts and designed meta-optics for each. The positive and negative
images were computationally subtracted afterward to produce the net
convolution. Two-dimensional phase maps of an example set of sub-optics are
shown in Fig. 2(b). The phase maps were implemented by silicon nitride pillars
which are 750 nm tall but varied widths corresponding to their relative phase
delay. Scanning electron microscope (SEM) images of the fabricated optics are
also shown for comparison, highlighting the fabrication quality. All sixteen
optics (corresponding to 8 convolutional kernels) were fabricated on a single
substrate (more details in Materials and Methods), allowing convolution from
16 different kernels in a single image capture. A photograph of the fabricated
meta-optic is shown in Fig. 2(b). Each kernel meta-optic is of size 470 $\mu$m
$\times$ 470 $\mu$m and center-to-center distance of 705 $\mu$m. The optics
are arranged in two rows of eight optics for a total footprint of 5.6 mm
$\times$ 1.4 mm.

To verify the performance of the optics, we measured the PSF of each sub-optic
using a single-mode fiber as a point light source, as shown in the Fig. 2(a).
The PSFs from all 16 sub-optics are simultaneously captured by the two-
dimensional CMOS camera (more details in Materials and Methods). Accordingly,
the convolved images from all 16 sub-optics are also captured at the same time
when we replaced the single mode fiber with the display, as described in Fig.
2(a). Exemplary electronic convolutional kernels, which represent the ground
truth PSFs for the optically-implemented kernels, are shown in Fig. 2(d). We
also present the simulated PSFs from the meta-optics using angular spectrum
propagation [25, 26]. The experimentally measured PSF shown in Fig. 2(d) match
well to the simulated PSFs, confirming fabrication accuracy. However, due to
the constraints on physically realizable PSFs, there are notable differences
between the ground truth PSFs and experimentally measured PSFs. To correct for
these differences, as well as slight noise and misalignments in the optical
system, we introduce a calibration layer to the computational backend, further
discussed in Section 4.2.

###  2.3 Hybrid Network Classification Results

To address optical noise and misalignment affecting image classification
performance, an additional calibration layer is introduced to adjust optical
representations for compatibility with the computational backend.
Specifically, this calibration layer was fine-tuned with only 10% of the
training dataset and ensures that the computational backend does not need to
be retrained. Therefore, the electronic backend of the hybrid network consists
of the calibration layer followed by the original compressed electronic
backend.

We compare three CNN architectures (AlexNet-Mod, compressed electronic
network, and hybrid optical-electronic network) in Table 1. The AlexNet-Mod
achieves classification accuracy exceeding 98% on both training and testing
datasets. The number of MAC operations of this network is 17 million with 8
bit precision (details are shown in the Supplementary Information, Section
S1). The compressed electronic CNN achieves greater than 96% accuracy; this 2%
decline reflects the inherent challenges of compressing multiple layers into a
single layer. Primarily due to the compression of the convolution layers, the
number of MAC operations is reduced to 228,672. The hybrid network, which
integrates the optical convolution layer with the calibration layer and single
fully connected layer electronic backend, experimentally achieves
classification accuracy of 93.9% (± 0.25%) and 93.4% (± 0.22%) on the training
and testing datasets, respectively, and requires only 85,824 MAC operations,
which is 0.5% and 37% of that required for AlexNet-Mod and the compressed
electronic networks, respectively.

Table 1: Classification Results Network Architecture | Train (%) | Test (%) | MAC Operations
---|---|---|---
AlexNet-Mod | 98.9 ± 0.33 | 98.4 ± 0.32 | 17,268,224
Compressed electronic CNN (without KD) | 84.2 ± 0.47 | 82.1 ± 0.69 | 228,672
Compressed electronic CNN (KD) | 97.2 ± 0.35 | 96.2 ± 0.29 | 228,672
Hybrid CNN (KD) | 93.9 ± 0.25 | 93.4 ± 0.22 | 85,824

Figure 3 illustrates the confusion matrices for three neural network
configurations. Each matrix visually represents the model’s tested performance
across different classes (in this case, digits labeled 0 through 9), with the
true labels on the rows and the predicted labels on the columns. The multi-
layer electronic CNN, AlexNet-Mod, displays high values along the diagonal,
exceeding 98.1% accuracy on each class. The compressed electronic CNN, while
having a slight decline in diagonal values, still demonstrates robust
classification accuracy with a minimum of 94.3%. The hybrid network exhibits a
more diverse range of values along the diagonal, with some classes exhibiting
lower predictive accuracy compared to the compressed electronic network. We
attribute this slight decline to noise in the optical experiment, which may be
due to optics fabrication, camera sensor noise, and optical noise due to
vibrations that cannot be fully compensated by the calibration layer. Despite
these factors, the hybrid network still performs reasonably well with the
network correctly predicting each class with a minimum of 87.6% accuracy. This
indicates that the hybrid network maintains a reasonable level of accuracy
against these noises and discrepancies.

Figure 3: Confusion matrices for different network architectures. (a)
Classification results for AlexNet-Mod (multiple-layer electronic CNN). (b)
Classification results for the all-electronic CNN compressed without using
knowledge distillation. (c) Classification results for the all-electronic CNN
compressed with knowledge distillation. (d) Classification results for the
hybrid optical-electronic CNN.

##  3 Discussion

###  3.1 Ablation Study and Principal Component Analysis

To understand the contribution of each component of the hybrid optoelectronic
CNN, we perform an ablation study and principal component analysis. In the
ablation study, we evaluate the classification accuracy when using only the
electronic backend structure for classification. We summarize the ablation
study results in Table 2. For a fair comparison, the backend layer here is the
same structure as used in the hybrid network but re-optimized for the best
performance in the absence of any convolutional frontend. Specifically, we
compare the performance of a backend layer only (a single fully-connected
layer), the calibration and backend layers together (two fully-connected
layers), and the entire hybrid network. For a single fully-connected layer
alone, an accuracy of 89% was attained, which is less than that of the hybrid
network; this highlights the utility of the optical frontend. Further, for two
fully-connected layers (representing the calibration and backend layer)
without any convolutional frontend, the accuracy is reduced to 84%. This can
be attributed to the fact that two layers are fully-connected and, in the
absence of nonlinear activation functions, tend to converge towards a ‘saddle
point’ in the optimization landscape [27].

Table 2: Network Ablation Study Configuration | Train accuracy | Test accuracy
---|---|---
Backend Only | 89% | 87%
Calibration + Backend | 84% | 80%
Optics + Calibration + Backend | 94% (+5% / +10%) | 93% (+6% / +13%)

We hypothesize that the calibration layer only re-maps the optical
representations and does not improve the performance of the electronic backend
alone. To examine this, we further analyze the effect of the calibration layer
and the overall performance of the hybrid network as compared to an all-
electronic network using Principal Component Analysis (PCA). PCA is a
statistical method widely used in various fields, especially in analyzing the
quality of neural networks [28], to project original, high-dimensional data
into a new, simpler coordinate system for explicit interpretation.
Specifically, PCA computes the eigenvalue decomposition of the covariance
matrix or the singular value decomposition of input data to determine the
principal direction (known as “principal components"). Principal component 1
denotes the axis of maximum variance, encapsulating the most substantial
relationships among variables, while principal component 2, orthogonal to
principal component 1, captures the second most significant variance
direction. Notably, the first two principal components typically contain the
most crucial information. In this study, we compress the output dimensions
from electronic and optical convolutional layers into two principal
components, respectively, to compare the classification efficacy of each
approach; this is observed by comparing the clusters observed in PCA
visualizations [29].

In Fig. 4, we use PCA to show that the raw experimental data do identify the
fundamental components necessary for classification, but that the calibration
function is necessary to shift optical representations to a form that is
compatible with the pre-designed electronic backend. As shown in Fig. 4a and
4c, we observe that both the all-electronic CNN outputs and the uncalibrated
hybrid network experimental results can effectively distinguish between
different classes due to the clustering behavior of specific classes, e.g.
light blue and navy blue. This clustering behavior indicates that despite any
observable shifts in the PCA plot, the fundamental capacity of the hybrid
network to classify data remains comparable to that of all-electronic
networks. However, due to differences between the optical experiment output
and the expected input to the electronic backend, directly using the original
backend network results in a notable drop in accuracy, down to 16.3%. The
calibration layer is designed to re-calibrate the outputs from the optical
convolution layer back to the original outputs, thereby enabling the use of
the original backend without retraining. As shown in Fig. 4b and 4c, the
calibrated experiment result exhibits very similar clustering behavior to the
all-electronic network, further demonstrating that the hybrid network
classification is comparable to that of the all-electronic network.

Figure 4: PCA of the hybrid CNN. (a) PCA of the uncalibrated experimental
hybrid CNN classification data. (b) PCA of the calibrated experimental data,
which has been re-mapped and exhibits clustering behavior similar to that of
the compressed electronic CNN data. (c) PCA of the compressed electronic CNN
data.

###  3.2 PSF-Engineered Meta-Optics

We emphasize two advantages of our PSF-engineering method to perform optical
convolution. Firstly, we highlight the simplicity of the optical system, as
this method requires only a display, a single layer of optic, and a camera,
making it compact and simple to execute. Incoherent illumination is used, so
this approach can be applied to real-world image classification scenarios.
Secondly, we highlight the ease of integration with the optimized electronic
system. One of the challenges faced by optical computing is that electronic
computing is already extremely powerful, having had decades of research and
development into algorithms and hardware [3]. In our approach, the optics are
designed to implement the electronically-optimized kernels. These kernels may
be modified to reflect improvements in electronic CNN models and
architectures, and the optics can accordingly be adapted to implement
convolutions with arbitrary kernel matrices.

Furthermore, the Gerchberg-Saxon (GS) algorithm [24] used to design the optics
is a well-established technique. The GS algorithm is an iterative phase
retrieval algorithm to determine the phase (in the optic plane) that produces
an intensity pattern in another desired plane (the focal plane). In other
words, the meta-optics are phase-only holograms producing the desired PSFs as
their images. Due to the fact that only amplitude, and not phase, contributes
to the intensity pattern, the iteratively designed phase masks are not unique.
More details on the implementation of the GS algorithm are available in the
Supplement. In the Supplement, we also discuss an alternative design method
based on automatic differentiation, which also produces viable optics but we
found the GS-designed optics to produce slightly brighter, clearer images.

There are, however, two major limitations of the PSF-engineering approach. One
limitation is that there is no guarantee that the desired PSF is physically
realizable. That is, a single phase mask that satisfies the desired amplitude
constraints may not exist. However, by introducing an electronic calibration
layer, the resultant PSF does not need to be perfect in order to effectively
classify the data. Alternatively, to ensure physically realizable PSFs, one
could adopt an end-to-end optimization scheme wherein the phase mask is
simultaneously optimized with a backend. However, training this large phase
mask (on the order of $10^{5}-10^{6}$ unit cells per kernel optic) is
prohibitively costly and the design space is potentially too large to attain
convergence. In contrast, separately training the electronic convolutional
kernels and the optics are both reasonable steps, which as demonstrated are
effective when combined.

A second limitation of the described approach is that we assume the PSF is
spatially invariant. In reality, light from different spatial locations on the
imaging object intersects the meta-optic at various angles of incidence,
resulting in a different PSF for the off-axis rays. In contrast, we design the
optics assuming a normal incident illumination. To mitigate this discrepancy,
we ensure the incoming angles of incidence are relatively small by placing the
display far away from the optics (90 mm) relative to the focal length (2.4
mm). Therefore, for a displayed image size of 8 mm $\times$ 8 mm, we ensure
that the maximum deviation from normal incidence is $3.6^{\circ}$, and
therefore the assumption of spatial invariance is reasonable for our system.
However, for a large field of view imaging system, we may need to explicitly
model the spatially varying PSF. We note that Wei et al. [14] use
reparameterization techniques to design spatially varying kernel optics and
report higher classification accuracy using this method (73.8%) versus
designing optics with the assumption of spatial invariance (71.6 %) on the
CIFAR-10 dataset. In another variation of a PSF-engineering technique, Zheng
et al. [30] engineer the PSF of polarization-sensitive meta-optics to provide
an array of focal spots which produce images of intensities relative to the
kernel weights.

###  3.3 Outlook - Computational Effectiveness of the Hybrid Convolutional
Neural Network

The number of MAC operations required of a network serves as a metric of
computational complexity which is independent of the employed hardware
technology. In a modern digital system, one MAC operation consumes
approximately $1pJ$ [21, 18], so the hybrid network is expected to reduce the
required power to classify an input from $17\mu J$ to $85nJ$ based on the
reduction in MAC operations. The latency of such a classification task is also
expected to decrease proportionately. For input which is already in the
optical domain, the power and latency required to capture an image and convert
it to digital input is the same regardless of the network. Specifically, the
hand-written digits of MNIST were captured using a standard camera, and the
optical frontend of our network uses a standard camera sensor with meta-optics
replacing the refractive camera lens.

The benefit of optically implementing the convolutional step becomes more
significant as the number of input pixels is increased. The electronic
computational complexity of each convolutional layer is determined by the
height and width of the input images ($H,W$), as well as the size of the
kernels ($k$), and is $\mathcal{O}(HWk^{2})$ [16]. However, the computational
complexity decreases to $\mathcal{O}(1)$ in the optical convolutional layer.
For example, when the MNIST dataset’s typical image size of 28$\times$28
pixels is increased to 100$\times$100 pixels, the electronic computational
complexity of each convolutional layer is expected to increase $12.76$ times
(assuming the kernel size remains unchanged), but an optically implemented
convolution would not incur any increase in computation time. Therefore, as
the resolution of real-world images continues to increase, hybrid networks
such as the one described offer a promising solution to scaling problems
incurred by all-electronic networks.

In summary, we demonstrate single-layer optical convolution with an electronic
backend to achieve similar accuracy as AlexNet-Mod on MNIST hand-written digit
classification, with 99.5% reduction in computational complexity. To
circumvent the nonlinearity of AlexNet-Mod, we use knowledge distillation to
compress the CNN into linear layers which are then implemented in a hybrid
format. As a further innovation, we implement the convolution optically via
engineering the PSF of meta-optics, which results in a more compact and
resilient optical frontend than the commonly used $4f$ lens system and does
not require coherent illumination or polarization control. This hybrid
approach integrates seamlessly with existing CNN architectures, utilizing
simple optical design and requiring no re-training of the electronic backend
to classify the data in experiment. This approach is also suitable for scaling
to higher-resolution datasets; unlike in all-electronic networks where the
convolution time scales with the number of input pixels, for optical
convolution the processing time is independent of the resolution of the
dataset. This work serves as a baseline for other optical and hybrid neural
networks for higher bandwidth as well as lower power and latency in
increasingly prevalent CNN applications.

##  4 Materials and Methods

###  4.1 Transfer knowledge to linear networks

The knowledge distillation (KD) algorithm is designed to compress neural
networks. KD accomplishes this by transferring knowledge from a larger, pre-
trained network (referred to as the ‘teacher model’) to a more compact network
(referred to as the ‘student model’). In our implementation, we use AlexNet-
Mod as the teacher network and a linear electronic network as the student
network. The student network only comprises a single CNN coupled with a single
fully connected layer. Straightforwardly training this linear network tends to
easily converge to suboptimal saddle points; however, with the knowledge
distillation approach, the student network converges faster and obtains better
performance than it would achieve without the knowledge distillation training.

The KD algorithm optimizes the linear student network by combining two types
of losses: temperature loss and student loss. Similar to other conventional
losses in image classification, KD uses the training labels as “hard labels”
and computes the probabilities distribution vector $p^{hl}$, where each
element $p_{i}^{hl}$ in the vector corresponds to the probability of the
current input belonging to the class $i$. The softmax function is used to
compute probabilities

$p_{i}^{hl}=exp(z_{i})/\sum{exp(z_{i})}$ (1)

where $z_{i}$ is the student logits after the last fully connected layer.

Then, KD treats the prediction of the teacher model as ‘soft labels’ that
inform the student model since training with hard labels only is a sensitive
process, especially for compact linear networks. KD includes a softening
parameter, $T$, named as the distillation temperature for teacher
probabilities. Therefore, for each input, at the same time of computing
$p^{hl}$ with the student model, KD computes the soft probabilities vector,
$p^{sl}$, with the teacher model according to

$p_{i}^{sl}=exp(y_{i}/T)/\sum{exp(y_{i}/T)}$ (2)

Therefore, the total loss is then calculated as a weighted summation of the
two losses

$\mathcal{L}(x,\Phi)=\alpha\mathcal{L}_{C}(y,p^{hl})+(1-\alpha)\mathcal{L}_{k}((p^{sl,t};T=\tau),(p^{sl,s};T=\tau))$
(3)

where $x$ corresponds to the input, $y$ is the training data, $\Phi$ are the
student model weights, $\mathcal{L}_{C}$ is the cross-entropy loss function,
$\mathcal{L}_{k}$ is the Kullback-Leibler (KL) Divergence Loss function [31],
$p^{hl}$ corresponds to the student model hard predictions, $p^{sl,s}$
corresponds to the student predictions under given teacher model probabilities
$p^{sl,t}$, and $\alpha$ is the weighting parameter.

###  4.2 Calibrating Optical Experiment Results with Limited Data

The calibration function is designed to remap the optical convolution outputs
to align with those of the previously trained backend. This addition addresses
a variety of differences which may occur between the optical and electronic
counterparts, including scaling, translation, rotation, and optical noise.
With the addition of the calibration layer, the weights of the fully connected
layer are preserved, and the optical frontend can be integrated into the
existing network framework without retraining the backend or fine-tuning the
optical alignment.

Specifically, the calibration layer is a fully connected layer. The loss
function used in this process is defined:

$\mathcal{L}=min(f_{calibrate}(\textit{ON}),\textit{EN})$ (4)

where $ON$ is the network with the optical experiment results and $EN$ is the
all-electronic network. This approach aims to refine the experiment output to
align more closely with the pre-designed electronic network. To prevent
overfitting, we strategically limit our training to only 10% of the available
data, ensuring that our model remains efficient [32, 33]. The calibration
layer addresses diverse types of noise encountered in the optical system, thus
ensuring a more robust hybrid network.

###  4.3 Meta-optics Design

The meta-optics are designed with our experiment setup in mind. Due to the
sensitivity of our camera (GT-1930C) and available light sources, we design
the optics specifically for 525 nm illumination. For all electromagnetic
simulations, we use a simulation grid size of 586 nm to be both comparable to
the wavelength of the light and evenly divisible by the size of the camera
pixels (5.86 $\mu$m per pixel). Each sub-optic is square, 800x800 simulation
pixels in size, which provide compact footprint and reasonable computation
time. While it is not necessary to propagate the electric fields on a sub-
wavelength grid, it is necessary to design the meta-optic scatterers with sub-
wavelength periodicity. Therefore, we divide each meta-optic pixel into a 2x2
block of square meta-optic scatterers each with a period of 293 nm. With 750
nm SiN (n = 2.06) pillars, we select a set which provides 0 to 2$\pi$ phase
shift at the desired wavelength. The scatterer unit cells were simulated using
S4 RCWA [34]. More details on the meta-optic design are available in the
Supplement.

###  4.4 Meta-optics Fabrication

The convolutional meta-optics are fabricated on a silicon nitride layer on a
quartz substrate. We first deposited silicon nitride on a double-side polished
quartz wafer using plasma-enhanced chemical vapor deposition (Oxford; Plasma
Lab 100). Then we patterned on a positive-tone resist (ZEP-520A) using e-beam
lithography (JEOL; JBX6300FS). We used alumina as a hard mask for etching the
silicon nitride layer, so we deposited the alumina using e-beam evaporator
(CHA; SEC-600) and did liftoff with 1-methyl-2-pyrrolidinone. We etched the
silicon nitride layer with a plasma etcher (Oxford; PlasmaLab 100, ICP-180)
using fluorine-based gases. In order to minimize the stray light of the meta-
optics, we blocked the light except for the 16 kernel sub-optics by putting
apertures around each one using photolithography (Heidelberg; DWL66+) and
metal deposition followed by a liftoff process.

###  4.5 Optical Measurements

Two-dimensional point spread functions of 16 different optical kernels are
measured simultaneously with a simple measurement setup. Single-mode optical
fiber-coupled light source acts as a point source, and the meta-optics are
placed 92 mm apart from the source. As we put the meta-optics on a three-axis
linear stage and a kinetic mount with two rotation adjusting knobs, both the
position and angle of the meta-optics can be well-defined with respect to the
designed setup. Then we put a high-resolution color camera (GT-1930C with 5.86
$\mu$m per pixel resolution) 2.4 mm from the meta-optics to collect the point-
spread functions of the kernels. We simply replaced the light source from the
single-mode optical fiber to a micro-display presenting MNIST dataset of
handwritten digits to get the convolved images. The camera could capture all
16 convolved images from different kernels at the same time, and we used
Python code to automatically collect convolved images from 10,000 number of
MNIST dataset.

## Supplementary Information

The online version contains supplementary materials.

## Code Availability

The code and data generated and/or analyzed are available from the
corresponding author upon reasonable request.

## Acknowledgements

The research is supported by National Science Foundation (NSF-ECCS-2127235 and
EFRI-BRAID-2223495). Part of this work was conducted at the Washington
Nanofabrication Facility/ Molecular Analysis Facility, a National
Nanotechnology Coordinated Infrastructure (NNCI) site at the University of
Washington with partial support from the National Science Foundation via
awards NNCI-1542101 and NNCI-2025489.

## Author Contributions

A.W.S., J.X., and M.C. contributed equally to this work. E.S. and A.M.
conceived of the idea and provided funding. S.C. was involved in designing
preliminary convolutional meta-optics and knowledge-distillation approaches.
J.X. and E.S. developed the knowledge distillation approach. J.X. trained the
neural network and analyzed the experiment data. A.W.S. developed the PSF-
engineering approach and designed the meta-optics. M.C. fabricated the meta-
optics and conducted the optical experiments. L.H. wrote the script to
automate the experiment. J.F. advised on the fabrication and experiment. J.F.,
E.S., and A.M. provided project management and edited the manuscript.

## Competing Interests

A.M. is a co-founder of Tunoptix, which aims to commercialize meta-optics
technology.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “Imagenet classification with deep convolutional neural networks”  In _Advances in neural information processing systems_ 25, 2012
  * [2] X. Li et al.  “Performance analysis of gpu-based convolutional neural networks.”  In _45th ICPP_ , 2016, pp. 67–76
  * [3] P.. McMahon  “The Physics of Optical Computing.”  In _Nat. Rev. Physics_ 5, 2023, pp. 717–734
  * [4] J. Chang et al.  “Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.”  In _Scientific Reports_ 8, 2018, pp. 12324
  * [5] S. Colburn, Y. Chu, A. Majumdar and E. Shlizerman  “Optical frontend for a convolutional neural network.”  In _Applied Optics_ 58.12, 2019, pp. 3179–3186
  * [6] M. Yang et al.  “Optical convolutional neural network with atomic nonlinearity.”  In _Optics Express_ 31.10, 2023, pp. 16451–16459
  * [7] L. Cutrona, E. Leith, C. Palermo and L. Porcello  “Optical data processing and filtering systems.”  In _IRE Trans. Inf. Theory_ 6.3, 1960, pp. 386–400
  * [8] J.. Mait, G.. Euliss and R.. Athale  “Computational imaging.”  In _Adv. Opt. Photon._ 10, 2018, pp. 409–483
  * [9] Carlos Mauricio Villegas Burgos, Tianqi Yang, Yuhao Zhu and A. Vamivakas  “Design framework for metasurface optics-based convolutional neural networks”  In _Appl. Opt._ 60.15 Optica Publishing Group, 2021, pp. 4356–4365
  * [10] Z. Hu et al.  “High-Throughput Miltichannel Parallelized Diffraction Convolutional Neural Network Accelerator.”  In _Laser Photonics Rev._ 16, 2022, pp. 2200213
  * [11] W. Lohmann and W.. Rhodes  “Two-pupil synthesis of optical transfer functions”  In _Appl. Opt._ 17.7, 1978, pp. 1141–1151
  * [12] J.. Mait and W.. Rhodes  “Two-pupil synthesis of optical transfer functions: 2: Pupil function relationships”  In _Appl. Opt._ 25.12, 1986, pp. 2003–2007
  * [13] J.. Mait  “Pupil-function design for complex incoherent spatial filtering”  In _JOSA_ 4.7, 1987, pp. 1185–1193
  * [14] K. Wei et al.  “Spatially Varying Nanophotonic Neural Networks.”  In _Arxiv_ , 2023, pp. 2308.03407
  * [15] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “ImageNet Classification with Deep Convolutional Neural Networks (AlexNet) ImageNet Classification with Deep Convolutional Neural Networks (AlexNet) ImageNet Classification with Deep Convolutional Neural Networks”
  * [16] Jinlin Xiang, Shane Colburn, Arka Majumdar and Eli Shlizerman  “Knowledge distillation circumvents nonlinearity for optical convolutional neural networks”  In _Applied Optics_ 61.9 Optica Publishing Group, 2022, pp. 2173–2183
  * [17] A. Ryou et al.  “Free-space Optical Neural Network Based on Thermal Atomic Nonlinearity.”  In _Photon. Res._ 9, 2021, pp. B128–B134
  * [18] T. Wang et al.  “Image sensing with multilayer nonlinear optical neural networks.”  In _Nature Photonics._ 17, 2023, pp. 408–415
  * [19] H. Zheng et al.  “Meta-optic accelerators for object classifiers.”  In _Science Advances_ 8.30, 2022
  * [20] T. Zhou et al.  “Large-scale Neuromorphic Optoelectronic Computing with a Reconfigurable Diffractive Processing Unit.”  In _Nature Photonics_ 15, 2021, pp. 367–373
  * [21] L. Huang et al.  “Photonic advantage of optical encoders.”  In _Nanophotonics._ , 2023
  * [22] Şaban Öztürk, Umut Özkaya, Bayram Akdemir and Levent Seyfi  “Convolution kernel size effect on convolutional neural network in histopathological image processing applications”  In _2018 International Symposium on Fundamentals of Electrical Engineering (ISFEE)_ , 2018, pp. 1–5  IEEE
  * [23] Han Cai, Chuang Gan, Ji Lin and Song Han  “Network augmentation for tiny deep learning”  In _arXiv preprint arXiv:2110.08890_ , 2021
  * [24] R.. Gerchberg  “A Practical Algorithm for the Determination of Phase from Image and Diffraction Plane Pictures.”  In _Optik_ 35, 1972, pp. 237–246
  * [25] J.. Goodman  “Introduction to Fourier Optics”  Englewood, Colorado: Robers & Company Publishers, 2005
  * [26] K. Matsushima and T. Shimobaba  “Band-Limited Angular Spectrum Method for Numerical Simulation of Free-Space Propagation in Far and Near Fields”, 2009, pp. 19662–19673
  * [27] Arthur Jacot et al.  “Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity”  In _arXiv preprint arXiv:2106.15933_ , 2021
  * [28] Ji Ma and Yuyu Yuan  “Dimension reduction of image deep feature using PCA”  In _Journal of Visual Communication and Image Representation_ 63 Elsevier, 2019, pp. 102578
  * [29] Gordana Ivosev, Lyle Burton and Ron Bonner  “Dimensionality reduction and visualization in principal component analysis”  In _Analytical chemistry_ 80.13 ACS Publications, 2008, pp. 4933–4944
  * [30] H. Zheng et al.  “Multichannel meta-imagers for accelerating machine vision.”  In _Nature Nanotechnology._ 59, 2024
  * [31] Geonseok Seo, Jaeyoung Yoo, Jaeseok Cho and Nojun Kwak  “Kl-divergence-based region proposal network for object detection”  In _2020 IEEE International Conference on Image Processing (ICIP)_ , 2020, pp. 2001–2005  IEEE
  * [32] Jinlin Xiang and Eli Shlizerman  “TKIL: Tangent Kernel Optimization for Class Balanced Incremental Learning”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3529–3539
  * [33] Yang Zheng, Jinlin Xiang, Kun Su and Eli Shlizerman  “Bi-maml: Balanced incremental approach for meta learning”  In _arXiv preprint arXiv:2006.07412_ , 2020
  * [34] V. Liu and S. Fan  “S4: A free electromagnetic solver for layered periodic structures.”  In _Computer Physics Communications._ 183, 2012, pp. 2233–2244


# Federated Named Entity Recognition

Joel Mathew, Dimitris Stripelis  José Luis Ambite
USC Information Sciences Institute

###### Abstract

We present an analysis of the performance of Federated Learning in a
paradigmatic natural-language processing task: Named-Entity Recognition (NER).
For our evaluation, we use the language-independent CoNLL-2003 dataset as our
benchmark dataset and a Bi-LSTM-CRF model as our benchmark NER model. We show
that federated training reaches almost the same performance as the centralized
model, though with some performance degradation as the learning environments
become more heterogeneous. We also show the convergence rate of federated
models for NER. Finally, we discuss existing challenges of Federated Learning
for NLP applications that can foster future research directions.

Federated Named Entity Recognition

Joel Mathew, Dimitris Stripelis and José Luis Ambite USC Information Sciences
Institute

##  1 Introduction

To protect personal and proprietary data from illegal access and malicious
use, new data regulatory frameworks (e.g., GDPR, CCPA, PIPL) have been
recently enacted that govern data access, processing and analysis. When also
taking into account the increasing amount of data being generated across
various domains, new tools are required to perform meaningful large scale data
analysis while at the same complying to the newly introduced data privacy and
security legislations.

One potential solution towards secure and private data analysis is Federated
Learning McMahan et al. (2017); Kairouz et al. (2021); Yang et al. (2019).
Federated Learning has emerged as a standard computational paradigm for
distributed machine learning that allows geographically distributed data
sources to efficiently train machine learning models while their raw data
remain at their original location. Compared to traditional machine learning
algorithms that require all training data to be aggregated at a centralized
location, Federated Learning relaxes the need of raw data migration and
instead pushes the training of the machine learning model down to each source.
During federated training only the locally trained model parameters are shared
with the outside world and a centralized entity is responsible to aggregate
these parameters to compute the federation model.

Due to its inherent data security and privacy benefits, Federated Learning has
seen wide adoption across many different disciplines and applications ranging
from smart transportation Liu et al. (2020a), finance Liu et al. (2020b) and
healthcare Sheller et al. (2020); Rieke et al. (2020). In this work, we study
Federated Learning in a more generalized setting within the context of Natural
Language Processing (NLP) with a focus on the Named Entity Recognition (NER)
task.

NER is an important tool for textual analysis that has been applied across
various domains, including information extraction, question-answering, and
semantic annotation Marrero et al. (2013). However, some domains require the
textual data to remain private; for example, intelligence applications or
analyses of email data across different organizations. These private-text
domains motivate our research into federated learning approaches for NLP
problems, such as Named Entity Recognition.

##  2 Related Work

Federated Learning was originally introduced in the seminal work of McMahan et
al. (2017) for user data in mobile phones. In their work, the authors proposed
the first synchronous federated learning algorithm, termed Federated Average
(FedAvg), where a set of clients is randomly selected at every federation
round to train the global model on their local dataset under the coordination
of a central server who is responsible to aggregate the local models into the
global model and distribute the respective local training workload to every
client. Following this learning approach, recently many other federated
training approaches Kairouz et al. (2021); Li et al. (2020); Yang et al.
(2019) have been proposed both for mobile devices (cross-device) and large-
scale organizations (cross-silo).

Other approaches have provided convergence guarantees for the original FedAvg
algorithm over non-IID data Li et al. (2019) while others have decoupled the
federated optimization problem into global and local optimization Wang et al.
(2021); Reddi et al. (2020) with linear Mitra et al. (2021); Karimireddy et
al. (2020) and sub-linear convergence guarantees Li et al. (2020); Wang et al.
(2020) in the presence of different client learning constraints, such as
system (different computational capabilities) and statistical heterogeneities
(different local dataset distributions).

Even though many federated training approaches have been proposed most of them
focus in the context of computer vision and only a handful of them provide a
federated solution tailored for NLP applications Liu et al. (2021); Lin et al.
(2021). For instance, federated learning has been applied to tackle the
problem of next word keyboard prediction Hard et al. (2018); Stremmel and
Singh (2021); Yang et al. (2018), speech recognition Leroy et al. (2019) and
health text mining Liu et al. (2019). In the context of sequence tagging and
named entity recognition recent works have investigated the feasibility of a
federated learning solution on medical named entity extraction Ge et al.
(2020) and extraction of personally identifiable information elements from
text documents Hathurusinghe et al. (2021).

For the NER task that we investigate in this work, one of the first successful
uses of neural networks was the Bi-LSTM Hochreiter and Schmidhuber (1997) and
CRF Lafferty et al. (2001) models, with both models combined into a single
state-of-the-art deep learning model architecture Lample et al. (2016) that
exhibited superior performance. Other recent works have proposed large
transformer-based models such as BERT Devlin et al. (2019) and XLM Lample and
Conneau (2019). However, for understanding the impact of federated training in
the learning performance of the NER task, in this work we employ the Bi-LSTM-
CRF model, similar to Mathew et al. (2019).

##  3 NER Model

We use a Bi-LSTM layer which is fed a concatenation of 300-dimension GloVe
Pennington et al. (2014) word embedding and a character embedding that is
trained on the data as input. We use dropout during training, set at 0.5. The
output of the Bi-LSTM model is then fed into a CRF layer in order to capture
neighboring tagging decisions. The CRF layer produces scores for all possible
sequences of tags over which we apply the Softmax function to produce the
output tag sequence. Figure 1 shows the architecture of the deep learning
model used throughout training.111https://github.com/guillaumegenthial/tf_ner
The total number of trainable parameters are 322,690.

Figure 1: NER Model Architecture.

##  4 Experiments

For all learning environments (centralized and federated) the random seed was
set to 1990. We use Vanilla SGD as the solver of the centralized model and the
local model in the federation with a learning rate of 0.01 and batch size of
20.

Federated Training. Our federated environments follow a centralized learning
topology Bellavista et al. (2021); Yang et al. (2019); Kairouz et al. (2021)
where a single aggregator (server) is responsible to orchestrate the execution
of the participating clients. In our experiments, we consider full client
participation at every round. We test the performance of the federated model
on federation environments consisting of 8, 16 and 32 clients. Each client
trained on its local dataset for 4 local epochs in-between federation rounds
and each federated experiment was run for a total number of 200 rounds. When
merging the local models at the server, we used FedAvg as the merging
function. During training clients shared the kernel and bias matrices of the
LSTM and dense layers and the transition matrix of the CRF layer. All
federated environments were run using the Metis framework Stripelis and Ambite
(2021).

Federated Data Distributions. The CoNLL-2003 Sang and De Meulder (2003) is a
language-independent newswire dataset developed for the named entity
recognition task. The dataset consists of 20,744 sentences (14041 training,
3250 validation, 3453 test) and contains entities referring to locations
(LOC), organizations (ORG), people (PER), and miscellaneous (MISC). 222Total
number of entities in the training and validation sets combined, LOC: 8977,
ORG: 7662, PER: 8442, MISC: 4360. For test, LOC: 1668, ORG: 1661, PER: 1617,
MISC: 702 We measure that classification performance for all entities. The
original tagging scheme is BIO (beginning-intermediate-other). When extracting
the named entities two subtasks need to be solved, finding the exact
boundaries of an entity, and the entity type. The metrics used to evaluate
correct tag (entity) predictions are Precision, Recall and F1. All models were
evaluated on the same original test dataset.

To generate the federated environments that we investigate in this work, we
split the dataset into equal (Uniform) and unequal (Skewed) sized partitions.
For the Uniform enviroments, we combine the training and validation datasets
of the original dataset and split them into approximately equally sized
partitions for 8, 16 and 32-clients, such that each partition (client) has
almost the same proportion of different tag types. The proportion of tags in
each split is approximate, as it can also be seen in Figure 2, since any
sentence can have any number of different tags. For the 32-clients and Skewed
environment, we randomly partitioned the combined training and validation
datasets into 32 splits over an increasing amount of data points (200 to 872
sentences). This increased the data heterogeneity across all partitions with
certain partitions containing more unique entity mentions (compared to the
Uniform environments); see also Figure 3(d).

Figure 2 presents the total number of location, organization and person tags
at each client within each federation environment.333The Miscellaneous (MISC)
tag has a similar distribution. For Uniform environments, a similar amount of
tags has been assigned to each client. For the Skewed environment, the
distribution of tags follows a left-skewed assignment. A similar distribution
pattern can be observed for tags present in only one client, which we call
unique tags. Figure 3 shows the numbers of unique location, organization and
person entities each client contains. Finally, Figure 4 shows how many clients
contain each entity, that is, how many tags appear simultaneously at K number
of clients (e.g., K=5 gives how many tags appear in exactly 5 clients).

(a)

(b)

(c)

(d)

Figure 2: CoNLL-2003 dataset: B-LOC, B-ORG and B-PER entity (tag) distribution
for each client within each federation environment.

(a)

(b)

(c)

(d)

Figure 3: CoNLL-2003 dataset: Number of unique B-LOC, B-ORG and B-PER entity
(tag) distribution for each client within each federation environment.

(a)

(b)

(c)

(d)

Figure 4: CoNLL-2003 dataset B-LOC, B-ORG and B-PER entity (tag) common occurrences across clients for each federation environment (log scale). Environment | Precision | Recall | F1-score
---|---|---|---
Centralized | 90.91 | 90.12 | 90.51
8-clients-Uni | 88.85 | 88.30 | 88.57
16-clients-Uni | 88.99 | 87.69 | 88.34
32-clients-Uni | 88.52 | 86.15 | 87.32
32-clients-Ske | 88.43 | 86.37 | 87.39
Table 1: Final learning performance of centralized vs federated models. The
Clients-Uni and Clients-Ske environments refer to the federation environments
with uniform and skewed assignment of training samples across clients. Figure
5: NER model learning performance in centralized and federated settings.

Results. Table 1 shows the final learning performance of each model for each
learning environment. As expected, the centralized model outperforms the
federated models. However, the performance degradation is small ($\sim$2-3
percentage points). The degradation increases as the federation environments
become more challenging, that is, with the number of clients and the
heterogeneity of the data. As the number of clients increases, the amount of
data available for local training decreases, which makes federated training
harder. Nonetheless, federated learning remains effective even with larger
number of clients and skewed distributions.

Figure 5 shows the convergence rate of the federated models, as F1-score over
federation rounds. For the centralized model we plot the best F1 score as a
horizontal line. The harder the federation environment, the more training
(federation) rounds the federated model needs to reach an acceptable
performance.

##  5 Discussion

We have empirically shown that Federated Learning can be successfully applied
in the context of NLP to train deep learning models for Named-Entity
Recognition. Across the federated environments that we investigate, the final
NER model can reach an acceptable performance when compared to its centralized
counterpart. As the federated learning environments become more challenging
due to increased statistical heterogeneity and reduced data amount per client,
a moderate performance decrease occurs, and more training rounds are required
for the federation to converge.

Our immediate future work focuses on training NLP models when the text at each
site is private. A critical pre-processing step for training deep NLP models
is the creation of a common vocabulary/dictionary that maps each token to a
unique identifier. In a centralized setting where all the data are located at
a central repository the creation of a such a vocabulary is straightforward.
However, in a federated setting where the global dataset is split across
multiple clients with each client having its own private local dataset, the
creation of such a vocabulary is more challenging. Due to privacy concerns,
clients need to always keep their data at their original location. They cannot
reveal either sentences or individual words/tokens to others. For example,
consider an intelligence domain where person names and locations under
investigation at a site must be protected. Therefore constructing a common
representation that includes all federation tokens needs to be performed in a
secure and private way. In our experiments, we assumed that a publicly
available vocabulary exists. We are currently investigating hashing methods
that can provide a global, but private, vocabulary for federated training in
NLP applications.

## References

  * Bellavista et al. (2021) Paolo Bellavista, Luca Foschini, and Alessio Mora. 2021.  Decentralised learning in federated deployment environments: A system-level survey.  _ACM Computing Surveys (CSUR)_ , 54(1):1–38.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ge et al. (2020) Suyu Ge, Fangzhao Wu, Chuhan Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2020.  Fedner: Privacy-preserving medical named entity recognition with federated learning.  _arXiv preprint arXiv:2003.09288_.
  * Hard et al. (2018) Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018.  Federated learning for mobile keyboard prediction.  _arXiv preprint arXiv:1811.03604_.
  * Hathurusinghe et al. (2021) Rajitha Hathurusinghe, Isar Nejadgholi, and Miodrag Bolic. 2021.  A privacy-preserving approach to extraction of personal information through automatic annotation and federated learning.  _arXiv preprint arXiv:2105.09198_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9:1735–1780.
  * Kairouz et al. (2021) Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2021.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2):1–210.
  * Karimireddy et al. (2020) Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020.  Scaffold: Stochastic controlled averaging for federated learning.  In _International Conference on Machine Learning_ , pages 5132–5143. PMLR.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _ICML_.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California. Association for Computational Linguistics.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  In _NeurIPS_.
  * Leroy et al. (2019) David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. 2019.  Federated learning for keyword spotting.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6341–6345. IEEE.
  * Li et al. (2020) Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020.  Federated learning: Challenges, methods, and future directions.  _IEEE Signal Processing Magazine_ , 37(3):50–60.
  * Li et al. (2019) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019.  On the convergence of fedavg on non-iid data.  In _International Conference on Learning Representations_.
  * Lin et al. (2021) Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. 2021.  Fednlp: Benchmarking federated learning methods for natural language processing tasks.  _arXiv preprint arXiv:2104.08815_.
  * Liu et al. (2019) Dianbo Liu, Dmitriy Dligach, and Timothy Miller. 2019.  Two-stage federated phenotyping and patient representation learning.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, page 283. NIH Public Access.
  * Liu et al. (2021) Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. 2021.  Federated learning meets natural language processing: A survey.  _arXiv preprint arXiv:2107.12603_.
  * Liu et al. (2020a) Yi Liu, JQ James, Jiawen Kang, Dusit Niyato, and Shuyu Zhang. 2020a.  Privacy-preserving traffic flow prediction: A federated learning approach.  _IEEE Internet of Things Journal_ , 7(8):7751–7763.
  * Liu et al. (2020b) Yuan Liu, Zhengpeng Ai, Shuai Sun, Shuangfeng Zhang, Zelei Liu, and Han Yu. 2020b.  Fedcoin: A peer-to-peer payment system for federated learning.  In _Federated Learning_ , pages 125–138. Springer.
  * Marrero et al. (2013) Mónica Marrero, Julián Urbano, Sonia Sánchez-Cuadrado, Jorge Morato, and Juan Miguel Gómez-Berbís. 2013.  Named entity recognition: fallacies, challenges and opportunities.  _Computer Standards & Interfaces_, 35(5):482–489.
  * Mathew et al. (2019) Joel Mathew, Shobeir Fakhraei, and José Luis Ambite. 2019.  Biomedical named entity recognition via reference-set augmented bootstrapping.  _arXiv preprint arXiv:1906.00282_.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pages 1273–1282. PMLR.
  * Mitra et al. (2021) Aritra Mitra, Rayana Jaafar, George Pappas, and Hamed Hassani. 2021.  Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients.  _Advances in Neural Information Processing Systems_ , 34.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Reddi et al. (2020) Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and Hugh Brendan McMahan. 2020.  Adaptive federated optimization.  In _International Conference on Learning Representations_.
  * Rieke et al. (2020) Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. 2020.  The future of digital health with federated learning.  _NPJ digital medicine_ , 3(1):1–7.
  * Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  _arXiv preprint cs/0306050_.
  * Sheller et al. (2020) Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al. 2020.  Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data.  _Scientific reports_ , 10(1):1–12.
  * Stremmel and Singh (2021) Joel Stremmel and Arjun Singh. 2021.  Pretraining federated text models for next word prediction.  In _Future of Information and Communication Conference_ , pages 477–488. Springer.
  * Stripelis and Ambite (2021) Dimitris Stripelis and José Luis Ambite. 2021.  Semi-synchronous federated learning.  _arXiv preprint arXiv:2102.02849_.
  * Wang et al. (2021) Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. 2021.  A field guide to federated optimization.  _arXiv preprint arXiv:2107.06917_.
  * Wang et al. (2020) Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  _Advances in neural information processing systems_ , 33:7611–7623.
  * Yang et al. (2019) Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019.  Federated machine learning: Concept and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2):1–19.
  * Yang et al. (2018) Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. 2018.  Applied federated learning: Improving google keyboard query suggestions.  _arXiv preprint arXiv:1812.02903_.