# Latent Space Imaging

Matheus Souza<EMAIL_ADDRESS>Yidan Zheng
<EMAIL_ADDRESS>Kaizhang Kang<EMAIL_ADDRESS>Yogeshwar
Nath Mishra<EMAIL_ADDRESS>Qiang Fu<EMAIL_ADDRESS>Wolfgang Heidrich<EMAIL_ADDRESS>Visual Computing Center,
King Abdullah University of Science and Technology, Thuwal, 23955, Makkah,
Saudi Arabia

###### Abstract

Digital imaging systems have classically been based on brute-force measuring
and processing of pixels organized on regular grids. The human visual system,
on the other hand, performs a massive data reduction from the number of photo-
receptors to the optic nerve, essentially encoding the image information into
a low bandwidth latent space representation suitable for processing by the
human brain. In this work, we propose to follow a similar approach for the
development of artificial vision systems.

Latent Space Imaging is a new paradigm that, through a combination of optics
and software, directly encodes the image information into the semantically
rich latent space of a generative model, thus substantially reducing bandwidth
and memory requirements during the capture process.

We demonstrate this new principle through an initial hardware prototype based
on the single pixel camera. By designing an amplitude modulation scheme that
encodes into the latent space of a generative model, we achieve compression
ratios from 1:100 to 1:1,000 during the imaging process, illustrating the
potential of latent space imaging for highly efficient imaging hardware, to
enable future applications in high speed imaging, or task-specific cameras
with substantially reduced hardware complexity.

###### keywords:

Computational Imaging, Generative Models, Single Pixel Imaging

##  1 Main

Conventional imaging systems are based on the brute-force sampling of scene
information on regular grids. Biological vision systems operate very
differently: for example the human visual system has around 120 million rods
and 7 million cones [1], but the information from these photoreceptors is
distilled by retinal processing into neural signals that are transmitted over
the optic nerve with only 0.7-1.7 million axions [2]. This compression forms
an encoding of the image information into a latent space that is compatible
with the neural architecture of the human brain.

Here we propose Latent Space Imaging (LSI), a new paradigm that aims to bring
this concept to the realm of cameras and artificial vision systems.
Specifically, we propose to utilize the combination of linear optical encoding
for drastic data reduction, paired with a small amount of non-linear
computational processing to directly encode image information into the latent
space of a generative model. Overall, this combination drastically reduced the
amount of sensor data that needs to be captured relative to a conventional 2D
image sensor.

Latent Space Imaging builds and improves upon the recent trend for co-
designing both optics and reconstruction algorithms. This joint optimization
has yielded favorable outcomes in computational imaging and downstream
computer vision tasks. These include color image restoration [3], microscopy
[4, 5, 6], monocular depth imaging [7, 8, 9, 10], super-resolution, extended
depth of field [11, 12], time-of-flight imaging [13], and high-dynamic range
imaging [14], among others. The enhancements observed are attributed to the
ability of such systems to _optically encode_ relevant scene information into
sensor data, with the computational post-processing acting as a decoder. The
crucial innovation in LSI is that the code represents the rich latent space of
a generative model, with that model acting as the decoder module.

While there are a number of possible ways to optically implement the LSI
paradigm, our first prototype utilizes the single-pixel imaging (SPI) concept
[15, 16, 17, 18]. SPI was originally developed under the Compressed Sensing
(CS) framework [19], and indeed LSI and CS share the common goal of pre-
sensing data reduction. However, where CS utilizes a generic (usually random)
encoding of the image, LSI uses codes that are specifically designed to embed
the image into the rich latent space of a generative model in the form of a
StyleGAN [20, 21, 22]. StyleGANs are renowned for their ability to synthesize
highly realistic face images, editing and image-to-image translation
applications. This capability stems from their semantically rich,
disentangled, well-behaved, and _compact_ latent space, making them an ideal
architecture for LSI.

To showcase our framework, we tackle the particularly challenging task of full
image reconstruction of human faces with the aim of accurate identity
retrieval. Considering the distinctiveness of facial features, which present
considerable challenges for existing compressed imaging techniques state-of-
the-art usually suffer from blurred features and an overall loss of detail.
LSI on the other hand consistently produces detailed, aesthetically pleasing
faces that remain true to the original signal. Our experiments on this
particularly challenging task illustrate potential for even higher compression
ratios in easier tasks such as object detection or classification. Overall,
the simplicity of LSI offers vast potential for expansion across a broad
spectrum of applications in computer vision.

##  2 Results

###  2.1 Latent Space Imaging

a b c e d Figure 1: (a) Traditional imaging methods require a large number of
measurements and sensors with extensive pixel counting capabilities. b)
Compressed sensing techniques typically act as low-pass filters, and when the
measurements are compressed below $1\%$ of the original pixel counting they
fail to reconstruct highly detailed images, even with the aid of sophisticated
deep neural networks during the reconstruction process. (c) By encoding the
image signal into the latent space of a generative model, surprisingly high
compression rates can be achieved, while maintaining distinct fine details.
(d) A linear optical and a non-linear digital encoder are jointly optimized to
acquire compressed data, and encode it into the latent space provided by the
inversion network $\mathcal{E}$ by minimizing the loss function
$\mathcal{L}_{lat}$. During reconstruction, the compressed data is fed into
the generative model $\mathcal{G}$ (StyleGAN2) in order to reconstruct the
image. The lock symbol represents a fixed model that is not being optimized.
(e) We illustrate one possible implementation of the Latent Space Imaging
technique using a single-pixel framework. An objective lens focuses the image
onto the Digital Micromirror Device (DMD), which is responsible for
implementing the learned mask to spatially modulate the incoming signal. This
is followed by a relay lens, which focuses the modulated signal onto a
photodiode (SPD) responsible for integrating the signal. Utilizing time
multiplexing, we can retrieve the necessary measurements.

Traditional imaging captures a regular $m\times n$ grid of measurements (Fig.
1a), while Compressed Sensing (CS) uses a collection of random linear codes to
leverage the information redundancy inherent in real images (Fig. 1b) in
combination with a reconstruction software module. In contrast, we demonstrate
that meaningful information can be significantly more compressed in the latent
space of generative models (Fig. 1c). Our method allows for direct imaging
into this space, achieving high quality image reconstructions from only
$1.0\%$ to $0.1\%$ of the measurements compared to the original pixel count.
Unlike traditional approaches that learn to interpolate missing information,
LSI enables detailed reconstruction focusing on the extraction of key domain
features through a combined optical and digital encoding process.

As depicted by Fig. 1d, our objective is to learn an optically encoded mapping

$\begin{matrix}\textbf{L}=\mathcal{D}_{\theta}(\textbf{O}\cdot\textit{I}),\end{matrix}$
(1)

where the input image $\textit{I}\in\mathbb{R}^{mn}$ is mapped to a tensor
$\textbf{L}\in\mathbb{R}^{k\times l}$ representing the latent space of a
generative model. This mapping is achieved through the combination of a co-
designed (linear) optical encoder $\textbf{O}\in\mathbb{R}^{mn\times d}$ and a
digital (nonlinear) encoder $\mathcal{D}_{\theta}$, with $\theta$ representing
the set of parameters that configure the digital encoder.

The optical encoder linearly projects the image to an intermediate vector
$\textit{J}\in\mathbb{R}^{d}$ representing the actual measurements of the
signal, where $d$ is 100 to 1000 times smaller than the pixel count $mn$ in
the original image. This substantial compression ratio is achieved by
optimizing the projection alongside other components during the image
reconstruction process. Instead of merely matching pixel values, the process
targets the generative model’s latent representation L of the original signal.
By focusing on this compressed space, the essential features of the data
distribution within a particular domain are captured, effectively eliminating
redundancies inherent in the data.

However, due to the complex non-linear operations intrinsic to generative
models and the impracticality of realizing these non-linearities optically, an
additional digital encoder $\mathcal{D}_{\theta}$ is necessary to align J with
the entire latent space of interest. This alignment is achieved through a deep
neural network (DNN), which enhances the representation by effectively
addressing the non-linearities.

The digital encoder $\mathcal{D}_{\theta}$ (see Fig.1d) expands J to
$\textbf{L}\in\mathbb{R}^{512\times 18}$, aligning it with the chosen
generative model’s dimensionality. In our work, StyleGAN2 [21] is employed
since it performs robustly within specific domains, and its mechanisms for
inversion to latent space are well-studied [23].

The digital encoder employs a multi-level network inspired by advancements in
the inversion domain [24]. This approach leverages the detailed structural
levels of StyleGAN2 [21]. In this framework, the optically encoded vector J
passes through a series of linear layers and attention mechanisms, each
followed by activation functions. The depth of these stacks varies to match
the required level of detail. For the pre-trained StyleGAN2 on human faces,
there are 18 levels corresponding to different resolutions. The number of
stacks increases progressively from coarse to fine, aligning with low to high
resolutions, respectively.

After acquiring all detail levels, we obtain
$\textbf{L}\in\mathbb{R}^{512\times 18}$. A mixer block is then used to learn
a weighted mixture of information across these levels, which is subsequently
fed into the generative model $\mathcal{G}$ to output the image I’

$\begin{matrix}\textit{I'}=\mathcal{G}(\textbf{L}),\end{matrix}$ (2)

where $\textit{I'}\in\mathbb{R}^{mn\times 3}$, with $m$ and $n$ determined by
the capabilities of the generative model $\mathcal{G}$. The value 3
corresponds to the RGB color channels which are discussed in further detail
later in this section.

A pivotal aspect of the LSI framework is the identification of the optimal
latent space to serve as ground-truth during the optimization process. To
accomplish this, we employ a pre-trained encoder, $\mathcal{E}$, following the
method proposed by [24] for accessing the StyleGAN2 latent space. This encoder
provides an approximate ground-truth latent representation,
$\mathcal{E}(\textit{I})$, which serves as the target for optimizing each
entry of the O matrix and the $\mathcal{D}_{\theta}$. The optimization is
driven by the loss function $\mathcal{L}_{lat}$

$\begin{matrix}\mathcal{L}_{lat}=\left\|\mathcal{D}_{\theta}(\textbf{O}\cdot\textit{I}))-\mathcal{E}(\textit{I})\right\|_{1}\end{matrix}$
(3)

Making the encoding L close to $\mathcal{E}(\textit{I})$ guarantees that it is
inside the StyleGAN average space and diminishes hallucination caused by only
matching an $\ell_{2}$ norm at the pixel image level. To further enhance
facial fidelity, we incorporate additional loss functions. Detailed
information on these loss functions can be found in the supplementary
material.

a b c d Figure 2: (a) Simulated results of our framework compared to the
ground truth. The process involves optically encoding input images using
learned masks. The resulting intermediate latent space vector is visualized
through the “sea urchin” diagram, where each spike represents one measured
variable. (b) Compression efficiency of the LSI framework demonstrated through
side-by-side comparisons. The left side of the panel shows the ground truth,
while the right side illustrates the progression from 512 to 64 measurements,
representing compression rates from approximately $0.8\%$ to $0.1\%$ of the
total pixel count. (c) Visual comparison across different imaging strategies:
the input image, LSI, LSI with Variational Autoencoder (LSI-VAE), and a
representative method from Compressive Sensing (CS) employing Fourier Single-
Pixel Imaging (FSI). The FSI image is reconstructed using a deep learning
algorithm from the AuSamNet [25] framework for fixed circular mask. This panel
highlights the effectiveness of our imaging strategy in retrieving detailed
features. (d) Quantitative comparison of identity retrieval capabilities
between LSI and CS, using is the dot product between features extracted from
CurricularFace [26], a specialized model for face recognition. This plot shows
that LSI excels in high-compression settings but saturates at lower ratios,
whereas CS exhibits poorer performance at extreme compression levels but
improves substantially as the number of measurements increases.

Fig. 2a showcases the identity preservation capabilities of our model
reconstructing images at a resolution of $256\times 256$ with $256$
measurements. Latent Space Imaging is able to retrieve eye colors, expression,
shape, skin tone with less than $0.4\%$ of the original pixel count. This is
achieved by designing O to retrieve specific features from the domain that we
are targeting, thus pushing it to focus on more important characteristics on
facial identification as well as the landmarks present in the underlying
distribution learned by the generative model. This property, together with
compressing the target space to be flat, drives the reconstruction process to
not only interpolate under-sampled information but to produce an output close
to the original detailed face.

Direct learning in the latent space is also an efficient method for handling
color images, compared with traditional compressive sensing designs. In this
approach, color is already implicitly represented in the latent space and does
not require explicit modeling. O integrates the signal over the wavelengths to
generate our measurement, which in simulation means to sum over the RGB
channels. Typically, explicit modeling would increase the number of
measurements necessary for reconstruction or necessitate some trade-off
between spatial and spectral resolution. Latent Space Imaging (LSI)
effectively bypasses these complexities, enabling going to further compressive
levels utilizing only $512,256,128$, and $64$ measurements. Fig. 2b provides
an example of detailed presence even in such critical situations. This
contrasts with traditional compressive sensing strategies, where limited
measurements lead to reconstruction algorithms over-smoothing the images,
resulting in the loss of critical identity features such as the shape of eyes,
beards, and other distinguishing landmarks. Fig. 2c provides visual evidence
of this phenomenon, while Fig. 2d quantitatively demonstrates the differences.
In this analysis, we compare Latent Space Imaging (LSI) against two other
methods: LSI with a Variational Autoencoder (LSI-VAE) and Fourier Single-Pixel
Imaging (FSI) [18]. For the FSI, we employ fixed circular masks coupled with a
deep learning model for signal reconstruction, adhering to the procedures set
forth in Huang et al.[25]. Detailed methodologies and additional comparisons
are elaborated in supplementary material.

###  2.2 Physical Setup

Realizing the LSI framework involves implementing the optical encoder hardware
along with the digital encoder software module. Among a range of different
alternatives, we opt for the single-pixel camera approach as the hardware
framework for our prototype. In this context, O spatially modulates the
incoming image I through matrix multiplication (denoted by $\cdot$ in Equation
1). Each row of the optical encoder corresponds to a learned mask, and each
column represents one measurement.

Utilizing the straight-through estimator (STE) [27], we enable the gradients
to update the optical encoder, tailoring it for specific feature extraction
while ensuring its physical viability. This involves maintaining O’s entries
as positive, bounded between [0, 1], and quantized. To enhance the system’s
light efficiency and control over subtle changes, an energy efficiency loss is
strategically added during the later stages of training. This key adjustment
finely tunes the encoding process, ensuring each pattern is distinctly
counted, thereby boosting the system’s capability to discriminate between
different patterns effectively. As demonstrated in Fig.3a, with the energy
loss applied, the histogram showcases a broad spectrum of occupancies,
reflecting a high diversity of patterns. In contrast, without this
modification, the patterns are more uniform in their occupancy levels,
exhibiting less variability. Additionally, Fig.3b illustrates that the masks
projected onto the DMD following this loss application result in patterns that
strikingly resemble faces, highlighting the system’s focus on facial landmarks
for specific domain encoding. This enhancement not only augments detection
capabilities but also prevents minor intensity changes in the projected image
from going unnoticed.

To experimentally verify Latent Space Imaging, we constructed a proof-of-
concept setup as depicted in Fig. 1a. In this setup, a scene is projected onto
a Digital Micromirror Device (DMD TI DLP4500) using an objective lens. The
image is then spatially modulated by the optimized patterns. The modulated
output is captured by a relay lens, which focuses it onto a single-pixel
detector (ThorLabs PDA100A2). This detector acts as an integrator, measuring a
single value that represents the integrated intensity of the entire modulated
image. A computer controls and synchronizes the operation of the DMD and the
measurement signals, enabling the time-multiplexing of all measurements that
compose the compressed signal.

Fig. 3c demonstrates the reconstruction quality of our experimental setup
using a monitor as the source image. For such a task, we utilized the LSI
trained to reconstruct images based on $256$ measurements. fine-tuned our
model with 200 images from the FFHQ dataset, and used 100 unseen images from
CelebAHQ to validate our results. This step accounts for non-modeled noise and
sensor non-linearities. More details on the supplementary material.

a c b Figure 3: (a) Shows the histogram of pixel occupancy, comparing
scenarios where the energy loss function is and is not utilized. The use of
energy modeling enhances the dynamic range of the measurements captured by the
single sensor. (b) Displays representative patterns after using the energy
loss function. (c) Demonstrates real-world captures using the setup depicted
in Fig. 1c. These images are acquired with only 256 measurements, which
constitutes merely approximately $0.4\%$ of the total pixel count, based on an
image resolution of $256\times 256$.

##  3 Discussion

Compared to other imaging techniques, Latent Space Imaging (LSI) offers
substantial benefits in domain specific scenarios with tight requirements for
hardware complexity, memory and bandwidth requirements, or frame rate. As Fig.
2b illustrates, LSI is capable of keeping details even in a compression ratio
1:1,000, represented by the 64 out of 65536 measurements case, were the
overall facial structure, expression, eyes and skin are well preserved.

The LSI reconstruction power inherits the advantages of a well-trained
generative model specialized in a specific domain. In this approach, we
leverage a pre-trained StyleGAN2, optimized to cover the entire distribution
of the dataset. Consequently, its latent space learns to represent the most
meaningful and essential information within this domain. This capability
enables high-quality measurements with high compression ratios. However, it
also inherently possesses the limitations of the underlying generative model
and the inversion network.

Therefore, the model may fail to reconstruct very specific features not
present during training, such as hats, glasses, makeup, or other distinct
image details, and may struggle to generalize to a multi-domain approach.
Additional analysis about challenging scenarios and how different generative
models can be utilized on out framework can be found at supplementary
materials.

###  3.1 Alternative Physical Implementations

In our prototype, we showcase the physical implementation using binary
structural masking with a Digital Micromirror Device (DMD), where the light
source is a monitor projecting the images. The SPI framework is an established
platform for prototyping and testing different compressed imaging approaches,
since the DMD acts as a programmable encoding element, making it easy to
experimentally validate different code patterns and compression ratios.

However, SPI also has several downsides that limit its practicality for real
imaging systems, including its form factor, and, most notably, its difficulty
in dealing with moving scenes. However, the recent efforts in optical
computing [28] have resulted in a range of alternative options for
implementing the required optical encoding layer. For example diffractive
optical elements have been used to implement both convolutional [29] and fully
connected linear layers [30], and meta-surfaces have been demonstrated for
shift variant convolution kernels [31]. Due to the linear nature of light, all
these methods are fundamentally limited to linear operators. However, as we
show with the SPI setup, linear optical encoders are sufficient for achieving
high compression rates when combined with a small digital encoder. As such,
any of the recently proposed optical compute frameworks could be used instead
of the SPI framework, although typically this would require “freezing” the
optical encoder into hardware. Nonetheless, for special purpose imaging
systems, such an approach could present a more practical avenue than the SPI
framework.

###  3.2 Relationship to Biological Vision

As mentioned in the introduction, LSI is inspired by biological vision, in
particular the human visual system (HVS). Specifically, in the HVS,
photoreceptors sense the spatial distribution of light, retinal ganglion cells
(RGCs) encode these spatial distributions into a compressed latent space
transmitted over the optic nerve, and finally the visual cortex decodes and
processes the information. LSI can be seen as analogous to this process: in
our single pixel camera setup, the pixels of the spatial light modulator take
on the role of the photoreceptors – they define the spatial resolution of the
sensed image. The optical codes shown on the SLM, combined with the digital
encoder take on the role of the RGCs, and both encode and compress this
information, while the StyleGAN takes on the role of the visual cortex.

The encoding module is therefore composed of a combination of a (linear)
optical computing part and a non-linear digital encoder. The digital encoder
is needed since optical computing [28] is in practice limited to linear
operators. Nonetheless, our work shows that even with linear optical
computing, data reduction in the range of 100:1 to 1,000:1 is feasible even
for the challenging task of full image restoration. Easier tasks such as
object detection or classification will likely be possible with substantially
higher compression ratios.

##  4 Future Potentials of LSI and Conclusion

a b c d Figure 4: Visualization of image and latent space representations
using UMAP. (a) UMAP projection of images reconstructed by our pipeline, using
images from the test set of CelebA-HQ, colored by gender and highlighting the
image 172631 and its neighborhood. (b) UMAP projection of the corresponding
latent space representations, colored by the same categories. (c)
Visualization of the chosen image in the image space with its four closest
neighbors. (d) The same example image with its four closest neighbors in the
latent space, highlighting stronger semantic similarities among neighbors. We
fit the UMAP for 5 nearest neighbors for each point, with a small minimum
distance of 0.001 to ensure that points are tightly packed, preserving local
relationships and creating distinct clusters.

Latent Space Imaging provides a general framework for imaging within a
compressed environment, but its utility is not limited to this application
alone. This technique opens up new avenues for exploring deeper interactions
and potential innovations in image processing and analysis.

Previous studies have delved into the semantics within GAN models, uncovering
boundaries in the latent space that distinguish various image attributes [32,
33, 34]. Direct visualization of these models’ latent spaces allows for a
natural observation of these structures, offering significant advantages for
various applications without the need for additional complex deep learning
frameworks to extract these semantics.

To further investigate this, we analyzed the latent space representation L of
each image in the CelebA-HQ [35] test set using our pipeline, along with
images I’ reconstructed by StyleGAN2. We performed UMAP [36] feature reduction
to visualize them in a 2D map. Fig.4a shows the image representation, while
Fig.4b illustrates the latent space representation.

The latent representation forms clusters based on image features. For
instance, we highlighted males, females, and one specific image along with its
neighborhood in different colors. Males tend to cluster on one side, separated
from females, and image 172631 clusters close to images with similar
characteristics (e.g., hair color, age, glasses, skin tone). This clear
semantic division is not observable in the image-based representation. To
illustrate this, we examined the four closest neighbors in each
representation. It is evident that the neighbors in the latent space share
more semantic similarities (see Fig. 4c,d).

This analysis not only underscores that latent space representations enable a
novel compressed imaging process, but also demonstrates their potential as
powerful semantic extractors. Such capabilities suggest that exploring these
representations could yield significant insights and advancements in the field
of generative models, potentially leading to more effective and nuanced image
processing techniques in the future.

Besides a semantic analysis, the LSI framework here leverages a single-pixel
approach. This implementation can be engineered for high frame rate due to low
bandwidth requirements, and utilizing state-of-the-art DMDs would provide an
out-of-box improvement for the already proposed setup. As discussed, Latent
Space Imaging (LSI) could also be effectively implemented using a variety of
other optical computing paradigms, such as diffractive optical elements and
meta-surfaces (see Sec.3.1). These alternatives to the traditional Single
Pixel Imaging (SPI) framework can manage optical encoding with high
efficiency. Furthermore, enhancing the digital decoder through research into
different generative model implementations holds promise for significantly
improving both the versatility and efficacy of LSI systems. This approach
could lead to more practical and adaptable imaging solutions, particularly
suited for dynamic environments.

##  5 Methods

We utilized PyTorch to design and validate the LSI framework. All training
details, hyperparameters and $\mathcal{D}_{\theta}$ architecture detailed
description and discussion can be found in the supplementary material.

###  5.1 Optical Encoder Optimization

In the O optimization process, we employ the straight-through estimator (STE)
[27], a technique also favored (VQ-VAEs) [37, 38, 39]. During the forward
pass, values are quantized and constrained before computing J. However, during
back-propagation, gradients are allowed to flow through the non-quantized
version, facilitating weight updates. Importantly, we ensure that the entries
of O remain positive and are bounded between $[0,1]$. Although using
complementary patterns [40] allows for the inclusion of negative values,
empirical experiments have shown that this approach does not yield significant
improvements and necessitates double the measurements.

Initially, masks are generated from a uniform distribution and are then
binarized through a quantization process. This process results in binary
patterns that achieve an even total counting of 0s and 1s, theoretically
maximizing bit entropy and steering the system towards an optimal solution
[41]. This distribution presumes the sensor is sensitive enough to discern
very fine differences, a demanding prerequisite as each pattern will have
similar total intensity. This issue is addressed using the energy loss
mechanism, which was introduced in Section 2.2 and is further detailed in the
supplementary material.

###  5.2 Digital Encoder Optimization

Inspired by inversion networks, our approach tailors the digital encoder
specifically for our unique imaging task. Unlike traditional inversion
processes used for image editing, our encoder is designed for an optically
compressed inversion, aligning the characteristics of the latent vector with
the optical system’s constraints. This method allows our encoder to generalize
across any image within the domain, without the need for training on specific
data points.

In the broader context of the traditional inversion process, a latent array is
retrieved from a pre-trained generative model to accurately reproduce the
original image. This enables not only spatial adjustments, such as translation
and rotation, but also semantic modifications including age, hair color, and
expression changes. The inversion can be achieved through various methods: it
may involve a full optimization for each data point [42, 21, 43], the use of
an encoder trained to produce latent representations in a zero-shot manner
[44, 24], or a combination of these strategies [45, 46, 47].

###  5.3 Datasets

To effectively exploit the latent space of generative models, we have narrowed
our focus to the domain of human facial recognition. Specifically, we aim to
retrieve identities from pictures of human faces. For this purpose, we
utilized two datasets: FFHQ [20] and CelebAHQ [35]. The FFHQ dataset was
exclusively employed for training, while CelebAHQ was used both for training
and testing—comprising 28,000 images for training and 2,000 for testing.
Combined with FFHQ, this resulted in a total of 98,000 face images used for
training. This approach ensures a robust dataset that enhances the reliability
and effectiveness of our model in capturing and reconstructing facial
identities.

Both datasets contain background information that could pose a challenge due
to the already compressed measurements. To address this, we further refine the
domain by applying Segment Anything (SAM) [48] to remove background elements
and prioritize identity retrieval. Moreover, the versatility of LSI allows it
to be easily extended to other specific domains by adopting generative models
trained on different datasets.

##  6 Code Availability

The code will be made available together with all checkpoints to ensure
reproducibility.

##  7 Data Availability

The CelebA-HQ dataset is accessible on GitHub at
https://github.com/tkarras/progressive_growing_of_gans. The FFHQ dataset can
be found at https://github.com/NVlabs/ffhq-dataset. The AFHQ can be obtained
at https://github.com/clovaai/stargan-v2/tree/master.

Supplementary Material

##  S1 Digital Encoder Architecture

a b Figure S1: (a) Overview of all the blocks involved in the digital encoder
architecture. The three steps capture coarse, middle, and fine details, with
each step depending on the output of the previous one. (b) Illustration of the
MIX process. The latent vector is statically projected to capture interactions
among different levels of detail.

After acquiring the measurements J from the co-optimized optical encoder O, we
employ a digital encoder network $\mathcal{D}_{\theta}$. The objective of this
model is to match the intermediate latent space produced by the measurements
with the actual latent able to reproduce the image of interest I.

The original StyleGAN2 implementation receives a random noise vector
$\textit{Z}\in\mathbb{R}^{512}$ and utilizes a mapping network to project this
to different levels of details, result in $\textbf{L}\in\mathbb{R}^{512\times
l}$, $l$ may vary depending the desired image resolution that the generative
models was trained to generate. In the case of face images, $l$ is equal to
18, finally we want O to map $\textit{J}\in\mathbb{R}^{d}$ to
$\textbf{L}\in\mathbb{R}^{512\times 18}$. $d$ corresponds to our measurement
count, therefore, going from 512 to 64.

As depicted by Fig. S1a, the intermediate latent goes first through 3
different set of blocks from left to right. The first one produce the first 3
elements of L out of 18, each one of them only take into account the initial
J. The middle block, besides J, it also takes the concatenation of
$\textit{L}_{1},\textit{L}_{2}$ and $\textit{L}_{3}$ through the MIX module,
as input, outputting $\textit{L}_{4}$ to $\textit{L}_{7}$. Finally, the last
block follows the same idea, it concatenates the output of the middle one and
pass through another MIX block, the output of this is summed together with the
original measurement and it passes through the block to finally result in
$\textit{L}_{8}$ to $\textit{L}_{18}$. The idea here is to make the optical-
aware inversion network match the multi-level structure of StyleGAN2,
following a coarse-to-fine approach.

In the end, the latents from all detail levels pass through the MIX block. The
purpose of the MIX blocks is to facilitate interactions and learn a weighted
mixture across multiple detail levels. The model incorporates the spatial
gating unit proposed in [49]. Within the MIX block (see Fig.S1b), the features
are projected to a higher dimension, after which the resulting tensor ${x}$ is
split into two parts, ${u}$ and ${v}$, along the channel dimension. This
division allows for separate processing paths within the block. ${v}$ is
normalized and linearly projected using a weights matrix and bias vector to
capture interactions in a static manner, as the weights matrix remains
unchanged after training because it does not depend on the input. The final
element-wise multiplication of ${u}$ and the projected ${v}$ modulates the
information flow, controlling which parts of ${u}$ are allowed to pass
through—similar to a gating mechanism. Finally, this gated output is merged
with the input through a residual connection, preserving the original
information and enhancing gradient flow during back-propagation.

##  S2 Loss Functions and Training Details

In this work, we trained the optical O and digital $\mathcal{D}_{\theta}$
encoders while keeping the generative model, StyleGAN2, frozen. To jointly
optimize these components, we utilized an off-the-shelf inversion network pre-
trained to invert images to the StyleGAN2 latent space of a specific domain
[24]. This network is used to compute a latent similarity loss function
($\mathcal{L}_{lat}$), making the output of the $\mathcal{D}_{\theta}$ similar
to the latent space approximated by this model. This process is described in
Equation 3 of the main text.

Additionally, we aim to match the image quality at the pixel level and enforce
identity similarity. For this task, we used the identity loss
($\mathcal{L}_{id}$), which computes the cosine distance between feature maps
extracted by ArcFace [50], a facial recognition network. We also utilized the
$\ell_{2}$ norm ($\mathcal{L}_{l2}$) and LPIPS ($\mathcal{L}_{pips}$) [51] to
enforce pixel-wise and perceptual similarities, respectively.

The total loss function during training can be summarized as follows:

$\mathcal{L}_{total}=\lambda_{lat}\mathcal{L}_{lat}+\lambda_{id}\mathcal{L}_{id}+\lambda_{pips}\mathcal{L}_{pips}+\lambda_{l2}\mathcal{L}_{l2}$
(S1)

After convergence, we added an additional loss term
($\lambda_{energy}\mathcal{L}_{energy}$) to account for the intensity
diversity among the masks.

$\begin{matrix}\mathcal{L}_{energy}=\frac{1}{d}\sum_{j}^{d}\left|\sum_{i}^{mn}\mathcal{\textbf{O}}_{i,j}-\epsilon_{j}\right|\end{matrix}$
(S2)

Here, $\epsilon$ represents the ground-truth energy level, heuristically
designed to correspond to a certain percentage of pixels set to one. This
percentage increases incrementally from $10\%$ to $90\%$, with a step of
$1\%$. The patterns are shuffled to avoid any undesired structure.

To optimize O and $\mathcal{D}_{\theta}$, we employ distinct optimizers: Lion
[52] for O and Ranger (combination of Lookahead [53] and the Rectified Adam
[54]) for $\mathcal{D}_{\theta}$, each with a learning rate of $10^{-4}$. The
weighting coefficients $\lambda$ for each loss function are set at
$1,0.5,0.8,$ and $1$, in the respective order that they appear in Equation S1.
Additionally, for the energy loss component, we set $\lambda$ to 3. Regarding
batch sizes, we typically use 32 for face datasets. However, for the AFHQ [55]
dataset, as discussed in Sec. S4, the batch size is reduced to 8 due to the
limited number of images available. For all experiments we utilize a single
NVIDIA A100.

##  S3 Experimental setup

Our experimental setup utilizes a single-pixel imaging configuration as
follows: Face images are displayed on a high-dynamic-range Eizo CG3145
monitor. These images are then focused onto the DMD of the Texas Instruments
DLPLCR4500EVM evaluation board using a Canon EF 35mm f/2 IS USM objective
lens. We interpolate our learned $256\times 256$ patterns to optimally fill
the largest possible area, adjusting the scale to compensate for the diamond-
shaped micromirror grid of the DMD. This adjustment ensures full utilization
of the height within a central square region. The DMD spatially modulates the
image by reflecting patterns according to an optimized encoding matrix (O).
The resultant patterned light is captured by a photodiode (PDA100A2) through a
Canon EF 85mm f/1.8 USM relay lens. An analog-to-digital converter (NI 6001)
converts the received analog signals into digital form. The digital data are
then received by a connected PC, which is also responsible for synchronizing
all components involved in the experimental setup.

To enhance the congruence between our measurements and simulations, we employ
a white image to determine a global scaling deviation and apply a correction
factor. This approach effectively reduces the gap between simulation and
measurement, yet some inherent challenges persist due to non-linear behaviors,
minor deviations in DMD reflection angles, and sensor noise, which complicate
the precise replication of simulated conditions. However, by leveraging the
controlled environment of our experimental setup, we are able to
systematically gather real-world training data and fine-tune
$\mathcal{D}_{\theta}$.

The fine-tuning process involves selecting a subset of 200 training images and
measuring their outputs using our configured setup. We employ the loss
functions specified in Equation S1, while reducing the learning rate to
$10^{-5}$, to optimize performance and accuracy in real-world applications.
Fig. S2 illustrates the effect of fine-tuning.

Figure S2: Demonstrates the effect of the fine-tuning process on our model.
Post fine-tuning, the average ID similarity of the 100 test images increases
from $0.25$ to $0.31$. This enhancement brings the values in line with those
observed in our simulation results, indicating a closer match and improved
performance.

##  S4 Additional Results

We present additional LSI results encompassing a diverse set of images, along
with the range of compression levels studied in this work. Our methodology
involves reconstructing images with a resolution of $256\times 256$ pixels and
examining compression rates that result in retaining 512, 256, 128, and 64
total measurements.

In Fig. S3 a consistent retention of facial details is observed even with
increasing compression, highlighting the effectiveness of our approach even at
higher compression rates. Targeting the latent space addresses the over-smooth
appearance typical of strong sub-sampling strategies, preserving textures and
achieving a realistic look instead of creating flat reconstructions. Here, we
emphasize the persistence of facial expressions, with key features such as
eyebrow shape, beard, and smile being well reconstructed.

Figure S3: Additional results for the simulation case accounting for all set
of compression rates proposed in this work, 512, 256, 128 and 64 measurements
which corresponds to approximately 0.8$\%$,0.4$\%$,0.2$\%$ and 0.1$\%$ of the
total pixel count.

Additional domains were explored in Fig. S4, we utilize our simulated pipeline
with 512 measurements to reconstruct cats and dogs images from AFHQ dataset
[55]. These datasets imposed additional challenge because they are very small
compared with FFHQ, with only 5000 training images. However, LSI is capable to
faithfully reconstruct such domain pictures.

Figure S4: Illustrates another set of reconstructions from our simulated
results, highlighting the versatility of our methods across various domains,
such as cats and dogs.

In Fig. S5 we also expand the examples of reconstructions performed by our
physical setup utilizing the single-pixel approach for the 256 measurements
case.

Figure S5: Showcase another set of reconstructions performed by the proposed
physical setup.

##  S5 Failure Cases and Limitations

As we are focusing on faces and our underlying generative model is primarily
trained with frontal-centered faces under typical daily lighting, strong
deviations from these conditions are not properly modeled and result in strong
hallucinations. Fig. S6a shows some failure conditions. From these
observations, we note that accessories like glasses and hats are not modeled
by the generative model and cannot be reconstructed. Additionally, other parts
of the body are also not reproducible, and faces oriented far from a frontal
view add additional burden to the reconstruction process. Similar behavior is
observed in other works utilizing an encoder network to perform inversion when
a purely software-based network is employed. In our case, we are dealing with
a more challenging scenario where a real spatial encoder compresses the image
to a very limited amount of data. Fig. S6b demonstrates how significant
features can be lost in challenging scenes when compressing to over $99.9\%$.
We notice that the mouth is not open as in the original image. This behavior
can be explained by the underlying distribution learned by the generative
model, where few images have the mouth in such a position. Consequently, our
model struggles to reconstruct it accurately.

a b Figure S6: (a) Illustrates the conditions where LSI fails to reproduce the
scene. Demonstrate different challenging scenarios like accessories, face
orientation, and color reproduction. (b) Exemplify a case where extreme
compression hurt the expression reconstruction.

##  S6 Latent Space Image in the context of Compressive Sensing

Latent Space Imaging (LSI) aims to retrieve rich compressed information from a
specific domain, directly targeting the latent space of the underlying
reconstruction model. The concept can be used in different imaging modalities.
Here, we showcase the compression aspect of LSI utilizing a single-pixel
approach. Therefore, we compare against traditional compressive sensing (CS)
strategies.

First, it is essential to state that some CS ideas in the literature are
introduced only in simulation and may make unrealistic assumptions that
prevent actual hardware implementation. Works like [56, 57, 58, 59], to name a
few, provide valuable insights on how to improve compressive sensing
reconstructions. However, they are not applicable because they have
measurement matrices that are not feasible in the real world due to negative
entries, or they do not consider challenges like quantization. Here, we are
interested in methods that are feasible in an experimental setup.

In this context, DCAN [60] and follow-up works optimize their binary patterns
together with a deep learning model for reconstruction in the spatial domain.
Other solutions shift to the Fourier domain as proposed by [18], leveraging
the fact that information is usually concentrated on low-frequencies, enabling
under-sampling strategies with high-quality reconstructions but suffering when
dealing with low sampling rates [61]. Recent Fourier single-pixel imaging
(FSI) techniques try to overcome such issue, using a more advanced network as
a reconstruction method, improving the heuristics for the frequency domain
filters designing and jointly optimizing the masks and reconstruction
algorithms [25, 62, 63].

Finally, to assess how LSI reconstruction quality is compared with SPI, we
retrain FSI models using a fixed circle mask and optimize it with the deep
learning reconstruction using AuSamNet pipeline [25]. To also analyze the
performance when utilizing more sophisticated deep learning strategies from a
CS perspective, we retrain SAUNet [59] imposing the quantization constraints.
We train and evaluate using the same dataset as LSI and a similar compression
rate.

#### AuSamNet and FSI-DL [25]

Both are Fourier basis methods with deep learning reconstruction algorithms;
the first optimizes the mask, and the second utilizes a fixed heuristically
designed circular mask. The patterns $P_{\phi}$ are generated using the ideal
proposed by [18]

$P_{\phi}(x,y;f_{x},f_{y})=a+b\cos(2\pi f_{x}x+2\pi f_{y}y+\phi),$ (S3)

where $(x,y)$ represents the 2D Cartesian coordinates in the scene, $a$
symbolizes the average intensity distribution, $b$ stands for the amplitude of
the Fourier basis pattern, and $(f_{x},f_{y})$ indicates the non-zero spatial
frequency points within the optimized mask. Furthermore, $\phi$ denotes the
initial phase, adopting three steps phase shifting of $0$, $2\pi/3$, and
$4\pi/3$.

This work also utilizes a color filter array (CFA) and demosaic process to
reduce the number of measurements necessary to retrieve color images,
computing the Fourier transform of the CFA and multiplying it with the mask.

When evaluating under our heavily compressed setting, AuSamNet cannot optimize
the mask, leading to unstable results. FSI-DL is capable of reconstructing the
images but produces low-quality results as demonstrated by Fig. S7.

#### SAUNet [59]

This approach proposes a 2D measurement system and an unfolding network as a
reconstruction algorithm. Their measurement can be defined by the equation
$Y=HXW^{T}$, where $H$ and $W$ are learned during the optimization process,
and $X$ is the input. To enable a fair comparison, we clamped and then
quantize their measurement matrix during training to be binary, following the
same procedure adopted by our methods. We add a normalization layer after
utilizing $H$ and $W$ to avoid exploding gradients and instability.

Similar to FSI-DL, SAUNet can reconstruct images but struggles to maintain
fine details, resulting in lower quality outputs (see Fig. S7). It is
important to note that we trained SAUNet following the methodology outlined in
their work. Like many other compressive sensing approaches, we utilized the
YCrCb color space. In this approach, the objective is to reproduce the
luminance component, while the chrominance components were added during
inference using ground-truth values. This approach gives them an advantage, as
they would in any practical setting require additional measurements to
reproduce color images or model color filters in conjunction with the
modulation masks, as proposed by AuSamNet [25].

Figure S7: LSI qualitatively compared against compressive sensing approaches
across several compression ratios.

###  S6.1 Different Generative Models

Recent advancements in diffusion models have significantly improved the
generation of high-quality, text-guided images, their success root also in
leveraging the latent space [64]. This process utilizes a VAE as the encoder
to the latent space and decoder to back to image, which can also be applied in
LSI. However, such generative models still depend on spatial structures, with
a latent space of $4\times 32\times 32$ for $256\times 256$ images. To
leverage the encoding power of VAE, we replaced the StyleGAN generator with
the VAE decoder and used the encoder to provide a new ground-truth
$\mathcal{E}(\textit{I})$ for our target latent space, then solving the
Equation 3 in the main text. The spatial structure in this latent space,
coupled with its training for multi-domain applications, leads to some
traditional compressive sensing issues, where high-frequency information is
often lost during reconstruction (see Fig.3c in the main text). However,
because our approach directly images to a compressed latent space, it
outperforms other methods like Fourier single-pixel imaging. The compactness
of the StyleGAN latent space exemplifies its superior suitability over VAE for
the application demonstrated here.

##  S7 Evaluation Metrics

It is well established that traditional metrics such as PSNR and SSIM do not
correlate well with human visual perception [51, 65, 66]. These metrics
prioritize pixel-level accuracy and are largely insensitive to aspects of
perceptual quality, failing to account for high-level structural information.
Seminal studies, such as [51], have demonstrated that phenomena like blurring,
which significantly impact perceptual experience, often lead to minimal
changes in the L2 norm. This limitation becomes particularly pronounced in the
context of heavily compressed image reconstructions, where PSNR and SSIM
inadequately reflect the extent of perceived quality deterioration.

In this work, we evaluate our methods in terms of identity similarity and FID
[67]. The first one, is the dot product between features extracted from
CurricularFace [26], a specialized model for face recognition, which is
different from the one utilized during training to compute the ID loss, in
order to guarantee that we are not optimizing for specific evaluation metric.

Finally, Table S1 presents the evaluation metrics for LSI, FSI-DL, and SAUNet.
It is evident that our method produces superior perceptual quality,
particularly in heavily compressed cases where other methods fail entirely.

Table S1: Comparison of ID and FID scores across different methods and resolutions (top result shown in bold). | 512 | 256 | 128 | 64
---|---|---|---|---
ID $\uparrow$ | FID $\downarrow$ | ID $\uparrow$ | FID $\downarrow$ | ID $\uparrow$ | FID $\downarrow$ | ID $\uparrow$ | FID $\downarrow$
LSI | 0.32 | 35.57 | 0.31 | 35.65 | 0.27 | 37.19 | 0.22 | 37.29
FSI-DL | 0.12 | 107.48 | 0.07 | 118.79 | 0.04 | 134.40 | 0.02 | 173.71
SAUNet | 0.24 | 58.96 | 0.14 | 75.53 | 0.09 | 104.23 | 0.06 | 107.35

## References

  *   * Molday and Moritz [2015] Molday, R.S., Moritz, O.L.: Photoreceptors at a glance. Journal of Cell Science 128(22), 4039–4045 (2015)
  * Jonas et al. [1992] Jonas, J.B., Schmidt, A.M., Müller-Bergh, J., Schlötzer-Schrehardt, U., Naumann, G.: Human optic nerve fiber count and optic disc size. Investigative Ophthalmology & Visual Science 33(6), 2012–2018 (1992)
  * Chakrabarti [2016] Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. NIPS’16, pp. 3089–3097. Curran Associates Inc., Red Hook, NY, USA (2016)
  * Nehme et al. [2020] Nehme, E., Freedman, D., Gordon, R., Ferdman, B., Weiss, L., Alalouf, O., Naor, T., Orange, R., Michaeli, T., Shechtman, Y.: DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 1–7 (2020)
  * Kellman et al. [2019] Kellman, M., Bostan, E., Chen, M., Waller, L.: Data-driven design for Fourier ptychographic microscopy. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–8 (2019)
  * Shechtman et al. [2016] Shechtman, Y., Weiss, L., Backer, A., Lee, M., Moerner, W.: Multicolour localization microscopy by point-spread-function engineering. Nat. Photonics 10 (2016)
  * Chang and Wetzstein [2019] Chang, J., Wetzstein, G.: Deep optics for monocular depth estimation and 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10193–10202 (2019)
  * Haim et al. [2018] Haim, H., Elmalem, S., Giryes, R., Bronstein, A.M., Marom, E.: Depth estimation from a single image using deep learned phase coded mask. IEEE Transactions on Computational Imaging 4(3), 298–310 (2018)
  * He et al. [2018] He, L., Wang, G., Hu, Z.: Learning depth from single images with deep neural network embedding focal length. IEEE Transactions on Image Processing 27(9), 4676–4689 (2018)
  * Wu et al. [2019] Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A., Veeraraghavan, A.: PhaseCAM3D—learning phase masks for passive single view depth estimation. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–12 (2019). IEEE
  * Sitzmann et al. [2018] Sitzmann, V., Diamond, S., Peng, Y., Dun, X., Boyd, S., Heidrich, W., Heide, F., Wetzstein, G.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37(4), 1–13 (2018)
  * Sun et al. [2021] Sun, Q., Wang, C., Fu, Q., Dun, X., Heidrich, W.: End-to-end complex lens design with differentiate ray tracing. ACM Trans. Graph. 40(4) (2021)
  * Chugunov et al. [2021] Chugunov, I., Baek, S.-H., Fu, Q., Heidrich, W., Heide, F.: Mask-tof: Learning microlens masks for flying pixel correction in time-of-flight imaging. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9116–9126 (2021)
  * Sun et al. [2020] Sun, Q., Tseng, E., Fu, Q., Heidrich, W., Heide, F.: Learning Rank-1 diffractive optics for single-shot high dynamic range imaging. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 1386–1396 (2020)
  * Duarte et al. [2008] Duarte, M.F., Davenport, M.A., Takhar, D., Laska, J.N., Sun, T., Kelly, K.F., Baraniuk, R.G.: Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine 25(2), 83–91 (2008)
  * Katz et al. [2009] Katz, O., Bromberg, Y., Silberberg, Y.: Compressive ghost imaging. Applied Physics Letters 95(13) (2009)
  * Song et al. [2023] Song, K., Bian, Y., Wu, K., Liu, H., Han, S., Li, J., Tian, J., Qin, C., Hu, J., Xiao, L.: Single-pixel imaging based on deep learning (2023)
  * Zhang et al. [2015] Zhang, Z., Ma, X., Zhong, J.: Single-pixel imaging by means of fourier spectrum acquisition. Nat. Commun. 6(1), 6225 (2015)
  * Donoho [2006] Donoho, D.L.: Compressed sensing. IEEE Transactions on Information Theory 52(4), 1289–1306 (2006)
  * Karras et al. [2019] Karras, T., Laine, S., Aila, T.: A Style-Based Generator Architecture for Generative Adversarial Networks (2019)
  * Karras et al. [2020] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proc. CVPR (2020)
  * Karras et al. [2021] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. Advances in Neural Information Processing Systems 34, 852–863 (2021)
  * Xia et al. [2022] Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., Yang, M.-H.: GAN inversion: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (2022)
  * Richardson et al. [2021] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
  * Huang et al. [2023] Huang, W., Wang, F., Zhang, X., Jin, Y., Situ, G.: Learning-based adaptive under-sampling for fourier single-pixel imaging. Opt. Lett. 48(11), 2985–2988 (2023)
  * Huang et al. [2020] Huang, Y., Wang, Y., Tai, Y., Liu, X., Shen, P., Li, S., Li, J., Huang, F.: Curricularface: adaptive curriculum learning loss for deep face recognition. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 5901–5910 (2020)
  * Hubara et al. [2016] Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. Adv. Neural Inform. Process. Syst. 29 (2016)
  * Wetzstein et al. [2020] Wetzstein, G., Ozcan, A., Gigan, S., Fan, S., Englund, D., Soljačić, M., Denz, C., Miller, D.A., Psaltis, D.: Inference in artificial intelligence with deep optics and photonics. Nature 588(7836), 39–47 (2020)
  * Chang et al. [2018] Chang, J., Sitzmann, V., Dun, X., Heidrich, W., Wetzstein, G.: Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8(1), 12324 (2018)
  * Lin et al. [2018] Lin, X., Rivenson, Y., Yardimci, N.T., Veli, M., Luo, Y., Jarrahi, M., Ozcan, A.: All-optical machine learning using diffractive deep neural networks. Science 361(6406), 1004–1008 (2018)
  * Wei et al. [2023] Wei, K., Li, X., Froech, J., Chakravarthula, P., Whitehead, J., Tseng, E., Majumdar, A., Heide, F.: Spatially varying nanophotonic neural networks. arXiv preprint arXiv:2308.03407 (2023)
  * Xu and Zheng [2021] Xu, J., Zheng, C.: Linear semantics in generative adversarial networks. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9351–9360 (2021)
  * Collins et al. [2020] Collins, E., Bala, R., Price, B., Susstrunk, S.: Editing in style: Uncovering the local semantics of gans. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 5771–5780 (2020)
  * Shen et al. [2020] Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of gans for semantic face editing. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9243–9252 (2020)
  * Karras et al. [2018] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: Int. Conf. Learn. Represent. (2018)
  * McInnes et al. [2018] McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
  * van den Oord et al. [2018] Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural Discrete Representation Learning (2018)
  * Razavi et al. [2019] Razavi, A., Oord, A., Vinyals, O.: Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019)
  * Mentzer et al. [2024] Mentzer, F., Minnen, D., Agustsson, E., Tschannen, M.: Finite scalar quantization: VQ-VAE made simple. In: The Twelfth International Conference on Learning Representations (2024)
  * Xu et al. [2024] Xu, Y., Lu, L., Saragadam, V., Kelly, K.F.: A compressive hyperspectral video imaging system using a single-pixel detector. Nat. Commun. 15(1), 1456 (2024)
  * Lin et al. [2022] Lin, M., Ji, R., Xu, Z., Zhang, B., Chao, F., Lin, C.-W., Shao, L.: SiMaN: Sign-to-Magnitude Network Binarization (2022)
  * Abdal et al. [2019] Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: How to embed images into the StyleGAN latent space? In: IEEE Conf. Comput. Vis. Pattern Recog. (2019)
  * Creswell and Bharath [2018] Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems 30(7), 1967–1974 (2018)
  * Tov et al. [2021] Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph. 40(4), 1–14 (2021)
  * Zhu et al. [2020] Zhu, J., Shen, Y., Zhao, D., Zhou, B.: In-domain GAN inversion for real image editing. In: Eur. Conf. Comput. Vis. (2020)
  * Dinh et al. [2022] Dinh, T.M., Tran, A.T., Nguyen, R., Hua, B.-S.: HyperInverter: Improving StyleGAN inversion via hypernetwork. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 11389–11398 (2022)
  * Alaluf et al. [2022] Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.: HyperStyle: StyleGAN inversion with hypernetworks for real image editing. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 18511–18521 (2022)
  * Kirillov et al. [2023] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.-Y., et al.: Segment anything. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4015–4026 (2023)
  * Liu et al. [2021] Liu, H., Dai, Z., So, D., Le, Q.V.: Pay attention to MLPs. Adv. Neural Inform. Process. Syst. 34, 9204–9215 (2021)
  * Deng et al. [2019] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4690–4699 (2019)
  * Zhang et al. [2018] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 586–595 (2018)
  * Chen et al. [2024] Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al.: Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems 36 (2024)
  * Zhang et al. [2019] Zhang, M., Lucas, J., Ba, J., Hinton, G.E.: Lookahead optimizer: k steps forward, 1 step back. In: Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F., Fox, E., Garnett, R. (eds.) Adv. Neural Inform. Process. Syst., vol. 32. Curran Associates, Inc., ??? (2019)
  * Liu et al. [2019] Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Han, J.: On the variance of the adaptive learning rate and beyond. In: Int. Conf. Learn. Represent. (2019)
  * Choi et al. [2020] Choi, Y., Uh, Y., Yoo, J., Ha, J.-W.: Stargan v2: Diverse image synthesis for multiple domains. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
  * Zhang and Ghanem [2018] Zhang, J., Ghanem, B.: Ista-net: Interpretable optimization-inspired deep network for image compressive sensing. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 1828–1837 (2018)
  * Mou et al. [2022] Mou, C., Wang, Q., Zhang, J.: Deep generalized unfolding networks for image restoration. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 17399–17410 (2022)
  * Song et al. [2023] Song, J., Mou, C., Wang, S., Ma, S., Zhang, J.: Optimization-inspired cross-attention transformer for compressive sensing. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 6174–6184 (2023)
  * Wang and Yuan [2023] Wang, P., Yuan, X.: Saunet: Spatial-attention unfolding network for image compressive sensing. In: Proceedings of the 31st ACM International Conference on Multimedia, pp. 5099–5108 (2023)
  * Higham et al. [2018] Higham, C.F., Murray-Smith, R., Padgett, M.J., Edgar, M.P.: Deep learning for real-time single-pixel video. Sci. Rep. 8(1), 2369 (2018)
  * Zhang et al. [2017] Zhang, Z., Wang, X., Zheng, G., Zhong, J.: Fast fourier single-pixel imaging via binary illumination. Sci. Rep. 7(1), 12029 (2017)
  * Rizvi et al. [2019] Rizvi, S., Cao, J., Zhang, K., Hao, Q.: Improving imaging quality of real-time fourier single-pixel imaging via deep learning. Sensors 19(19) (2019)
  * Yang et al. [2021] Yang, X., Jiang, P., Jiang, M., Xu, L., Wu, L., Yang, C., Zhang, W., Zhang, J., Zhang, Y.: High imaging quality of fourier single pixel imaging based on generative adversarial networks at low sampling rate. Optics and Lasers in Engineering 140, 106533 (2021)
  * Rombach et al. [2021] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models (2021)
  * Ledig et al. [2017] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4681–4690 (2017)
  * Blau and Michaeli [2018] Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 6228–6237 (2018)
  * Heusel et al. [2017] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv. Neural Inform. Process. Syst. 30 (2017)