# SHACIRA: Scalable HAsh-grid Compression
for Implicit Neural Representations

Sharath Girish
University of Maryland
<EMAIL_ADDRESS>Abhinav Shrivastava
University of Maryland
<EMAIL_ADDRESS>Kamal Gupta
University of Maryland
<EMAIL_ADDRESS>
###### Abstract

Implicit Neural Representations (INR) or neural fields have emerged as a
popular framework to encode multimedia signals such as images and radiance
fields while retaining high-quality. Recently, learnable feature grids
proposed by Müller et al. [1] have allowed significant speed-up in the
training as well as the sampling of INRs by replacing a large neural network
with a multi-resolution look-up table of feature vectors and a much smaller
neural network. However, these feature grids come at the expense of large
memory consumption which can be a bottleneck for storage and streaming
applications. In this work, we propose SHACIRA, a simple yet effective task-
agnostic framework for compressing such feature grids with no additional post-
hoc pruning/quantization stages. We reparameterize feature grids with
quantized latent weights and apply entropy regularization in the latent space
to achieve high levels of compression across various domains. Quantitative and
qualitative results on diverse datasets consisting of images, videos, and
radiance fields, show that our approach outperforms existing INR approaches
without the need for any large datasets or domain-specific heuristics. Our
project page is available at https://shacira.github.io.

Figure 1: We demonstrate the effectiveness of SHACIRA for two tasks. The left
column shows a gigapixel image at $21450\times 56718$ resolution (cropped for
visualization) encoded using Instant-NGP [1], JPEG2000 [2], and SHACIRA
(ours). The right column reconstructs NeRF [3] from 2D images and their camera
poses using Instant-NGP [1], VQAD [4], and SHACIRA. For each example, we zoom
into two crops to compare different methods. We show overall PSNR and size
required by each method. SHACIRA can capture high-resolution details with a
smaller storage size in a task-agnostic way (only 2D/3D examples shown here).

##  1 Introduction

In today’s digital age, large quantities of data in different modalities
(images, audio, video, 3D) is created and transmitted every day. Compressing
this data with minimal loss of information is hence an important problem and a
number of techniques have been developed in the last few decades to address
this challenging problem. While the conventional methods such as JPEG [5] for
images, HEVC [6] for videos excel at encoding signals in their respective
domains, coordinate-based implicit neural representations (INR) or Neural
Fields [7] have emerged as a popular alternative for representing complex
signals because of their ability to capture high frequency details, and
adaptability for diverse domains. INRs are typically multi layer perceptrons
(MLPs) optimized to learn a scalar or vector field. They take a coordinate
(location and/or time) as input and predict a continuous signal value(s) as
output (such as pixel color/occupancy). Recently, various works have adapted
INRs to represent a variety of signals such as audio [8], images [9, 10, 11,
12], videos [13, 14], shapes [15, 16], and radiance fields [3, 17].
Unsurprisingly, several methods have been proposed to compress INRs using
quantization [18, 10, 14], pruning, or a combination of both [13]. The focus
of these works is to compress the weights of the MLP, which often leads to
either a big drop in the reconstruction quality, or slow convergence for high
resolution signals.

In this work, we consider a different class of INR approaches that employ
learnable multi-resolution feature grids [19, 1]. These feature grids store
feature vectors at different coordinate locations with varying Level-Of-Detail
(LOD). The features from different levels (or resolutions) are concatenated
and passed through a tiny MLP to reconstruct the required signal. This shifts
the burden of representing the signal to the feature grid instead of the MLP.
Such methods have shown to be effective in approximating complex signals such
as 3D scenes and gigapixel images with high fidelity [1] and fast training
time (since the cost of lookup is very small). However, the size of the
feature grids can be very large which is not memory efficient and impractical
for many real-world applications with network bandwidth or storage
constraints.

We propose an end-to-end learning framework for compressing such feature grids
without any loss in reconstruction performance. Our feature grid consists of
quantized feature vectors and parameterized decoders which transform the
feature vectors into continuous values before passing them to MLP. We use an
entropy regularization loss on the latent features to reduce the size of the
discrete latents without significantly affecting the reconstruction
performance. To address the discretization gap inherent to this discrete
optimization problem, we employ an annealing approach to the discrete latents
which improves the training stability of the latents, converging to better
minima. Both entropy regularization and reconsutruction objective can be
trained jointly in an end-to-end manner without requiring post-hoc
quantization, pruning or finetuning stages. Further, the hierarchical nature
of feature grids allows scaling to high dimensional signals unlike pure MLP-
based implicit methods.

As seen in Figure 1, the proposed approach is able to compress feature-grid
methods such as Instant-NGP [1] with almost an order of magnitude while
retaining the performance in terms of PSNR for gigapixel images and 3D scenes
from the RTMV dataset [20]. We also conduct extensive quantitative experiments
and show results on standard image compression benchmarks such as the Kodak
dataset outperforming the classic JPEG codec as well as other implicit methods
in the high compression regime. Our approach can even be trivially extend to
videos, performing competitively with video-specific INR methods such as NeRV
[13], without explicitly exploiting the inherent temporal redundancy present
in videos. The key contribution of our work is to directly compress the
learnable feature grid with proposed entropy regularization loss and highlight
its adaptibility to diverse signals. We summarize our contributions below:

  * •

We introduce an end-to-end trainable compression framework for implicit
feature grids by maintaining discrete latent representations and parameterized
decoders.

  * •

We provide extensive experiments on compression benchmarks for a variety of
domains such as images, videos, and 3D scenes showing the generalizability of
our approach outperforming a variety of INR works.

Figure 2: Overview of our approach: We maintain latent representations which
are quantized and decoded using parameterized decoders to obtain a hash
table/feature-grid at different levels. We then index the input coordinate
into the hash table to obtain feature vectors. The feature vectors are then
concatenated and passed through an MLP to obtain the output signal.

##  2 Related work

Learned image/video compression: A large number of neural compression works
for images consist of an autoencoder framework [21, 22] which transform/encode
a data point to a latent code and decode the quantized latent to obtain a
reconstruction of the data signal. The autoencoder is typically trained in an
end-to-end fashion on a large training dataset by minimizing a rate-distortion
objective. Numerous extensions to these works introduce other improvements
such as hyperpriors [23], autoregressive modeling [24], Gaussian mixture
likelihoods and attention modules [25], improved inference [26]. Another set
of works extends this framework for videos as well [27, 28, 29, 30] exploiting
the inherent temporal redundancy. These approaches achieve impressive
compression results outperforming classical codecs in their respective
domains. In contrast, we focus on a different class of works involving
implicit neural representations (INRs) which overfit a network to each
datapoint and store only the network weights to achieve data compression.

INRs and application to data compression: INRs [31] are a rapidly growing
field popularized for representing 3D geometry and appearance [15, 3, 32, 33]
and have since been applied to a wide variety of fields such as GANs [34, 35],
image/video compression [9, 10, 13, 14, 18], robotics [36] and so on. Meta-
learning on auxiliary datasets has been shown to provide good initializations
and improvements in reconstruction performance while also greatly increasing
convergence speed for INRs [16, 11, 18]. Our approach can similarly benefit
from such meta-learning stages but we do not focus on it and rather highlight
the effectiveness of our approach to compress feature-grid based INRs and its
advantages over MLP-based INRs. Perhaps the closest work to our approach is
that of VQAD [4] which performs a vector quantization of these feature grids
learning a codebook/dictionary and its mapping to the feature grid in 3D
domain. They however learn a fixed-size codebook without any regularization
loss and do not perform well for high-fidelity reconstructions as we discuss
in Section 4.

Model compression: As INRs represent data as neural networks, they transform
the data compression problem to a model compression one. Many works exist for
model compression involving pruning for achieving high levels of sparsity [37,
38, 39, 40] or quantization for reducing the number of bits necessary [41, 42,
43]. Another line of works [44, 45] perform compression similar to [23] using
quantized latents with entropy regularization losses. These methods, however,
are specific to convolutional networks and are not trivially extensible to
compress INRs.

##  3 Approach

Our goal is to simultaneously train and compress feature-grid based implicit
neural networks. Section 3.1 provides a brief overview of feature-grid INRs
proposed in [1]. Section 3.2 describes various components of our approach
while Section 3.3 discusses compressing feature grids. Our approach for end-
to-end feature grid compression is outlined in Equation 9 and also illustrated
in Figure 2.

###  3.1 Feature-grid INRs

INRs or neural fields [7] typically learn a mapping
$g_{\phi}(\mathbf{x}):\mathbb{R}^{d}\rightarrow~{}\mathbb{R}^{c}$ where $g$ is
an MLP with parameters $\phi$. Input $\mathbf{x}\in\mathbb{R}^{d}$ to the MLP
are $d$ dimensional coordinates, where $d=2$ in the case of images, $3$ in the
case of videos, or $5$ in the case of radiance fields. $c$-dimensional output
can represent RGB colors or occupancy in space-time for the given input
coordinates. Such methods are able to achieve high-quality reconstructions,
however, suffer from long training times and scalability to high-resolution
signals. A few works have suggested utilizing fixed positional encodings of
input coordinates [46] to be able to reconstruct complex high-resolution
signals but the training time of these networks still poses a big challenge.
To alleviate this issue, [1, 19] proposed storing the input encodings in the
form of a learnable feature grid. The feature grid allows for significant
speed-up in the training time by replacing a large neural network with a
multi-resolution look-up table of feature vectors and a much smaller neural
network. We now provide a brief overview of Instant-NGP [1].

For ease of explanation, for the rest of this section, we will assume input is
a 2D signal $\mathbf{x}\in\mathbb{R}^{2}$. However, all the techniques we
discuss can be directly applied to the 3D case. In this framework, we
represent the feature grid by a set of parametric embeddings $\mathbf{Z}$.
$\mathbf{Z}$ is arranged into $L$ levels representing varying resolutions or
Levels-Of-Detail (LOD). More formerly, each level has its own embedding matrix
$\mathbf{Z}_{l}$, and hence
$\mathbf{Z}=\\{\mathbf{Z}_{1},\dots,\mathbf{Z}_{l}\\}$. The number of feature
vectors in the embedding matrix $\mathbf{Z}_{l}$ depends on the resolution of
the level $R_{l}$. For coarser resolutions, we allow $\mathbf{Z}_{l}$ to
consist of $R_{l}^{2}$ rows, but for finer resolutions, we cap the maximum
number of rows in the matrix to $T$. Hence for $\mathbf{x}\in\mathbb{R}^{2}$,

$\mathbf{Z}_{l}\in\mathbb{R}^{T_{l}\times F},\textnormal{ where
}T_{l}=\textnormal{min}(R_{l}^{2},T)$ (1)

Here $F$ is the dimension of feature vectors and is kept fixed for all levels.
For a given input $\mathbf{x}$, we can obtain the 4 closest corner indices
$\\{tl,tr,bl,br\\}$ within the $R_{l}\times R_{l}$ grid. Each of the corner
index maps to an entry in $\mathbf{Z}_{l}$. This mapping is direct when
$R_{l}^{2}\leq T$ and uses a hashing function [47] otherwise. The feature
vector $\mathbf{f}_{l}$ at level $l$ for the input $\mathbf{x}$ is then
obtained by a simple bilinear interpolation of feature vectors of the corner
indices, _i.e_.

$\mathbf{f}_{l}(\mathbf{x})=\text{interp}(\mathbf{Z}_{l}[tl],\mathbf{Z}_{l}[tr],\mathbf{Z}_{l}[bl],\mathbf{Z}_{l}[br])$
(2)

Note that in the case of 3D input, we consider 8 closest corner indices, and
perform a trilinear interpolation. $\mathbf{f}_{l}(\mathbf{x})$ is
concatenated across different levels to obtain the overall feature vector
$\mathbf{f}(\mathbf{x})\in\mathbb{R}^{FL}$, which is passed as input to the
neural network $g_{\phi}$.

$\widehat{\textbf{y}}=g_{\phi}(\text{concat}[\mathbf{f}_{1}(\mathbf{x}),\dots,\mathbf{f}_{L}(\mathbf{x})])$
(3)

Here $\widehat{\textbf{y}}$ is the final prediction of INR for the input
$\mathbf{x}$. Since the concatenation, indexing, and interpolation operations
are differentiable, parameters $\\{\phi,\mathbf{Z}\\}$ can be optimized using
any gradient based optimizer by minimizing a loss function
$\mathcal{L}(\widehat{\textbf{y}},\textbf{y})$ between the predicted
$\widehat{\textbf{y}}$ and ground-truth signal y. $\mathcal{L}$ can be any
differentiable loss function such as the Mean Squared Error (MSE). The MLP is
typically very small in comparison to MLP-based INRs. Thus, $\phi$ consists of
far fewer parameters than $\mathbf{Z}$. Such an INR design allows for much
faster training and inference as the cost of indexing into the feature grid is
quite small.

###  3.2 Feature-grid reparameterization

While feature-grid INRs can converge faster than pure-MLP approaches, the
space required to store all the parameters at different levels can rise very
rapidly if we want high-fidelity reconstructions for high-resolution inputs.
This makes them unsuitable for resource-constrained applications. We thus aim
to reduce the storage size of these feature grids. To this effect, we propose
to maintain discrete or quantized latent representations
$\mathbf{Q}_{l}\in\mathbb{R}^{T_{l}\times D}$ for each embedding
$\mathbf{Z}_{l}\in\mathbb{R}^{T_{l}\times F}$. The latents, consisting of only
integer values, can be of any dimension $D$ with a larger $D$ allowing for
greater representation power at the cost of storage size.

In order to map these discrete latent features $\mathbf{Q}_{l}$ to the
continuous features in the embedding table $\mathbf{Z}_{l}$, we propose a
parameterized decoder $d_{\theta}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{F}$.
While it is possible to use separate complex decoders for each level, in
practice, we observed that a single shared decoder parameterized as a linear
transform across all L levels works pretty well and has a minimal impact on
the training/inference times and fidelity of reconstructions.

Note that the quantized latents $\mathbf{Q}$ are no more differentiable (here
we dropped the subscript $l$ without loss of generality for notational
simplicity). In order to optimize these quantized latents, we maintain a
continuous proxy parameters $\widehat{\mathbf{Q}}$ of the same size as
$\mathbf{Q}$. $\mathbf{Q}$ is obtained by rounding $\widehat{\mathbf{Q}}$ to
the nearest integer. To make this operation differentiable, we utilize the
Straight-Through Estimator [48](STE). In STE, we use quantized weights
$\mathbf{Q}$ during the forward pass, but use the continuous proxies to
propagate the gradients from $\mathbf{Q}$ to $\widehat{\mathbf{Q}}$ during the
backward pass.

STE serves as a simple differentiable approximation to the rounding operation
but leads to a large rounding error $\|\mathbf{Q}-\widehat{\mathbf{Q}}\|$. To
overcome this issue, we utilize an annealing approach [26] to perform a soft
rounding operation. We represent $\mathbf{Q}$ by either rounding up (denoted
as $\lceil.\rceil$) or down (denoted as $\lfloor.\rfloor$)
$\widehat{\mathbf{Q}}$ to the nearest integer. Using one-hot gates
$b\in\\{0,1\\}$, where $b=0$ corresponds to rounding up, $b=1$ corresponds to
rounding down, we can represent
$\mathbf{Q}=b\lfloor\widehat{\mathbf{Q}}\rfloor+(1-b)\lceil\widehat{\mathbf{Q}}\rceil$.
The gate $b$ is sampled from a soft relaxed 2-class distribution

$\displaystyle\text{Prob}(b=0)\propto\exp\left\\{-\operatorname{\tanh^{-1}}{\left(\widehat{\mathbf{Q}}-\lfloor\widehat{\mathbf{Q}}\rfloor\right)}/\tau\right\\}$
$\displaystyle\text{Prob}(b=1)\propto\exp\left\\{-\operatorname{\tanh^{-1}}{\left(\lceil\widehat{\mathbf{Q}}\rceil-\widehat{\mathbf{Q}}\right)}/\tau\right\\}$
(4)

where $\tau>0$ represents the temperature parameter. $\widehat{\mathbf{Q}}$
approaching either $\lfloor\widehat{\mathbf{Q}}\rfloor$ or
$\lceil\widehat{\mathbf{Q}}\rceil$ thus increases the likelihood of sampling
to that respective value. In the beginning of the training, $\tau=1$ and is a
poor approximator of rounding operation but provides more stable gradients. As
training progresses, $\tau$ is annealed towards zero, so that the random
distribution converges to a deterministic one at the end of training. The
gradients are propagated through the samples using the Gumbel
reparameterization trick [49].

###  3.3 Feature-grid compression

To further improve compression levels, we minimize the entropy of the latents
using learnable probability models [23]. We note that our learned latents
$\mathbf{Q}$ or $\widehat{\mathbf{Q}}$ are of dimensions $T\times D$. We can
interpret these latents as consisting of $T$ samples from a discrete $D$
dimensional probability distribution. We introduce $D$ probability models one
for each latent dimension $d\in\\{1,\dots,D\\}$,
$P_{d}:\mathbb{R}\rightarrow[0,1]$. We discuss the exact form of $P_{d}$ in
the supplementary material. With these probability models, we can minimize the
length of a bit sequence encoding $\widehat{\mathbf{Q}}$ by minimizing the
self-information loss or the entropy of $\widehat{\mathbf{Q}}$ [50]

$\mathcal{L}_{I}(\widehat{\mathbf{Q}})=-\frac{1}{T}\sum_{d=1}^{D}\sum_{i=1}^{T}\textnormal{log}_{2}\left(P_{d}\left(\widehat{\mathbf{Q}}[i,d]+n\right)\right)$
(5)

where $n\sim\mathcal{U}[-1,1]$ represents the uniform random distribution to
approximate the effects of quantization, and $\mathcal{L}_{I}$ is the self-
information loss.

###  3.4 End-to-end optimization

We provide an overview of our approach in Fig. 2. We maintain continuous
learnable latents $\widehat{\mathbf{Q}}$. The proposed annealing approach
(Sec. 3.1) progressively converges the continuous $\widehat{\mathbf{Q}}$ to
the discrete $\mathbf{Q}$. The approach is made differentiable using the
gumbel reparameterization trick and straight-through estimator. $\mathbf{Q}$
is passed through the decoder $d_{\theta}$ with parameters $\theta$ to obtain
the feature grid table $\mathbf{Z}$. We then index $\mathbf{Z}$ at different
levels/resolutions using the coordinates x to obtain a concatenated feature
vector $\mathbf{f}$ which is passed through an MLP $g_{\phi}$ to obtain the
predicted signal $\widehat{\textbf{y}}$.

$\displaystyle\mathbf{Q}$
$\displaystyle=\text{discretize}(\widehat{\mathbf{Q}})$ (6)
$\displaystyle\mathbf{Z}$ $\displaystyle=d_{\theta}(\mathbf{Q})$ (7)
$\displaystyle\widehat{\mathbf{y}}$
$\displaystyle=g_{\phi}\left(\text{concat}\left[\text{interp}(\mathbf{Z},\mathbf{x})\right]\right)$
(8)

Our framework is thus fully differentiable in the forward and backward pass.
For a given signal $\mathbf{y}$ and its corresponding input coordinate grid
$\mathbf{x}$, we optimize the parameters $\phi$ of MLP $g_{\phi}$, discrete
feature grid $\widehat{\mathbf{Q}}$, discrete to continuous decoder $\theta$,
and the probability models $\\{P_{d}\\}$ in an end-to-end manner by minimizing
the following rate distortion objective

$\mathcal{L}_{\textnormal{MSE}}(\widehat{\mathbf{y}},\textbf{y})+\lambda_{I}\mathcal{L}_{I}(\widehat{\mathbf{Q}})$
(9)

where $\lambda_{I}$ controls the rate-distortion optimization trade-off. Post
training, the quantized latents are stored losslessly using algorithms such as
arithmetic coding [51] utilizing the probability tables from the density
models $\\{P_{d}\\}$.

##  4 Experiments

We apply our INR framework to images, videos, and radiance fields. We outline
our experimental setup in Sec. 4.1. Sec. 4.2, Sec. 4.3, and Sec. 4.4 provide
results on compression for images, radiance fields, and videos respectively.
Sec. 4.5 illustrates the application of our approach for progressive
streaming. Sec. 4.6 discusses the convergence speeds of our approach. Sec. 4.7
ablates the effect of entropy regularization and annealing. Additional
experiments and ablations are provided in the supplementary material.

###  4.1 Experimental details and setup

We show image compression performance on the Kodak dataset [52] consisting of
24 images of resolution $512\times 768$. To show our scalability to larger
images, we provide results on higher resolution images varying from 2M pixels
to 1G pixels. We use the Kaolin-Wisp library [53] for our experiments on
radiance fields. In addition to the Lego scene in Fig. 1, we provide results
on 10 bricks scenes from the RTMV dataset [20] which contains a wide variety
of complex 3D scenes. For videos, we benchmark on the UVG dataset [54]
consisting of seven 1080p resolution videos consisting of 600 frames each at
120fps. Additionally, we extract the first frame from these 7 videos to create
a 7-image dataset, UVG-F, for benchmarking image compression at $1080\times
1920$ resolution against other popular INR compression approaches. We
primarily measure distortion in terms of the PNSR (dB) and rate in terms of
Bits-Per-Pixel (BPP) (or model size for neural fields).

We fix the minimum grid resolution to 16 and vary the number of per-level
entries T, the number of levels L, and the maximum grid resolution for various
modalities to control the rate/distortion trade-off. We fix the number of
hidden MLP layers to 2 with the ReLU activation. We fix the batch size (number
of coordinates per iteration) to be $2^{18}$ for all our experiments except
Kodak where we pass the full coordinate grid. The entropy regularization
parameter is set to $1.0e^{-4}$ for all our experiments unless specified
otherwise. The parameters are optimized using the Adam optimizer [55]. We
provide additional experimental details in the supplementary material.

Figure 3: Comparison of our approach on the Kodak image dataset with classical
(dashed), RDAE (dotted), INR (solid) methods. We outperform state-of-the-art
INR approaches bridging the gap to classical and RDAE methods. Table 1: Image
compression at varying resolutions. We compare against implicit network
methods of INGP [1], Positional [18], SIREN [8], and the auto-encoder based
work, RDAE [23]. We achieve high values of PSNR, maintaining similar quality
reconstructions as INGP while requiring far fewer bits ($4-9{\times}$).

Image |  Method |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  BPP $\downarrow$
---|---|---|---|---
UVG-F ($1920\times 1080$) |  Positional [18] |  33.17 |  0.86 |  1.52
|  JPEG [5] |  36.98 |  0.91 |  0.76
|  RDAE [23] |  34.23 |  0.93 |  0.76
|  Ours |  37.74 |  0.92 |  0.76
SMACS ($4630\times 4537$) |  INGP [1] |  34.61 |  0.86 |  0.18
|  JPEG [5] |  34.77 |  0.86 |  0.18
|  RDAE [23] |  34.06 |  0.89 |  0.40
|  Ours |  34.90 |  0.86 |  0.04
Cosmic-Cliffs ($8441\times 14575$) |  INGP [1] |  38.78 |  0.96 |  0.63
|  SIREN [8] |  27.32 |  0.90 |  0.14
|  JPEG [5] |  38.38 |  0.95 |  0.29
|  RDAE [23] |  37.90 |  0.97 |  0.38
|  Ours |  38.79 |  0.96 |  0.11
Pearl ($23466\times 20000$) |  INGP [1] |  29.62 |  0.84 |  1.00
|  JPEG [5] |  29.10 |  0.84 |  0.29
|  Ours |  29.44 |  0.84 |  0.12
Tokyo ($21450\times 56718$) |  INGP [1] |  31.87 |  0.82 |  0.39
|  JPEG [5] |  31.16 |  0.82 |  0.18
|  Ours |  31.62 |  0.82 |  0.06

Figure 4: Qualitative results on Kodak and UVG-F: We obtain much higher
quality reconstructions capturing finer detail compared to [18] or JPEG.
Notice the blocking and discoloration artifacts present in JPEG which are
significant at lower BPP values. Figure 5: Compression result visualization on
the Cosmic-Cliffs image for 4 methods. We obtain similar PSNR and
reconstruction quality as Instant-NGP while ${\sim}6{\times}$ smaller. SIREN
fails to fit high frequency information leading to blurry patches as seen.
JPEG on the other hand suffers from blocking artifacts and discoloration
leading to drop in reconstruction quality. Figure 6: Evaluation on V8 from the
RTMV dataset ($1600\times 1600$ resolution). We obtain ${\sim}60{\times}$
compression compared to Instant-NGP but with a PSNR drop of $1dB$. We
outperform VQAD obtaining higher PSNR at similar model size. We also obtain
much finer reconstructions when compared with VQAD as shown in the zoomed
patches.

###  4.2 Scalable image compression

We visualize results on the Kodak benchmark in Fig. 3. We outperform the MLP-
based INR approaches of COIN [9], COIN++ [10] by a significant margin at all
bitrates. We outperform [18], which utilizes positional encodings and weight
quantization, at higher bitrates while also having much lesser encoding times
(as we show in Sec. 4.6) We show qualitative results on one of the images from
the KODAK dataset in Fig. 4. We obtain higher quality reconstructions
($28.66\rightarrow 34.68$ PSNR) capturing fine details while also requiring
fewer bits for storage ($1.02\rightarrow 0.88$ BPP) as compared to [18]. We
however, observe slightly lower performance in the low BPP regime where
uncompressed MLP weights represent a significant fraction(${\sim}25\%$) of the
total size. We hypothesize that at lower bit rates, compression of MLPs can
achieve further reduction in model size. Our work focuses only on the
compression of feature grids and can potentially benefit from the
complimentary works on MLP compression.

We additionally outperform JPEG for the full range of BPP with a much larger
gap at lower bitrates. Notice the blocking artifacts in Fig. 4 for JPEG while
we achieve consistent reconstructions. However, the classical methods of
JPEG2000, BPG and the autoencoder based work of [23] continue to outperform
our method, especially in the higher BPP regime. Nevertheless, we reduce the
gap between INRs and autoencoder in the low-dimensional image space.

In order to understand the scalability of various image compression methods
with image resolution, we compress images in the UVG-F dataset ($1920\times
1080)$ and 4 images with increasing resolution. Results are summarized in
Table 1. We outperform [18] by a large margin on the UVG-F dataset with an
improvement of over 4.5dB while also requiring $2\times$ fewer bits. This is
also observed in Fig. 4 (right column) where we capture much finer high
frequency details compared to the positional encoding approach of [18]. We
marginally outperform JPEG as well in the high BPP regime which again exhibits
minor blocking artifacts. Perhaps, the most surprising result is the
performance of the Rate-Distortion AutoEncoder (RDAE) based approach [23]
which does not scale very well to large dimensions. We obtain a 3.5dB
improvement in PSNR while maintaining a similar BPP albeit with slightly lower
SSIM score.

For higher image resolutions, we continue to observe negligible drops in
performance compared to INGP [1] while achieving $4{-}9\times$ smaller
bitrates. We qualitatively visualize results on the Cosmic-cliffs image (with
a resolution of $8441\times 14575$) in Fig. 5. We achieve a similar PSNR as
Instant-NGP (38.78 dB) with ${\sim}6{\times}$ reduction in bitrates. MLP-based
INRs such as SIRENs [8] are unable to capture the high frequency detail
resulting in blurry images and achieve low PSNR (27.32 dB). JPEGs also lead to
a large drop in performance in the similar low BPP regime(${\sim}0.15$).

Figure 7: Our multiresolution compressed representations can be transmitted at
varying LODs at inference time (without any retraining) making it suitable for
applications with progressive streaming.

###  4.3 Radiance fields compression

We now turn to the application of SHACIRA to neural radiance fields or NeRFs.
We compare our approach against the baseline Instant-NGP [1], mip-NERF [56]
and VQAD [4], a codebook based feature-grid compression approach. We evaluate
on 10 brick scenes from the RTMV dataset by training each approach for 600
epochs and summarize the results in Table 3. We marginally outperform the
baseline INGP in terms of all the reconstruction metrics of PSNR, SSIM and
LPIPS while also achieving ${\sim}48\times$ compression requiring an average
of only 1MB per scene. We also outperform mip-NERF performing better on PSNR
and SSIM while reducing the storage size. For a better comparison with VQAD
(based off of NGLOD-NERF [19]), we scale down T, the maximum number of hashes.
We see that we obtain $>20\%$ lower model size at 0.43 MB compared to 0.55 MB
of VQAD while slightly better in the reconstruction metrics. We also see a
clear improvement over VQAD for the LEGO NeRF scene visualized in Figure 1.
VQAD has around 1.5dB PSNR drop while still being ${\sim}7{\times}$ larger in
terms of storage size. Additionally, VQAD fails to scale for higher bitwidth
due to memory constraints even on an NVIDIA RTX A100 GPU with 40GB GPU RAM.

To better illustrate the 3 approaches of INGP, VQAD and SHACIRA, we train on
the full resolution V8 scene ($1600~{}\times~{}1600$) for 600 epochs,
visualizing the results in Fig. 6. We outperform VQAD (+1dB) at similar model
size and obtain ${\sim}60\times$ compression compared to INGP, but with 1dB
drop in PSNR. Nevertheless, we reconstruct the scene with fewer artifacts and
consistency in the intricate shapes, compared to VQAD as highlighted in the
patches.

Table 2: Comparison against various video INR approaches on the UVG dataset.
We outperform NIRVANA with higher PSNR and lower BPP while obtaining slightly
lower PSNR than NeRV at $3\times$ reduction in model size.

Method |  Encoding Time $\downarrow$ |  PSNR $\uparrow$ |  BPP $\downarrow$
---|---|---|---
SIREN |  15 hours |  27.20 |  0.28
NeRV |  3.5 hours |  35.54 |  0.66
NIRVANA |  4 hours |  34.71 |  0.32
Ours |  6.5 hours |  35.01 |  0.21

Table 3: Comparison of various methods on scenes in the RTMV dataset. We
marginally outperform the baseline Instant-NGP in terms of all metrics while
also achieving a $48\times$ compression.

Method |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Storage $\downarrow$
---|---|---|---|---
NeRF |  28.28 |  0.9398 |  0.0410 |  2.5MB
mip-NERF |  31.61 |  0.9582 |  0.0214 |  1.2MB
NGLOD-NERF |  32.72 |  0.9700 |  0.0379 |  $\approx$20MB
VQAD |  31.45 |  0.9638 |  0.0468 |  0.55MB
Ours |  31.46 |  0.9657 |  0.0428 |  0.43MB
Instant-NGP |  31.88 |  0.9690 |  0.0381 |  48.9MB
Ours |  32.14 |  0.9704 |  0.0348 |  1.03MB

###  4.4 Video compression

Next, we apply our approach to video compression as well. As a baseline, we
compare against SIREN a coordinate-based INR. We also compare against NeRV
[13], a video INR based approach which takes a positional encoding as input
and predicts frame-wise outputs for a video. We also compare against another
video INR, NIRVANA [14], which is an autoregressive patch-wise prediction
framework for compressing videos. We run the baselines on the UVG dataset,
with the results shown in Table 2.

We outperform SIREN by a significant margin with an almost $+7dB$ gain in PSNR
and $25\%$ lesser BPP and shorter encoding times. This is to be expected as
SIREN fails to scale to higher dimensional signals such as videos usually with
more than $10^{9}$ pixels. We also outperform NIRVANA achieving higher PSNR
and lower BPP albeit at longer encoding times. Compared to NeRV, we obtain a
0.5dB PSNR drop but achieve $3{\times}$ compression in model size. We would
like to add that our current implementation utilizes PyTorch and the encoding
time can be reduced significantly using fully fused CUDA implementation ([1]
demonstrated that efficient lookup of the hashtables can be an order of
magnitude faster than a vanilla Pytorch version). Additionally, our approach
is orthogonal to both baselines and provides room for potential improvements.
For instance, compressed multi-resolution feature grids can be used to replace
positional embedding for NeRV as well as coordinate-based SIREN, and provide
faster convergence and better reconstruction for high dimensional signals.

###  4.5 Streaming LOD

[4] show the advantage of feature-grid based INRs for progressive streaming at
inference time due to their multi-resolution representation capabilities.
Since our compression framework consists of latents at different LODs as well,
they can be progressively compressed with varying LOD yielding reconstructions
with different resolution. Formally, for
$\mathbf{Q}~{}{=}~{}\\{\mathbf{Q}_{1},\mathbf{Q}_{2},...\mathbf{Q}_{L}\\}$ we
can reconstruct the signal at LOD $l$ by only passing the first $l$ latents
while masking out the finer resolutions. This can be applied directly during
inference without any additional retraining. We visualize the effect of such
progressive streaming in Fig. 7. We obtain better reconstruction with
increasing bitrates or the latent size. This is especially beneficial for
streaming applications as well as for easily scaling the model size based on
the storage or bandwidth constraints.

Figure 8: Comparison of convergence speeds of our feature-grid approach with
[18] utilizing MLP-based INRs. We obtain higher PSNR at a much faster rate for
a similar BPP range.

###  4.6 Convergence speeds

We now compare the convergence speeds of our feature-grid based approach with
that of [18] which is an MLP-based INR with positional encoding. We summarize
the results for an image in the Kodak dataset in Fig.8. We measure encoding
times on an NVIDIA RTX A6000 GPU for the full length of the training for both
approaches. We obtain higher PSNR at a much faster rate than [18] at a similar
BPP range (hue value in color bar). While [18] reduces BPP (0.84 at 600s) with
higher encoding times, their PSNR remains stagnant at 32.5dB. In contrast, our
approach achieves this PSNR and BPP (0.85) within just 53s achieving more than
a $10\times$ speedup in convergence. Additionally, we achieve higher PSNR with
longer encoding times reaching 34dB in 430s while maintaining a similar BPP
(0.87).

###  4.7 Effect of entropy regularization and annealing

In this section, we analyze the effect of entropy regularization and
annealing. We pick the Jockey image from UVG-F ($1080\times~{}1920$) for our
analysis. We set the default values of latent and feature dimensions to $1$.
For the analysis, we compare trade-off curves by increasing the number of
entries from $2^{13}$ to $2^{17}$ in multiples of 2 which naturally increases
the number of parameters and also the PSNR and BPP. Note that better trade-off
curves indicate shifting upwards (higher PSNR) and to the left (lower BPP).

Figure 9 shows the effect of entropy regularization. The absence of it,
corresponding to a value of $0.0$ shows a drop in performance compared to
values of $1.0e^{-4}$ and higher. We see that the network performance is
fairly stable in this range with much higher values of $4.0e^{-4}$ showing
small drops. This shows that entropy regularization using the defined
probability models helps in improving the PSNR-BPP trade-off curve by
decreasing entropy (or model size/BPP) with no drop in network performance
(PSNR).

Figure 10 shows the effect of annealing. We vary the duration of annealing as
a fraction of the total training duration. No annealing corresponds to the
value $0.0$ and has a large drop in the trade-off curve compared to higher
values. This shows that annealing performs better than standard STE (Straight
Through Estimator) alone and is an important component of our approach for
quantizing the feature grid parameters. Increasing the period of annealing
shows consistent improvements in the trade-off curve.

Figure 9: Effect of entropy regularization. In absence of entropy
regularization, corresponding to the value $0.0$, there is a drop in the PSNR-
BPP trade-off curve. Figure 10: Effect of annealing. Increasing the fractional
period of annealing improves the trade-off curve highlighting its importance
for quantization approximation compared to stand-alone STE (Straight Through
Estimator) with an annealing period of 0.0.

##  5 Conclusion

We proposed SHACIRA, a general purpose, end-to-end trainable framework for
learning compressed INRs or neural fields for images, videos, and NeRFs. Our
method adds a minimal overhead over feature-grid based methods of training
INRs. We make this possible by parameterizing the feature grid of INRs with
discrete latents and a decoder. Applying entropy regularization on discrete
latents ensures that we can learn and maintain compressed quantized weights. A
tiny decoder allows us to still operate in the continuous space for training
and sampling with high fidelity. Our method requires no posthoc training or
quantization and can be easily added to existing INR pipelines. We conduct an
extensive evaluation on images, videos, and 3D datasets to show the efficacy
of our approach in these domains. Compared to MLP-based INRs, we scale well
for high resolution signals capturing high frequency details and converge
faster. An additional benefit of our approach is that it allows reconstruction
of the signal at different levels-of-details without retraining, which is
especially beneficial for streaming applications. While the compressed network
is memory efficient, it does not offer any inference speedups compared to the
uncompressed network and is an interesting avenue for further research. Meta
learning the feature grid for even faster convergence is another direction for
future work.

## References

  * [1] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  * [2] David S Taubman and Michael W Marcellin.  Jpeg2000: Standard for interactive imaging.  Proceedings of the IEEE, 90(8):1336–1357, 2002.
  * [3] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [4] Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler.  Variable bitrate neural fields.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  * [5] Gregory K Wallace.  The jpeg still picture compression standard.  IEEE transactions on consumer electronics, 38(1):xviii–xxxiv, 1992\.
  * [6] Vivienne Sze, Madhukar Budagavi, and Gary J Sullivan.  High efficiency video coding (hevc).  In Integrated circuit and systems, algorithms and architectures, volume 39, page 40. Springer, 2014.
  * [7] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  Computer Graphics Forum, 2022.
  * [8] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  * [9] Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet.  Coin: Compression with implicit neural representations.  arXiv preprint arXiv:2103.03123, 2021.
  * [10] Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliński, Yee Whye Teh, and Arnaud Doucet.  Coin++: Data agnostic neural compression.  arXiv preprint arXiv:2201.12904, 2022.
  * [11] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng.  Learned initializations for optimizing coordinate-based neural representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846–2855, 2021.
  * [12] Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker.  Modulated periodic activations for generalizable local functional representations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14214–14223, 2021.
  * [13] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava.  Nerv: Neural representations for videos.  Advances in Neural Information Processing Systems, 34:21557–21568, 2021.
  * [14] Shishira R Maiya, Sharath Girish, Max Ehrlich, Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxiang Wu, Chen Wang, and Abhinav Shrivastava.  Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling.  arXiv preprint arXiv:2212.14593, 2022.
  * [15] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [16] Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  Metasdf: Meta-learning signed distance functions.  Advances in Neural Information Processing Systems, 33:10136–10147, 2020.
  * [17] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  Advances in Neural Information Processing Systems, 33:15651–15663, 2020.
  * [18] Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari.  Implicit neural representations for image compression.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 74–91. Springer, 2022.
  * [19] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3d shapes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358–11367, 2021.
  * [20] Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, and Stan Birchfield.  Rtmv: A ray-traced multi-view synthetic dataset for novel view synthesis.  IEEE/CVF European Conference on Computer Vision Workshop (Learn3DG ECCVW), 2022, 2022.
  * [21] Johannes Ballé, Valero Laparra, and Eero P Simoncelli.  End-to-end optimized image compression.  arXiv preprint arXiv:1611.01704, 2016.
  * [22] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár.  Lossy image compression with compressive autoencoders.  arXiv preprint arXiv:1703.00395, 2017.
  * [23] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston.  Variational image compression with a scale hyperprior.  arXiv preprint arXiv:1802.01436, 2018.
  * [24] David Minnen, Johannes Ballé, and George D Toderici.  Joint autoregressive and hierarchical priors for learned image compression.  Advances in neural information processing systems, 31, 2018.
  * [25] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto.  Learned image compression with discretized gaussian mixture likelihoods and attention modules.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7939–7948, 2020.
  * [26] Yibo Yang, Robert Bamler, and Stephan Mandt.  Improving inference for neural image compression.  Advances in Neural Information Processing Systems, 33:573–584, 2020\.
  * [27] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao.  Dvc: An end-to-end deep video compression framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11006–11015, 2019.
  * [28] Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomczak, and Taco S Cohen.  Video compression with rate-distortion autoencoders.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7033–7042, 2019.
  * [29] Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici.  Scale-space flow for end-to-end optimized video compression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503–8512, 2020.
  * [30] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte.  Learning for video compression with hierarchical quality and recurrent enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6628–6637, 2020.
  * [31] Kenneth O Stanley.  Compositional pattern producing networks: A novel abstraction of development.  Genetic programming and evolvable machines, 8:131–162, 2007.
  * [32] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939–5948, 2019.
  * [33] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019.
  * [34] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628–8638, 2021.
  * [35] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34:852–863, 2021\.
  * [36] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.  Neural descriptor fields: Se (3)-equivariant object representations for manipulation.  In 2022 International Conference on Robotics and Automation (ICRA), pages 6394–6400. IEEE, 2022.
  * [37] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  arXiv preprint arXiv:1803.03635, 2018.
  * [38] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin.  Stabilizing the lottery ticket hypothesis.  arXiv preprint arXiv:1903.01611, 2019.
  * [39] Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry S Davis, and Abhinav Shrivastava.  The lottery ticket hypothesis for object recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 762–771, 2021.
  * [40] Trevor Gale, Erich Elsen, and Sara Hooker.  The state of sparsity in deep neural networks.  arXiv preprint arXiv:1902.09574, 2019.
  * [41] Xiaofan Lin, Cong Zhao, and Wei Pan.  Towards accurate binary convolutional neural network.  Advances in neural information processing systems, 30, 2017.
  * [42] Frederick Tung and Greg Mori.  Clip-q: Deep network compression learning by in-parallel pruning-quantization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7873–7882, 2018.
  * [43] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin.  Training with quantization noise for extreme model compression.  arXiv preprint arXiv:2004.07320, 2020.
  * [44] D. Oktay et al.  Scalable model compression by entropy penalized reparameterization.  In ICLR, 2020.
  * [45] Sharath Girish, Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava.  Lilnetx: Lightweight networks with extreme model compression and structured sparsification.  ArXiv, abs/2204.02965, 2022.
  * [46] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [47] Matthias Teschner, Bruno Heidelberger, Matthias Müller, Danat Pomerantes, and Markus H Gross.  Optimized spatial hashing for collision detection of deformable objects.  In Vmv, volume 3, pages 47–54, 2003.
  * [48] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [49] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  arXiv preprint arXiv:1611.01144, 2016.
  * [50] Claude E Shannon.  A mathematical theory of communication.  The Bell system technical journal, 27(3):379–423, 1948.
  * [51] Ian H Witten, Radford M Neal, and John G Cleary.  Arithmetic coding for data compression.  Communications of the ACM, 30(6):520–540, 1987.
  * [52] Kodak lossless true color image suite.  http://r0k.us/graphics/kodak/8.
  * [53] Towaki Takikawa, Or Perel, Clement Fuji Tsang, Charles Loop, Joey Litalien, Jonathan Tremblay, Sanja Fidler, and Maria Shugrina.  Kaolin wisp: A pytorch library and engine for neural fields research.  https://github.com/NVIDIAGameWorks/kaolin-wisp, 2022.
  * [54] Alexandre Mercat, Marko Viitanen, and Jarno Vanne.  Uvg dataset: 50/120fps 4k sequences for video codec analysis and development.  In Proceedings of the 11th ACM Multimedia Systems Conference, pages 297–302, 2020.
  * [55] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint:1412.6980, 2014.
  * [56] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [57] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.

##  Appendix A Probability models

We define the probability models similar to [23]. The underlying probability
density of the latents $\widehat{\mathbf{Q}}$ is defined by its Cumulative
Density Function (CDF) $c:\mathbb{R}\rightarrow[0,1]$, with the constraints:

$c(-\infty)=0,\quad c(\infty)=1,\quad\frac{\partial c(x)}{\partial x}\geq 0$
(10)

This is represented using MLPs which take in a real valued scalar and output a
CDF value between 0 and 1. Each dimension in $\widehat{\mathbf{Q}}$ is
represented by a separate model. To satisfy the monotonicity constraint, [23]
use a combination of tanh and softplus activations for each layer of the MLP.
A sigmoid activation is used at the final layer to constrain the CDF between 0
and 1. To model the true distribution, the standard uniform distribution
$n\sim\mathcal{U}[-0.5,0.5]$ is convolved with the density model to derive the
Probability Mass Function (PMF) of the latents as

$P_{d}(x)=c\left(\widehat{\mathbf{Q}}_{d}+\frac{1}{2}\right)-c\left(\widehat{\mathbf{Q}}_{d}-\frac{1}{2}\right)$
(11)

The entropy regularization loss is then the self information loss given by

$\mathcal{L}_{I}(\widehat{\mathbf{Q}})=-\frac{1}{T}\sum_{d=1}^{D}\sum_{i=1}^{T}\textnormal{log}_{2}\left(P_{d}(x)\right)$
(12)

##  Appendix B Experimental settings

The latent dimension is set to 1 for images and video and 2 for 3D
experiments. The feature dimension obtained after decoding the latents is set
to 1 for Kodak and UVG-F, 2 for high resolution (giga-pixel) images and UVG
for videos, and 4 for 3D experiments. Since we do not compress MLP weights but
include their floating-point size in our PSNR-BPP tradeoff calculation, we
vary the hidden dimension according to the signal resolution. For images, we
set the dimension to be 16 for Kodak, 48 for UVG-F and SMACS, and 96 for the
remaining high-resolution images. For videos and 3D, we set the layer size to
128. The number of layers is fixed to 2 for all cases. Note that higher layer
size generally leads to better PSNR at the cost of a proportionally higher BPP
but further gains can be obtained by compressing these weights as well. This
is orthogonal to our direction of compression of the feature grid itself.

For all our experiments, we initialize the decoder parameters with a normal
distribution $\mathcal{N}(0,0.1)$, latents with $\mathcal{U}(-0.01,0.01)$, MLP
weights with the Xavier initialization [57], and probability model parameters
as in [23].

We use the Adam optimizer for jointly optimizing all network parameters. We
set the learning rate of MLP parameters to be $1.0e^{-3}$ for Kodak and
$5.0e^{-4}$ for all other experiments. The learning rate for the probability
models is fixed at $1.0e^{-4}$ following [45]. The decoder learning rate is
set to $0.01$ for images, video and $0.1$ for 3D experiments. We set the
learning rate of the latents to be $0.01$ for Kodak, UVG-F, and UVG, $0.05$
for 3D, and $0.1$ for the other higher-resolution images. We observe that the
training is not very susceptible to variations in the initialization or
learning rates and the values are obtained from a coarse search for each
signal domain.

##  Appendix C Sensitivity Analysis

In this section, we analyze the effect of various components of our pipeline.
We pick the Jockey image ($1080\times 1920$) from UVG-F as the benchmark for
our analysis. We set the default values of entropy regularization to
$1.0e^{-4}$, latent and feature dimensions to $1$ each, MLP width to $48$, MLP
depth to $2$, and annealing period to $0.95$. Each subsection below analyzes
varying a single parameter while keeping others fixed to their default values.
For the analysis, we compare tradeoff curves by increasing the number of
entries from $2^{13}$ to $2^{17}$ in multiples of 2 which provides a natural
way of increasing the number of parameters and subsequently the PSNR and BPP.
Note that better tradeoff curves indicate shifting upwards (higher PSNR) and
to the left (lower BPP).

|  |
---|---|---
(a) |  (b) |  (c)
Figure 11: Effect of latent and feature dimension and number of LODs. Increasing the latent dimension leads to a much higher size at smaller increases in PSNR. Optimal value of feature dimension (or decoded latents) is 1 or 2. Increasing the number of LODs shifts the curve upwards and to the right yielding no difference in tradeoff curves. (a) |  (b)
---|---
Figure 12: Effect of MLP Width and Depth. Increasing MLP hidden layer size
(width) is detrimental due to a large increase in uncompressed parameters (and
BPP) with a small increase in PSNR. Increasing the number of hidden layers
beyond 2 has little effect on the tradeoff curve.

###  C.1 Effect of Latent and Feature Dimension

We ablate the number of LODs, the number of latent dimensions, and feature
dimensions after decoding the latents. Results are shown in Fig.11. We see
that increasing the feature dimension from 1 to 2 does not significantly alter
the tradeoff curve while increasing it to 4 leads to a small drop in
performance. On the other hand, increasing the latent dimension has a strong
impact on the tradeoff curve as it directly impacts the number of latents
entropy encoded. BPP visibly increases with higher latent dimension but leads
to only modest gains in PSNR. Increasing the number of LODs shifts the curve
upwards and to the right but has no overall impact in terms of improving PSNR
for a fixed BPP.

###  C.2 Effect of MLP Width and Depth

Finally, we analyze the effect of the hidden dimension (width) of the MLP as
well as the number of layers (depth) in Fig.12. Increasing the depth from 1 to
2 shows a marginal improvement while further increases do not have a large
effect. This shows that the MLP’s representation capability caps at a certain
value as the majority of parameters are present in the feature grid.
Increasing MLP width on the hand leads to a clear drop in performance as the
number of uncompressed parameters in the MLP increases approximately
quadratically leading to a larger BPP (shifting to the right) but with only
small gains in PSNR (small shift up).

Figure 13: Qualitative results on Kodak. We significantly outperform JPEG in
the low BPP regime (${+}4dB$) at similar BPP (${\sim}0.26$). JPEG fails to
capture the intricate details in the image such as the window of the building
or the clouds in the sky, exhibiting severe artifacts.

##  Appendix D Feature grid visualization

Figure 14: Visualization of learned features (right) for Beauty image (left)
from UVG-F. Feature maps at increasing LODs (left-to-right then top-to-bottom)
capture finer details in the image highlighting the hierarchical features
learned for each signal. Figure 15: Application of our approach for super-
resolution of the Jockey image in the UVG-F dataset. We obtain sharper images
compared with standard upsampling methods at 2x upsampling factor ($2160\times
3840$).

We visualize the learned latents after training on the Beauty image from UVG-F
in Fig.14. With increasing LOD or feature resolution (from left-to-right and
top-to-bottom), we see that finer details of the image are captured. Thus, the
latents represent the image features at different scales/levels. This can be
particularly useful for downstream tasks which may require features at
different scales. Additionally, such a hierarchy enables the application of
our method in streaming scenarios with higher bitrates leading to higher PSNR
(as also discussed in Sec.4.5). Beyond a certain level, we see that the
features become less informative globally. This is due to the fact that the
grid resolution at finer levels is larger than the number of entries in the
latents which is fixed. Multiple locations in the grid map to the same entry
in the latent space.

##  Appendix E Image superresolution

We show the capability of our approach to perform super-resolution of images
by providing the input coordinate grids at the desired resolution. Fig.15
shows results on the super-resolution of the Jockey image from UVG-F
($1080\times 1920$ resolution) by a factor of 2. We obtain slightly sharper
images as compared to standard upsampling methods such as linear, cubic or
nearest neighbor interpolation.

##  Appendix F Additional visualizations

We show additional visualizations of the Pearl ($23466\times 20000$) and SMACS
images ($4630\times 4537$) in Fig.16 (top row and bottom row respectively). We
see that we continue to obtain high quality reconstructions and achieve
similar PSNR compared to Instant-NGP while being much smaller in storage size
(in terms of BPP). Significant artifacts and discoloration are also visible
for the traditional JPEG while still requiring more memory than our approach.

Figure 16: Visualization of Pearl and SMACS images. For Pearl, we maintain a
similar PSNR as Instant-NGP [1] while being ${\sim}9{\times}$ smaller. For
SMACS, we marginally outperform them with more than a $4{\times}$ compression
factor. JPEG exhibits visual discoloration artifacts leading to a much lower
PSNR even with lower compression factors.