# Deep Learning for HDR Imaging: State-of-the-Art and Future Trends

Lin Wang, and Kuk-Jin Yoon L. Wang and K.-J. Yoon are with the Visual
Intelligence Lab, Department of Mechanical Engineering, Korea Advanced
Institute of Science and Technology, 291 Daehak-ro, Guseong-dong, Yuseong-gu,
Daejeon 34141, Republic of Korea. E-mail: {wanglin,
<EMAIL_ADDRESS>received May 15, 2021; revised August 26, 2021.
(Corresponding author: Kuk-Jin Yoon)

###### Abstract

High dynamic range (HDR) imaging is a technique that allows an extensive
dynamic range of exposures, which is important in image processing, computer
graphics, and computer vision. In recent years, there has been a significant
advancement in HDR imaging using deep learning (DL). This study conducts a
comprehensive and insightful survey and analysis of recent developments in
deep HDR imaging methodologies. We hierarchically and structurally group
existing deep HDR imaging methods into five categories based on (1)
number/domain of input exposures, (2) number of learning tasks, (3) novel
sensor data, (4) novel learning strategies, and (5) applications. Importantly,
we provide a constructive discussion on each category regarding its potential
and challenges. Moreover, we review some crucial aspects of deep HDR imaging,
such as datasets and evaluation metrics. Finally, we highlight some open
problems and point out future research directions.

###### Index Terms:

High-dynamic-range (HDR) imaging, deep learning (DL), convolutional neural
networks (CNNs)

##  1 Introduction

High dynamic range (HDR) imaging, an important field in image processing,
computer graphics/vision, and photography, is a technique that allows a
greater dynamic range of exposures than traditional imaging techniques. It
aims to accurately represent a wide range of intensity levels captured in real
scenes, ranging from sunlight to shadows [1, 2]. Using HDR imaging, real-world
lighting can be captured, stored, transmitted, and fully used in various
applications without the need to linearize the signal and handle clamped
values [3]. Conventional HDR imaging mainly uses special HDR cameras to
capture HDR images [4, 5, 6]; however, these cameras are prohibitively
expensive for general users. An alternative is to create HDR content from
virtual environments using rendering tools. However, this approach is mostly
explored in the entertainment industry, e.g., gaming and virtual reality (VR)
[7, 4, 3].

In addition to the aforementioned approaches, a common method is to
reconstruct HDR images from the visual content captured by low-dynamic-range
(LDR) cameras using specially designed algorithms. Among these algorithms,
there are two commonly explored methods. The first is to generate HDR content
by fusing multiple LDR images of the same scene at different exposure times
[8, 3, 9]. However, as capturing LDR contents at different exposures requires
the use of certain software/hardware technology, it is often difficult to
create datasets. Therefore, the second approach is to generate HDR content
from a single-exposure image [10, 11, 12, 13, 14].

Deep learning (DL) has recently been applied to HDR imaging [15]. DL-based HDR
imaging methods often achieve state-of-the-art (SoTA) performances on various
benchmark datasets. Deep neural network (DNN) models have been developed based
on diverse architectures, ranging from convolutional neural networks (CNNs)
[10, 16, 9] to generative adversarial networks (GANs) [17, 18, 19]. In
general, SoTA-DNN-based methods differ in terms of five major aspects: network
design that considers the number and domain of input LDR images [10, 14, 9],
purpose of HDR imaging in multitask learning [20, 21], different sensors being
used to obtain deep HDR imaging [22, 23, 24], novel learning strategies [17,
25, 26], and practical applications [27, 28, 29].

Figure 1: Hierarchical and structural taxonomy of HDR imaging with deep
learning.

This study provides a comprehensive and systematic overview of recent
developments in deep HDR imaging methods. Previous surveys [30, 4, 31, 5, 6]
focused on conventional HDR imaging algorithms, especially deghosting methods
in dynamic scenes [5, 31], multiple-exposure fusion in the image and radiance
domain [4, 6], and stack-based algorithms for HDR reconstruction [30]. Unlike
these existing surveys, this study highlights the importance of DL and
investigates recent advances in deep HDR imaging, both methodically and
comprehensively.

The main contributions of this study are three-fold: (I) We provide a
comprehensive overview of deep HDR imaging methods, including the problem
definition, datasets, evaluation metrics, a taxonomy, and applications. (II)
We conduct an analytical study of recent trends in deep HDR imaging
hierarchically and structurally and offer insights into the opportunities and
challenges of each category. (III) We discuss the open challenges and problems
that HDR imaging faces on the whole and identify future directions to provide
guidance to further research in this area.

In the following sections, we discuss and analyze various aspects of recent
advances in deep HDR imaging. Fig. 1 shows the structural and hierarchical
taxonomy used in this study. The remainder of this paper is organized as
follows. In Sec. 2, we define the HDR imaging problem, datasets, and
evaluation metrics. In Sec. 3, we review and analyze SoTA-supervised methods,
including multiple-exposure and single-exposure HDR imaging/video methods.
Sec. 4 focuses on joint HDR imaging with other learning tasks, e.g., image
super-resolution. In Sec. 5, we investigate recent deep HDR imaging methods
using novel camera sensors. Sec. 6 covers methods that use novel learning
paradigms, e.g., meta-learning. Sec. 7 then scrutinizes the applications,
followed by Sec. 8, where we discuss open problems and future directions.

##  2 Problem, Datasets, and Evaluation

###  2.1  Definition of Deep HDR Imaging

The LDR camera usually sets an appropriate exposure time $\Delta t$ and relies
on the camera response function (CRF) $f_{CRF}$ to map the irradiance $E$ of
scenes to the LDR image $x$ [32].

$x=f_{CRF}(E\Delta t)$ (1)

Fundamentally, deep HDR imaging can be regarded as the process of
reconstructing HDR image content from the LDR counterpart(s) using DNNs. The
aim is to learn a mapping function $M$ with parameters $\theta$ that maps a
set of multi-exposure LDR images $X=\\{x_{1},x_{2},\cdots,x_{n}\\}$ to an HDR
image $y$ [33], where $n$ is the number of LDR images. For single-exposure HDR
imaging, the set size is set to one. Mathematically, this is formulated as
follows:

$Y=M(X;\theta).$ (2)

TABLE I: Summary of publicly available benchmark datasets. N/A: not available. Dataset | Amount | Data type | GT | Resolution | Scene details
---|---|---|---|---|---
Kalantari et al.[9] | 89 (train + test) | Real | Yes | 1500 × 1000 | Multi-exposure with 100 dynamic scenes
Sen et al.[34] | 8 (test only) | Real | No | 1350 × 900 | Multi-exposure with dynamic scenes
Tursun et al. [35] | 16 (test only) | Real | No | 1024 × 682 | Multi-exposure with indoor and outdoor scenes
Endo et al. [36] | 1043 (train only) | Synthetic | Yes | 512 × 512 | Single exposure to indoor and outdoor scenes
Eilertsen et al. [10] | 96 (test only) | Synthetic | Yes | 1024 × 768 | Single exposure to static scenes
Zhang et al. [16] | 41222 (train + test) | Synthetic + Real | Yes | 128 × 64 | Single-exposure panorama scenes
HDREye [37] | 46 (test only) | Synthetic | Yes | N/A | Static scenes (both indoor and outdoor)
Lee et al. [38] | 96 (train + test) | Synthetic | Yes | 512 × 512 | Static scenes with different exposure levels
Prabhakar et al. [39] | 582 (train + test) | Real | Yes | 1-4 Megapixels | Dynamic scenes with real-life object motions
Cai et al. [40] | 4413 (train + test) | Synthetic | Yes | 3 K-6K | Multi-exposure static indoor and outdoor scenes
Liu et al. [14] | N/A (train + test) | Synthetic + Real | Yes | 1536 × 1024 | Single exposure to static scenes
Kim et al. [41] | 59818 frames | Synthetic | Yes | 4 K-UHD | YouTube video for HDR + super-resolution

Therefore, deep HDR imaging restores scene irradiance using feature learning
from LDR images. The reconstructed HDR images must satisfy three criteria:
high contrast ratio, high bit depth, and preserved details. Fig. 2 shows the
pipelines used for deep HDR imaging. The first pipeline, shown in Fig. 2(a)
aims to create an HDR content by learning from multi-exposure LDR images with
DNNs [9, 33]. Specifically, multi-exposure LDR images can be mapped to the
irradiance domain using DNNs based on the inverse CRF and exposure time
($\Delta t$) (or exposure value (EV)[32]) from Eq. (1) and Eq. (2).
Accordingly, an HDR image can be reconstructed from DNNs, which are trained to
align and merge dynamic information from the LDR images. It is to be noted
that alignment is only required for dynamic scenes. By contrast,
reconstructing HDR images from a single-exposure LDR image is much simpler, as
shown in Fig. 2(b), where an inverse tone-mapping network is learned to
generate an HDR image reflecting natural illumination. However, as a single-
exposure LDR image conveys limited exposure information, learning the HDR
visual content from it is an ill-posed problem [3, 32]. Therefore, the
reconstructed HDR image has a limited dynamic range, which indicates that the
visual quality of the HDR images is inferior to that of the multiple-exposure
LDR images. As HDR images are usually displayed after tone mapping, most DNN-
based methods optimize learning on tone-mapped HDR images, which is more
effective than direct computation in the HDR domain [9, 33, 2]. A typical
approach for learning deep HDR imaging was proposed by [9]: the $\mu$-law,
which is commonly used for range compression in audio processing. In Sec. 3,
we conduct a comprehensive and analytical study on existing deep HDR imaging
methods.

Figure 2: DNN-based HDR imaging approaches. Learning to reconstruct an HDR
image (a) by aligning, merging, and fusing multi-exposure LDR images; (b) by
using an inverse one-mapping network from single exposure LDR images.

###  2.2 Datasets for Deep HDR Imaging

The dataset is important for the success of deep HDR imaging methods. Various
benchmark datasets available for deep HDR imaging differ in exposure levels,
dataset sizes, image resolution, quality, scene diversity, etc. Some datasets
obtain LDR images by simulating the CRF or using a CRF database, whereas
others contain real-world LDR images with ground-truth (GT) HDR images. In
addition, some datasets have been developed for multiple-exposure fusion (MEF)
[42, 43], video HDR [44, 45], and HDR imaging with novel sensors [22, 46].
Table I summarizes the commonly used benchmark datasets for deep HDR imaging,
including the data size, data type, availability of the GT, spatial
resolution, and scene details.

Regarding synthetic HDR datasets, HDR GT images are collected from diverse
real-world scenes. For single-exposure HDR image reconstruction, the most
common approach is to imitate the LDR formation pipeline using different CRF
functions from a set of CRF databases [14, 36] or a virtual camera to capture
a number of random regions of the scene based on randomly selected camera
curves [10, 47, 38]. In [14], to create a real-world single-image HDR dataset,
600 amateurs were instructed to capture scenes with multiple exposures using
an LDR camera placed on a steady tripod. The LDR exposure stacks were fused to
obtain HDR images using Photomatix 111https://www.hdrsoft.com/.

Regarding multi-exposure HDR image reconstruction, the dataset collection
process from [9] is representative. For a static scene, HDR image generation
is similar to that in [14]. For the dynamic scene, the HDR image is aligned
with the reference middle-exposure LDR images. Using the reference LDR images,
the up- and down-exposed LDR images are captured by asking the subject to move
simulating camera motion or no motion. More details are provided in the
supplementary material.

###  2.3 HDR Image Quality Assessment (IQA)

Evaluation of the HDR image quality is also essential for deep HDR imaging. In
low-level vision, IQA comprises subjective and objective computation
approaches [48]. We now introduce several commonly used IQA methods, including
statistical measures and HDR-oriented IQA measures.

####  2.3.1 Difference Measures and Statistical-Oriented Metrics

Peak Signal-to-Noise Ratio (PSNR). PSNR is a commonly used IQA to assess the
quality of reconstruction of lossy transformations [49, 36]. In HDR imaging,
PSNR measures the difference between the GT HDR image and the reconstructed
HDR image both before and after tone mapping using the $\mu$-law.

Mean Squared Error (MSE). MSE is the simplest yet most widely used full-
reference quality metric, computed by averaging the squared intensity
differences of reconstructed HDR and GT HDR image pixels before and after tone
mapping using the $\mu$-law.

####  2.3.2 Structural Similarity Measures

Structural similarity (SSIM) measures the quality of an image based on pixel
statistics to model luminance (using the mean), contrast (variance), and
structure (cross-correlation) [50]. In deep HDR imaging, SSIM measures the
structural differences between the reconstructed HDR and GT HDR image pixels
before and after tone mapping using the $\mu$-law. Other structural similarity
measures include the multi-scale SSIM index (MS-SSIM) and MEF-SSIM [51] for
image fusion quality measures.

####  2.3.3 Tailored HDR Statistical Measures

In HDR imaging, HDR-VDP-2 [52], which is an updated version of HDR-VDP [53],
is regarded as an effective metric for measuring the visual quality of HDR
images. HDR-VDP-2 compares the GT HDR image and the reconstructed HDR image,
and predicts visibility and quality. For a detailed discussion on HDR-VDP-2,
readers can refer to [52]. Moreover, HDR-VQM [54] was designed to evaluate the
quality of HDR videos.

####  2.3.4 Deep Learning–Based Evaluation Metrics

Compared with conventional HDR image reconstruction methods, DL-based methods
are more capable of recovering the dynamic range of images, even in extreme
scenes. This has inspired many attempts to employ DNNs for the IQA of HDR
images. [48, 55, 56, 57, 58, 59]. Among these methods, [58, 55, 59] are full-
reference methods that assess the perceptual quality of generated HDR images.
By contrast, [56, 48] are non-reference methods that measure the reconstructed
HDR image quality by quantifying the extracted low- and high-level features
from DNNs.

####  2.3.5 HDR Imaging Challenge

New trends in image restoration and enhancement (NTIRE) [60], in conjunction
with CVPR 2021, started the first HDR imaging challenge. It focused on
reconstructing an HDR image from a single (Track 1) or multiple (Track 2)
exposure LDR images. The workshop utilized existing HDR datasets, e.g., [9],
and newly curated HDR datasets for evaluation. The performance was measured by
PSNR using the ground truth.

##  3 Supervised HDR Reconstruction

Figure 3: Deep multi-exposure HDR imaging methods. (a) Optical flow-based
(net) LDR image alignment. (b) Direct concatenation of features of LDR images
to merge them and reconstruct the HDR image. (c) Correlation-guided LDR
feature alignment, followed by merging and reconstruction. (d) Image
translation-based LDR image alignment. (e) Reconstruction of the HDR image
from static LDR image fusion.

###  3.1 Multi-Exposure HDR Image Reconstruction

Insight: Reconstructing an HDR image from multiple-exposed LDR images in
dynamic scenes follows two steps: (1) learn to align the LDR images; (2) merge
them to get an HDR image.

The common method of obtaining an HDR image is to capture a series of LDR
images at different exposures and merge them to reconstruct an HDR image [61,
9]. For a dynamic scene, the input LDR images must first be aligned. This step
is crucial, yet challenging, for multi-exposure HDR imaging pipelines. The
quality of the alignment is usually reflected in the reconstructed HDR images.
In other words, a less reasonable alignment often leads to visible artifacts
in the pixel regions of low correspondences. We analyze these methods, as
shown in Fig. 3.

####  3.1.1 Optical Flow-based Image Alignment

A common method of alignment for LDR images is the use of optical flow
algorithms or networks. The aligned LDR images are fed to the DNNs to
reconstruct an HDR image, as depicted in Fig. 3(a). Given a sequence of LDR
images with low, medium, and high exposures, Kalantari et al. [9] proposed a
representative approach aligning low- and high-exposure images to a medium-
exposure image, called the reference image, using the classical optical flow
algorithm [62]. The aligned LDR images are taken as inputs to a DNN to
reconstruct an HDR image supervised by the GT HDR image. Yan et al. [63] also
followed this pipeline. However, to enrich the image information, their method
takes a sequence of LDR and HDR images obtained by gamma correction as the
inputs at different scales and adopts three sub-networks to obtain the
corresponding HDR images. Peng et al. [64] and Prabhakar et al. [39] argued
that classical optical flow algorithms could lead to considerable misalignment
errors. They proposed the alignment of low- and high-exposure images with the
reference images using SoTA optical flow networks, e.g., FlowNet[65].

####  3.1.2 Direct Feature Concatenation

Optical flow algorithms may provide unsatisfactory performance in aligning LDR
images with large-scale foreground motion. Therefore, Wu et al. [49] proposed
a representative framework including three encoder networks, a merger network,
and a decoder network, as shown in Fig. 3(b). The decoders are trained to
encode the LDR images at different exposures into latent feature spaces. Next,
the encoded LDR images are fed to a merger to learn the aligned features.
Finally, the decoder reconstructs the HDR image. A similar strategy was
adopted in [66]; however, in contrast to [49], LDR sub-networks are added to
recover three static LDR images corresponding to the HDR image. The feedback,
i.e., cycle, reconstruction of LDR images adds more constraints on the forward
path, which benefits the restoration of the motion areas in the HDR image.
Another end-to-end single processing pipeline was proposed by Chaudhari et al.
[67]; individual encoders are designed to map the raw color filter array (CFA)
data of different exposures to learn intermediate LDR features. Metwaly et al.
[68] also employed three encoders to extract features of three LDR images at
different exposures, similar to [49]. The extracted features are concatenated
and fed into a merger network. A pivotal contribution of this method is the
attention mask, designed to enable the network to focus on the parts of the
scene with considerable motion to avoid ghosting effects after the decoder.
However, Niu et al. [69] argued that using individual encoders cannot fully
exploit multiscale contextual information from features. Therefore, they
proposed multiscale LDR encoders to extract visual features at different
scales, similar to the strategy proposed by [63]. The extracted features are
fused at different scales via residual learning before being fed to a merger
network.

TABLE II: Deep multi-exposure HDR reconstruction employed by some representative methods. N/A: not available. Method | Publication | No. of Exposures | Alignment | Merge | Scene types | Loss functions
---|---|---|---|---|---|---
Kalantari [9] | Siggraph’17 | 3 | Classical optical flow | Image fusion | Dynamic | L2
Prabhakar [39] | ICCP’19 | 3 | Flow+Refine nets | Feature fusion | Dynamic | L2 + MS-SSIM
Yan [63] | WACV’19 | 3 |  Same as [9] |  Multiscale image fusion | Dynamic | L1
Wu [49] | ECCV’18 | 3 | Direct feature cat. | Feature fusion | Dynamic | L2
Yan [66] | Neurocomput. | 3 | Direct feature cat. | Feature fusion | Dynamic | L2 + Cycle losses
Chaudhari [67] | WACV’20 |  $\geq$3 | Feature encoding | Feature fusion | Dynamic | L2
Niu [69] | TIP’21 | 3 |  Multiscale feature cat. |  Multiscale feature fusion | Dynamic |  Two L1 losses \+ GAN loss
Yan [33] | CVPR’19 | 3 |  Attention-guided feature alignment |  Dilated dense feature merge | Dynamic | L1
Yan [8] | TIP’20 | 3 | Direct feature cat. |  non-local + triple-pass feature correlations | Dynamic | L2
Choi [61] | Sensor’20 | 3 |  Pyramid-attention feature alignment |  Dilated dense feature merge | Dynamic | L1
Pu [70] | ACCV’20 | 3 |  Multiscale pyramidal alignment |  Residual dense feature merge | Dynamic | L1 + Gradient
Rosh [71] | ICIP’19 | 3 | Image translation |  Multi-exposure image merge | Dynamic | L1 + Perceptual
Prabhakar[72] | CVPR’17 | 2 | N/A |  Unsupervised feature fusion | Static | SSIM
Ma[73] | TIP’19 |  $\geq$3 | N/A |  Guided weight map filtering | Static | FEM-SSIM
Xu[43] | TIP’20 | 2 | N/A |  Self-attention \+ local detail cat. | Static |  MSE + GAN \+ Gradient

####  3.1.3 Correlation-Guided Feature Alignment

In reality, object motion in a dynamic scene can be diverse. This makes it
difficult to suppress ghosting effects in flow-based HDR image reconstruction
methods. Recent DL-based methods have been explored to determine the
correlation between the LDR inputs of different exposures and motions. Using
the attention mechanism [74] is a representative approach to these problems,
as shown in Fig. 3(c). Yan et al.[33] generally followed the pipeline of [49].
However, their method applies the attention modules in the encoding phase to
achieve better feature alignment. In particular, attention modules help
exclude less useful components caused by misalignment and saturation in
individual encoders. This is done by feeding under- or over-exposed LDR images
(non-reference LDR images) and the reference LDR image to two attention
modules. Attention maps are then obtained to refine the feature maps of the
non-reference LDR images. Consequently, misaligned and saturated non-reference
LDR images are excluded, preventing less useful features from entering the
merging network.

Pu et al. [70] and Choi et al.[61] explored the alignment of the features of
LDR images using the pyramid modules in the encoding stage. This is
conceptually similar to that in [33]. In particular, the method in [70]
proposes a multiscale pyramidal feature alignment strategy. This is more
flexible and robust in handling motion and disparity. The method in [61]
explores the inter-attention of the self-similarity of pixels in LDR images. A
dual excitation block is also designed to recalibrate the features spatially
and channel-wise in the merging phase. By contrast, the method proposed by
[75] employs non-local modules [74] in the merging phase instead of the
encoding phase. The encoded features are directly concatenated and fed into
the merging network. The method proposed by [76] uses only two LDR images (one
under-exposed and one over-exposed) and adopts a homograph network (encoder)
to warp the under-exposed images to the overexposed images. Then, an attention
module is employed to reduce the misaligned features before being fed to the
merging network. In the NTIRE 2021 HDR Challenge [60], ADNet [77] proposes the
alignment of LDR images with a pyramid, cascading, and deformable (PCD) module
and adaptively fuses them with a spatial attention module.

####  3.1.4 Image Translation-Based Alignment

Although optical flow-based HDR imaging methods can reconstruct ghost-free HDR
images in conditions with varied motions and different exposures, these
methods tend to generate considerable artifacts. Moreover, the end-to-end
feature concatenation-based methods successfully reconstruct HDR images;
however, they generate less realistic details in highly saturated regions.
Some studies have attempted to translate low- and high-exposure images to the
reference LDR image using DNNs under the supervision of the reference image,
as depicted in Fig. 3(d). The method proposed by Rosh et al. [71] is a
representative method in which image translation networks are designed to
generate two LDR images resembling the reference LDR image. The three LDR
images are then fed to a merging network to reconstruct an HDR image; however,
the two stages are not learned in an end-to-end manner. Lee et al. [78]
extended [71] and generated accurately aligned LDR images by blending
information from multi-exposure images using encoder–decoder network
structures. The alignment and merging stages can be trained in an end-to-end
manner.

####  3.1.5 Deep Static Exposure Fusion

Compared with multi-exposure HDR imaging in dynamic scenes, merging LDR images
with static scenes is relatively simple. The main challenge is to find methods
that merge LDR images at different exposures and address the artifacts and
loss of textural details in the reconstructed HDR images. In the literature,
MEF is a method of merging multi-exposure images. The MEF strategy has been
broadly studied for image enhancement and HDR imaging. This study reviews and
analyzes representative MEF methods targeted at HDR imaging using DNNs. For
details on more general image fusion techniques, refer to [79].

Figure 4: Deep single-exposure HDR imaging methods. (a) Directly learning a
domain transformation autoencoder from a single LDR image. (b) Generating
bracketed (up- and over-exposed) LDR image stacks and then reconstructing an
HDR image. (c) Using a recurrent (efficient) learning network structure. (d)
Learning camera-imaging pipeline with DNNs. (e) Learning camera sensing with
DNNs (neural sensing encoder).

DeepFuse, proposed by Prabhakar et al. [72], is a representative learning
approach for static multi-exposure image fusion. To achieve this fusion,
DeepFuse collects a large dataset of multi-exposure image stacks for training
and reduces the need for GT images. The unsupervised DL framework for the MEF
uses a no-reference image quality loss function. This baseline has been
updated by many recent studies [73, 42, 80, 43]. MEF-Net, proposed by Ma et
al. [73], focuses on static LDR image fusion of arbitrary spatial resolution
and exposure levels. It typically combines single-image super-resolution
(SISR) and HDR imaging by feeding a low-resolution (LR) version of the input
sequence to a neural network for weight map prediction. The predicted weight
maps are jointly upsampled using a guided filter. The final HDR image is
calculated by weighted fusion with high-resolution (HR) LDR images. MEF-GAN
proposed by Xu et al. [43] is targeted at multi-exposure image fusion based on
GANs [81, 82]. As the luminance of the multi-exposure images varies greatly
with the spatial location in LDR images, the self-attention mechanism [75] is
adopted in the generator to learn the attention-driven and long-range
dependency of the LDR images. FusionDN [42] extends DeepFuse and proposes an
unsupervised and unified densely connected network for image fusion tasks,
including LDR image fusion. Given two LDR images, FusionDN learns to generate
a fused HDR image. This is achieved by a weight block, which can obtain two
image-driven weights as the measurement of information preservation in the
features of different LDR images. Thus, losses of similarities based on
weights can be applied without using GT images.

####  3.1.6 Potential and Challenges

From the aforementioned review, one crucial yet meaningful step for multi-
exposure HDR imaging is LDR image alignment and merging, as summarized in
Table II. Aligning LDR images using optical flow is a traditional approach;
however, this alignment method is error-prone and less effective with large
motions. By contrast, correlation-guided feature alignment is more flexible
and effective. Therefore, the SoTA methods mostly explore the correlation of
features, e.g., using the attention mechanism to exclude misaligned features.
However, correlation-guided feature alignment is sensitive to over-saturated
regions, which often causes the loss of textual details owing to the exclusion
of features. Some methods, e.g., [33], employ attention in the encoding phase,
and other methods, e.g., [75], learn the correlation in the merging phase.
Nonetheless, more theoretical studies are needed to determine the phase that
is most appropriate for learning correlations.

Many challenges remain in this field of research. First, a robust image or
feature alignment must be guaranteed. The learning correlation of LDR images
is a promising research direction; however, it usually requires considerable
computational cost. Second, many labeled datasets are required to train DNNs
and make them robust. In multi-exposure HDR imaging, it is relatively
expensive to obtain cameras and equipment for scene capture. Therefore, future
research needs to investigate data-efficient learning. One promising direction
is to explore knowledge transfer [83] or semi-supervised learning [84, 85].
Third, as DNNs are adopted for learning salient features, inference latency is
inevitable in most SoTA methods. However, real-time HDR imaging is preferred
for real-life applications. Thus, it is crucial to develop lightweight DL-
based frameworks for balancing performance. We further discuss these problems
in Sec. 8.

###  3.2 Deep Single-Exposure HDR Imaging Approaches

Insight: Single-exposure HDR image reconstruction aims to recover missing
information in the saturated areas of an LDR image and reconstruct an HDR
image using a neural network.

Despite the success of deep HDR image reconstruction by combining a set of LDR
images at different exposures, this paradigm still has limitations; it has to
handle misalignment with reference images due to scene motion or requires
specialized optical systems. Single-exposure HDR image reconstruction avoids
these limitations. The distinct advantage of single-exposure HDR
reconstruction is that it can work with images captured with a standard camera
or even recover the full dynamic range of legacy LDR content [47].
Consequently, single-exposure HDR reconstruction has attracted attention in
the research community.

####  3.2.1 Direct Learning from a Single LDR Image

The most straightforward method is to directly learn from a single LDR image
using encoder–decoder network structures, as shown in Fig. 4(a). HDRCNN,
proposed by Eilertsen et al. [10], is a representative approach for
reconstructing an HDR image from a single LDR image. In particular, a hybrid
LDR encoder and HDR decoder, which operate in the log domain, are used to
reconstruct an HDR image. To improve the generalization of complex saturation
scenes, HDR images from existing datasets are used to simulate diverse LDR
exposures. Santos et al. [47] identified the disadvantages of applying the
same convolutional filters to well-exposed and saturated pixels in [10],
causing ambiguity during training and leading to checkerboard and halo
artifacts in the HDR images. To solve this problem, their method involves
masking the saturated areas, thus reducing the contribution of the features
from these areas. Meanwhile, a perceptual loss is also introduced to help
reconstruct visually pleasing textures. By contrast, ExpandNet [86] takes a
different viewpoint; this is a multiscale autoencoder architecture that aims
to learn a different level of detail from an LDR image. The learned multilevel
features are merged to reconstruct an HDR image. The method in [87] proposes a
DNN-based approach to learn the local and global features for HDR imaging at a
high resolution. These seminal methods have inspired many attempts to improve
the diversity of exposure to LDR images. Moriwaki et al. [88] found that using
the reconstruction loss, e.g., mean square error (MSE) loss, could often cause
blur effects and the loss of semantic details in HDR images. To solve these
problems, perceptual loss and adversarial loss are added to improve the
perceptual quality. The method in [89] also proposes a novel cosine similarity
loss, normalizing the HDR images and distributing the pixel value of HDR
images.

TABLE III: Deep single-exposure HDR imaging by some representative methods. Method | Input | CRF |  Image stacking |  Under-exposed regions |  Over-exposed regions | Merge |  Learning highlight | Loss
---|---|---|---|---|---|---|---|---
Eilertsen [10] |  2D Image | Fixed CRF | ✗ | ✓ | ✗ | ✗ |  Autoencoder with skip connections |  L2 (Illuminance + reflection)
Endo [39] | CRF database | ✗ | ✓ | ✓ | ✓ | 3D Autoencoder | L1 + L1
Marnerides [86] | ✗ | ✗ | ✓ | ✓ | ✓ | Multi-branch CNN | L1 + Cosine
Santos [47] | ✗ | ✗ | ✓ | ✓ | ✗ | Feature masking CNN |  L1 + Style + Perceptual
Lee [11] | ✗ | ✓ | ✓ | ✓ | ✓ | Recursive learning | L1 + GAN
Kim [20] |  Linear inverse CRF | ✓ | ✓ | ✓ | ✓ | Recursive learning |  Global + local \+ refine
Khan[12] | ✗ | ✗ | ✓ | ✓ | ✗ | Feedback RNN | L1 + Perceptual
Liu [14] |  Deep inverse CRF | ✗ | ✓ | ✓ | ✗ |  Dequantization \+ Hallucination \+ Refinement | Joint losses
Zeng [15] | ✗ | ✗ | ✓ | ✓ | ✗ |  Image-adaptive 3D LUT CNN |  L2 + GAN + Regularization
Zhang [16] |  3D Panorama | ✗ | ✗ | ✓ | ✓ | ✗ |  3D panorama Autoencoder \+ Regression | L1 + Render
Yang [90] | ✗ | ✗ | ✓ | ✓ | ✗ |  3D Panorama Reconstruction + Correction | L2 + LDR
Metzler [91] |  Neural Sensor | ✗ | ✗ | ✓ | ✓ | ✗ |  Optical encoder + Electronic decoder | L2 + Regularization
Martel [92] | Fixed CRF | ✗ | ✓ | ✓ | ✗ |  Electronic encoder + Digital decoder | L2

####  3.2.2 Generating Bracketed LDR Image Stacks

In reality, single-exposure HDR reconstruction suffers from a lack of quality
training data and is an ill-posed problem. To this end, Endo et al. [36]
proposed a representative framework based on a 2D encoder and 3D decoder with
skip connections to generate bracketed LDR image stacks (up-exposed and down-
exposed) from an HDR image using the camera response function (CRF) database.
As such, an HDR image is reconstructed from the bracketed LDR images, as shown
in Fig. 4(b). This method produces natural tones without introducing visible
noise or the color of the saturated regions. Motivated by [36], Lee et al.
[38] generated LDR stacks with a two-branch network. In particular, three up-
exposed and three down-exposed LDR stacks are generated sequentially. Thus,
six LDR images are obtained and merged to generate an HDR image. Similar
approaches were also proposed by [11, 20]. However, multi-exposure stacks are
generated using GANs [81, 82]. Cai et al. [40] collected a dataset containing
both up- and down-exposed LDR stacks generated from one LDR image by varying
the exposure values (EVs). The dataset is then used to train a neural network
to reconstruct an HDR image. Instead of generating multiple exposures, An et
al. [93] argued that a single image could contain pixel-varying exposure for a
scene. Thus, an image is changed to the Bayer mode and multiple sub-images are
generated according to each exposure and merged to reconstruct an HDR image
using a neural network.

####  3.2.3 Computationally Efficient Learning

Although the HDR image quality can be improved by increasing network depth or
adding more losses, it involves considerable computational costs. FHDR,
proposed by Khan et al. [12], exploits the power of the feedback mechanism,
and proposes a recurrent neural network (RNN)-based frameworks (see Fig.
4(c)). As such, low-level features are guided by high-level features in
multiple iterations, leading to better reconstruction with few parameters. In
contrast to [12], a different perspective of the feedback mechanism was
proposed in [90]; the method involves learning to generate HDR images first.
Next, the outputs are used to generate LDR images reciprocally via a
correction network. As the correction network can be removed after training,
there is no additional inference cost. Zeng et al. [15] found that, in the
camera imaging pipeline, 3-dimensional lookup tables (3D LUTs) are important
for manipulating the color and tone of photos. Thus, they proposed learning
image-adaptive 3D LUTs together with a small network to reconstruct high-
resolution HDR images effectively. The method is shown to produce satisfactory
results with fewer computation costs. A more detailed analysis is provided in
Sec. 8.

####  3.2.4 Learning Camera Imaging Pipeline

The major challenge for HDR image reconstruction from a single image is to
restore the missing details in the under- or over-exposed regions in the LDR
images. The details are missing because of the quantization and saturation of
the camera sensors. Therefore, modeling the imaging pipeline is a pivotal step
in effective HDR image reconstruction. SingleHDR, proposed by Liu et al. [14],
is a representative framework for modeling the HDR-to-LDR formation process
into three sub-steps: dynamic range clipping, non-linear mapping with a CRF,
and quantization, as shown in Fig. 4(d). Based on these intuitions, tailored
DNN models are built to sequentially learn the three reverse steps. The sub-
networks are jointly fine-tuned in an end-to-end manner to reconstruct an HDR
image. SingleHDR has been demonstrated to perform favorably against methods
generating bracketed LDR image stacks, e.g., [11].

####  3.2.5 Learning Neural Sensors

In a single-exposure HDR imaging pipeline, the biggest challenge is to
correctly restore the saturated regions of the LDR images. The aforementioned
approaches have addressed this problem by generating LDR images from the CRF
database, directly modeling the reverse CRF, or designing efficient networks.
However, these methods do not consider the problems within the sensors. Recent
methods have attempted to model sensor processing using DNNs, which is a
promising direction in HDR imaging, as depicted in Fig. 4(e). In particular,
the method proposed by Metzler et al. [91] introduces an optical encoder to
encode optical HDR information of the lens. Meanwhile, an electronic decoder
network is designed to decode optically encoded information to reconstruct an
HDR image. A similar approach was proposed in [94]. A modulation method is
designed to learn spatially optical coding information; however, the overall
framework is an end-to-end learning method. Martel et al. [92] introduced a
differential neural sensor to optimize per-pixel shutter functions, jointly
learned by a neural network in an end-to-end manner. Modeling the exposure
function enables the sensor to capture blurred LDR images, which are then used
to reconstruct HDR images.

####  3.2.6 Potential and Challenges

Based on the aforementioned analysis, various aspects of deep single-exposure
HDR imaging methods have been studied. The rapid development of methodologies
has resulted in significant performance enhancements. Table III summarizes
representative studies. Deep single-exposure HDR imaging has several
advantages. First, multi-exposure HDR imaging excludes the alignment problem
of LDR images and suffers from the ghosting effect to a lesser degree. Second,
it is more flexible in applications and simplifies data collection. Third, it
is computationally efficient.

There are several challenges in this direction. First, it is difficult to
estimate the saturated regions of the LDR images. Although learning optical
sensors [94, 92, 91] improves the estimation of the saturated pixels, it
requires more complex camera settings and hardware. Future research could
simplify the hardware settings and learning frameworks. Moreover, mainstream
methods attempt to generate LDR images under diverse illuminance conditions;
however, in reality, complex saturated pixels are still difficult to handle
using a CRF database or learning CRF. It might be more promising to
simultaneously combine the camera imaging pipeline and CRF to reinforce a more
robust estimation of the saturated pixels in the LDR images. Furthermore, most
methods resort to pixel-wise loss, e.g., L1 loss, for optimization. Although
some methods include adversarial loss and perceptual loss to enhance
perceptual quality, these losses sometimes lead to less realistic results. We
discuss learning strategies in Sec. 3.6.

###  3.3 Deep 3D Panorama HDR imaging

Insight: Deep 3D Panorama HDR imaging aims to reconstruct an HDR panorama by
learning the lighting from a single LDR image.

Unlike the general 2D image, a 3D panorama represents the physical world in a
3D wide-range view [95]. Owing to its wide-range view, lighting in a scene
often varies spatially, making it more challenging to estimate 3D lighting for
HDR imaging. Zhang et al.[16] proposed a representative method that aimed to
learn inverse tone mapping from an LDR panorama image to reconstruct an HDR
panorama image in outdoor scenes. The framework typically follows an
encoder–decoder structure that reconstructs HDR images and estimates sun
elevation. By contrast, Gardner et al. [96], Gkitsas et al. [97], and Song et
al.[95] focused on indoor panorama HDR imaging by learning parametric
lighting. In particular, in [96], lighting is represented as a set of discrete
3D lights with geometric and photometric parameters, which are estimated using
DNN learning from a single image on a dataset of environment maps annotated
with depth. In [97], in contrast, the lighting incoming from all directions at
a 3D location is represented by learning from a single LDR image and a
selected 2D pixel. The core idea is to decompose the panorama HDR imaging
process into three subtask networks: geometry estimation, LDR completion, and
LDR-to-HDR mapping. The networks are trained sequentially and then fine-tuned
in an end-to-end manner.

Discussion: Compared with 2D HDR imaging, there are fewer studies on 3D
panorama HDR imaging. 3D panorama HDR imaging is a complex task that requires
a robust algorithm that can correctly estimate spatially varying lighting. The
biggest difficulty in learning 3D panorama HDR imaging is the lack of
qualitative and quantitative real-world labeled datasets. Most methods rely on
synthetic datasets and learn a parametric lighting model in multiple steps.
Such learning methods are expensive and data-driven. Future research may
consider bridging 2D HDR imaging with 3D panorama HDR imaging using knowledge
transfer [83, 98]. As it is relatively easy to collect labeled datasets in the
2D domain and there are many pretrained network models, the methods that
facilitate 3D panorama HDR imaging can be explored.

###  3.4 Deep Stereo-Based HDR Imaging

Insight: Deep stereo-based HDR imaging aims to reconstruct an HDR image from a
pair of multi-exposure stereo LDR images.

Stereo-based HDR image reconstruction has been widely studied. Lin et al.[99]
proposed the first stereo-based HDR imaging framework, which includes five
steps: CRF estimation, stereo matching, image warping, hole-filling, and HDR
image fusion. This baseline framework has been regarded as a basic paradigm
for stereo HDR imaging. However, it involves multiple processing steps, and
the reconstructed HDR image is highly dependent on the performance of each
processing step. Recently, DNNs have been applied to stereo-based HDR imaging.
The method proposed by Chen et al. [100] is a representative framework for
stereo HDR with an LDR stereo imaging system trained using a GAN, as shown in
Fig. 5. This framework consists of two steps: exposure view transfer and image
fusion. The left-view (LV) under-exposed image is translated to an LV over-
exposed LDR image in the first step. Similarly, the right-view (RV) over-
exposed image is translated into an RV under-exposed image. In the image
fusion step, the input LV LDR image and the translated LV LDR image are fed to
a fusion generator to generate an LV HDR image. The same step is applied to
the RV images. Thus, HDR LV and RV image pairs are obtained. By contrast, the
method in [32] follows the conventional pipeline of [99] and replaces it with
three learning modules: an exposure calibration network, a hole-filling
network, and an HDR fusion network. Consequently, the proposed method better
addresses the challenges of distorted brightness and color attributes and
detail loss in fused HDR images.

Figure 5: Deep stereo-based HDR imaging method [100].

Discussion and Challenges: Based on the aforementioned analysis, only a few
methods exist in this research domain. The SoTA method [32] replaces the
conventional pipeline [99] with DNNs; however, it complicates learning and
increases the computational cost. Moreover, as many modules are used, each
module needs to be trained and fine-tuned to reconstruct an HDR image in an
end-to-end manner. Importantly, exposure calibration and texture filling are
challenging steps. They often cause a loss of texture details and visual
artifacts in HDR images. Therefore, further research is needed to address
these problems. Finally, the method in [100] simplifies the overall pipeline
using GAN; however, it is only focused on one-side (either LV or RV) HDR
imaging without considering exposure calibration and image wrapping. It could
be beneficial to apply image translation methods, e.g., [101] to LV and RV
wrapping and use the attention mechanism in the merging phase to better
exclude misaligned features.

###  3.5 Deep Video HDR

Insight: Deep video HDR aims to learn clean HDR image content and temporally
consistent frames from an input LDR video.

Unlike deep HDR imaging, deep video HDR has received relatively less attention
in the community. The main reason is that video HDR must address the temporal
consistency. In this situation, the reference image at each frame has
different exposures, which poses a challenge for image alignment at each
frame.

####  3.5.1 HDR Videos with Single Alternating Exposure

A representative method was proposed by Kalantari et al. [102]. This method is
an updated version of an earlier work [9] for HDR video reconstruction from
LDR sequences with alternating exposures; it includes two steps. As each frame
has different exposures, directly using off-the-shelf optical flow algorithms
leads to suboptimal HDR video reconstruction. Therefore, the first step is to
align the neighboring frames with the current frame using an optical flow
network. The estimated flows are used to warp the neighboring frames and
produce a set of aligned images. Finally, a merging network is designed to
merge the aligned images and reconstruct the final HDR frames. The method in
[103] is another representative method that directly reconstructs HDR videos
from LDR videos. The LDR-to-HDR mapping is similar to single-image HDR methods
[36, 13], where gamma correction is applied to convert the LDR video into real
scenes. The transformed real scenes are fed into a generator under adversarial
learning. A generator with a 3D convolutional autoencoder is designed to
address the flickering problem caused by temporal inconsistencies. Moreover,
intrinsic loss is introduced to maintain the luminance and color information
of the output video. The methods proposed by Kim et al. [18, 41] are also
related to video HDR with super-resolution. However, their methods follow a
single-exposure HDR imaging pipeline without considering temporal consistency.
This issue is discussed in Sec. 4.1.

####  3.5.2 HDR Videos with Multiple Alternating Exposures

Despite significant progress in deep multi-exposure HDR imaging, deep video
HDR with multiple alternating exposures remains challenging. Chen et al. [44]
and Jiang et al. [45] proposed two recent representative video HDR methods. In
particular, the method in [44] is based on a coarse-to-fine framework, with
multiple alternating exposures as inputs. In the coarse reconstruction phase,
the optical flow algorithm in [62] is used to align the neighboring frames
with the reference frame blended by a weight estimation network. In the
refinement phase, the features of the neighboring frames are aligned to the
reference frame using deformable convolutions. Finally, the features are
temporally fused to reconstruct an HDR video. By contrast, the method in [45]
uses tri-exposure quad-Bayer sensors. The sensors spatially extend each color
Bayer filter to four neighboring pixels. The method employs a feature fusion
module that merges features in the feature space to handle motion blur. An
attention-based temporal denoising module is proposed to reduce noise and
maintain temporal coherence. Meanwhile, a super-resolution module is designed
to enhance spatial resolution.

####  3.5.3 Discussion and Challenges

Based on the aforementioned analysis, research on video HDR using DL is still
rare in the literature. The main challenge is dealing with temporal
inconsistencies. It is common to use optical flow to align the neighboring
frames with the reference frame; however, this leads to errors in the
saturated regions. Although multiple alternating-exposure video HDR methods
aim to reconstruct HDR videos in a coarse-to-fine manner, many modules are
needed, which increases the computational cost. More importantly, a factor
that hinders the progress of HDR video reconstruction is the lack of high-
quality datasets. Although some synthetic datasets have been built [44, 45],
there exists a considerable domain gap with real-world data. Future research
must consider these problems, and investigate the potential of self-supervised
learning and transfer learning for reconstructing HDR videos. Moreover,
effective temporal consistency loss functions as studied in many video
reconstruction problems are worth exploring.

###  3.6 Optimization and Loss Functions

Optimization and loss functions are important for learning effective HDR
imaging. In this section, we analyze the loss optimization strategies used for
HDR image reconstruction. Tables II and III summarize the commonly used loss
functions. As loss functions are crucial for deep HDR imaging, we conducted
experiments to validate the effectiveness of the commonly used loss functions.
Owing to space limitations, the details of the loss functions and the
experimental results validating the effectiveness of loss functions are
discussed in the supplementary material.

Figure 6: Deep HDR imaging with SR methods. (a) SR is performed first and then
the SR HDR image is reconstructed. (b) HDR image is reconstucted first and
then SR is performed. (c) LDR SR, LR HDR imaging, and HDR SR are perfomed
jointly. (d) LDR images are divided into base and detail layers and fusion of
the two branches are learned. (d) SR HDR image is reconstructed from multi-
exposure LDR images with optical flow and motion compensation.

###  3.7 Examining SoTA deep HDR imaging methods

In the previous sections, we have decomposed the HDR imaging methods based on
the number of inputs (Sec.3.1 and Sec.3.2), image modality (Sec. 3.4, and Sec.
3.3), and loss functions (Sec. 3.6). Most of the SoTA deep HDR imaging methods
can be attributed to combining these components. We have summarized some
representative methods and their key strategies in Table.II and Table III.
This section presents the quantitative and qualitative results of some
representative methods on benchmark datasets. All statistics are derived from
the original papers (the source codes are not publicly available) or
calculated using official models.

Comparison of the Representative Deep Multi-exposure HDR Methods. First, we
perform a comparison of the representative deep multi-exposure HDR methods
based on the taxonomy of Table II. Due to lack of space, the experiments are
provided in Sec.4.1 of the supplementary material.

Comparison of the Representative Deep single image HDR methods. We then
present the experimental results for the representative deep single-image HDR
methods, as summarized in Table III, on two benchmark datasets. The
experimental results are provided in Sec.4.2 of the supplementary material.

##  4 Deep HDR Imaging with other Tasks

###  4.1 Deep HDR Imaging with Super-Resolution (SR)

Insight: Deep HDR imaging with SR methods aim to directly reconstruct high-
resolution (HR) HDR images from a large amount of low-resolution (LR) LDR
image counterparts.

In this section, we provide a perceptive analysis of HDR imaging with SR. A
question can be raised: Why do we need to combine HDR with SR? Although ultra-
high-definition (UHD) images deliver more realistic visual information, they
are less accessible to most users. By contrast, one can easily obtain many LR
LDR image resources; therefore, it is important to convert the LR LDR images
into HR LDR images. From this perspective, the goal is to solve two visual
learning problems: LDR-to-HDR reconstruction and LR-to-HR mapping (SR). Image
SR using DL has successfully enhanced performance [104, 105]. Similar to HDR
imaging, image SR is an ill-posed problem [106, 85] because multiple HR images
exist for an LR image. As the degradation process of LR images is usually
complex, the DNN-based methods mostly rely on manually designed degradation
kernels [107, 105]. Hence, the simultaneous learning of HDR and SR is a
complex problem. In LDR images, local variations in contrast and textural
details are missing unlike in HDR images. Meanwhile, LR images are degraded by
complex degradation kernels with reduced resolution, causing the loss of high-
frequency details [41]. Therefore, it is crucial to recover high-frequency
details, contrast, and signal amplitude while enhancing the spatial
resolution. In general, HDR imaging with SR can be divided into two learning
strategies: sequential learning and joint learning. Subsequently, we
comprehensively analyze SoTA methods.

####  4.1.1 Sequential Learning

Most methods are based on a single image in sequential learning and follow the
learning strategy in the image SR. A natural solution of HDR with image SR is
sequentially connecting one to the other. However, it is imperative to
determine whether the first step should be image SR (see Fig. 6(a)) or LDR-to-
HDR mapping (see Fig. 6(b)). Considering this critical problem, [108, 21]
studied the effect of different connections of modules on the quality of HDR
image SR. In particular, in [108], an image is first decomposed into a
luminance component (Y) and chromatic component (UV). Then, two frameworks
(HDR-SR and SR-HDR systems) are established to examine learning efficacy.
Among many solutions, it has been shown that the best results can be obtained
when processing the luminance component (Y) of the input with a single image
and then feeding the enhanced HDR luminance to the single-image SR network
trained by only the luminance component. [21] also compared the results of
sequential learning with the proposed joint learning framework, which we will
discuss in the following section. Their study shows that sequential learning
causes an accumulation of errors.

####  4.1.2 Joint Learning

Joint learning is the most explored approach to HDR imaging with SR in SoTA
methods [109, 110, 21, 41, 18]. Although these approaches have different
perceptions, they share common features. We examine these methods in this
section.

Single-exposure HDR with SR. Kim et al.[21] proposed a representative
framework for joint single-exposure HDR and SR, as shown in Fig. 6(c). The
proposed framework includes LDR image SR, LR HDR image reconstruction, and
joint HDR SR via feature concatenation. This baseline was improved by [110,
41], where an LDR image is decomposed into the base layer (illumination) and
detail layer (reflectance). Subsequently, two branches are trained based on
these layers. Finally, the deep features from the two branches are
concatenated to reconstruct an HR HDR image, as shown in Fig. 6(d). A new
branch is added in [18], namely, the image reconstruction module, to the
framework of [21] to jointly reconstruct the HR HDR image using GAN [81].

Multi-exposure HDR with SR. As single-exposure HDR methods have critical
limitations, joint learning of multi-exposure HDR and SR is worth exploring.
However, based on this review, few studies have focused on this direction
[111, 112]. Ashwath et al. [112] proposed a unified framework without joint
branches, taking three LR images as inputs, similar to reference-based HDR
methods (see Fig. 6(e)). The LR images are first aligned using optical flow
and fed to a fusion network to generate an LR feature map. The LR feature map
is then fed to a bilateral-guided upsampler to reconstruct an HR HDR image. By
contrast, the method in [111] takes two LR images (under- and over-exposed)
with a two-branch learning framework in which each branch aims to learn the
under- and over-exposed information recursively. Finally, the information is
fused to reconstruct an HR HDR image.

###  4.2 Deep HDR Imaging with Denoising

Insight: As there exists noise in the under-/over-exposed regions of an LDR
image, it is necessary to remove the noise while reconstructing an HDR image.

####  4.2.1 Single-Exposure Noise Removal

The under-/over-exposed regions of the LDR images may exhibit unexpected
noise, affecting the HDR imaging quality. Therefore, SoTA methods attempt to
remove noise from the HDR images. Noise2Noise, proposed by [113], is a
representative framework for noise removal in image restoration without using
clean data. The method was shown to be effective for HDR imaging of noisy LDR
images. Godard et al. [114] employed RNNs for burst image denoising. This
method handles the noise in HDR imaging better, especially under low-light
conditions. Recently, [115] proposed a spatially dynamic learning network for
HDR imaging with denoising and dequantization. This work won 2nd place in the
NITRE2021 single-exposure HDR imaging challenge [60].

####  4.2.2 Multi-Exposure Noise Removal

Accougalan et al. [116] proposed a representative framework for noise removal
by merging multi-exposure LDR images in dynamic scenes. Interlacing is a
common problem in dual-ISO images. Therefore, a joint deinterlacing and
denoising framework is designed to reconstruct a noise-free HDR image. The
framework is similar to that of joint HDR and SR [21], where a network with
two branches is used to learn from under-exposed and over-exposed images. The
outputs from the two branches are merged to obtain an HDR image. The method in
[117] aims to reconstruct clean and noise-free HDR videos from a dual-exposure
sensor using a UNet [118]. As there is a lack of clean GT HDR videos, blur and
noise are generated manually.

Discussion: From the aforementioned analysis, there are few denoising methods
specially designed for single-exposure HDR image reconstruction. This might be
because when applying the SoTA denoising frameworks to HDR imaging directly,
the reconstructed HDR image tends to lose details [116, 14]. However, more
tailored frameworks are required, as the noise at different exposure levels
may be different when reconstructing the multi-exposure HDR image. Unlike HDR
imaging with SR, few studies have considered joint HDR imaging and image
denoising. One reason for this is the difficulty in modeling real-world noise
in LDR images. Future research should consider this problem, as real-world LDR
images contain considerable noise induced by the sensors in the under-/over-
exposed regions. Moreover, similar to HDR imaging with SR, sequential and
joint learning effectiveness needs to be studied.

###  4.3 Deep HDR Imaging with Deblurring

Insight: In the extreme imaging conditions, e.g., dark scenes, long exposures
often cause blur effects in the LDR images. Thus, learning HDR imaging must
often consider image deblurring.

Image deblurring using DNNs has been actively studied in image restoration
[119]. Regarding HDR imaging, the LDR images are often degraded by blur due to
the long exposure in extreme conditions. Therefore, some studies have
investigated deblurring for HDR imaging. [120] found that visible light
sources and bright highlight regions often appear as elongated streaks when
blurred. To this end, a DNN framework was proposed to reconstruct an HDR image
from a blurred LDR image. However, it is difficult to restore the saturated
regions for single-exposure HDR imaging. [91] interpreted single-exposure HDR
reconstruction as jointly training an optical encoder and electronic decoder.
In the framework, the points are parameterized by the point spread function
(PSF) of the lens. During the inference, a blurred LDR image is fed to the
network to reconstruct an HDR image. Similarly, [92] introduced a differential
neural sensor to optimize per-pixel shutter functions, jointly learned with a
DNN in an end-to-end manner. Modeling the exposure function enables the sensor
to capture real blurred LDR images used to reconstruct HDR images.

Discussion: Blur is an important factor to be considered in deep HDR imaging.
The methods in [91, 92] model image sensing with learnable functions to obtain
real blurred LDR images. However, these methods are limited to single-exposure
HDR imaging. Modeling image sensing could be more challenging for multi-
exposure HDR imaging in dynamic scenes. It is possible to combine blind image
deblurring methods, e.g., [121], with an HDR reconstruction pipeline. Future
research should also consider joint HDR with blind image deblurring.

##  5 Deep HDR Imaging with Novel Sensors

Figure 7: Examples of events in HDR image reconstruction. (a) HDR images from
[104]. From left to right: Dark APS images, embedded LR events, reconstructed
LR HDR images, restored LR HDR images, and SR HDR images. (b) HDR images of
[122] (fourth column), where 2nd and 3rd columns show the reconstructed LR
images of [23] and [123], respectively. Figure 8: Event-based deep HDR
methods. (a) Event-to-HDR image reconstruction. (b) Event-guided HDR image
reconstruction [22].

###  5.1 Deep HDR Imaging with Neuromorphic Cameras

Insight: Neuromorphic cameras are novel sensors that possess much higher
dynamic range than general frame-based cameras, which is promising and
advantageous for HDR imaging.

Neuromorphic cameras are bio-inspired sensors with a paradigm shift in the
manner in which visual information is acquired. Event cameras sample light
based on scene dynamics rather than a fixed frame rate, which is unrelated to
the viewed scenes. Neuromorphic cameras are different in principle from
conventional cameras [124, 125]. Unlike conventional cameras that capture
intensity values at a fixed frame rate, event cameras sense changes in
intensity asynchronously at the time they occur. Neuromorphic cameras, such as
DAVIS240C [126], have clear advantages over conventional cameras.
Specifically, they have a very high HDR ($140dB$ vs. $60dB$). Active pixel
sensor (APS) frames (a type of frame-based intensity sensor) are less
invisible in low-light conditions; however, event cameras capture the
intensity changes and output convincing visual details. These distinct
advantages of event cameras make them promising for robotics, autonomous
driving, and wearable applications in scenarios challenging for standard
cameras, such as HDR and high-speed motion [124].

####  5.1.1 Reconstruction-Based HDR Imaging

As shown in Fig. 8(a), APS frames are less visible when most pixels are under-
exposed, but events reflect the scene details with clear edges of objects. To
fully explore these visual details, Wang et al. [23, 122] proposed a
representative framework for reconstructing HDR intensity images and videos
from event data using a conditional generative adversarial network (cGAN).
Concurrently, Rebecq et al.[123] employed an RNN to reconstruct temporally
consistent HDR-intensity videos. Inspired by [23], Zhang et al.[127], and Zhe
et al. [128], further investigated the reconstruction of HDR images from event
data under low-light (night) scenes. Some studies further extended the
approach of [123] by either minimizing network parameters [129], improving the
generalization of simulated event data [130], or adding optical flow [131] to
impose photometric consistency [132] for HDR video reconstruction. Although
these methods are effective, the reconstructed HDR results are in LR as event
cameras, such as [126], are in LR. To overcome these challenges, Mostafavi et
al. [133] and Wang et al. [104, 134] proposed the reconstruction of HR HDR
intensity images from LR event data in a supervised and unsupervised learning
manner. Visual examples of super-resolved HDR images are shown in Fig. 7.
Moreover, [135] leveraged the HDR property of the event camera and proposed an
event-based synthetic aperture imaging method by combining spiking neural
networks (SNNs) and CNNs. The proposed method collects confident light
information from occluded targets even under extreme lighting conditions,
making the reconstruction of scenes feasible for HDR. Inspired by the HDR
image reconstruction methods [104, 23], Wang et al.[98, 136] proposed to
jointly reconstruct HDR images and learn end-tasks under either under-exposed
or over-exposed conditions via knowledge distillation [83]. Moreover, [137]
proposed a recurrent neural network (RNN) for reconstructing high-speed HDR
videos from events.

Potential and challenges: Event cameras show better HDR capability than frame-
based cameras, enabling the aforementioned methods to reconstruct HDR images
from events. The reconstructed HDR images bridge the event and intensity
domains and can be potentially applied to other vision and graphic tasks.
However, some challenges remain. First, reconstructing HDR images using DNNs
induces considerable inference latency [123, 138]. Second, as events
predominantly reflect the edge information of scenes and suffer from noise due
to abrupt intensity changes in extreme lighting conditions, the reconstructed
HDR images are degraded by noise and loss of textural details. In other words,
crucial visual information may be missing during the reconstruction stage.
Finally, the aforementioned methods, except for [98, 136], only consider HDR
image reconstruction, with less consideration of its application to end tasks.
Future research should consider jointly learning HDR reconstruction and end
tasks to improve efficiency.

####  5.1.2 Neuromorphic camera-Enhanced HDR Imaging

The other line of research aims to directly leverage events to guide the
reconstruction of HDR images from LDR image content, as illustrated in Fig.
8(b). Han et al. [22] proposed a representative framework for single-exposure
HDR reconstruction by adding the intensity map information of an event camera.
In particular, a hybrid camera system is set up to jointly take a single HR
LDR image and an LR intensity map generated from the events. The proposed HDR
reconstruction framework addresses the challenges of gaps in resolution,
dynamic range, and color representation between two types of sensors and
images. Moreover, Wang et al. [138] proposed a unified framework by
simultaneously denoising, deblurring, and super-resolving the LDR APS images
to reconstruct clean HR HDR images. The idea is to integrate the sparse events
into a sparsity network, where the LDR APS images and events are jointly
learned to reconstruct HDR images. [139] proposed a learning framework to
estimate the ambient light using an event camera based on its advantage of
obtaining HDR images.

Discussion: Based on our review, only three event-guided deep HDR imaging
methods have been proposed in the literature. Han et al. [22] focused on
single-exposure HDR image reconstruction guided by an event-generated
intensity map. There are three critical problems associated with this method.
First, the intensity map is obtained by running off-the-shelf intensity
reconstruction networks, such as [123]. Therefore, there is no direct
optimization of HDR image reconstruction and intensity map acquisition.
Second, it is limited to single-exposure HDR image reconstruction; this limits
its ability to handle diverse LDR scenes. Finally, the HR image sensor and LR
event sensor are not perfectly aligned, distorting the reconstructed HDR
images. Future research directions should consider end-to-end HDR image
reconstruction by jointly learning intensity reconstruction and LDR-to-HDR
image mapping. Importantly, multi-exposure HDR reconstruction approaches,
enhanced by multiple event stacks, should be considered to process more
complex saturated regions of the LDR images. Moreover, a cross-modality HDR
imaging method is explored. The corresponding research question is whether it
is possible to directly fuse image and event data and learn a unified HDR
imaging framework without relying on intensity reconstruction.

###  5.2 Deep HDR Imaging with Thermal Sensors

Insight: Thermal or infrared (IR) cameras can highlight thermal objects of the
scene and output contour information in a night-time environment, enabling HDR
imaging via intensity reconstruction or feature fusion with LDR images.

####  5.2.1 Thermal to HDR Image Generation

IR cameras possess distinct HDR capabilities, especially under low-light
conditions. Accordingly, translating an IR image into an HDR color image could
be a good solution for enhancing scene perception at night [140]. However, as
IR images are grayscale and only reflect the object contour, transforming IR
images into HDR color images is challenging. Therefore, recent DNN-based
methods [24, 141, 140] aim to learn transformation mapping from the IR image
domain to the HDR color image domain. To better preserve the texture
information from IR images, many methods have been developed using GANs [81,
82]. The method, proposed by Kuang et al. [24], is a representative framework
in which a coarse-to-fine generator was used to preserve the textural details
of IR images and generate color HDR images in a supervised manner, as shown in
Fig. 9(a). By contrast, IR-GVI in [140] assumes that no GT color HDR images
are available. IR-GVI thus proposes an unsupervised method to map IR images to
HDR grayscale images and colorize these images, as shown in Fig. 9(b). The
method in [141] considers GAN-based approaches to be complex and proposes a
deep HDR color image reconstruction framework based on the registration with
optical image pairs. This method reconstructs high-quality HDR color images
and preserves thermal profiles even at night.

####  5.2.2 Thermal and Image Fusion

The other line of research explores fusing IR and LDR images for color HDR
image reconstruction [142], similar to the multi-exposure HDR imaging pipeline
[10, 33]. The intuition is that IR images possess grayscale HDR content, but
in a different domain than LDR images. Therefore, the challenge is to merge
heterogeneous image content or features to reconstruct HDR images. Based on
our review, two approaches are commonly used. The first approach involves
stacking IR and LDR images, as shown in Fig. 9(c) before feeding the results
to the reconstruction network [143, 17, 144] using GAN. The second strategy is
based on feature fusion [144, 145], as shown in Fig. 9(d). Notably, the method
in [144] employs both image content fusion and feature fusion. The image
content is first decomposed and then fused to be fed into a network. Feature
fusion based on the VGG network [146] is applied to fuse the image features.

###  5.3 Deep Modulo Camera-based HDR Imaging

Conventional cameras fail to capture the over-exposed or under-exposed regions
in a scene, which limits the dynamic range of the output. Therefore, HDR
reconstruction may lead to considerable artifacts. The modulo camera concept
was first proposed by [147], aiming to mitigate the limited dynamic range of
conventional cameras using a novel, practical sensor. The sensor resets pixels
to zero whenever the maximal value is reached in the exposure time to avoid
saturation. In [148], HDR image reconstruction using neural priors was
proposed. The important feature of this method is that it requires no training
data for the dynamic range enhancement of low-light images. The method
proposed by Zhou et al.[149] reformulates the unwrapping of a modulo image
into a series of binary labeling problems and proposes a DL-based framework to
estimate the binary rollover mask of the modulo images iteratively. To date,
deep HDR imaging methods using modulo cameras are still limited. Future
endeavors could be directed toward this direction.

Figure 9: Thermal camera-based HDR methods. (a) Supervised color HDR
reconstruction method (b) Unsupervised color HDR reconstruction methods with
two steps [140]. (c) IR and LDR image concatenation-based HDR reconstruction
methods (d) IR and LDR feature concatenation-based HDR reconstruction method.

###  5.4 Discussion and Challenges

Generating HDR images from IR images using GANs is the most common approach;
however, the results are usually worse than those based on fusion methods.
This is because some visual information might be lost in the generation
process, or IR images possess limited visual details; that is, contours of
objects are degraded by noise in the LDR conditions. Moreover, recovering
color information remains challenging because it is an ill-posed problem.
Importantly, generating HDR images leads to considerable inference latency in
high-level tasks, such as object detection.

IR cameras show distinct HDR capabilities and are promising for HDR
reconstruction, as shown in Fig. 10. However, some challenges remain. First,
because IR cameras output only grayscale images, most fusion-based methods can
only reconstruct grayscale HDR intensity images. Although DenseFuse [145] can
reconstruct color images, color information is not naturally encoded and
recovered. Most fusion algorithms are based on GANs and require many IR image
training datasets; however, high-quality datasets remain scarce. Our review
identifies that most fusion methods concatenate two images or image patch
inputs to DNNs. By contrast, fusing features is another approach, as in [145];
however, this approach treats all features equally, which may not be
reasonable in low-light conditions as some pixels in the images are saturated.
Therefore, it would be better to filter pixels by considering the degree of
exposure in LDR regions.

##  6 HDR with Novel Learning Strategies

###  6.1 HDR Imaging with Unsupervised Learning

Insight: Unsupervised learning reduces the need for GT data and is promising
for LDR image fusion and HDR image reconstruction.

Unsupervised MEF. MEF is a type of unsupervised learning that has been
actively studied in static LDR image fusion. The MEF aims to fuse LDR images
with varying exposures into a single HDR image. DeepFuse [72] is a
representative unsupervised method that contains three types of layers:
feature extraction, fusion, and reconstruction layers. The framework is
learned without a reference image using an objective function based on the
SSIM image quality measure. The objective function maximizes the structural
consistency between the fused image and each input image. [19] and [150]
proposed GAN-based unsupervised frameworks, inspired by CycleGAN [101], to
learn LDR image fusion. [151] and [152] explored the correspondence of source
LDR images. As such, the similarity between the fused output and the source
images is adaptively preserved.

Unsupervised LDR-to-HDR Mapping. HDR imaging without GT data is challenging,
especially for single-exposure HDR image reconstruction. Based on our review,
there are no unsupervised methods for single-exposure HDR image
reconstruction. For multi-exposure HDR imaging, Li et al. [153] proposed a
representative method called UPHDR-GAN to relax the constraint of paired data.
The framework is built on GAN, where the generator takes three LDR images and
aims to reconstruct an HDR image. The discriminator distinguishes the tone-
mapped image from an unpaired real HDR image. Additionally, perceptual loss is
used to preserve the semantic information.

Figure 10: Visual results of IR-based methods. (a) HDR content for image
reconstruction [24]. (b) HDR results from IR and LDR image fusion [145]. From
left to right: IR, LDR, and HDR results.

Discussion: In our review, most unsupervised methods are targeted at FEM-HDR,
and direct LDR-to-HDR mapping methods are rare. Although it is possible to
reconstruct an HDR image reconstruction from multi-exposure LDR images without
using paired data [153], the reconstruction quality is worse than that of
supervised methods. Moreover, there is no unsupervised method for single-
exposure HDR imaging. Future research may explore more efficient learning in
this direction. It could be beneficial to apply contrastive learning [154] and
domain adaptation [155] to unsupervised HDR imaging.

###  6.2 HDR Imaging with GAN

Insight: Using GAN for HDR imaging can help to estimate the true data
distribution of light intensity and local contrast..

The key problem in HDR imaging is learning the true data distribution of the
light intensity and local contrast. Using L1 or L2 losses to minimize the
difference between the HDR image and GT cannot effectively address the
problem. By contrast, GANs can learn the true data distribution [81, 82, 156].
Recently, the use of GANs for HDR imaging has attracted attention in the
community. In the previous sections, we briefly mentioned the use of GANs for
HDR imaging. In this section, we explicitly analyze the technical details of
GANs for HDR imaging. We assume that the readers are familiar with the basics
of GANs and skip the definitions and formulations. For an introduction to
GANs, refer to [81].

GANs can be applied to HDR imaging as follows: the HDR reconstruction network
can be regarded as the generator; then, a discriminator is defined to discern
whether it is the generated image or GT. [157, 13] first proposed to add an
adversarial loss [81] to help train the UNet-based generator [118] for
learning an inverse tone mapping. [158] then adopted cGAN [159] to learn HR
and high-quality tone mapping. Considering that the problem of single-exposure
HDR imaging is ill-posed, [160] proposed generating multi-exposure stacks
using cGAN. A cycle consistency loss [101] was also used to preserve the
exposure level. [18] proposed a GAN-based method for both HDR and SR, where
RaGAN [161] was adopted as a basic adversarial loss for stable training. All
of these methods focus on single-exposure HDR imaging. By contrast, [43, 69]
applied GANs to multi-exposure HDR imaging. Specifically, the method in [43]
takes under- and over-exposed images and feeds them into two independent
feature learning branches, which are then concatenated to reconstruct an HDR
image. A discriminator is used to distinguish whether the fused image was from
the GT. The method in [69] uses three LDR images, similar to the reference-
based HDR methods, and feeds them to a multiscale LDR image encoder. The
extracted features are gradually aggregated to larger scales and finally
concatenated to reconstruct an HDR image. A hypersphere adversarial loss [162]
was used to train the generator based on the PatchGAN discriminator [163].

Discussion: Although GAN-based HDR methods have shown promising results, some
challenges remain. The first challenge is the stability of the training,
especially when the LDR data distribution is diverse. Although these methods
focus on inverse tone mapping–based (single-exposure) HDR image reconstruction
using GAN, they are limited under more complex LDR conditions. It has been
shown to achieve multi-exposure HDR image reconstruction with GANs [69, 43];
however, only a single discriminator is used to distinguish the entire
generated HDR image content from the GT. Indeed, an additional discriminator
can distinguish the local regions of the LDR images from the HDR images.
Moreover, it is also feasible to deploy a discriminator to facilitate learning
using the learned feature representations [164].

###  6.3 HDR Imaging with Transfer Learning

Insight: Transfer learning improves the learning of HDR by transferring
knowledge from a related learned task.

Recent advancements in DL have shown the possibility of transferring
information from a learned task to a target task [83]. This DL strategy has
also been studied for HDR imaging. [165] leveraged transfer learning to
overcome the lack of sufficiently large HDR datasets. This demonstrates that
transferring knowledge from image classification on ImageNet considerably
improves single-exposure HDR reconstruction. [25] showed the possibility of
transferring the learned information from a related larger database to a
smaller one. In particular, AlexNet [166] is used to extract features that are
adapted to the images of a small database by reducing the dimensionality of
the feature vector.

Discussion and Potential: From the aforementioned analysis, it is apparent
that transfer learning has been less explored for HDR imaging. [165, 25]
transferred the learned features from the image classification task and
adapted them to the learning from limited datasets. Recent developments in
transfer learning methodologies have been shown to be effective for learning
more robust networks and model compression [83]. One intuitive research
direction is to fully utilize the trained models using labeled HDR datasets to
learn a robust student model on an unlabeled dataset. Moreover, it is also
meaningful to aggregate the knowledge from the networks learned on multiple
datasets to learn a unified model to address more complex LDR image content.
Additionally, the visual information of GT HDR images is less exploited in
existing methods; however, it is feasible to transfer the feature information
from the HDR image to the reconstruction network learning LDR image content.
Finally, in multi-exposure HDR imaging, the multi-exposed images are only
aligned to the reference image; nonetheless, the visual information of the
non-reference image was less excavated. It might be a good solution to build a
mutual learning framework [167] among student peers to learn more effective
knowledge.

###  6.4 HDR Imaging with Meta-Learning

Insight: Meta-learning enables quick learning of nonlinear mapping in HDR
imaging given a few specific LDR examples.

In HDR image reconstruction, there exists a nonlinear mapping between the
radiance in a scene and the recorded pixel values of an LDR image [41]. The
SoTA DL methods, e.g., [14, 10], assume that there exists one consistent
nonlinear mapping in single-task learning for all possible scenes. However,
every scene has unique nonlinear mapping. Thus, [26] proposed a meta-learning
framework in which the ground of meta-parameters is set to learn the common
patterns in the nonlinear mapping. It is reasonable to learn nonlinear mapping
to better adapt to a specific HDR image reconstruction task.

Potential and Challenges. From the aforementioned analysis, it is obvious that
the potential of meta-learning remains unexplored. Although [26] proposed a
framework using the popular meta-learning algorithm MAML [168], the HDR images
still suffer from artifacts caused by the less effective loss functions that
handle color saturation problems. Future research should explore better meta-
learning algorithms to capture nonlinear mapping. The method in [169] proposes
a meta-transfer learning framework to solve the SISR problem, where the DNNs
are first trained using a large-scale dataset. Then, meta-transfer learner is
built into the internal information within a single image. From [169], it is
possible to combine transfer learning with meta-learning to learn a more
robust nonlinear mapping for HDR imaging. Moreover, it is worth exploring a
few-shot meta-transfer learning framework [170] to adapt to the HDR image
reconstruction network.

###  6.5 HDR Imaging with the Attention Mechanism

Insight: Attention enables the HDR reconstruction network to focus on the most
relevant regions of LDR images.

The attention mechanism has been shown to enhance the performance of many
computer vision problems. Readers can refer to the seminal work [74] for more
details. The attention mechanism for HDR imaging has also been actively
studied. [171] applied the channel-wise attention to capture the feature
interdependences between channels. The attention module is built using global
average pooling to obtain the global distribution. [33] proposed learnable
attention modules to guide the merging process of multiple exposures. In
particular, attention modules generate soft attention maps to evaluate the
importance of different image regions in reconstructing an HDR image. The
feature merge using attention guidance was shown to be effective in obtaining
ghost-free HDR results. The method of attention guidance for the multi-
exposure merge was further enriched by [75], which exploited the nonlocal
correlation in the inputs between the encoder and decoder of the UNet
structure [118]. The non-local block helps build dependency between different
exposures, thus removing the ghosting artifacts for HDR imaging.
Interestingly, to better align the multi-exposed LDR images, [172] proposed a
multiscale contextual attention module to obtain multiscale attention feature
maps for alignment. By contrast, [173] focused on static multi-exposure HDR
image reconstruction and proposed the use of channel-wise attention, similar
to [171], to learn the channel-wise statistics for the under-/over-exposed
regions in the LDR images.

Discussion: From the aforementioned analysis, we identified that the attention
mechanism is effective for multi-exposure mergers regardless of dynamic or
static scenes. Using the attention mechanism leads to fewer ghost artifacts
than when using an optical flow. However, as pointed out by [173], using
attention mechanisms still has limitations for LDR images with fewer sharp
edges and textures. Increasing the effectiveness of attention mechanisms
requires more parameters for the reconstruction networks, thus increasing the
computational cost. Therefore, lightweight attention networks that balance the
performance and computation costs should be considered.

###  6.6 HDR Imaging with Deep Reinforcement Learning

Insight: Deep reinforcement learning can effectively learn multiple local
exposure operations based on policy gradient.

Deep reinforcement learning is widely used in many visual learning tasks. HDR
imaging has been applied to learn multiple local exposure operations [174]. To
apply the policy gradient algorithm, the LDR image is decomposed into multiple
sub-images. For different sub-images, different exposures are used according
to the policy network. To simplify the framework, adversarial learning is
adopted, where the discriminator is regarded as a value function. This simple
yet novel reinforced adversarial learning approach shows superior visual
results for HDR image reconstruction.

Discussion: Currently, there is only one method using deep reinforcement
learning to exploit multiple local exposure operations in a single image for
HDR image reconstruction. Although the learning strategy is different, this is
similar to generating multi-exposure LDR image stacks from a single LDR image.
Future research should consider MEF methods using reinforcement learning.
Furthermore, in [174], sub-images are obtained using image segmentation.
However, this method only considers class-level and non-exposure-level
regions. Therefore, it is beneficial to segment images based on exposure
information to better learn over-/under-exposed regions.

##  7 Applications

HDR imaging can generate images with a greater range of illuminance than
general LDR images, resulting in a better visual representation of detail in
highlights or shadows [175]. Thus, it is of great interest to apply HDR
imaging to address some graphics/vision problems in real-life applications. In
this section, we review SoTA research on the applications of imaging
techniques. HDR imaging techniques have been applied to computer vision tasks,
such as object recognition, scene segmentation, depth estimation, and visual
odometry. In computer graphics, it is mainly used for estimating lighting and
illumination.

Scene Recognition and Detection. HDR images naturally reflect light patterns
and textural details in dark and bright backgrounds. In a representative
study, Wang et al. [175, 27] applied HDR imaging to traffic light recognition
for autonomous driving. In their approach, traffic light candidates were
detected from the low-exposure frames and accurately classified using a DNN in
the high-exposure frames. [106] presented a DNN method for detecting the
source of HDR images from LDR images.

Conversely, existing object detection networks trained on general LDR images
cannot detect relevant objects in adverse visual conditions. [28] proposed an
approach to detect objects using HDR imaging under extreme lighting conditions
based on existing CNN frameworks. The approach does not require HDR datasets
and expensive retraining procedures because it uses HDR-to-LDR mapping
techniques, which preserve the salient highlight, shadow, and color details.
Existing DNN models use mapped LDR images to extract relevant features for
object detection. [176] introduced a DNN-based framework for saliency
detection of HDR content. The framework follows a similar idea as [28]: first,
HDR images are transformed to LDR images; next, neural networks detect the
salient regions. Recently, [177] proposed a joint learning approach in which
exposure control is trained with object detection in an end-to-end manner.

Semantic Segmentation. Semantic segmentation is an important per-pixel
classification task. Recently, HDR imaging has been used to improve the
performance of DNNs. [29] proposed a preliminary method that leveraged HDR
data to learn segmentation networks. It generates LDR images from HDR images
using various one-mapping operators. The network can be trained using both LDR
and HDR images. Domain adaptation is used for training synthetic and real-
world datasets. [178] presented a per-pixel classification method for
identifying cloud types based on HDR sky images. Their method uses a patch-
based feature extraction approach, which extracts the representative feature
vectors used for labeling.

Stereo and Depth Estimation. As mentioned in Sec. 3.4, many attempts have been
made to obtain HDR images from stereo LDR image pairs. This section analyzes
some recent DL-based approaches that use HDR imaging for stereo matching and
depth estimation. The method in [32] is a representative study using multi-
exposure LDR images for stereo matching, and the HDR fusion network is learned
using the warped LDR images. [179] introduced a depth estimation method from a
short burst of multi-exposed LDR images. The estimated depth could align the
LDR images, which are necessary for achieving HDR or image fusion.

Illumination Estimation. Illumination is important for determining the
appearance of a scene. Correspondingly, research has focused on predicting HDR
lighting conditions using LDR images. [180, 181, 182, 183] are representative
methods aiming to estimate the illumination in either indoor or outdoor
environments using DNNs. The core pipeline of these methods is to train DNNs
to achieve image-to-illumination mapping to model varying light conditions.
However, learning illumination requires many high-quality datasets with the
corresponding illumination conditions; thus, most methods explore indirect or
multistep approaches to achieve this goal.

##  8 Discussion and New Perspectives

Pros and Cons of DL-based Methods vs. the Prior Arts. The common approach in
the early methods [34, 184, 185] are to generate HDR images by aligning
multiple LDR images captured with different exposures via optimization.
However, these methods have two distinct disadvantages. First, they failed to
work on complex backgrounds and large motions. Second, optical flow-based
alignment algorithms cannot generate new contents in the saturation and
occlusion regions. By contrast, DL-based methods [49, 9, 33] better address
the artifacts induced in large foreground motions. Moreover, DNN-based optical
flow algorithms can better align the LDR images and generate new content in
the saturated and occlusion regions. Overall, the DL-based methods showed
significant performance gains.

By contrast, traditional single-image HDR imaging methods estimate the density
of light sources to expand the dynamic range [186, 187, 188, 189]. However,
these methods cannot recover the brightness of the saturated regions as they
do not utilize contexts in the images. Therefore, prior arts suffer from
considerable artifacts and show less plausible performance, which can be
verified in the experimental studies in the supplementary material. By
contrast, DL-based methods [10, 14, 47] systematically utilize contextual
information and produce better results in saturated regions. Although DL-based
methods are promising, they are computationally more expensive, and it is
necessary to create LDR scenes with diverse exposure information to
reconstruct HDR images.

Exposure Bracket Selection for Deep HDR Imaging. Exposure bracket selection is
an important factor in deep HDR imaging. This depends on several key factors.
The first is the dynamic range of a scene. The exposure value (EV) determines
exposure compensation, and is a term used for bracketing. Therefore, EV+0 is
the exposure of the camera’s default value. ‘EV-2’ means that its exposure is
‘minus two shots’. EV+2 means that its exposure is ‘plus two shots’. Three
exposure shots at -2/0/+2 EV can usually handle most scenes, which is a common
setting in representative works, e.g., [49, 33, 9]. However, scenes with more
over-exposed pixels, such as pixels with sunlight, may require more over-
exposed LDR shots (e.g., 5 shots), and vice versa. Although more brackets
contribute to better HDR imaging quality, the computation costs increase
accordingly. Therefore, most deep HDR imaging methods consider three brackets,
which are merged into a single HDR.

Feature Representations in Network Design. Feature representation is an
important factor in HDR imaging, especially in multi-exposure HDR image
reconstruction. Learning features from LDR images using the attention
mechanism are common; however, it requires more computations. A possible
solution is to use more efficient yet effective attention modules, e.g.,
[190], to learn more correlated features. Another direction is to explore
contrastive learning [191] to learn better feature representations from LDR
images. Moreover, the potential of GT HDR images has not yet been fully
exploited. Existing methods only leverage HDR images for supervision; however,
they can be used to learn privileged knowledge, as in [160]. Therefore, it is
possible to add a removable network that encodes the feature representations
of HDR images to guide the learning of LDR image features.

Data-efficient Learning. A challenge in deep HDR imaging is the need for
large-scale labeled datasets to train the DNN models. However, for multi-
exposure HDR image reconstruction, building large-scale datasets is expensive
and challenging. Therefore, it is necessary to explore more data-efficient
deep HDR imaging methods. One promising direction is to explore self-
supervised or unsupervised learning to better explore visual information from
unlabeled LDR images. In this situation, it is crucial to learn more reliable
visual information based on data augmentation or adversarial examples [192].
Another possible solution is to explore data-free knowledge transfer [193] to
reduce the requirements for training data. Finally, the recent development of
few-shot learning [194] may be a way to learn robust HDR imaging networks,
reducing the need for labeled data.

Computationally Efficient Learning. Based on our review, the SoTA multi-
exposure DHR image reconstruction methods include feature encoding, merging,
and reconstruction. The encoding phase relies on learning to exclude
misaligned features using attention or homography; the merging phase also
needs to avoid harmful features. This inevitably leads to a large number of
convolutional operations to ensure a robust performance. Consequently, DNNs
are extremely deep and large. However, real-time methods are preferred in
practice. Thus, there is an urgent need to develop lightweight HDR imaging
models. One solution is to compress the learned models using network pruning
[195] or knowledge distillation [83]. Another solution is to replace the
standard convolution with depth-wise convolution or inverted residual
structures [196].

Potential of Cross-Task Consistency. Research on HDR imaging with multitask
learning has become more practical. To learn multiple tasks, existing methods
mostly rely on sequential or joint learning. However, these methods incur
considerable computational costs and require high-quality datasets. For
instance, joint HDR and SR inevitably add the requirement for HR cameras to
capture HR scenes. Recent progress in multitask learning [197] shows that the
predictions of multiple tasks for the same image are not independent but are
expected to be consistent. Therefore, future research may explore the
intrinsic consistency between HDR imaging and other tasks. Importantly, new
loss constraints are required to ensure cross-task consistency in HDR imaging.

Potential of Novel Sensors. In Sec. 5, we reviewed and analyzed deep HDR
imaging techniques using recently developed sensors, such as event cameras and
thermal sensors. These sensors usually capture scenes with a relatively high
HDR. There have been some attempts to reconstruct HDR images using these
sensors; however, existing methods should be further improved and studied. For
instance, although the method in [22] uses the reconstructed intensity maps to
enhance the LDR-to-HDR image mapping, intensity generation is not
differentiable; thus, the intensity maps cannot fully convey the HDR
information of the event data. EvDistill [98] is a method for joint learning
of intensity reconstruction and end tasks in an end-to-end manner in LDR
scenes. This might be a potential direction for event-guided HDR image
reconstruction. Another solution might be to directly fuse the intermediate
features of events and those of the LDR images using multimodality fusion
techniques, such as [198].

Potential of GNNs for HDR Imaging. Attention and non-local feature aggregation
[75, 33] have been shown to be effective for deep HDR imaging. However, these
methods exploit only the similar patches of LDR images within the same scale.
Although multiscale dense networks are important for learning the illuminance
information in LDR images, they only aggregate the feature information in
local regions. IGNN [199] adopts graph neural networks (GNNs), which aggregate
similar cross-scale patches of the LR images in a graph, to complement
external information learned from the training dataset. In HDR imaging, it has
been shown that recurrence and correlation of local patches of LDR images are
crucial for either exposure alignment or merging. Therefore, future research
may explore the potential of GNNs to learn dense feature information from LDR
images by exploring the cross-scale recurrence property in graphs.

Toward Real-World HDR. Deep HDR imaging is a practical technique, but it is
limited to real-world scenarios. For example, it suffers from unexpected
saturated regions in the LDR images and limited exposure levels. Although many
methods have achieved reasonable performance on simulated datasets, their
performance dramatically drops on more complex real-world LDR scenes. Research
should consider the lack of generalization of DNNs. One solution is to apply
unsupervised domain adaptation to adapt the simulated exposure information to
DNN learning real-world LDR images. Another solution is to aggregate knowledge
from multiple DNNs [83] trained on various LDR contents and transfer the
knowledge to the DNN learning real-world LDR images.

##  9 Conclusion

We reviewed the major technical details and applications of deep HDR imaging.
We formally defined the problem and introduced a hierarchical and structural
taxonomy of deep HDR imaging methods. For each category in the taxonomy, we
reviewed the current technical status, potential, and challenges. Drawing
connections among these approaches highlights new active research fields and
is likely to identify new research problems and methods by exploring the
potential of each paradigm. Based on our review and analysis, we discussed the
ways in which the existing challenges can be overcome and the research gaps
can be addressed.

## References

  * [1] Wikipedia. High-dynamic-range imaging. [Online]. Available: https://en.wikipedia.org/wiki/High-dynamic-range_imaging.
  * [2] J. Chang. [Online]. Available: https://github.com/JimmyChame/The-State-of-the-Art-in-HDR-Deghosting.
  * [3] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, _Advanced high dynamic range imaging_. CRC press, 2017\.
  * [4] G. Tiwari and P. Rani, “A review on high-dynamic-range imaging with its technique,” _IJSIP_ , vol. 8, no. 9, pp. 93–100, 2015.
  * [5] O. T. Tursun, A. O. Akyüz, A. Erdem, and E. Erdem, “The state of the art in hdr deghosting: A survey and evaluation,” in _CGF_ , vol. 34, no. 2, 2015, pp. 683–707.
  * [6] A. K. Johnson, “High dynamic range imaging-a review,” _IJIP_ , vol. 9, no. 4, p. 198, 2015.
  * [7] L. Wang and K.-J. Yoon, “Co-designmr: A mr-based interactive workstation design system supporting collaboration,” _arXiv preprint_ , 2019.
  * [8] Q. Yan, L. Zhang, Y. Liu, Y. Zhu, J. Sun, Q. Shi, and Y. Zhang, “Deep hdr imaging via a non-local network,” _IEEE TIP_ , vol. 29, pp. 4308–4322, 2020\.
  * [9] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes.” _ToG_ , vol. 36, no. 4, pp. 144–1, 2017.
  * [10] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “Hdr image reconstruction from a single exposure using deep cnns,” _ACM TOG_ , vol. 36, no. 6, pp. 1–15, 2017.
  * [11] S. Lee, G. H. An, and S.-J. Kang, “Deep recursive hdri: Inverse tone mapping using generative adversarial networks,” in _European Conference on Computer Vision_ , 2018, pp. 596–611.
  * [12] Z. Khan, M. Khanna, and S. Raman, “Fhdr: Hdr image reconstruction from a single ldr image using feedback network,” _arXiv preprint_ , 2019.
  * [13] S. Ning, H. Xu, L. Song, R. Xie, and W. Zhang, “Learning an inverse tone mapping network with a generative adversarial regularizer,” in _IEEE ICASSP_. IEEE, 2018, pp. 1383–1387.
  * [14] Y.-L. Liu, W.-S. Lai, Y.-S. Chen, Y.-L. Kao, M.-H. Yang, Y.-Y. Chuang, and J.-B. Huang, “Single-image hdr reconstruction by learning to reverse the camera pipeline,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1651–1660.
  * [15] H. Zeng, J. Cai, L. Li, Z. Cao, and L. Zhang, “Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time,” _IEEE TPAMI_ , 2020.
  * [16] J. Zhang and J.-F. Lalonde, “Learning high dynamic range from outdoor panoramas,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2017, pp. 4519–4528.
  * [17] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, “Fusiongan: A generative adversarial network for infrared and visible image fusion,” _Information Fusion_ , vol. 48, pp. 11–26, 2019.
  * [18] S. Y. Kim, J. Oh, and M. Kim, “Jsi-gan: Gan-based joint super-resolution and inverse tone-mapping with pixel-wise task-specific filters for uhd hdr video,” in _AAAI_ , vol. 34, no. 07, 2020, pp. 11 287–11 295.
  * [19] Z. Yang, Y. Chen, Z. Le, and Y. Ma, “Ganfuse: a novel multi-exposure image fusion method based on generative adversarial networks,” _NCA_ , pp. 1–13, 2020.
  * [20] J. H. Kim, S. Lee, S. Jo, and S.-J. Kang, “End-to-end differentiable learning to hdr image synthesis for multi-exposure images,” _AAAI_ , 2020.
  * [21] S. Y. Kim and M. Kim, “A multi-purpose convolutional neural network for simultaneous super-resolution and high dynamic range image reconstruction,” in _ACCV_. Springer, 2018, pp. 379–394.
  * [22] J. Han, C. Zhou, P. Duan, Y. Tang, C. Xu, C. Xu, T. Huang, and B. Shi, “Neuromorphic camera guided high dynamic range imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1730–1739.
  * [23] L. Wang, Y.-S. Ho, K.-J. Yoon _et al._ , “Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 081–10 090.
  * [24] X. Kuang, J. Zhu, X. Sui, Y. Liu, C. Liu, Q. Chen, and G. Gu, “Thermal infrared colorization via conditional generative adversarial network,” _Infrared Physics & Technology_, vol. 107, p. 103338, 2020.
  * [25] V. A. Kumar, S. Gupta, S. S. Chandra, S. Raman, and S. S. Channappayya, “No-reference quality assessment of tone mapped high dynamic range (hdr) images using transfer learning,” in _QoMEX_. IEEE, 2017, pp. 1–3.
  * [26] E. Pan and A. Vento, “Metahdr: Model-agnostic meta-learning for hdr image reconstruction,” _arXiv preprint_ , 2021.
  * [27] J.-G. Wang and L.-B. Zhou, “Traffic light and vehicle signal recognition with high dynamic range imaging and deep learning,” in _Deep Learning: Algorithms and Applications_. Springer, 2020, pp. 157–192.
  * [28] R. Mukherjee, M. Melo, V. Filipe, A. Chalmers, and M. Bessa, “Backward compatible object detection using hdr image content,” _IEEE Access_ , vol. 8, pp. 142 736–142 746, 2020.
  * [29] M. Weiher, “Domain adaptation of hdr training data for semantic road scene segmentation by deep learning,” 2019.
  * [30] O. Gallo and P. Sen, “Stack-based algorithms for hdr capture and reconstruction,” in _High Dynamic Range Video_. Elsevier, 2016, pp. 85–119.
  * [31] A. Srikantha and D. Sidibé, “Ghost detection and removal for high dynamic range images: Recent advances,” _SPIC_ , vol. 27, no. 6, pp. 650–662, 2012\.
  * [32] Y. Chen, G. Jiang, M. Yu, Y. Yang, and Y.-S. Ho, “Learning stereo high dynamic range imaging from a pair of cameras with different exposure parameters,” _IEEE TCI_ , vol. 6, pp. 1044–1058, 2020.
  * [33] Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [34] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, “Robust patch-based hdr reconstruction of dynamic scenes.” _ToG_ , vol. 31, no. 6, pp. 203–1, 2012.
  * [35] O. T. Tursun, A. O. Akyüz, A. Erdem, and E. Erdem, “An objective deghosting quality metric for hdr images,” in _CGF_ , vol. 35, no. 2, 2016, pp. 139–152.
  * [36] Y. Endo, Y. Kanamori, and J. Mitani, “Deep reverse tone mapping,” _ACM TOG (SIGGRAPH ASIA)_ , vol. 36, no. 6, Nov. 2017.
  * [37] H. Nemoto, P. Korshunov, P. Hanhart, and T. Ebrahimi, “Visual attention in ldr and hdr images,” in _VPQM_ , no. CONF, 2015.
  * [38] S. Lee, G. H. An, and S.-J. Kang, “Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image,” _IEEE Access_ , vol. 6, pp. 49 913–49 924, 2018.
  * [39] K. R. Prabhakar, R. Arora, A. Swaminathan, K. P. Singh, and R. V. Babu, “A fast, scalable, and reliable deghosting method for extreme exposure fusion,” in _ICCP_. IEEE, 2019, pp. 1–8.
  * [40] J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” _TIP_ , vol. 27, no. 4, pp. 2049–2062, 2018\.
  * [41] S. Y. Kim, J. Oh, and M. Kim, “Deep sr-itm: Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3116–3125.
  * [42] H. Xu, J. Ma, Z. Le, J. Jiang, and X. Guo, “Fusiondn: A unified densely connected network for image fusion,” in _AAAI_ , vol. 34, no. 07, 2020, pp. 12 484–12 491.
  * [43] H. Xu, J. Ma, and X.-P. Zhang, “Mef-gan: multi-exposure image fusion via generative adversarial networks,” _TIP_ , vol. 29, pp. 7203–7216, 2020.
  * [44] G. Chen, C. Chen, S. Guo, Z. Liang, K.-Y. K. Wong, and L. Zhang, “Hdr video reconstruction: A coarse-to-fine network and a real-world benchmark dataset,” _arXiv preprint_ , 2021.
  * [45] Y. Jiang, I. Choi, J. Jiang, and J. Gu, “Hdr video reconstruction with tri-exposure quad-bayer sensors,” _arXiv preprint_ , 2021.
  * [46] V. V. Kniaz, V. A. Knyaz, J. Hladuvka, W. G. Kropatsch, and V. Mizginov, “Thermalgan: Multimodal color-to-thermal image translation for person re-identification in multispectral dataset,” in _European Conference on Computer Vision Workshop_ , 2018, pp. 0–0.
  * [47] M. S. Santos, T. I. Ren, and N. K. Kalantari, “Single image hdr reconstruction using a cnn with masked features and perceptual loss,” _arXiv preprint_ , 2020.
  * [48] S. Jia, Y. Zhang, D. Agrafiotis, and D. Bull, “Blind high dynamic range image quality assessment using deep learning,” in _ICIP_. IEEE, 2017, pp. 765–769.
  * [49] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in _European Conference on Computer Vision_ , 2018, pp. 117–132.
  * [50] P. Hanhart, M. V. Bernardo, M. Pereira, A. M. Pinheiro, and T. Ebrahimi, “Benchmarking of objective quality metrics for hdr image quality assessment,” _EURASIP JIVP_ , vol. 2015, no. 1, pp. 1–18, 2015.
  * [51] K. Ma, K. Zeng, and Z. Wang, “Perceptual quality assessment for multi-exposure image fusion,” _TIP_ , vol. 24, no. 11, pp. 3345–3356, 2015.
  * [52] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” _TOG_ , vol. 30, no. 4, pp. 1–14, 2011.
  * [53] R. Mantiuk, S. J. Daly, K. Myszkowski, and H.-P. Seidel, “Predicting visible differences in high dynamic range images: model and its calibration,” in _SPIE_ , vol. 5666, 2005, pp. 204–214.
  * [54] M. Narwaria, M. P. Da Silva, and P. Le Callet, “Hdr-vqm: An objective quality measure for high dynamic range video,” _SPIC_ , vol. 35, pp. 46–60, 2015\.
  * [55] A. Artusi, F. Banterle, F. Carra, and A. Moreno, “Efficient evaluation of image quality via deep-learning approximation of perceptual metrics,” _TIP_ , vol. 29, pp. 1843–1855, 2019.
  * [56] Q. He, D. Li, T. Jiang, and M. Jiang, “Quality assessment for tone-mapped hdr images using multi-scale and multi-layer information,” in _ICMEW_. IEEE, 2018, pp. 1–6.
  * [57] Y. Kinoshita and H. Kiya, “Deep inverse tone mapping using ldr based learning for estimating hdr images with absolute luminance,” _arXiv preprint_ , 2019\.
  * [58] A. Choudhury and S. Daly, “Hdr image quality assessment using machine-learning based combination of quality metrics,” in _GlobalSIP_. IEEE, 2018, pp. 91–95.
  * [59] H. Ziaei Nafchi, “Efficient and effective objective image quality assessment metrics,” Ph.D. dissertation, École de technologie supérieure, 2018.
  * [60] E. Pérez-Pellitero, S. Catley-Chandar, A. Leonardis, and R. Timofte, “Ntire 2021 challenge on high dynamic range imaging: Dataset, methods and results,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop_ , 2021, pp. 691–700.
  * [61] S. Choi, J. Cho, W. Song, J. Choe, J. Yoo, and K. Sohn, “Pyramid inter-attention for high dynamic range imaging,” _Sensors_ , vol. 20, no. 18, p. 5102, 2020.
  * [62] C. Liu _et al._ , “Beyond pixels: exploring new representations and applications for motion analysis,” Ph.D. dissertation, MIT, 2009.
  * [63] Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in _WACV_. IEEE, 2019, pp. 41–50.
  * [64] F. Peng, M. Zhang, S. Lai, H. Tan, and S. Yan, “Deep hdr reconstruction of dynamic scenes,” in _ICIVC_. IEEE, 2018, pp. 347–351.
  * [65] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2015, pp. 2758–2766.
  * [66] Q. Yan, B. Wang, L. Zhang, J. Zhang, Z. You, Q. Shi, and Y. Zhang, “Towards accurate hdr imaging with learning generator constraints,” _Neurocomputing_ , vol. 428, pp. 79–91, 2021.
  * [67] P. Chaudhari, F. Schirrmacher, A. Maier, C. Riess, and T. Köhler, “Merging-isp: Multi-exposure high dynamic range image signal processing,” _arXiv preprint_ , 2019.
  * [68] K. Metwaly and V. Monga, “Attention-mask dense merger (attendense) deep hdr for ghost removal,” in _ICASSP_. IEEE, 2020, pp. 2623–2627.
  * [69] Y. Niu, J. Wu, W. Liu, W. Guo, and R. W. Lau, “Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions,” _TIP_ , vol. 30, pp. 3885–3896, 2021.
  * [70] Z. Pu, P. Guo, M. S. Asif, and Z. Ma, “Robust high dynamic range (hdr) imaging with complex motion and parallax,” in _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * [71] G. R. KS, A. Biswas, M. S. Patel, and B. P. Prasad, “Deep multi-stage learning for hdr with large object motions,” in _ICIP_. IEEE, 2019, pp. 4714–4718.
  * [72] K. R. Prabhakar, V. S. Srikar, and R. V. Babu, “Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs.” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , vol. 1, no. 2, 2017, p. 3.
  * [73] K. Ma, Z. Duanmu, H. Zhu, Y. Fang, and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” _TIP_ , vol. 29, pp. 2808–2819, 2019.
  * [74] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _NIPS_ , 2017\.
  * [75] Q. Yan, L. Zhang, Y. Liu, Y. Zhu, J. Sun, Q. Shi, and Y. Zhang, “Deep hdr imaging via a non-local network,” _TIP_ , vol. 29, pp. 4308–4322, 2020.
  * [76] S.-Y. Chen and Y.-Y. Chuang, “Deep exposure fusion with deghosting via homography estimation and attention learning,” in _ICASSP_. IEEE, 2020, pp. 1464–1468.
  * [77] Z. Liu, W. Lin, X. Li, Q. Rao, T. Jiang, M. Han, H. Fan, J. Sun, and S. Liu, “Adnet: Attention-guided deformable convolutional network for high dynamic range imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern RecognitionW_ , 2021, pp. 463–470.
  * [78] S.-H. Lee, H. Chung, and N. I. Cho, “Exposure-structure blending network for high dynamic range imaging of dynamic scenes,” _IEEE Access_ , vol. 8, pp. 117 428–117 438, 2020.
  * [79] H. Kaur, D. Koundal, and V. Kadyan, “Image fusion techniques: a survey,” _ACME_ , pp. 1–23, 2021.
  * [80] J.-L. Yin, B.-H. Chen, Y.-T. Peng, and C.-C. Tsai, “Deep prior guided network for high-quality image fusion,” in _ICME_. IEEE, 2020, pp. 1–6.
  * [81] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014, pp. 2672–2680.
  * [82] L. Wang, W. Cho, and K.-J. Yoon, “Deceiving image-to-image translation networks for autonomous driving with adversarial perturbations,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1421–1428, 2020.
  * [83] L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [84] J. E. Van Engelen and H. H. Hoos, “A survey on semi-supervised learning,” _Machine Learning_ , vol. 109, no. 2, pp. 373–440, 2020.
  * [85] L. Wang and K.-J. Yoon, “Semi-supervised student-teacher learning for single image super-resolution,” _Pattern Recognition_ , vol. 121, p. 108206, 2022\.
  * [86] D. Marnerides, T. Bashford-Rogers, J. Hatchett, and K. Debattista, “Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content,” in _CGF_ , vol. 37, no. 2. Wiley Online Library, 2018, pp. 37–49.
  * [87] M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” _TOG_ , vol. 36, no. 4, pp. 1–12, 2017.
  * [88] K. Moriwaki, R. Yoshihashi, R. Kawakami, S. You, and T. Naemura, “Hybrid loss for learning single-image-based hdr reconstruction,” _arXiv preprint_ , 2018\.
  * [89] Y. Kinoshita and H. Kiya, “itm-net: deep inverse tone mapping using novel loss function considering tone mapping operator,” _IEEE Access_ , vol. 7, pp. 73 555–73 563, 2019.
  * [90] X. Yang, K. Xu, Y. Song, Q. Zhang, X. Wei, and R. W. Lau, “Image correction via deep reciprocating hdr transformation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1798–1807.
  * [91] C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for single-shot high-dynamic-range imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1375–1385.
  * [92] J. N. Martel, L. K. Mueller, S. J. Carey, P. Dudek, and G. Wetzstein, “Neural sensors: Learning pixel exposures for hdr imaging and video compressive sensing with programmable sensors,” _TPAMI_ , vol. 42, no. 7, pp. 1642–1653, 2020.
  * [93] V. G. An and C. Lee, “Single-shot high dynamic range imaging via deep convolutional neural network,” in _APSIPA ASC_. IEEE, 2017, pp. 1768–1772.
  * [94] M. M. Alghamdi, Q. Fu, A. K. Thabet, and W. Heidrich, “Reconfigurable snapshot hdr imaging using coded masks and inception network,” 2019.
  * [95] S. Song and T. Funkhouser, “Neural illumination: Lighting prediction for indoor environments,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6918–6926.
  * [96] M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J.-F. Lalonde, “Deep parametric indoor lighting estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7175–7183.
  * [97] V. Gkitsas, N. Zioulis, F. Alvarez, D. Zarpalas, and P. Daras, “Deep lighting environment map estimation from spherical panoramas,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern RecognitionW_ , 2020, pp. 640–641.
  * [98] L. Wang, Y. Chae, S.-H. Yoon, T.-K. Kim, and K.-J. Yoon, “Evdistill: Asynchronous events to end-task learning via bidirectional reconstruction-guided cross-modal knowledge distillation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 608–619.
  * [99] H.-Y. Lin and W.-Z. Chang, “High dynamic range imaging for stereoscopic scene representation,” in _ICIP_. IEEE, 2009, pp. 4305–4308.
  * [100] Y. Chen, M. Yu, K. Chen, G. Jiang, Y. Song, Z. Peng, and F. Chen, “New stereo high dynamic range imaging method using generative adversarial networks,” in _ICIP_. IEEE, 2019, pp. 3502–3506.
  * [101] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2223–2232.
  * [102] N. K. Kalantari and R. Ramamoorthi, “Deep hdr video from sequences with alternating exposures,” in _CGF_ , vol. 38, no. 2. Wiley Online Library, 2019, pp. 193–205.
  * [103] Y. Xu, L. Song, R. Xie, and W. Zhang, “Deep video inverse tone mapping,” in _BigMM_. IEEE, 2019, pp. 142–147.
  * [104] L. Wang, T.-K. Kim, and K.-J. Yoon, “Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8315–8325.
  * [105] A. Shocher, N. Cohen, and M. Irani, ““zero-shot” super-resolution using deep internal learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3118–3126.
  * [106] J. Wang, H. Wang, X. Zhu, and P. Zhou, “A deep learning approach in the dct domain to detect the source of hdr images,” _Electronics_ , vol. 9, no. 12, p. 2053, 2020.
  * [107] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _European Conference on Computer Vision_ , 2018, pp. 286–301.
  * [108] J. S. Park, J. W. Soh, and N. I. Cho, “High dynamic range and super-resolution imaging from a single image,” _IEEE Access_ , vol. 6, pp. 10 966–10 978, 2018.
  * [109] H. Zeng, X. Zhang, Z. Yu, and Y. Wang, “Sr-itm-gan: Learning 4k uhd hdr with a generative adversarial network,” _IEEE Access_ , vol. 8, pp. 182 815–182 827, 2020.
  * [110] J. W. Soh, J. S. Park, and N. I. Cho, “Joint high dynamic range imaging and super-resolution from a single image,” _IEEE Access_ , vol. 7, pp. 177 427–177 437, 2019.
  * [111] X. Deng, Y. Zhang, M. Xu, S. Gu, and Y. Duan, “Deep coupled feedback network for joint exposure fusion and image super-resolution,” _TIP_ , vol. 30, pp. 3098–3112, 2021.
  * [112] B. Ashwath and R. V. Babu, “Towards practical and efficient high-resolution hdr deghosting with cnn,” 2019.
  * [113] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2noise: Learning image restoration without clean data,” _ICML_ , 2018.
  * [114] C. Godard, K. Matzen, and M. Uyttendaele, “Deep burst denoising,” in _European Conference on Computer Vision_ , 2018, pp. 538–554.
  * [115] X. Chen, Y. Liu, Z. Zhang, Y. Qiao, and C. Dong, “Hdrunet: Single image hdr reconstruction with denoising and dequantization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop_ , 2021, pp. 354–363.
  * [116] A. O. Akyüz _et al._ , “Deep joint deinterlacing and denoising for single shot dual-iso hdr reconstruction,” _TIP_ , vol. 29, pp. 7511–7524, 2020.
  * [117] U. Çoğalan, M. Bemana, K. Myszkowski, H.-P. Seidel, and T. Ritschel, “Hdr denoising and deblurring by learning spatio-temporal distortion models,” _arXiv preprint_ , 2020.
  * [118] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [119] J. Koh, J. Lee, and S. Yoon, “Single-image deblurring with neural networks: A comparative survey,” _CVIU_ , 2021.
  * [120] M. Aittala and F. Durand, “Burst image deblurring using permutation invariant convolutional neural networks,” in _European Conference on Computer Vision_ , 2018, pp. 731–747.
  * [121] P. Wieschollek, M. Hirsch, B. Scholkopf, and H. P. A. Lensch, “Learning blind motion deblurring,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , Oct 2017.
  * [122] M. Mostafavi, L. Wang, and K.-J. Yoon, “Learning to reconstruct hdr images from events, with applications to depth and flow prediction,” _IJCV_ , pp. 1–21, 2021.
  * [123] H. Rebecq, R. Ranftl, V. Koltun, and D. Scaramuzza, “High speed and high dynamic range video with an event camera,” _TPAMI_ , 2019.
  * [124] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza, “Event-based vision: A survey,” _IEEE TPAMI_ , vol. PP, 2020.
  * [125] Y. Chae, L. Wang, and K.-J. Yoon, “Siamevent: Event-based object tracking via edge-aware similarity learning with siamese networks,” _arXiv preprint arXiv:2109.13456_ , 2021.
  * [126] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, “A 240$\times$ 180 130 db 3 $\mu$s latency global shutter spatiotemporal vision sensor,” _JSSC_ , vol. 49, no. 10, pp. 2333–2341, 2014.
  * [127] S. Zhang, Y. Zhang, Z. Jiang, D. Zou, J. Ren, and B. Zhou, “Learning to see in the dark with events,” 2020.
  * [128] J. Zhe, “Dark photo reconstruction by event camera,” in _ICVRV_. IEEE, 2019, pp. 113–117.
  * [129] C. Scheerlinck, H. Rebecq, D. Gehrig, N. Barnes, R. Mahony, and D. Scaramuzza, “Fast image reconstruction with an event camera,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 156–163.
  * [130] T. Stoffregen, C. Scheerlinck, D. Scaramuzza, T. Drummond, N. Barnes, L. Kleeman, and R. Mahony, “Reducing the sim-to-real gap for event cameras,” in _European Conference on Computer Vision_. Springer, 2020.
  * [131] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Ev-flownet: Self-supervised optical flow estimation for event-based cameras,” _arXiv preprint_ , 2018\.
  * [132] F. Paredes-Vallés and G. C. de Croon, “Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [133] M. Mostafavi, J. Choi, and K.-J. Yoon, “Learning to super resolve intensity images from events,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE/CVF, 2020.
  * [134] L. Wang, T.-K. Kim, and K.-J. Yoon, “Joint framework for single image reconstruction and super-resolution with an event camera,” _TPAMI_ , no. 01, pp. 1–1, 2021.
  * [135] X. Zhang, L. Wei, L. Yu, W. Yang, and G.-S. Xia, “Event-based synthetic aperture imaging with a hybrid network,” _arXiv e-prints_ , 2021.
  * [136] L. Wang, Y. Chae, and K.-J. Yoon, “Dual transfer learning for event-based end-task prediction via pluggable event to image translation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2135–2145.
  * [137] Y. Zou, Y. Zheng, T. Takatani, and Y. Fu, “Learning to reconstruct high speed and high dynamic range videos from events,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2024–2033.
  * [138] B. Wang, J. He, L. Yu, G.-S. Xia, and W. Yang, “Event enhanced high-quality image recovery,” _European Conference on Computer Vision_ , 2020.
  * [139] Z. Chen, Q. Zheng, P. Niu, H. Tang, and G. Pan, “Indoor lighting estimation using an event camera,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 760–14 770.
  * [140] S. Liu, M. Gao, V. John, Z. Liu, and E. Blasch, “Deep learning thermal image translation for night vision perception,” _ACM TIST_ , vol. 12, no. 1, pp. 1–18, 2020.
  * [141] S. Goswami and S. K. Singh, “A novel registration & colorization technique for thermal to cross domain colorized images,” _arXiv preprint_ , 2021.
  * [142] J. Ma, Y. Ma, and C. Li, “Infrared and visible image fusion methods and applications: A survey,” _IF_ , vol. 45, pp. 153–178, 2019.
  * [143] J. Ma, P. Liang, W. Yu, C. Chen, X. Guo, J. Wu, and J. Jiang, “Infrared and visible image fusion via detail preserving adversarial learning,” _Information Fusion_ , vol. 54, pp. 85–98, 2020.
  * [144] H. Li, X.-J. Wu, and J. Kittler, “Infrared and visible image fusion using a deep learning framework,” in _ICPR_. IEEE, 2018, pp. 2705–2710.
  * [145] H. Li and X.-J. Wu, “Densefuse: A fusion approach to infrared and visible images,” _TIP_ , vol. 28, no. 5, pp. 2614–2623, 2018.
  * [146] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _ICLR_ , 2015.
  * [147] H. Zhao, B. Shi, C. Fernandez-Cull, S.-K. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in _ICCP_ , 2015, pp. 1–10.
  * [148] G. Jagatap and C. Hegde, “High dynamic range imaging using deep image priors,” in _ICASSP_. IEEE, 2020, pp. 9289–9293.