Faculty of Electrical Engineering

A Master’s Thesis in Mechatronics Engineering

Predicting Depth from Semantic Segmentation using
Game Engine Dataset

Mohammad Amin Kashi

Hamid D. Taghirad

Summer 2020

CHAPTER: ACKNOWLEDGEMENTS

I would like to express my gratitude to my supervisor Prof. Hamid D. Taghirad for the useful comments, remarks, and engagement through the learning process of this master thesis. I would also like to thank Advanced Robotics and Automated System (ARAS) team members for their comments on my work. Furthermore, I would like to thank my friends, specially Erfan Khaniki, for their encouragement during the research process.

Finally, I must express my very profound gratitude to my parents, my brother, and my sister for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. Thank you.
Tehran, September 26, 2020
Mohammad Amin Kashi

CHAPTER: ABSTRACT

Depth perception is fundamental for robots to understand the surrounding environment. As the view of cognitive neuroscience, visual depth perception methods are divided into three categories, namely binocular, active, and pictorial. The first two categories have been studied for decades in detail. However, research for the exploration of the third category is still in its infancy and has got momentum by the advent of deep learning methods in recent years.
In cognitive neuroscience, it is known that pictorial depth perception mechanisms are dependent on the perception of seen objects. Inspired by this fact, in this thesis, we investigated the relation of perception of objects and depth estimation convolutional neural networks. For this purpose, we developed new network structures based on a simple depth estimation network that only used a single image at its input. Our proposed structures use both an image and a semantic label of the image as their input. We used semantic labels as the output of object perception. The obtained results of performance comparison between the developed network and original network showed that our novel structures can improve the performance of depth estimation by 52% of relative error of distance in the examined cases.
Most of the experimental studies were carried out on synthetic datasets that were generated by game engines to isolate the performance comparison from the effect of inaccurate depth and semantic labels of non-synthetic datasets. It is shown that particular synthetic datasets may be used for training of depth networks in cases that an appropriate dataset is not available. Furthermore, we showed that in these cases, usage of semantic labels improves the robustness of the network against domain shift from synthetic training data to non-synthetic test data.

Keywords: Depth estimation, pictorial depth cues, semantic segmentation, synthetic dataset, deep convolutional neural network.

[type=acronym, nonumberlist, title=Symbols and Acronyms, style=mystyle, nogroupskip=true]

CHAPTER: INTRODUCTION

§ PREFACE

Depth perception is fundamental for robots to understand the surrounding environment. This perception guides robots to know where the objects in their environment are. Various sensors including SONAR, LIDAR, RADAR, and camera can be used to accommodate a robot with depth perception. Among these sensors, cameras are the cheapest and widely available in various locations such as factories, streets, shopping centers and systems such as mobile phones, cars, drones, surveillance camera systems. Furthermore, cameras are used for different purposes such as security, traffic monitoring, mapping, retail analysis, and industrial automation. Therefore, images of cameras may be widely used for depth estimation in different applications.

Due to its importance, depth estimation from images has received a large amount of attention in the literature. These researches are inspired by studying the mechanisms underlying human vision, as the whole computer vision is inspired by the human vision. Therefore, proposed algorithms may be distinguished into the same categories as that of the depth perception mechanisms of human vision.

§ TAXONOMY OF DEPTH ESTIMATION ALGORITHMS

In cognitive neuroscience, different sources of visual information about distance and depth are called depth cues and are divided into three kinds. The first and second kinds are called binocular cues and active cues, respectively. The third kind is called pictorial cues, which are used for SIDE[This classification is borrowed from <cit.>. In another classification depth cues are divided into three kinds of oculomotor cues, monocular cues, and binocular cues <cit.>.]. Using this classification, mechanisms of depth perception may be classified by their source of information.[In some papers, the term monocular depth estimation was used to describe the estimation of depth from pictorial cues. This designation is ambiguous because monocular depth estimation also applies to depth estimation from active cues. Therefore, only the term SIDE is considered in this thesis.]

§.§ Binocular Depth Cues

Binocular cues include image disparity and orientation of optical sensors. Since optical sensors are fixed relative to each other in a stereo camera, the later cue has not been investigated in computer vision. In contrast, the binocular disparity has been extremely studied (for reviews of works done from the mid-70's up to 2020 refer to barnard_computational_1982,dhond_structure_1989, scharstein_taxonomy_2002, zhou_review_2020). Since the relation of disparity map and depth map is known, in many of the proposed algorithms only estimation of disparity map is addressed.

§.§ Active Depth Cues

Active cues include accommodation information and motion parallax information. Several works proposed algorithms that use accommodation information to estimate the depth of a scene (e.g. kim_video_2016, hazirbas_deep_2018, maximov_focus_2020). For this purpose, a sequence of images is captured from the scene, while the amount of accommodation is changed. Relative to accommodation information, motion parallax information has a more active research area (for reviews of related works refer to triggs_bundle_2000, oliensis_critique_2000, hartley_multiple_2003, ozyesil_survey_2017). Depth estimation algorithms related to motion parallax are called SfM. Motion parallax is a special case of parallax[Difference or change in the apparent position or direction of an object as seen from two different points <cit.>] in which different views are obtained by motion at different moments. Another example of parallax occurs when a binocular vision system is used, while different views are obtained by two eyes or cameras at the same moment.

§.§ Pictorial Depth Cues

Between the three kinds of depth cues, the third kind contains the highest number of cues. Pictorial cues are:

*  Occlusion: Nearby objects occlude farther objects.

*  Relative position: By knowing the relative position of an object to another object that its depth is known, depth estimation of the first object becomes possible.

*  Size of objects: Image size of a specific object varies with distance.

*  Density of textures' gradient: It varies with distance.

*  Image quality: Contrast of an image decreases with distance because of light diffusion by particles in the air.

Until the last decade, few studies investigated the depth estimation using pictorial cues. With the widespread use of deep learning methods, more and more researches have been focused on pictorial cues.

Some works tried to estimate the depth from a specific cue in special situations.  proposed a numerical model that estimates the depth of an object, provided the defocused image and the sharp image of the object are available <cit.>. The parameters of the model should be recalculated for different cameras. In <cit.> the relative position cue is used to estimate the distance of a vehicle on a road. The proposed method first detects the contact point of the vehicle on the road, then by assuming a planar road and a known camera height and using geometry rules the distance of the contact point is calculated and is reported as the distance of the vehicle. Another representative research on this area is the use of a haze removal method as a tool for estimating the depth of a scene <cit.>.

By imposing the burden of extracting the pictorial cues on CNN, the majority of works are focused to directly estimate the depth.  proposed the first successful CNN to estimate depth from a single image <cit.>. They designed a two-stage architecture; the first stage makes a coarse and global prediction, while the second stage enhances this prediction locally. They also used a scale-invariant loss function to train the proposed architecture regardless of scale. This loss function becomes a standard metric in performance evaluation of depth estimation, e.g. in KITTI depth benchmark <cit.>. The later works tried to propose better models in terms of performance by using novel architectures laina_deeper_2016, fu_deep_2018, ren_deep_2019, lee_big_2019, new cost functions yin_enforcing_2019, diaz_soft_2019, li_monocular_2018, liebel_multidepth_2019, or multi-task architectures peng_wang_towards_2015, ji_joint_2016,jiao_look_2018, ochs_sdnet_2019, zama_ramirez_geometry_2019, zhang_pattern-affinitive_2019, lin_depth_2019, chen_towards_2019, kwak_novel_2020.

§ OBTAINING PICTORIAL CUES

Although various works addressed the problem of estimating depth from pictorial cues for SIDE, not enough attention has been paid to the relation of pictorial cues and depth estimation. In most of the proposed methods, a CNN is responsible for extracting the depth cues implicitly and estimating depth from those cues; however, most of the pictorial cues are high-level abstract concepts and logically need a separate CNN to extract them with high accuracy. For example, the recognition of the occlusion between two objects already needs the recognition of the two objects, as the determination of the position of one object relative to another object needs the recognition of the two objects. In a similar way, the size determination of an object already needs the recognition of that object, and the determination of density of a texture needs that the texture be detected first.

As mentioned above a common prerequisite for obtaining pictorial cues is to detect and locate different objects and textures in an image; a task that is somewhat similar to semantic segmentation. For the same reason the most of the proposed multi-task CNNs (or CRF) were designed to estimate depth and simultaneously segment the input image semantically peng_wang_towards_2015, ji_joint_2016,jiao_look_2018, ochs_sdnet_2019, zama_ramirez_geometry_2019, zhang_pattern-affinitive_2019, lin_depth_2019, chen_towards_2019, kwak_novel_2020. Simultaneous learning enables these models to better extract pictorial cues and estimate depth. However, it is plausible that a large aspect of the information, that semantic segments could have delivered to the model, is not used. As stated earlier, semantic segmentation is a prerequisite for obtaining pictorial depth cues; therefore, high-level abstract concepts can not be obtained from the semantic segments in these multi-task models, because the semantic segments are an output of the model and can not be processed anymore.

In some other works, semantic segments were used in other various methods.  used the edges of semantic segments as a guide for up-sampling of sparse depth data of LIDAR <cit.>.  used a pre-designed and pre-trained segmentation model in parallel with a pre-designed depth estimation model instead of designing a multi-task CNN. During training and test of the depth model, the segmentation model injects low-level and high-level features from different layers of its network to the corresponding layers of the depth model <cit.>.  proposed the first depth estimation model that uses predicted semantic segments as its input. They used an MRF for semantic segmentation and an MRF for predicting depth from both images and semantic segments <cit.>. Furthermore,  analyzed the mutual impact of depth and semantic segments in refining each other. They designed a CNN that uses an estimated depth map and a semantic segment map as its inputs and jointly refines them <cit.>.

§ RESEARCH GOALS, METHODS, AND RESULTS

With the assumption that semantic segmentation is a prerequisite for obtaining pictorial depth cues, this thesis investigates how the use of semantic segments as a raw data, like an input image, can potentially improve the performance of a SIDE CNN, and if there is an improvement, how significant it is. This thesis also investigates the effect of the use of semantic segments on the robustness of a SIDE model against domain shift.

Experimental Study on Estimation of Depth from Semantic Labels: For the first goal of the thesis, a SIDE CNN is chosen to be empowered by semantic segments as its input, to drive an SSIDE CNN[From here onwards, models that only use an image as their input for depth estimation are referred to as SIDE models, and models that use an image and a semantic label as their inputs for depth estimation are referred to as SSIDE models]. After the selection of a SIDE CNN, several similar networks are developed from it. These networks use semantic segments along with an image as inputs, in contrast to the original network that only uses an image as the input. Furthermore, the structures of the derived networks have some differences from the first network. The differences are formed to facilitate the utilization of extra data in semantic segments.

For the evaluation of performances, the driven SSIDE CNNs are compared with the original SIDE CNN on estimating depth on specific datasets. In fact, all of the experiment is carried out on a synthetic dataset to eliminate the altering effect of noisy and sparse depth labels and inaccurate semantic segments of non-synthetic datasets. At the end, some parts of the experiment are repeated on two other datasets, one synthetic and another non-synthetic, as described in <ref>. The results show that the SSIDE CNNs outperform the SIDE CNN by 18 percent of the absolute relative error in the examined situations (For complete results see <ref>).

Experimental Study on Robustness of SSIDE against Domain Shift: For the second objective of this thesis, an SSIDE model and a SIDE model are trained on a synthetic dataset (a combination of multiple synthetic datasets), and two other SIDE models are trained on two different non-synthetic datasets. Note that all SIDE models have the same architecture. All models are tested on the test-set of the two non-synthetic datasets, as described in <ref>. Results show that the SSIDE CNN performs like the two SIDE models that were trained on non-synthetic datasets, while the SIDE model that was trained on the synthetic dataset has a poor performance relative to others. In other words, use of semantic segments improves the robustness of the model against domain shift (For complete results see <ref>.).

Experimental Study on Use of Synthetic Data instead of Non-Synthetic Data: An SSIDE model needs both an image and semantic segments to estimate depth. The used datasets in the first experimental study include all needed data: image, semantic segments, and depth. The non-synthetic datasets in the final experimental study do not have semantic labels. To provide semantic segments for these datasets, one way is to train a segmentation model on a synthetic dataset and generate segmentation for these datasets. To ensure that synthetic datasets are capable of training a segmentation model to generate accurate semantic segments for non-synthetic datasets, a supplementary experimental study was carried out before the final experimental study. In this experimental study, multiple segmentation models with the same architecture are trained on different synthetic and non-synthetic datasets. Then all models are tested on non-synthetic datasets as described in <ref>. The results show that some of synthetic datasets can be more informative than some of non-synthetic datasets, i.e., the model that was trained on a synthetic dataset performed better than a model that was trained on a non-synthetic dataset (For complete results see <ref>.).

§ DATASETS

Due to the abundance and diversity of self-driving cars' datasets, this group of datasets were mainly used in this thesis. These datasets were captured with different sensors (RGB camera, LIDAR, RADAR, IMU, and GPS) and contain different data (RGB image, depth data, semantic segments, instance segments, ego velocity, ego position, optical flow) and were captured in different weather conditions and different cities. These datasets are available in non-synthetic types and synthetic types as well. However, in the first experimental study, the non-synthetic NYU Depth Dataset V2 <cit.> was used too.

For two reason, synthetic datasets were used in this these:

* the available non-synthetic datasets does not have enough or doesn't have any semantic labels at all (except NYU Depth Dataset V2).

* depth maps and semantic labels in non-synthetic datasets are not accurate and may alter the results of the experiments.

Note that you may find depth maps and semantic labels in a dataset, while they are not associated with a specific sample[By the definition of NUSCENES dataset: <cit.> A sample is data collected at (approximately) the same timestamp as part of a single LIDAR sweep <cit.>. ]. For example KITTI benchmark <cit.> has both semantic segmentation subset and depth subset, while they are linked to different samples. Only some of the synthetic datasets have these data synced at the same sample time.

Semantic labels in non-synthetic datasets are generated manually or automatically by AI-based software under human supervision, mainly because they are not perfect. However, generated semantic segments by a human or supervised by a human are not immune from error and suffer from inaccurate or false segmentation (<ref>).

RGB image in frame n

Semantic segments in frame n

Semantic segments in frame n+1

Semantic segments in frame n+2

[Inaccurate or false segmentation by a human]Inaccurate or false segmentation by a human. In these three consecutive frames, different labels are assigned to an object(inside the orange box), and inaccurate labels are assigned to another object such that the label exceeds the covered area by the object(inside the red box). The labels are from KITTI dataset.

Depth data in non-synthetic datasets are usually generated by LIDAR sensors for outdoor scenes and by RGB-D cameras for indoor scenes. If a LIDAR is used for depth generation, the sparse point clouds generated can be corrupted by measurement noise due to the internal noise of the sensor and environmental factors such as reflectivity, lighting, and precipitation <cit.>. The reflectivity of glasses and glossy surfaces also causes many 3D points to be missed <cit.>. Downfall of rain and snow blocks laser beams and create noise in the point cloud (<ref>). Another drawback of LIDAR is its sparsity of points which increases with distance. Generated depth labels by RGB-D cameras share the same drawbacks in addition to a limited range of view.

[Snowy point cloud collected by LIDAR]Snowy point cloud collected by LIDAR <cit.>

The existing issues of depth data and semantic labels in non-synthetic datasets do not exist in synthetic datasets, provided they are designed properly. It can be mentioned that synthetic data are completely correct. But this perfection can be considered superiority only when CNNs are capable of predicting high details and impurities in datasets cause them to malfunction.

§.§ Dataset Adaptation and Domain Adaptation

Since every dataset has its characteristics and distribution of data , in this thesis multiple datasets from both synthetic and non-synthetic types are examined. In this way the hypothesis of this thesis could be examined more accurately and independent of the impact of datasets' characteristics. Moreover, the difference in the distribution of data in these datasets exposes the used model to the domain shift. In other words, the used models are challenged to adapt to different domains.[*Transfer learning and domain adaptation refer to the situation where what has been learned in one setting (i.e., distribution P1) is exploited to improve generalization in another setting (say distribution P2). In transfer learning, the learner must perform two or more different tasks, but we assume that many of the factors that explain the variations in P1 are relevant to the variations that need to be captured for learning P2. In the related case of domain adaptation, the task (and the optimal input-to-output mapping) remains the same between each setting, but the input distribution is slightly different. <cit.>]. Recently, the problem of domain adaption from synthetic data to real data has attracted lots of attention and has been investigated in several works sun_virtual_2014, vazquez_virtual_2014, xu_domain_2014, busto_adaptation_2015, ros_synthia_2016, xu_hierarchical_2016, wrenninge_synscapes_2018, luo_taking_2019, nikolenko_synthetic_2019. Furthermore, this approach introduces a difficulty as well, namely dataset adaptation. Because of some specifics of datasets, they can not be used directly, and some adjustments should be applied to jointly use different datasets.

One necessary adjustment is unifying semantic segmentation policies. Almost every dataset has chosen a different policy in the classification of objects for semantic labels. The only way of unifying datasets is to merge subclasses to superclasses.

Another necessary adjustment is the unifying of the AFOV of all frames (including RGB images, semantic labels, and depth maps). No two datasets have the same AFOV. In photography when you zoom in to magnify the scene, AFOV reduces. Therefore, different AFOV leads to a different perception of distances. According to optic laws, the value of AFOV is a function of the camera's focal length and size of the camera's sensor <ref>.

[Relation of AFOV with focal length and sensor size]Relation of AFOV with focal length and sensor size. Yellow line is the sensor.

According to <ref> adjusting the AFOV is possible through changing the focal length or sensor size. However, changing the focal length of the camera or the sensor size is not possible after taking a photo. However, since changing the image size is the same as changing the sensor size, one can crop the image to decrease sensor size. Therefore, AFOV can only be decreased.

\begin{equation}
\label{eq:AFOV}
\text{AFOV} \left[ ^{\circ} \right] = 2 \times \tan ^{-1} { \left( \frac{h}{2f} \right)}
\end{equation}

CHAPTER: DEPTH FROM SEMANTIC LABELS

This chapter describes an experimental study that analyzes the performance of several SIDE and SSIDE CNN architectures. This study aims to investigate the probable advantages of SSIDE over SIDE.

CHAPTER: SYNTHETIC DATA INSTEAD OF NON-SYNTHETIC DATA

To ensure that synthetic datasets are capable of training a segmentation model to generate accurate semantic segments for non-synthetic images, an experimental study was carried out. This chapter describes this experimental study.

CHAPTER: ROBUSTNESS OF SSIDE AGAINST DOMAIN SHIFT

This chapter describes the final experimental study which is based on the results of the two previous experimental studies: first, it is assumed that an SSIDE CNN exists that outperforms its SIDE counterpart. Second, it is assumed that semantic segments generated for non-synthetic images by a CNN trained on synthetic datasets are accurate enough. The goal of this experimental study is to investigate the effect of the use of semantic segments on the robustness of a SIDE model against domain shift. In other words, the goal is to determine how much an SSIDE CNN outperforms its SIDE counterpart while both generalize concepts from synthetic data to non-synthetic data.

CHAPTER: CONCLUSIONS AND FUTURE WORK

§ CONCLUSIONS

In this thesis, we investigated how the use of semantic labels as raw data, like an input image, can potentially improve the performance of a SIDE network. We also investigated the effect of the use of semantic labels on the robustness of a SIDE model against domain shift.

In the first experimental study (<ref>), we experimented and evaluated various methods to reinforce a pre-designed SIDE network with the knowledge of the input image's semantic label. The results showed that by concatenation of the raw image and its semantic label and feeding them both to a well structured network can significantly improve the performance of the depth estimation. From this observations, it is concluded that the use of semantic labels can improve the performance of a SIDE network. This result and conclusion are aligned with the assumptions that semantic segments are a prerequisite for the perception of pictorial cues, and furthermore, noting that pictorial cues could be effective in the improvement of a SIDE network performance. This conclusion also agrees with the various works that reported on the improvement in the depth estimation by different use of semantic segmentation or semantic segments peng_wang_towards_2015, ji_joint_2016,jiao_look_2018, ochs_sdnet_2019, zama_ramirez_geometry_2019, zhang_pattern-affinitive_2019, lin_depth_2019, chen_towards_2019, kwak_novel_2020, schneider_semantically_2016, guizilini_semantically-guided_2020, liu_single_2010, jafari_analyzing_2017.

It is further concluded that the use of a semantic label is most effective when it is fed to a SIDE network through the input layer of the network and not through its higher levels. This conclusion theoretically implies that multi-task models, that try to simultaneously estimate the depth and segment the image, are not efficient. This is mainly because this kind of model generates the semantic segments in the last layers.

In the second experimental study, we examined different synthetic datasets and compared them to some non-synthetic datasets in terms of training a segmentation model. The obtained results showed that if a synthetic dataset is more diverse in its content relative to a non-synthetic dataset, then it is superior in training a segmentation network. Therefore, we concluded that regardless of the origin of the dataset (whether it is synthetic or non-synthetic), the more diverse a dataset is in its content, the better performance for the network is obtained. This conclusion lead us to generate semantic labels for non-synthetic images by means of synthetically trained models in the last experimental study.

Finally, in the last experimental study, an SSIDE model outperformed its SIDE counterpart, while both suffered from domain shift from synthetic data to non-synthetic data. The SSIDE model performed like other SIDE models that were trained on non-synthetic datasets. From these results, we concluded that the use of semantic segments increases the robustness of the model against domain shifts. Therefore, for the task of depth estimation, if an appropriate synthetic dataset related to the environment of the task would be available, without loosing too much accuracy one can use an SSIDE model along with the synthetic datasets to achieve suitable performance.

Unlike other works, in this thesis by isolating the experiments
against altering parameters the following conclusions are obtained.

* Use of semantic segments may significantly improve the accuracy of the depth estimation through well structured deep neural networks.

* The performance of a depth estimation network improves only when semantic segments are fed to it from the input layer.

* The use of semantic segments improves the robustness of a depth estimation model against domain shifts.

* The robustness of the model is maximized when it uses semantic edges.

These conclusions are derived by the use of ground truths instead of inaccurate predictions for semantic segments. Furthermore the synthetic datasets are analyzed to distinguish the accurate and more reliable synthetic dataset from inaccurate and noisy non-synthetic ones.

Considering all the conclusions reported in this thesis, leads to a practical application; a method that could possibly improve the performance of a SIDE model. In this method a SIDE network could be converted to an SSIDE network only by feeding the semantic edges of the input image.

§ FUTURE WORK

In this thesis, we used a simple SIDE as the base network to develop our models. However, this base network is not very powerful and the developed SSIDE networks do not have competitive performance compared to recent state-of-the-art methods. Therefore, a demanded future work could be the examination of the proposed method in this thesis to convert recently proposed SIDE networks to SSIDE networks.

In previous section we mentioned that multi-task methods that both estimate a depth map and a semantic label are not efficient. Another interesting future research could be to develop a multi-task network based on the M0 architecture, and to compare it with other developed SSIDE networks.

In the second experimental study, we observed that a synthetic dataset led to better performance compared to a non-synthetic dataset that had lower diversity in its content. In this experimental study, we only used synthetic datasets that seemed visually with high quality or more realistic, compared to other available synthetic datasets. This leads to the following questions: What would happen if we used low-quality synthetic datasets? Or, if two synthetic and non-synthetic datasets with the same diversity exist, what qualities should the synthetic dataset have to lead to a better performance of a network? These questions could be answered in future work.

In the last experimental study we used generated semantic labels for non-synthetic datasets to evaluate an SSIDE network. Definitely, the quality of generated semantic labels affected the performance of the network,. However, its extent is unknown to us. A probable future work related to this issue could be to investigate the relation of the accuracy of semantic labels to the performance of SSIDE networks.

CHAPTER: SEMANTIC SEGMENTATION

Semantic segmentation of an image is to assign a semantic class to every pixel of the image. Usually every semantic class is encoded with an RGB color, so that a semantic segmented image can be visualized in a color image and stored in an image-type format(<ref>).

RGB image

RGB mapping of semantic segments

one-hot encoding of road class

one-hot encoding of person class

[Different encodings for semantic segments]Different encodings for semantic segments.An example of an RGB image with its corresponding semantic labels from

But this type of encoding is not useful for training a CNN, because semantic segmentation is a classification task which needs an array of binary numbers as the ground truth for each pixel; 1 shows the existence of a class and 0 shows the absence of a class. Therefore an RGB mapping of semantic segments can be converted to a one-hot labels, i.e.,  a group of binary images (<ref>).

CHAPTER: COMMON SEMANTIC LABELS SET

[Common semantic labels set]Common semantic labels set. Each column shows the available semantic classes in a dataset by mentioning its RGB code. The last column shows a set of unified semantic classes. In tha last column an arrow shows that the class of the row is merged to the class that arrow points to.

Class name     VIPER     SYNTHIA-SF     Synscapes     Virtual KITTI     KITTI     Common set
1     Fence     [190, 153, 153]     [190, 153, 153]     [190, 153, 153]     -     [190, 153, 153]     $\to$       Unlabeled

2     Guard Rail     -     -     [180, 165, 180]     [250, 100, 255]     [180, 165, 180]     $\to$ Unlabeled

3     Wall     -     [102, 102, 156]     [102, 102, 156]     -     [102, 102, 156]     $\to$ building

4     Parking     -     -     [250, 170, 160]     -     [250, 170, 160]     $\to$ Road

5     Rail Track     [230, 150, 140]     -     [230, 150, 140]     -     [230, 150, 140]     $\to$ Road

6     Road     [128, 64, 128]     [128, 64, 128]     [128, 64, 128]     [100, 60, 100]     [128, 64, 128]             [128, 64, 128]

7     Sidewalk     [244, 35, 232]     [244, 35, 232]     [244, 35, 232]     -     [244, 35, 232]     [244, 35,         232]

8     Bridge     -     -     [150, 100, 100]     -     [150, 100, 100]     $\to$ building

9     Building     [70, 70, 70]     [70, 70, 70]     [70, 70, 70]     [140, 140, 140]     [70, 70, 70]     [70,       70, 70]

10     Tunnel     -     -     [150, 120, 90]     -     [150, 120, 90]     $\to$ building

11     Person     [220, 20, 60]     [220, 20, 60]     [220, 20, 60]     -     [220, 20, 60]     [220, 20, 60]

12     Bicyclist     [255, 0, 0]     [255, 0, 0]     [255, 0, 0]     -     [255, 0, 0]     $\to$ Person

13     Lane Marking - General     -     [157, 234, 50]     -     -     -     $\to$ Road

14     Sky     [70, 130, 180]     [70, 130, 180]     [70, 130, 180]     [90, 200, 255]     [70, 130, 180]             [70, 130, 180]

15     Terrain     [152, 251, 152]     [152, 251, 152]     [152, 251, 152]     [210, 0, 200]     [152, 251,         152]     [152, 251, 152]

16     Vegetation     [35, 142, 35]     [107, 142, 35]     [107, 142, 35]     [90, 240, 0]     [107, 142, 35]             [107, 142, 35]

17     Pole     [153, 153, 153]     [153, 153, 153]     [153, 153, 153]     [255, 130, 0]     [153, 153, 153]             [153, 153, 153]

18     Traffic Light     [250, 170, 30]     [250, 170, 30]     [250, 170, 30]     [200, 200, 0]     [250, 170,      30]     $\to$ Pole

19     Traffic Sign (Front)     [220, 220, 0]     [220, 220, 0]     [220, 220, 0]     [255, 255, 0]     [220,       220, 0]     $\to$ Pole

20     Trash Can     [81, 0, 81]     -     -     -     -     $\to$ Unlabeled

21     Bicycle     [119, 11, 32]     [119, 11, 32]     [119, 11, 32]     -     [119, 11, 32]     [119, 11, 32]

22     Boat     [50, 0, 90]     -     -     -     -     $\to$ Unlabeled

23     Bus     [0, 60, 100]     [0, 60, 100]     [0, 60, 100]     -     [0, 60, 100]     $\to$ Car

24     Car     [0, 0, 142]     [0, 0, 142]     [0, 0, 142]     [255, 127, 80]     [0, 0, 142]     [0, 0, 142]

25     Caravan     [0, 0, 90]     -     [0, 0, 90]     -     [0, 0, 90]     $\to$ Car

26     Motorcycle     [0, 0, 230]     [0, 0, 230]     [0, 0, 230]     -     [0, 0, 230]     $\to$ Bicycle

27     On Rails     [0, 80, 100]     [0, 80, 100]     [0, 80, 100]     -     [0, 80, 100]     $\to$ Car

28     Trailer     -     -     [0, 0, 110]     -     [0, 0, 110]     $\to$ Car

29     Truck     [0, 0, 70]     [0, 0, 70]     [0, 0, 70]     [160, 60, 60]     [0, 0, 70]     $\to$ Car

30     Unlabeled     [0, 0, 0]     [0, 0, 0]     [0, 0, 0]     [0, 0, 0]     [0, 0, 0]     [0, 0, 0]

31     ground     -     -     [81, 0, 81]     -     [81, 0, 81]     $\to$ Unlabeled

32     dynamic     [111, 74, 0]     -     [111, 74, 0]     -     [111, 74, 0]     $\to$ Unlabeled

33     plane     [0, 100, 100]     -     -     -     -     $\to$ Unlabeled

34     trash     [81, 0, 21]     -     -     -     -     $\to$ Unlabeled

35     chair     [168, 153, 153]     -     -     -     -     $\to$ Unlabeled

36     firehydrant     [173, 153, 153]     -     -     -     -     $\to$ Unlabeled

37     mobilebarrier     [180, 180, 100]     -     -     -     -     $\to$ Unlabeled

38     billboard     [150, 20, 20]     -     -     -     -     $\to$ Unlabeled

39     tree     [87, 182, 35]     -     -     [0, 199, 0]     -     $\to$ Unlabeled

40     Misc     -     -     -     [80, 80, 80]     -     $\to$ Unlabeled

CHAPTER: EXPERIMENTS RESULTS

§ SUPPLEMENTARY DATA FOR FIRST EXPERIMENT

RGB input image

Depth ground truth

Depth estimation by M18

Depth estimation by M20

[Depth estimation samples of M18 and M20]Depth estimation samples of M18 and M20

§ SUPPLEMENTARY DATA FOR SECOND EXPERIMENT

[Perception sample of Cityscapes U-Net on Cityscapes]Perception sample of Cityscapes U-Net on Cityscapes. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Cityscapes U-Net on Mapillary]Perception sample of Cityscapes U-Net on Mapillary. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Cityscapes U-Net on VIPER]Perception sample of Cityscapes U-Net on VIPER. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Cityscapes U-Net on SYNTHIA-SF]Perception sample of Cityscapes U-Net on SYNTHIA-SF. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Mapillary U-Net on Cityscapes]Perception sample of Mapillary U-Net on Cityscapes. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Mapillary U-Net on Mapillary]Perception sample of Mapillary U-Net on Mapillary. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Mapillary U-Net on VIPER]Perception sample of Mapillary U-Net on VIPER. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of Mapillary U-Net on SYNTHIA-SF]Perception sample of Mapillary U-Net on SYNTHIA-SF. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of SYNTHIA-SF U-Net on Cityscapes]Perception sample of SYNTHIA-SF U-Net on Cityscapes. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of SYNTHIA-SF U-Net on Mapillary]Perception sample of SYNTHIA-SF U-Net on Mapillary. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of SYNTHIA-SF U-Net on VIPER]Perception sample of SYNTHIA-SF U-Net on VIPER. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of SYNTHIA-SF U-Net on SYNTHIA-SF]Perception sample of SYNTHIA-SF U-Net on SYNTHIA-SF. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of VIPER U-Net on Cityscapes]Perception sample of VIPER U-Net on Cityscapes. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of VIPER U-Net on Mapillary]Perception sample of VIPER U-Net on Mapillary. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of VIPER U-Net on VIPER]Perception sample of VIPER U-Net on VIPER. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

[Perception sample of VIPER U-Net on SYNTHIA-SF]Perception sample of VIPER U-Net on SYNTHIA-SF. The top right image is the input RGB image, the other two top images are ground truth and prediction in a single semantic class and two bottom images are overall ground truth and prediction.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

American Cancer Society

abstractThis chapter describes how people judge how far objects are from themselves. Comparing the images in the two eyes and the orientations of the two eyes when looking at the object of interest provides depth information that is based solely on the viewing geometry, but the resolution of such information is limited. Information acquired from cues such as occlusion, motion parallax, and height in the visual field is based on assumptions that need not always be true, such as that objects are isotropic, static, and resting on horizontal surfaces. By combining the many sources of information in a clever manner people obtain quite reliable judgments that are not too sensitive to violations of the assumptions of the individual sources of depth information.
booktitleStevens' Handbook of Experimental Psychology and Cognitive Neuroscience
titleDepth Perception
ENSATION AND PERCEPTION has helped a myriad of students understand perceptual research and how the results of this research relate to everyday experience. A key strength of this text has always been its ability to illustrate concepts through examples and visuals. Dr. Goldstein and new co-author Dr. James Brockmole take students on an intriguing journey through the senses with both clarity and thoroughness, combining their extensive classroom experience and the most innovative research to create a visual, colorful text.Important Notice: Media content referenced within the product description or the product text may not be available in the ebook version.
noteGoogle-Books-ID: x5d4CgAAQBAJ
titleSensation and Perception
Education / Teaching Methods & Materials / General

journaltitleACM Comput. Surv.
titleComputational Stereo
Barnard and Fischler - 1982 - Computational Stereo.pdf:application/pdf

abstractThe authors review major recent developments in establishing stereo correspondence for the extraction of the 3D structure of a scene. Broad categories of stereo algorithms are identified on the basis of differences in imaging geometry, matching primitives, and the computational structure used. The performance of these stereo techniques on various classes of test images is reviewed, and possible directions of future research are indicated.<>
journaltitleIEEE Transactions on Systems, Man, and Cybernetics
noteConference Name: IEEE Transactions on Systems, Man, and Cybernetics
titleStructure from stereo-a review

Dhond and Aggarwal - 1989 - Structure from stereo-a review.pdf:application/pdf;IEEE Xplore Abstract Record:D
Layout,Cameras,Computer vision,Lighting,Image reconstruction,3D structure recovery,computational structure,computerised pattern recognition,computerised picture processing,Image analysis,imaging geometry,Iterative algorithms,matching primitives,Optical imaging,pattern recognition,reviews,Robotics and automation,stereo correspondence,structure extraction,Testing

abstractStereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods. Our taxonomy is designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can easily be extended to include new algorithms. We have also produced several new multi-frame stereo data sets with ground truth and are making both the code and data sets available on the Web. Finally, we include a comparative evaluation of a large set of today's best-performing stereo algorithms.
journaltitleInternational Journal of Computer Vision
titleA Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
zeliski - 2002 - A Taxonomy and Evaluation of Dense Two-Frame Stere.pdf:D
Scharstein and Szeliski - 2002 - A Taxonomy and Evaluation of Dense Two-Frame Stere.pdf:application/pdf

abstractStereo vision is a flourishing field, attracting the attention of many researchers. Recently, leveraging on the development of deep learning, stereo matching algorithms have achieved remarkable performance far exceeding traditional approaches. This review presents an overview of different stereo matching algorithms based on deep learning. For convenience, we classified the algorithms into three categories: (1) non-end-to-end learning algorithms, (2) end-to-end learning algorithms, and (3) unsupervised learning algorithms. We have provided a comprehensive coverage of the remarkable approaches in each category and summarized the strengths, weaknesses, and major challenges, respectively. The speed, accuracy, and time consumption were adopted to compare the different algorithms.
noteISSN: 1687-5265 Library Catalog: www.hindawi.com Pages: e8562323 Publisher: Hindawi Volume: 2020
titleReview of Stereo Matching Algorithms Based on Deep Learning
titleaddonComputational Intelligence and Neuroscience
typeReview Article
tereo Matching Algorithms Based on Deep.pdf:D
Zhou et al. - 2020 - Review of Stereo Matching Algorithms Based on Deep.pdf:application/pdf

abstractMany compelling video post-processing effects, in particular aesthetic focus editing and refocusing effects, are feasible if per-frame depth information is available. Existing computational methods to capture RGB and depth either purposefully modify the optics (coded aperture, light-field imaging), or employ active RGB-D cameras. Since these methods are less practical for users with normal cameras, we present an algorithm to capture all-in-focus RGB-D video of dynamic scenes with an unmodified commodity video camera. Our algorithm turns the often unwanted defocus blur into a valuable signal. The input to our method is a video in which the focus plane is continuously moving back and forth during capture, and thus defocus blur is provoked and strongly visible. This can be achieved by manually turning the focus ring of the lens during recording. The core algorithmic ingredient is a new video-based depth-from-defocus algorithm that computes space-time-coherent depth maps, deblurred all-in-focus video, and the focus distance for each frame. We extensively evaluate our approach, and show that it enables compelling video post-processing effects, such as different types of refocusing.
booktitle2016 Fourth International Conference on 3D Vision (3DV)
eventtitle2016 Fourth International Conference on 3D Vision (3DV)
titleVideo Depth-from-Defocus
n this paper, we propose `Deep Depth From Focus (DDFF)' as the first end-to-end learning approach to this problem. One of the main challenges we face is the hunger for data of deep neural networks. In order to obtain a significant amount of focal stacks with corresponding groundtruth depth, we propose to leverage a light-field camera with a co-calibrated RGB-D sensor. This allows us to digitally create focal stacks of varying sizes. Compared to existing benchmarks our dataset is 25 times larger, enabling the use of machine learning for this inverse problem. We compare our results with state-of-the-art DFF methods and we also analyze the effect of several key deep architectural components. These experiments show that our proposed method `DDFFNet' achieves state-of-the-art performance in all scenes, reducing depth error by more than 75% compared to the classical DFF methods.
journaltitlearXiv:1704.01085 [cs]
titleDeep Depth From Focus
Hazirbas et al. - 2018 - Deep Depth From Focus.pdf:application/pdf;arXiv.org Snapshot:D

abstractData-driven depth estimation methods struggle with the generalization outside their training scenes due to the immense variability of the real-world scenes. This problem can be partially addressed by utilising synthetically generated images, but closing the synthetic-real domain gap is far from trivial. In this paper, we tackle this issue by using domain invariant defocus blur as direct supervision. We leverage defocus cues by using a permutation invariant convolutional neural network that encourages the network to learn from the differences between images with a different point of focus. Our proposed network uses the defocus map as an intermediate supervisory signal. We are able to train our model completely on synthetic data and directly apply it to a wide range of real-world images. We evaluate our model on synthetic and real datasets, showing compelling generalization results and state-of-the-art depth prediction.
journaltitlearXiv:2005.09623 [cs]
shorttitleFocus on defocus
titleFocus on defocus: bridging the synthetic to real domain gap for depth estimation
Maximov et al. - 2020 - Focus on defocus bridging the synthetic to real d.pdf:application/pdf;arXiv.org Snapshot:D

Berlin, Heidelberg

abstractThis paper is a survey of the theory and methods of photogrammetric bundle adjustment, aimed at potential implementors in the computer vision community. Bundle adjustment is the problem of refining a visual reconstruction to produce jointly optimal structure and viewing parameter estimates. Topics covered include: the choice of cost function and robustness; numerical optimization including sparse Newton methods, linearly convergent approximations, updating and recursive methods; gauge (datum) invariance; and quality control. The theory is developed for general robust cost functions rather than restricting attention to traditional nonlinear least squares.
booktitleVision Algorithms: Theory and Practice
seriesLecture Notes in Computer Science
titleBundle Adjustment — A Modern Synthesis
Optimization,Bundle Adjustment,Gauge Freedom,Scene Reconstruction,Sparse Matrices

abstractI review current approaches to structure from motion (SFM) and suggest a framework for designing new algorithms. The discussion focuses on reconstruction rather than on correspondence and on algorithms reconstructing from many images. I argue that it is important to base experiments and algorithm design on theoretical analyses of algorithm behavior and on an understanding of the intrinsic, algorithm-independent properties of SFM optimal estimation. I propose new theoretical analyses as examples, which suggest a range of experimental questions about current algorithms as well as new types of algorithms. The paper begins with a review of several of the important multi-image-based approaches to SFM, including optimization, fusing (e.g., Kalman filtering), projective methods, and invariant-based algorithms. I suggest that optimization by means of general minimization techniques needs to be supplemented by a theoretical understanding of the SFM least-squares error surface. I argue that fusing approaches are essentially no more robust than algorithms reconstructing from a small number of images and advocate experiments to determine the limitations of fusing. I also propose that fusing may be one of the best reconstruction strategies in situations where few-image algorithms give reasonable results, and suggest that an experimental understanding of the properties of few-image algorithms is important for designing good fusing methods. I emphasize the advantages of an approach based on fusing image-pair reconstructions. With regard to the projective approach, I argue that its trade-off of simplicity versus accuracy/robustness needs more careful experimental examination, and I advocate more research on the effects of calibration error on Euclidean reconstruction. I point out the relative lack of research on adapting Euclidean approaches to deal with incomplete knowledge of the calibration. I argue that invariant-based algorithms could be more nonrobust and inaccurate, and not necessarily much faster, than an approach fusing two-image optimizations. Based on recent results showing that two-image reconstructions are nearly as accurate as multi-image ones, I suggest that the authors of invariants methods conduct careful comparisons of their algorithms to two-image-based results. The remainder of the paper discusses the issues involved in designing a generally applicable SFM algorithm. I argue that current SFM algorithms perform well only in restricted domains, and that different types of algorithms do well on quite different types of sequences. I present examples of three domains that are important in applications and describe three types of algorithms, each of which performs well in just one of the three domains. I advocate testing current algorithms on a wider variety of sequences to determine their limits of applicability. More generally, I propose that SFM is a messy problem and that it could require a flexible “intermediate-level” system incorporating a variety of different algorithms and sophisticated decision rules for combining them. The paper concludes with a general discussion of experiments, pointing out that real-image experiments are not always the best means of evaluating reconstruction algorithms.
journaltitleComputer Vision and Image Understanding
titleA Critique of Structure-from-Motion Algorithms
S1077314200908695.html:text/html;Submitted Version:D
Oliensis - 2000 - A Critique of Structure-from-Motion Algorithms.pdf:application/pdf

Cambridge University Press

abstractA basic problem in computer vision is to understand the structure of a real world scene given several images of it. Techniques for solving this problem are taken from projective geometry and photogrammetry. Here, the authors cover the geometric principles and their algebraic representation in terms of camera projection matrices, the fundamental matrix and the trifocal tensor. The theory and methods of computation of these entities are discussed with real examples, as is their use in the reconstruction of scenes from multiple images. The new edition features an extended introduction covering the key ideas in the book (which itself has been updated with additional examples and appendices) and significant new results which have appeared since the first edition. Comprehensive background material is provided, so readers familiar with linear algebra and basic numerical methods can understand the projective geometry and estimation algorithms presented, and implement the algorithms directly from the book.
noteGoogle-Books-ID: si3R3Pfa98QC
titleMultiple View Geometry in Computer Vision
Computers / Computer Graphics,Computers / Computer Science,Computers / Computer Vision & Pattern Recognition,Mathematics / Applied,Mathematics / Geometry / General,Technology & Engineering / Robotics

abstractThe structure from motion (SfM) problem in computer vision is the problem of recovering the three-dimensional ($3$D) structure of a stationary scene from a set of projective measurements, represented as a collection of two-dimensional ($2$D) images, via estimation of motion of the cameras corresponding to these images. In essence, SfM involves the three main stages of (1) extraction of features in images (e.g., points of interest, lines, etc.) and matching these features between images, (2) camera motion estimation (e.g., using relative pairwise camera positions estimated from the extracted features), and (3) recovery of the $3$D structure using the estimated motion and features (e.g., by minimizing the so-called reprojection error). This survey mainly focuses on relatively recent developments in the literature pertaining to stages (2) and (3). More specifically, after touching upon the early factorization-based techniques for motion and structure estimation, we provide a detailed account of some of the recent camera location estimation methods in the literature, followed by discussion of notable techniques for $3$D structure recovery. We also cover the basics of the simultaneous localization and mapping (SLAM) problem, which can be viewed as a specific case of the SfM problem. Further, our survey includes a review of the fundamentals of feature extraction and matching (i.e., stage (1) above), various recent methods for handling ambiguities in $3$D scenes, SfM techniques involving relatively uncommon camera models and image features, and popular sources of data and SfM software.
journaltitleActa Numer.
titleA Survey on Structure from Motion
Ozyesil et al. - 2017 - A Survey of Structure from Motion.pdf:application/pdf
Oxford University Press

booktitleOED Online
titleparallax, n.

abstractThis paper explains the use of a sharpening filter to calculate the depth of an object from a blurred image of it. It presents a technique which is independent of edge orientation. The technique is based on the assumption that a defocused image of an object is the convolution of a sharp image of the same object with a two-dimensional Gaussian function whose spread parameter (SP) is related to the object depth. A sharp image of an object is obtained from a defocused image of the same object by applying sharpening filters. The defocused and sharp images of the object are used to calculate the SP which is then related to the object depth. The paper gives experimental results which show the feasibility of employing sharpening filters for depth computation.
journaltitleOpt Express
titleA depth estimation algorithm with a single image

abstractThe paper presents a novel approach for distance estimation using a single camera as input. In particular we discuss a method for depth estimation using camera parameters and also image geometry. The system has been implemented for specific camera parameters. This is a novel concept which uses road geometry and point of contact on road. Using a single forward facing camera located typically near the rear view mirror, our method can estimate distance of any vehicle on the road. Keywords— Depth, dead zone, optical axis, oblique distance, vanishing point, field of view
titleDepth Estimation Using Monocular Camera

abstractWe consider the problem of estimating the relative depth of a scene from a monocular image. The dark channel prior, used as a statistical observation of haze free images, has been previously leveraged for haze removal and relative depth estimation tasks. However, as a local measure, it fails to account for higher order semantic relationship among scene elements. We propose a dual channel prior used for identifying pixels that are unlikely to comply with the dark channel assumption, leading to erroneous depth estimates. We further leverage semantic segmentation information and patch match label propagation to enforce semantically consistent geometric priors. Experiments illustrate the quantitative and qualitative advantages of our approach when compared to state of the art methods.
booktitleIEEE Winter Conference on Applications of Computer Vision
eventtitleIEEE Winter Conference on Applications of Computer Vision
noteISSN: 1550-5790
titleCombining semantic scene priors and haze removal for single image depth estimation
mage segmentation,Semantics,Estimation,Roads,single image depth estimation,g,Buildings,Channel estimation,dark channel assumption,dark channel prior,dual channel prior,haze removal,higher order semantic relationship,image matching,monocular image,patch match label propagation,Reliability,semantic scene priors,semantic segmentation information,semantically consistent geometric priors

abstractPredicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.
journaltitlearXiv:1406.2283 [cs]
titleDepth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen et al. - 2014 - Depth Map Prediction from a Single Image using a M.pdf:application/pdf;arXiv.org Snapshot:D

abstractWe present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10-100 Hz using a variety of sensor modalities such as high-resolution color and grayscale stereo cameras, a Velodyne 3D laser scanner and a high-precision GPS/IMU inertial navigation system. The scenarios are diverse, capturing real-world traffic situations, and range from freeways over rural areas to inner-city scenes with many static and dynamic objects. Our data is calibrated, synchronized and timestamped, and we provide the rectified and raw image sequences. Our dataset also contains object labels in the form of 3D tracklets, and we provide online benchmarks for stereo, optical flow, object detection and other tasks. This paper describes our recording platform, the data format and the utilities that we provide.
journaltitleInt. J. Rob. Res.
shorttitleVision meets robotics
titleVision meets robotics: The KITTI dataset
,laser,mobile robotics,SLAM,stereo,tracking,TODO

abstractThis paper addresses the problem of estimating the depth map of a scene given a single RGB image. We propose a fully convolutional architecture, encompassing residual learning, to model the ambiguous mapping between monocular images and depth maps. In order to improve the output resolution, we present a novel way to efficiently learn feature map up-sampling within the network. For optimization, we introduce the reverse Huber loss that is particularly suited for the task at hand and driven by the value distributions commonly present in depth maps. Our model is composed of a single architecture that is trained end-to-end and does not rely on post-processing techniques, such as CRFs or other additional refinement steps. As a result, it runs in real-time on images or videos. In the evaluation, we show that the proposed model contains fewer parameters and requires fewer training data than the current state of the art, while outperforming all approaches on depth estimation. Code and models are publicly available.
booktitle2016 Fourth International Conference on 3D Vision (3DV)
eventtitle2016 Fourth International Conference on 3D Vision (3DV)
titleDeeper Depth Prediction with Fully Convolutional Residual Networks
Laina et al. - 2016 - Deeper Depth Prediction with Fully Convolutional R.pdf:application/pdf;IEEE Xplore Abstract Record:D
7785097.html:text/html;Submitted Version:D
Laina et al. - 2016 - Deeper Depth Prediction with Fully Convolutional R.pdf:application/pdf
image resolution,Semantics,Three-dimensional displays,learning (artificial intelligence),Computer architecture,Computer vision,Estimation,image colour analysis,neural nets,RGB image,CNN,depth prediction,Depth prediction,estimation theory,Image resolution,optimisation,Optimization,convolution,depth map estimation,feature map learning,fully convolutional residual network,optimization,output resolution,reverse Huber loss,statistical distributions,value distribution

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
titleDeep Ordinal Regression Network for Monocular Depth Estimation

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
titleDeep Robust Single Image Depth Estimation Neural Network Using Scene Understanding

abstractEstimating accurate depth from a single image is challenging, because it is an ill-posed problem as infinitely many 3D scenes can be projected to the same 2D scene. However, recent works based on deep convolutional neural networks show great progress achieving plausible results. The networks are generally composed of two parts: an encoder for dense feature extraction and a decoder for predicting the desired depth. In the encoder-decoder schemes, repeated strided convolution and spatial pooling layers lower the spatial resolution of transitional outputs, and several techniques such as skip connections or multi-layer deconvolutional networks are adopted to effectively recover back to the original resolution. In this paper, for a more effective guidance of densely encoded features to desired depth prediction, we propose a network architecture that utilizes novel local planar guidance layers located at multiple stages in decoding phase. We show that the proposed method outperforms the state-of-the-art works with significant margin evaluating on challenging benchmarks. We also provide results from an ablation study to validate the effectiveness of the proposed core factors.
journaltitlearXiv:1907.10326 [cs]
shorttitleFrom Big to Small
titleFrom Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
Lee et al. - 2019 - From Big to Small Multi-Scale Local Planar Guidan.pdf:application/pdf;arXiv.org Snapshot:D

eventtitleProceedings of the IEEE International Conference on Computer Vision
titleEnforcing Geometric Constraints of Virtual Normal for Depth Prediction

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
titleSoft Labels for Ordinal Regression

abstractMonocular depth estimation is very challenging in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks, the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression-based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for monocular depth estimation, which is critical in achieving scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth scores to continuous depth values. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments have been conducted on the Make3D, NYU v2, and KITTI datasets and superior performance have been achieved on NYU v2 and KITTI datasets compared with current state-of-the-art methods, which shows the superiority of our method. Furthermore, experiments on the NYU v2 dataset reveal that our classification based model is able to learn the probability distribution of depth.
journaltitlePattern Recognition
titleMonocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference
S0031320318302097.html:text/html;Submitted Version:D
Li et al. - 2018 - Monocular depth estimation with hierarchical fusio.pdf:application/pdf
estimation,used,TODO,Deep convolutional neural network,Dilated convolution,Soft-weighted-sum-inference,g

abstractWe introduce MultiDepth, a novel training strategy and convolutional neural network (CNN) architecture that allows approaching single-image depth estimation (SIDE) as a multi-task problem. SIDE is an important part of road scene understanding. It, thus, plays a vital role in advanced driver assistance systems and autonomous vehicles. Best results for the SIDE task so far have been achieved using deep CNNs. However, optimization of regression problems, such as estimating depth, is still a challenging task. For the related tasks of image classification and semantic segmentation, numerous CNN-based methods with robust training behavior have been proposed. Hence, in order to overcome the notorious instability and slow convergence of depth value regression during training, MultiDepth makes use of depth interval classification as an auxiliary task. The auxiliary task can be disabled at test-time to predict continuous depth values using the main regression branch more efficiently. We applied MultiDepth to road scenes and present results on the KITTI depth prediction dataset. In experiments, we were able to show that end-to-end multitask learning with both, regression and classification, is able to considerably improve training and yield more accurate results.
booktitle2019 IEEE Intelligent Transportation Systems Conference (ITSC)
eventtitle2019 IEEE Intelligent Transportation Systems Conference (ITSC)
titleMultiDepth: Single-Image Depth Estimation via Multi-Task Regression and Classification
mage Depth Estimation via Mult.pdf:application/pdf
image segmentation,Semantics,learning (artificial intelligence),Training,Estimation,image classification,CNN,optimisation,Task analysis,used,advanced driver assistance systems,Autonomous vehicles,Convergence,convolutional neural nets,convolutional neural network architecture,Decoding,depth interval classification,depth value regression,driver information systems,KITTI depth prediction dataset,MultiDepth,multitask classification,multitask regression,neural net architecture,pose estimation,regression analysis,road scene understanding,robust training behavior,SIDE,single-image depth estimation,training strategy

abstractDepth estimation and semantic segmentation are two fundamental problems in image understanding. While the two tasks are strongly correlated and mutually beneficial, they are usually solved separately or sequentially. Motivated by the complementary properties of the two tasks, we propose a unified framework for joint depth and semantic prediction. Given an image, we first use a trained Convolutional Neural Network (CNN) to jointly predict a global layout composed of pixel-wise depth values and semantic labels. By allowing for interactions between the depth and semantic information, the joint network provides more accurate depth prediction than a state-of-the-art CNN trained solely for depth prediction [6]. To further obtain fine-level details, the image is decomposed into local segments for region-level depth and semantic prediction under the guidance of global layout. Utilizing the pixel-wise global prediction and region-wise local prediction, we formulate the inference problem in a two-layer Hierarchical Conditional Random Field (HCRF) to produce the final depth and semantic map. As demonstrated in the experiments, our approach effectively leverages the advantages of both tasks and provides the state-of-the-art results.
booktitle2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
eventtitle2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
noteISSN: 1063-6919
titleTowards unified depth and semantic prediction from a single image
n this paper, we argue that these two tasks provide complementary information, which therefore can be performed jointly to reinforce individual tasks in terms of both accuracy and speed. In particular, we propose an Elastic Conditional Random Field (E-CRF) deployed upon superpixel segmentations, which models the interdependency between depth and semantics to refine each other in an iterative manner. Differing from the traditional CRFs, E-CRF makes edges elastically hidden/emergent during inference to conduct fast Loopy Belief Propagation, while explicitly modeling the depth-label interdependency to achieve high inference accuracy. Moreover, the Structured Support Vector Machine (SSVM) is further introduced to drastically speed up the inference. We have conducted extensive evaluations on both Make3D and NYU benchmark datasets, which demonstrated that our E-CRF method significantly outperforms state-of-the-art techniques in terms of precision, while significantly accelerating the inference speed (2–3 orders of magnitude).
journaltitlePattern Recognition
seriesCompositional Models and Structured Learning for Visual Recognition
titleJoint Depth and Semantic Inference from a Single Image via Elastic Conditional Random Field
oint Depth and Semantic Inference from a Single I.pdf:D
Ji et al. - 2016 - Joint Depth and Semantic Inference from a Single I.pdf:application/pdf;ScienceDirect Snapshot:D
estimation,g,Semantic labeling,Conditional random field,Content analysis,Scene understanding,Structured Support Vector Machine,multi-task learning

Springer International Publishing

abstractMonocular depth estimation benefits greatly from learning based techniques. By studying the training data, we observe that the per-pixel depth values in existing datasets typically exhibit a long-tailed distribution. However, most previous approaches treat all the regions in the training data equally regardless of the imbalanced depth distribution, which restricts the model performance particularly on distant depth regions. In this paper, we investigate the long tail property and delve deeper into the distant depth regions (i.e. the tail part) to propose an attention-driven loss for the network supervision. In addition, to better leverage the semantic information for monocular depth estimation, we propose a synergy network to automatically learn the information sharing strategies between the two tasks. With the proposed attention-driven loss and synergy network, the depth estimation and semantic labeling tasks can be mutually improved. Experiments on the challenging indoor dataset show that the proposed approach achieves state-of-the-art performance on both monocular depth estimation and semantic labeling tasks.
booktitleComputer Vision – ECCV 2018
seriesLecture Notes in Computer Science
shorttitleLook Deeper into Depth
titleLook Deeper into Depth: Monocular Depth Estimation with Semantic Booster and Attention-Driven Loss
Jiao et al. - 2018 - Look Deeper into Depth Monocular Depth Estimation.pdf:application/pdf
used,TODO,g,Attention loss,Monocular depth,Semantic labeling,multi-task learning

Springer International Publishing

abstractAutonomous vehicles and robots require a full scene understanding of the environment to interact with it. Such a perception typically incorporates pixel-wise knowledge of the depths and semantic labels for each image from a video sensor. Recent learning-based methods estimate both types of information independently using two separate CNNs. In this paper, we propose a model that is able to predict both outputs simultaneously, which leads to improved results and even reduced computational costs compared to independent estimation of depth and semantics. We also empirically prove that the CNN is capable of learning more meaningful and semantically richer features. Furthermore, our SDNet estimates the depth based on ordinal classification. On the basis of these two enhancements, our proposed method achieves state-of-the-art results in semantic segmentation and depth estimation from single monocular input images on two challenging datasets.
booktitlePattern Recognition
seriesLecture Notes in Computer Science
titleSDNet: Semantically Guided Depth Estimation Network
chs et al. - 2019 - SDNet Semantically Guided Depth Estimation Networ.pdf:application/pdf
used,multi-task learning

Springer International Publishing

abstractDepth estimation from a single image represents a very exciting challenge in computer vision. While other image-based depth sensing techniques leverage on the geometry between different viewpoints (e.g., stereo or structure from motion), the lack of these cues within a single image renders ill-posed the monocular depth estimation task. For inference, state-of-the-art encoder-decoder architectures for monocular depth estimation rely on effective feature representations learned at training time. For unsupervised training of these models, geometry has been effectively exploited by suitable images warping losses computed from views acquired by a stereo rig or a moving camera. In this paper, we make a further step forward showing that learning semantic information from images enables to improve effectively monocular depth estimation as well. In particular, by leveraging on semantically labeled images together with unsupervised signals gained by geometry through an image warping loss, we propose a deep learning approach aimed at joint semantic segmentation and depth estimation. Our overall learning framework is semi-supervised, as we deploy groundtruth data only in the semantic domain. At training time, our network learns a common feature representation for both tasks and a novel cross-task loss function is proposed. The experimental findings show how, jointly tackling depth prediction and semantic segmentation, allows to improve depth estimation accuracy. In particular, on the KITTI dataset our network outperforms state-of-the-art methods for monocular depth estimation.
booktitleComputer Vision – ACCV 2018
seriesLecture Notes in Computer Science
titleGeometry Meets Semantics for Semi-supervised Monocular Depth Estimation
emantics for Semi-supervised Monoc.pdf:application/pdf
used,g,IMPORTANT,multi-task learning

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
titlePattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation
used,multi-task learning

abstractSemantic segmentation and depth estimation are two important tasks in computer vision, and many methods have been developed to tackle them. Commonly these two tasks are addressed independently, but recently the idea of merging these two problems into a sole framework has been studied under the assumption that integrating two highly correlated tasks may benefit each other to improve the estimation accuracy. In this paper, depth estimation and semantic segmentation are jointly addressed using a single RGB input image under a unified convolutional neural network. We analyze two different architectures to evaluate which features are more relevant when shared by the two tasks and which features should be kept separated to achieve a mutual improvement. Likewise, our approaches are evaluated under two different scenarios designed to review our results versus single-task and multi-task methods. Qualitative and quantitative experiments demonstrate that the performance of our methodology outperforms the state of the art on single-task approaches, while obtaining competitive results compared with other multi-task methods.
journaltitleSensors (Basel)
titleDepth Estimation and Semantic Segmentation from a Single RGB Image Using a Hybrid Convolutional Neural Network

abstractMonocular depth estimation is a challenging task in scene understanding, with the goal to acquire the geometric properties of 3D space from 2D images. Due to the lack of RGB-depth image pairs, unsupervised learning methods aim at deriving depth information with alternative supervision such as stereo pairs. However, most existing works fail to model the geometric structure of objects, which generally results from considering pixel-level objective functions during training. In this paper, we propose SceneNet to overcome this limitation with the aid of semantic understanding from segmentation. Moreover, our proposed model is able to perform region-aware depth estimation by enforcing semantics consistency between stereo pairs. In our experiments, we qualitatively and quantitatively verify the effectiveness and robustness of our model, which produces favorable results against the state-of-the-art approaches do.
booktitle2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
eventtitle2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
noteISSN: 2575-7075
shorttitleTowards Scene Understanding
titleTowards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation
hen et al. - 2019 - Towards Scene Understanding Unsupervised Monocula.pdf:application/pdf;IEEE Xplore Abstract Record:D
image segmentation,stereo image processing,image colour analysis,image representation,computer vision,used,g,2D images,3D from Single Image,3D space,alternative supervision,depth information,geometric properties,Grouping and Shape,pixel-level objective functions,region-aware depth estimation,Representation Learning,RGB-depth image pairs,Robotics + Driving,Scene Analysis and Understanding,scene understanding,SceneNet,Segmentation,semantic understanding,semantic-aware representation,semantics consistency,stereo pairs,unsupervised learning,unsupervised learning methods,unsupervised monocular depth estimation,multi-task learning

abstractModern image processing techniques use three-dimensional (3D) images, which contain spatial information such as depth and scale, in addition to visual information. These images are indispensable in virtual reality, augmented reality (AR), and autonomous driving applications. We propose a novel method to estimate monocular depth using a cycle generative adversarial network (GAN) and segmentation. In this paper, we propose a method for estimating depth information by combining segmentation. It uses three processes: segmentation and depth estimation, adversarial loss calculations, and cycle consistency loss calculations. The cycle consistency loss calculation process evaluates the similarity of two images when they are restored to their original forms after being estimated separately from two adversarial losses. To evaluate the objective reliability of the proposed method, we compared our proposed method with other monocular depth estimation (MDE) methods using the NYU Depth Dataset V2. Our results show that the benchmark value for our proposed method is better than other methods. Therefore, we demonstrated that our proposed method is more efficient in determining depth estimation.
noteNumber: 9 Publisher: Multidisciplinary Digital Publishing Institute
titleA Novel Method for Estimating Monocular Depth Using Cycle GAN and Segmentation
Kwak and Lee - 2020 - A Novel Method for Estimating Monocular Depth Usin.pdf:application/pdf

Springer, Cham

abstractWe present a novel method for accurate and efficient upsampling of sparse depth data, guided by high-resolution imagery. Our approach goes beyond the use of intensity cues only and additionally...
booktitlePattern Recognition
eventtitleGerman Conference on Pattern Recognition
titleSemantically Guided Depth Upsampling
ubmitted Version:D
Schneider et al. - 2016 - Semantically Guided Depth Upsampling.pdf:application/pdf

abstractSelf-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.
journaltitleICLR 2020
titleSemantically-Guided Representation Learning for Self-Supervised Monocular Depth

abstractWe consider the problem of estimating the depth of each pixel in a scene from a single monocular image. Unlike traditional approaches, which attempt to map from appearance features to depth directly, we first perform a semantic segmentation of the scene and use the semantic labels to guide the 3D reconstruction. This approach provides several advantages: By knowing the semantic class of a pixel or region, depth and geometry constraints can be easily enforced (e.g., “sky” is far away and “ground” is horizontal). In addition, depth can be more readily predicted by measuring the difference in appearance with respect to a given semantic class. For example, a tree will have more uniform appearance in the distance than it does close up. Finally, the incorporation of semantic features allows us to achieve state-of-the-art results with a significantly simpler model than previous works.
booktitle2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
eventtitle2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
noteISSN: 1063-6919
titleSingle image depth estimation from predicted semantic labels
EEE International Conference on Robotics and Automation (ICRA)
eventtitle2017 IEEE International Conference on Robotics and Automation (ICRA)
titleAnalyzing modular CNN architectures for joint depth prediction and semantic segmentation
mage segmentation,Semantics,Computer architecture,Computer vision,convolutional neural networks,cross-modality influence,depth estimation,Estimation,image colour analysis,joint depth prediction,Labeling,modular CNN architecture,modular neural network architecture,Network architecture,neural nets,NYU-Depth v2 benchmark,red-green-blue image,RGB image,semantic prediction maps,semantic segmentation,used,TODO

Berlin, Heidelberg

abstractWe present an approach to interpret the major surfaces, objects, and support relations of an indoor scene from an RGBD image. Most existing work ignores physical interactions or is applied only to tidy rooms and hallways. Our goal is to parse typical, often messy, indoor scenes into floor, walls, supporting surfaces, and object regions, and to recover support relationships. One of our main interests is to better understand how 3D cues can best inform a structured 3D interpretation. We also contribute a novel integer programming formulation to infer physical support relations. We offer a new dataset of 1449 RGBD images, capturing 464 diverse indoor scenes, with detailed annotations. Our experiments demonstrate our ability to infer support relations in complex scenes and verify that our 3D scene cues and inferred support lead to better object segmentation.
booktitleComputer Vision – ECCV 2012
seriesLecture Notes in Computer Science
titleIndoor Segmentation and Support Inference from RGBD Images
ilberman et al. - 2012 - Indoor Segmentation and Support Inference from RGB.pdf:application/pdf

abstractRobust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first published dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online at http://www.nuscenes.org.
journaltitlearXiv:1903.11027 [cs, stat]
titlenuScenes: A multimodal dataset for autonomous driving
Caesar et al. - 2020 - nuScenes A multimodal dataset for autonomous driv.pdf:application/pdf;arXiv.org Snapshot:D
ine Learning,Computer Science - Robotics,Statistics - Machine Learning
titlenuScenes dataset - Data Format

abstractA common problem in autonomous driving is designing a system that can operate in adverse weather conditions. Falling rain and snow tends to corrupt sensor measurements, particularly for lidar sensors. Surprisingly, very little research has been published on methods to de-noise point clouds which are collected by lidar in rainy or snowy weather conditions. In this paper, we present a method for removing snow noise by processing point clouds using a 3D outlier detection algorithm. Our method, the dynamic radius outlier removal filter, accounts for the variation in point cloud density with increasing distance from the sensor, with the goal of removing the noise caused by snow while retaining detail in environmental features (which is necessary for autonomous localization and navigation). The proposed method outperforms other noise-removal methods, including methods which operate on depth image representations of the lidar scans. We show on point clouds obtained while driving in falling snow that we can simultaneously obtain > 90% precision and recall, indicating that the proposed method is effective at removing snow, without removing environmental features.
booktitle2018 15th Conference on Computer and Robot Vision (CRV)
eventtitle2018 15th Conference on Computer and Robot Vision (CRV)
noteISSN: null
titleDe-noising of Lidar Point Clouds Corrupted by Snowfall
louds Corrupted by Snow.pdf:D
Charron et al. - 2018 - De-noising of Lidar Point Clouds Corrupted by Snow.pdf:application/pdf;IEEE Xplore Abstract Record:D
Image segmentation,Three-dimensional displays,3D outlier detection algorithm,adverse weather conditions,autonomous driving,autonomous localization,depth image representations,dynamic radius outlier removal filter,environmental features,falling rain,falling snow,image denoising,image representation,image thinning,Laser radar,lidar filtering,LIDAR point clouds,LIDAR scans,LIDAR sensors,Noise reduction,noise-removal methods,optical radar,point cloud density,point cloud processing,rain,smoothing methods,Smoothing methods,snow,Snow,snow noise,snow noise removal,snowy weather conditions,traffic engineering computing,Two dimensional displays,used

abstractAutonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g., KITTI [2] or Cityscapes [3], ApolloScape contains much large and richer labelling including holistic semantic dense point cloud for each site, stereo, per-pixel semantic labelling, lanemark labelling, instance segmentation, 3D car instance, high accurate location for every frame in various driving videos from multiple sites, cities and daytimes. For each task, it contains at lease 15x larger amount of images than SOTA datasets. To label such a complete dataset, we develop various tools and algorithms specified for each task to accelerate the labelling process, such as joint 3D-2D segment labeling, active labelling in videos etc. Depend on ApolloScape, we are able to develop algorithms jointly consider the learning and inference of multiple tasks. In this paper, we provide a sensor fusion scheme integrating camera videos, consumer-grade motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robust self-localization and semantic segmentation for autonomous driving. We show that practically, sensor fusion and joint learning of multiple tasks are beneficial to achieve a more robust and accurate system. We expect our dataset and proposed relevant algorithms can support and motivate researchers for further development of multi-sensor fusion and multi-task learning in the field of computer vision.
journaltitleIEEE Transactions on Pattern Analysis and Machine Intelligence
noteConference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence
titleThe ApolloScape Open Dataset for Autonomous Driving and its Application
Huang et al. - 2019 - The ApolloScape Open Dataset for Autonomous Drivin.pdf:application/pdf;IEEE Xplore Abstract Record:D
authors.html:text/html;Submitted Version:D
Wang et al. - 2019 - The ApolloScape Open Dataset for Autonomous Drivin.pdf:application/pdf
Image segmentation,Semantics,Three-dimensional displays,Labeling,Two dimensional displays,Task analysis,3D Understanding,Autonomous Driving,Large-scale Datasets,Scene/Lane Parsing,Self Localization,Videos

MIT Press

titleDeep Learning

abstractThe most successful 2D object detection methods require a large number of images annotated with object bounding boxes to be collected for training. We present an alternative approach that trains on virtual data rendered from 3D models, avoiding the need for manual labeling. Growing demand for virtual reality applications is quickly bringing about an abundance of available 3D models for a large variety of object categories. While mainstream use of 3D models in vision has focused on predicting the 3D pose of objects, we investigate the use of such freely available 3D models for multicategory 2D object detection. To address the issue of dataset bias that arises from training on virtual data and testing on real images, we propose a simple and fast adaptation approach based on decorrelated features. We also compare two kinds of virtual data, one rendered with real-image textures and one without. Evaluation on a benchmark domain adaptation dataset demonstrates that our method performs comparably to existing methods trained on large-scale real image domains.
shorttitleFrom Virtual to Reality
titleFrom Virtual to Reality: Fast Adaptation of Virtual Object Detectors to Real Domains
Sun and Saenko - 2014 - From Virtual to Reality Fast Adaptation of Virtua.pdf:application/pdf

abstractPedestrian detection is of paramount interest for many applications. Most promising detectors rely on discriminatively learnt classifiers, i.e., trained with annotated samples. However, the annotation step is a human intensive and subjective task worth to be minimized. By using virtual worlds we can automatically obtain precise and rich annotations. Thus, we face the question: can a pedestrian appearance model learnt in realistic virtual worlds work successfully for pedestrian detection in real-world images? Conducted experiments show that virtual-world based training can provide excellent testing accuracy in real world, but it can also suffer the data set shift problem as real-world based training does. Accordingly, we have designed a domain adaptation framework, V-AYLA, in which we have tested different techniques to collect a few pedestrian samples from the target domain (real world) and combine them with the many examples of the source domain (virtual world) in order to train a domain adapted pedestrian classifier that will operate in the target domain. V-AYLA reports the same detection accuracy than when training with many human-provided pedestrian annotations and testing with real-world images of the same domain. To the best of our knowledge, this is the first work demonstrating adaptation of virtual and real worlds for developing an object detector.
journaltitleIEEE Transactions on Pattern Analysis and Machine Intelligence
noteConference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence
titleVirtual and Real World Adaptation for Pedestrian Detection
mage Enhancement,Image Interpretation,Computer-Assisted,Image resolution,Interpolation,learning (artificial intelligence),Machine Learning,object detection,object detector,Pattern Recognition,Automated,pedestrian appearance model,pedestrian detection,Pedestrian detection,pedestrian samples,pedestrians,Pedestrians,photo-realistic computer animation,Photography,real-world images,realistic virtual worlds,Reproducibility of Results,Sensitivity and Specificity,source domain,Subtraction Technique,target domain,Testing,Training,User-Computer Interface,V-AYLA,virtual reality,virtual-world based training,Whole Body Imaging

abstractThe accuracy of object classifiers can significantly drop when the training data (source domain) and the application scenario (target domain) have inherent differences. Therefore, adapting the classifiers to the scenario in which they must operate is of paramount importance. We present novel domain adaptation (DA) methods for object detection. As proof of concept, we focus on adapting the state-of-the-art deformable part-based model (DPM) for pedestrian detection. We introduce an adaptive structural SVM (A-SSVM) that adapts a pre-learned classifier between different domains. By taking into account the inherent structure in feature space (e.g., the parts in a DPM), we propose a structure-aware A-SSVM (SA-SSVM). Neither A-SSVM nor SA-SSVM needs to revisit the source-domain training data to perform the adaptation. Rather, a low number of target-domain training examples (e.g., pedestrians) are used. To address the scenario where there are no target-domain annotated samples, we propose a self-adaptive DPM based on a self-paced learning (SPL) strategy and a Gaussian Process Regression (GPR). Two types of adaptation tasks are assessed: from both synthetic pedestrians and general persons (PASCAL VOC) to pedestrians imaged from an on-board camera. Results show that our proposals avoid accuracy drops as high as 15 points when comparing adapted and non-adapted detectors.
journaltitleIEEE Transactions on Pattern Analysis and Machine Intelligence
noteConference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence
titleDomain Adaptation of Deformable Part-Based Models
mage Processing,Computer-Assisted,learning (artificial intelligence),Machine Learning,Models,Theoretical,object classifiers,object detection,Object recognition,on-board camera,PASCAL VOC,pedestrian detection,pedestrians,prelearned classifier,regression analysis,SA-SSVM,self-adaptive DPM,self-paced learning strategy,source-domain training data,SPL strategy,structure-aware A-SSVM,Support Vector Machine,support vector machines,Support vector machines,synthetic pedestrians,target-domain training examples,Training,Video Recording,Walking

abstractFigure 1: Humans are perfect for annotating coarse viewpoints of objects in real images, but fail to estimate pose accurately at a fine level. 3D graphic models can be used to synthesize data at very accurate fine angles, but it is time-consuming to model all appearance variations present in real images. We therefore propose to leverage the abilities of humans of estimating coarse viewpoints and the pose accuracy of synthetic data.
titleAdaptation of Synthetic Data for Coarse-to-Fine Viewpoint Refinement
Busto et al. - 2015 - Adaptation of Synthetic Data for Coarse-to-Fine Vi.pdf:application/pdf

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
shorttitleThe SYNTHIA Dataset
titleThe SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

abstractA key topic in classification is the accuracy loss produced when the data distribution in the training (source) domain differs from that in the testing (target) domain. This is being recognized as a very relevant problem for many computer vision tasks such as image classification, object detection, and object category recognition. In this paper, we present a novel domain adaptation method that leverages multiple target domains (or sub-domains) in a hierarchical adaptation tree. The core idea is to exploit the commonalities and differences of the jointly considered target domains. Given the relevance of structural SVM (SSVM) classifiers, we apply our idea to the adaptive SSVM (A-SSVM; Xu et al., IEEE Trans Pattern Anal Mach Intell 36(12):2367–2380, 2014a), which only requires the target domain samples together with the existing source-domain classifier for performing the desired adaptation. Altogether, we term our proposal as hierarchical A-SSVM (HA-SSVM). As proof of concept we use HA-SSVM for pedestrian detection, object category recognition and face recognition. In the former we apply HA-SSVM to the deformable part-based model (DPM; Felzenszwalb et al., IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645, 2010) while in the rest HA-SSVM is applied to multi-category classifiers. We will show how HA-SSVM is effective in increasing the detection/recognition accuracy with respect to adaptation strategies that ignore the structure of the target data. Since, the sub-domains of the target data are not always known a priori, we shown how HA-SSVM can incorporate sub-domain discovery for object category recognition.
journaltitleInt J Comput Vis
titleHierarchical Adaptive Structural SVM for Domain Adaptation
Xu et al. - 2016 - Hierarchical Adaptive Structural SVM for Domain Ad.pdf:application/pdf

abstractWe introduce Synscapes – a synthetic dataset for street scene parsing created using photorealistic rendering techniques, and show state-of-the-art results for training and validation as well as new types of analysis. We study the behavior of networks trained on real data when performing inference on synthetic data: a key factor in determining the equivalence of simulation environments. We also compare the behavior of networks trained on synthetic data and evaluated on real-world data. Additionally, by analyzing pre-trained, existing segmentation and detection models, we illustrate how uncorrelated images along with a detailed set of annotations open up new avenues for analysis of computer vision systems, providing fine-grain information about how a model's performance changes according to factors such as distance, occlusion and relative object orientation.
journaltitlearXiv:1810.08705 [cs]
titleSynscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing
Wrenninge and Unger - 2018 - Synscapes A Photorealistic Synthetic Dataset for .pdf:application/pdf;arXiv.org Snapshot:D

abstractWe consider the problem of unsupervised domain adaptation in semantic segmentation. The key in this campaign consists in reducing the domain shift, i.e., enforcing the data distributions of the two domains to be similar. A popular strategy is to align the marginal distribution in the feature space through adversarial learning. However, this global alignment strategy does not consider the local category-level feature distribution. A possible consequence of the global movement is that some categories which are originally well aligned between the source and target may be incorrectly mapped. To address this problem, this paper introduces a category-level adversarial network, aiming to enforce local semantic consistency during the trend of global alignment. Our idea is to take a close look at the category-level data distribution and align each class with an adaptive adversarial loss. Specifically, we reduce the weight of the adversarial loss for category-level aligned features while increasing the adversarial force for those poorly aligned. In this process, we decide how well a feature is category-level aligned between source and target by a co-training approach. In two domain adaptation tasks, i.e., GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, we validate that the proposed method matches the state of the art in segmentation accuracy.
journaltitlearXiv:1809.09478 [cs]
shorttitleTaking A Closer Look at Domain Shift
titleTaking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation
Luo et al. - 2019 - Taking A Closer Look at Domain Shift Category-lev.pdf:application/pdf;arXiv.org Snapshot:D

abstractSynthetic data is an increasingly popular tool for training deep learning models, especially in computer vision but also in other areas. In this work, we attempt to provide a comprehensive survey of the various directions in the development and application of synthetic data. First, we discuss synthetic datasets for basic computer vision problems, both low-level (e.g., optical flow estimation) and high-level (e.g., semantic segmentation), synthetic environments and datasets for outdoor and urban scenes (autonomous driving), indoor scenes (indoor navigation), aerial navigation, simulation environments for robotics, applications of synthetic data outside computer vision (in neural programming, bioinformatics, NLP, and more); we also survey the work on improving synthetic data development and alternative ways to produce it such as GANs. Second, we discuss in detail the synthetic-to-real domain adaptation problem that inevitably arises in applications of synthetic data, including synthetic-to-real refinement with GAN-based models and domain adaptation at the feature/model level without explicit data transformations. Third, we turn to privacy-related applications of synthetic data and review the work on generating synthetic datasets with differential privacy guarantees. We conclude by highlighting the most promising directions for further work in synthetic data studies.
journaltitlearXiv:1909.11512 [cs]
titleSynthetic Data for Deep Learning
Nikolenko - 2019 - Synthetic Data for Deep Learning.pdf:application/pdf;arXiv.org Snapshot:D
ine Learning,Computer Science - Cryptography and Security

abstractDeeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers - 8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
booktitle2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
eventtitle2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
noteISSN: 1063-6919
titleDeep Residual Learning for Image Recognition
mage Recognition.pdf:application/pdf
Image segmentation,object detection,learning (artificial intelligence),Training,image recognition,Image recognition,neural nets,Visualization,image classification,CIFAR-10,COCO object detection dataset,COCO segmentation,Complexity theory,deep residual learning,deep residual nets,deeper neural network training,Degradation,ILSVRC & COCO 2015 competitions,ILSVRC 2015 classification task,ImageNet dataset,ImageNet localization,ImageNet test set,Neural networks,residual function learning,residual nets,VGG nets,visual recognition tasks

booktitleInternational Conference on Computer Vision (ICCV)
titleThe Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

abstractIn this work we present a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced that uses an extremely efficient over-segmentation. In doing so, the computational complexity of the Stixel inference algorithm is reduced significantly, achieving real-time computation capabilities with only a slight drop in accuracy. We evaluate the proposed approach in terms of semantic and geometric accuracy as well as run-time on four publicly available benchmark datasets. Our approach maintains accuracy on flat road scene datasets while improving substantially on a novel non-flat road dataset.
journaltitlearXiv:1707.05397 [cs]
shorttitleSlanted Stixels
titleSlanted Stixels: Representing San Francisco's Steepest Streets
Hernandez-Juarez et al. - 2017 - Slanted Stixels Representing San Francisco's Stee.pdf:application/pdf;arXiv.org Snapshot:D
abstractA random seed (or seed state, or just seed) is a number (or vector) used to initialize a pseudorandom number generator. For a seed to be used in a pseudorandom number generator, it does not need to be random. Because of the nature of number generating algorithms, so long as the original seed is ignored, the rest of the values that the algorithm generates will follow probability distribution in a pseudorandom manner. A pseudorandom number generator's number sequence is completely determined by the seed: thus, if a pseudorandom number generator is reinitialized with the same seed, it will produce the same sequence of numbers. The choice of a good random seed is crucial in the field of computer security. When a secret encryption key is pseudorandomly generated, having the seed will allow one to obtain the key. High entropy is important for selecting good random seed data.If the same random seed is deliberately shared, it becomes a secret key, so two or more systems using matching pseudorandom number algorithms and matching seeds can generate matching sequences of non-repeating numbers which can be used to synchronize remote systems, such as GPS satellites and receivers. Random seeds are often generated from the state of the computer system (such as the time), a cryptographically secure pseudorandom number generator or from a hardware random number generator.
notePage Version ID: 933429432
titleRandom seed
p?title=Random_seed   oldid=933429432

abstractThis paper introduces an updated version of the well-known Virtual KITTI dataset which consists of 5 sequence clones from the KITTI tracking benchmark. In addition, the dataset provides different variants of these sequences such as modified weather conditions (e.g. fog, rain) or modified camera configurations (e.g. rotated by 15 degrees). For each sequence, we provide multiple sets of images containing RGB, depth, class segmentation, instance segmentation, flow, and scene flow data. Camera parameters and poses as well as vehicle locations are available as well. In order to showcase some of the dataset's capabilities, we ran multiple relevant experiments using state-of-the-art algorithms from the field of autonomous driving. The dataset is available for download at https://europe.naverlabs.com/Research/Computer-Vision/Proxy-Virtual-Worlds.
journaltitlearXiv:2001.10773 [cs, eess]
titleVirtual KITTI 2
Cabon et al. - 2020 - Virtual KITTI 2.pdf:application/pdf;arXiv.org Snapshot:D
noteLibrary Catalog: colab.research.google.com
titleGoogle Colaboratory

Springer, Cham

abstractThere is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong...
booktitleMedical Image Computing and Computer-Assisted Intervention – MICCAI 2015
eventtitleInternational Conference on Medical Image Computing and Computer-Assisted Intervention
titleU-Net: Convolutional Networks for Biomedical Image Segmentation
ubmitted Version:D
Ronneberger et al. - 2015 - U-Net Convolutional Networks for Biomedical Image.pdf:application/pdf

eventtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition
titleThe Cityscapes Dataset for Semantic Urban Scene Understanding

abstractWe present a benchmark suite for visual perception. The benchmark is based on more than 250K high-resolution video frames, all annotated with ground-truth data for both low-level and high-level vision tasks, including optical flow, semantic instance segmentation, object detection and tracking, object-level 3D scene layout, and visual odometry. Ground-truth data for all tasks is available for every frame. The data was collected while driving, riding, and walking a total of 184 kilometers in diverse ambient conditions in a realistic virtual world. To create the benchmark, we have developed a new approach to collecting ground-truth data from simulated worlds without access to their source code or content. We conduct statistical analyses that show that the composition of the scenes in the benchmark closely matches the composition of corresponding physical environments. The realism of the collected data is further validated via perceptual experiments. We analyze the performance of state-of-the-art methods for multiple tasks, providing reference baselines and highlighting challenges for future research.
booktitle2017 IEEE International Conference on Computer Vision (ICCV)
eventtitle2017 IEEE International Conference on Computer Vision (ICCV)
titlePlaying for Benchmarks
Richter et al. - 2017 - Playing for Benchmarks.pdf:application/pdf;arXiv.org Snapshot:D
1709.html:text/html;IEEE Xplore Abstract Record:D
igh-resolution video frames,benchmark suite,Benchmark testing,benchmarks playing,diverse ambient conditions,Games,ground-truth data collection,high-level vision tasks,image annotation,image resolution,image segmentation,Image segmentation,image sequences,Layout,low-level vision tasks,object detection,object tracking,object-level 3D scene layout,optical flow,realistic virtual world,semantic instance segmentation,Semantics,statistical analyses,statistical analysis,stereo image processing,Three-dimensional displays,video signal processing,visual odometry,visual perception,used

abstractIn this paper, we consider convolutional neural networks operating on sparse inputs with an application to depth completion from sparse laser scan data. First, we show that traditional convolutional networks perform poorly when applied to sparse data even when the location of missing data is provided to the network. To overcome this problem, we propose a simple yet effective sparse convolution layer which explicitly considers the location of missing data during the convolution operation. We demonstrate the benefits of the proposed network architecture in synthetic and real experiments with respect to various baseline approaches. Compared to dense baselines, the proposed sparse convolution network generalizes well to novel datasets and is invariant to the level of sparsity in the data. For our evaluation, we derive a novel dataset from the KITTI benchmark, comprising over 94k depth annotated RGB images. Our dataset allows for training and evaluating depth completion and depth prediction techniques in challenging real-world settings and is available online at: www.cvlibs.net/datasets/kitti.
booktitle2017 International Conference on 3D Vision (3DV)
eventtitle2017 International Conference on 3D Vision (3DV)
noteISSN: 2475-7888
titleSparsity Invariant CNNs
Uhrig et al. - 2017 - Sparsity Invariant CNNs.pdf:application/pdf;IEEE Xplore Abstract Record:D
Three-dimensional displays,convolutional neural networks,image colour analysis,neural nets,used,KITTI benchmark,Convolution,annotated RGB images,baseline approaches,convolution operation,Convolutional neural networks,dense baselines,depth completion,depth prediction techniques,effective sparse convolution layer,evaluating depth completion,Kernel,Laser applications,Laser modes,network architecture,novel dataset,sparse convolution network,sparse inputs,sparse laser scan data,sparsity invariant CNN,Standards,traditional convolutional networks,training
abstractMoving the Community Forward, Together. Open sourcing Level 5's autonomous driving data.
noteLibrary Catalog: level5.lyft.com
titleLyft Level 5 AV Dataset 2019
titleaddonLyft Level 5

abstractWe introduce the Precise Synthetic Image and LiDAR (PreSIL) dataset for autonomous vehicle perception. Grand Theft Auto V (GTA V), a commercial video game, has a large detailed world with realistic graphics, which provides a diverse data collection environment. Existing works creating synthetic LiDAR data for autonomous driving with GTA V have not released their datasets, rely on an in-game raycasting function which represents people as cylinders, and can fail to capture vehicles past 30 metres. Our work creates a precise LiDAR simulator within GTA V which collides with detailed models for all entities no matter the type or position. The PreSIL dataset consists of over 50,000 frames and includes high-definition images with full resolution depth information, semantic segmentation (images), point-wise segmentation (point clouds), and detailed annotations for all vehicles and people. Collecting additional data with our framework is entirely automatic and requires no human annotation of any kind. We demonstrate the effectiveness of our dataset by showing an improvement of up to 5% average precision on the KITTI 3D Object Detection benchmark challenge when state-of-the-art 3D object detection networks are pre-trained with our data. The data and code are available at https://tinyurl.com/y3tb9sxy.
booktitle2019 IEEE Intelligent Vehicles Symposium (IV)
eventtitle2019 IEEE Intelligent Vehicles Symposium (IV)
noteISSN: 2642-7214
titlePrecise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception
Hurl et al. - 2019 - Precise Synthetic Image and LiDAR (PreSIL) Dataset.pdf:application/pdf;IEEE Xplore Abstract Record:D
image annotation,image resolution,image segmentation,Image segmentation,object detection,Three-dimensional displays,visual perception,Object detection,autonomous driving,Laser radar,optical radar,Two dimensional displays,used,Autonomous vehicles,3D object detection networks,Annotations,autonomous vehicle perception,computer games,diverse data collection environment,Grand Theft Auto V,GTA V,high-definition images,in-game raycasting function,KITTI 3D object detection benchmark challenge,LiDAR dataset,LiDAR simulator,point-wise segmentation,precise synthetic image and LiDAR dataset,PreSIL dataset,road vehicle radar,synthetic LiDAR data,video game

abstractWe introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
noteLibrary Catalog: www.semanticscholar.org
titleAdam: A Method for Stochastic Optimization

[heading=bibintoc, title=Bibliography]