# DeePaste - Inpainting for Pasting

Levi Kassel
The Hebrew University of Jerusalem
Jerusalem, Israel
<EMAIL_ADDRESS>Michael Werman
The Hebrew University of Jerusalem
Jerusalem, Israel
<EMAIL_ADDRESS>
###### Abstract

One of the challenges of supervised learning training is the need to procure
an substantial amount of tagged data. A well-known method of solving this
problem is to use synthetic data in a copy-paste fashion, so that we cut
objects and paste them onto relevant backgrounds. Pasting the objects naively
results in artifacts that cause models to give poor results on real data. We
present a new method for cleanly pasting objects on different backgrounds so
that the dataset created gives competitive performance on real data. The main
emphasis is on the treatment of the border of the pasted object using
inpainting. We show state-of-the-art results both on instance detection and
foreground segmentation.

Figure 1: This is the main pipeline of our system. First we paste objects on
the background image (a). Second, we create the blending gap mask (b). We use
the trained inpainting model to fill the blending gap pixels (c). In (d) you
can see the final image. The pixels are filled with the right context (e).
Figure 2: The pasting in the top image is over the white masks. The inpainting
is on the gray area.

##  1 Introduction

In recent years, with the dramatic development of detection and segmentation
in computer vision with the help of AI, the demand has increased to put these
capabilities into existing systems such as mobile apps, autonomous vehicles,
robots, etc. One of the biggest problems with state of the art systems is the
amount of tagged data needed for each task or scene to train them.

For each new task, as well as for any change in the environment, such as a
change in the objects you wish to detect or detecting objects in a different
scene, one is required to create thousands or more tagged images based on the
same task or scene. Since data tagging is very labor intensive, deploying
those tasks is not carried out in most cases.

In recent years more and more researchers have been able to overcome this
challenge by creating synthetic data from synthetically rendered scenes and
objects[15][34][16][31], so that they render objects and scenes for training
detection and segmentation systems. Although these systems bring promising
results, they require a high level of graphics know how and a lot of effort to
make objects and scenes look real .

Moreover, models that train on such data find it difficult to give
satisfactory results on real data because of the change in the statistics of
the image[5][29]. To overcome this difficulty and to still generate tagged
data faster, more and more researchers are working on composing real images
[9][48][11]. The general idea is to cut real objects and paste them on real
scenes, thus getting free tagged images that fit the environment in which we
want to work.

However, if we use this paradigm naively - we will get poor results on real
data. This is because existing systems are more sensitive to local region
based features than to global scene layout. The implication is that when we
paste an object on an image we are creating subtle pixel artifacts on the
background images. This phenomenon prevents the system from generalizing to
real images. Dwibedi et al. [9] create a framework that combines all kinds of
simple blending and blurring types on the objects and achieves surprisingly
impressive results.

In this article, we present a new method of blending to overcome the
difficulty of local-level realism. We use the power of inpainting trained on
relevant scenes and thus succeed in filling in the gaps in required areas - in
our case - the area between the object and the background, the blending gap.

Because we know the environment in which we want to work, we train an
inpainting model that knows how to fill the blending gap so that the detection
and segmentation systems can be generalized to real data.

In recent years there have also been works on improving the blending and
improving the harmonization of objects that are pasted on a background image
such as Deep Harmonization[42][7] or Deep Blending [47][52]. These systems
were not compared because the purpose of these systems is graphical, and
perform the learning on each image and object individually[52] \- which makes
the purpose of creating a large dataset non-feasible. What we do is different.
The images we produce are not graphically perfect - but are good enough to
cheat the learning systems so that they can generalize well on real data.

In this article, we evaluate this method and its effectiveness and show that
it achieves state of the art results on instance detection and foreground
segmentation.

Figure 3:  Sample pairs of blending gap filling from the CDnet 2014
dataset[44] in thermal (a), baseline (b,e), night videos (c), and low frame-
rate (d) scenes.

##  2 Related Work

Instance detection is where we try to locate a particular object within a
target image. Early approaches, such as [6] tried to solve this task by using
local features of the desired object image such as SIFT [26], FAST [8], or
SURF [4], and try to match them with local features extracted from the target
image. These methods did not work well when the objects we wanted to detect
were partially occluded or did not have enough features [14][17].

Another well-known computer vision task is foreground segmentation (also
called background subtraction), determining which pixels belong to the
background and which to the foreground, mainly from videos. Early approaches,
such as [39][38][43][3], were mainly based on analyzing the displacement of
certain parts of the image and trying to decide if it is a displacement of a
foreground object or a natural change that occurs in the background such as
shading, change in lighting, or natural displacement of dynamic objects
belonging to the background.

Modern methods in these two areas are mainly deep learning systems based on
convolution network architectures so that the extraction of the features is
done more semantically even in difficult cases [33][32] [24][45]. Most of the
best performing systems are supervised - which means that they learn directly
from tagged images and are trained end to end.

In recent years, with the availability of powerful hardware, these methods are
more suitable for real-time applications and thus the demand is increasing to
use these capabilities in more and more areas like robots, navigation, and
surveillance [46][28].

These methods require large amounts of annotated data. There is a bottleneck
as each environment and scene requires a collection of annotated data which
significantly slows the deployment of these systems.

One method that solves this problem is to create a synthetic dataset. One
setting where synthetic data can be generated is by rendering the environment
and thus automatically creating a dataset tailored to the specific problem we
want to solve[15][34]. A big advantage of this method is that the space of
possibilities is large - you can create any environment you want and you can
change the objects and background simply as needed [16][31]. Unfortunately,
when trying to generalize those systems from synthetic data to real would
images - we encounter poor performance due to the difference in the statistics
between the images space[5][29].

Another method to produce synthetic data is by composing real images. In
general, the method works by taking existing objects, which you want to detect
or to segment, cut them, with a corresponding image mask, select a background
image, and paste the object in a certain position in the background image.
There are certain algorithms where you can automatically choose where to paste
an object to maintain the realism of the created image [41][20]. This process
is very scalable and you can easily create a variety of data for the required
purpose.

Dwibedi et al.[9] took this process and examined it on the task of instance
segmentation. In their paper, they notice that modern deep learning based
systems care more about local region-based features for detection than the
global scene layout. This is why it is very important to pay attention to how
the pasting process takes place. When we naively place objects in scenes
subtle pixel artifacts are introduced into the images. As these minor
imperfections in the pixel space feed forward deeper into the layers of a
ConvNet [22], they lead to noticeably different features and the training
algorithm focuses on these discrepancies to detect objects, often ignoring to
model their complex visual appearance. To overcome this problem they used
blending methods such as Gaussian Blurring and Poisson editing [30] and found
that the use of a blending method in the pasting process can alter the
detection performance dramatically. In addition, they found that if they
synthesize the same scene with the same object placement only varying the type
of blending used makes the training algorithm invariant to these blending
factors and improves performance by a big margin.

Kassel et al[20] used this framework in foreground segmentation. They used a
weak foreground segmenter to extract objects from the training images and
insert them in their original position into a background image. It is
especially pertinent to static cameras as the objects found are automatically
in the right location, being of the right size, color, and shape, and in the
right lighting conditions.

Another task in computer vision that is related to image synthesis is image
inpainting. The main idea of image inpainting is to fill missing pixels in the
image. The main difficulty in this task is to synthesize visually realistic
and semantically plausible pixels for the missing regions that are coherent
with existing ones.

Early work on this subject [2][13] attempted to solve this task by searching
for similar areas in the image and attempting to paste patches into the
missing holes while maintaining global consistency. These works work well
especially in images that are characterized by stationary textures but are
limited to non-stationary data such as natural images[35]. In recent years,
deep learning has also entered this field and GAN-based[12] approaches have
yielded promising results[18] [23]. Yang et al. [49] and [50] devised feature
shift and contextual attention operations, respectively, to allow the model to
borrow feature patches from distant areas of the image. Another work by Yu et
al. [51] tries to handle irregular holes by filling using gated convolutions.

##  3 Approach Overview

As in Cut, Paste, and Learn [9], we focus only on the process of pasting
objects on the background images, as part of the creation of synthetic data.
From their paper, it can be seen that there is a high correlation between the
pasting process and how the system focuses on the appearance of the pasted
object and so manages to generalize well on real data. As mentioned, if you
paste the object somehow on the background image, there are pixel artifacts in
the gap between the object and the background image which greatly affects how
the system treats real data. In other words, the object is fine, the
background is fine, but the border between them causes problems. It can be
seen that if we perform even just Gaussian smoothing at the edges of the
object - the generalization significantly increases[9]. In this paper, we used
this insight, and the power of inpainting, to make this connection as natural
as possible in terms of convolution networks.

Here we list the stages we did to enhance the pasting process.

  * •

Train an inpainting model We train a model to learn how to paste objects on
background images in a natural way. We train an inpainting model that learns
the statistics of the scene and can fill designated holes naturally . We tune
our model in filling the blending gap as well as in filling irregular holes in
this environment.

  * •

Paste objects on background When pasting the object on the background image we
automatically create a blending gap mask that separates the object from the
background image, Figure 1 (b).

  * •

Fill the blending gap  We use the inpainting model trained in Stage 1 to fill
in the blending gap, Figures 1 (c,d).

##  4 Approach Details

We now present our approach in detail.

###  4.1 Train an inpainting model

To fill the blending gap correctly we need to use a model that can fill this
area so that it will be as natural as possible. this model needs to have the
capability to harmoniously fill many _irregular_ holes in the image. For that
we chose to train the DeepFill-V2 inpainting model[51], which is designed
especially to deal with filling _irregular_ structures. This model uses a
simple algorithm to automatically generate random free-form masks on-the-fly
during training. So that the model will succeed to fill the blending gap
correctly we train it on the background images. In addition to enhancing the
inpainting model’s capabilities specifically on the pasting work, we add to
the training set some fixed mask around real objects, with a weak segmenter,
such that filling the blending gap will be fine-tuned.

Category | Coca | Coffee | Honey | Hunt’s | Mahatama | Nature | Nature | Palmolive | Pop | Pringles | Red | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---
| Cola | Mate | Bunches | Sauce | Rice | V1 | V2 | Orange | Secret | BBQ | Bull |
No blending | 64.3 | 87.4 | 83.5 | 57.9 | 61.4 | 92.3 | 79.4 | 59.2 | 54.8 | 44.4 | 32.2 | 65.2
Gaussian Blurring | 65.4 | 86.4 | 80.4 | 69.3 | 64.3 | 90.3 | 83.4 | 59.3 | 59.4 | 66.4 | 42.5 | 69.8
Poission | 63.6 | 83.2 | 67.3 | 55.3 | 29.2 | 85.5 | 66.4 | 60.3 | 70.8 | 50.3 | 19.0 | 59.2
Inpaint(Ours) | 70.3 | 90.4 | 84.0 | 73.7 | 62.4 | 94.3 | 83.5 | 73.4 | 65.4 | 68.3 | 44.5 | 73.7
All Blend+Same Image(No Inpaint) | 77.3 | 91.0 | 79.3 | 69.3 | 67.2 | 92.9 | 81.2 | 65.2 | 77.1 | 71.8 | 40.2 | 73.9
All Blend+Same Image(With Inpaint) | 80.4 | 93.0 | 84.6 | 73.3 | 72.9 | 94.1 | 88.4 | 75.4 | 78.9 | 66.8 | 47.9 | 77.8
Table 1:  Evaluation results on the GMU Dataset [10] from models that were
trained with different pasting techniques. Figure 4:  Sample synthetically
generated images from the instance detection setting. Each column refers to a
different blending technique. We add the inpainting mask column to show which
pixels where filled in with the inpainting model.

###  4.2 Paste objects on background

To paste the object on the background image we need to have an image of the
object together with its corresponding mask. Our method is agnostic to how we
find the segmentation mask of the object we need to paste. If you do not have
the ground truth of the mask, you can use pre-trained foreground segmenters
such as[19] depending on the task required. Appropriate augmentations can now
be performed on the object such as rotation, scaling, illumination, etc. Then
you have to choose where to paste the object on the image. There are settings
where a random placement[9] is sufficient and there are settings where the
location is important for improving the performance of the model[20].

The blending gap around object mask, $\mathcal{M}$, is produced in the
instance segmentation case as $Dilate(\mathcal{M})\setminus
Erode(\mathcal{M})$ and in the foreground segmentation as
$Dilate(\mathcal{M})\setminus\mathcal{M}$ as the background of the object is
correct for the scene and we do not have to worry about an incorrect
background leaking into the object, Figure 2.

One of the advantages of our method is that even if the object segmentation is
not that accurate at the border of the object, where mistakes are prone to
happen - our method can handle it.

###  4.3 Fill the blending gap

Once we have the background image along with the objects pasted on it with the
blending gap along with the corresponding inpainting mask (Figure 1 b), they
can be inserted as input into the inpainting model that we trained in the
first step. The inference process is fast and is performed for all images in
the environment in the same way. The results may not be perfect in terms of
graphics, but they do exactly the job we are interested in - cause the
learning models to focus on appearance and not recognize that it is a pasted
object.

Unsupervised Foreground Segmentation methods Comparison
---
| Baseline | Night | Low Frame | Thermal
|  | Videos | Rate |
BSUV-Net 2.0 | 0.962 | 0.585 | 0.790 | 0.893
SemanticBGS | 0.960 | 0.501 | 0.788 | 0.821
IUTIS-5 | 0.956 | 0.529 | 0.774 | 0.830
Ours | 0.991 | 0.752 | 0.893 | 0.902
Table 2:  F-measure comparison of state of the art unsupervised methods from
CDnet dataset.

##  5 Experiments

Our method is general and can be applied in creating synthetic data for a wide
variety of tasks. This paper explored the capabilities of our method in two
main computer vision tasks: instance detection and foreground segmentation.

###  5.1 Training and Evaluation on Instance Detection

To test our method in instance detection task we used the Dwibedi et al. [9]
environment as in the cut, paste, and learn paper. We use a total of 33 object
instances from the BigBIRD Dataset [37] overlapping with the 11 instances from
GMU Kitchen Dataset [10]. For the foreground/background masks of the objects,
we use a pretrained foreground segmentation network [19]. We use backgrounds
from the UW Scenes Dataset [21]. We trained a inpainting model for this task
on 1000 images, 600 of them were from the various background images from the
UW Scenes Dataset, and another 400 randomly picked images from the BigBIRD
Dataset from which all the objects that we pasted are. All the images were
trained on inpainting irregular holes . We generated a synthetic dataset with
approximately 6000 images using all the augmentations they suggested e.g.
scale, rotation, etc. where position and the background were chosen randomly.
Each background appears roughly few times in the generated dataset with
different objects.

We created 4 different synthetic datasets that differ in terms of the pasting
process e.g. No blending, Gaussian blurring, Poisson editing, and Inpainting
(ours). In addition, we generated another dataset called All Blend + same
image e.g. synthesize the same scene with the same object placement, and only
vary the type of blending used to make the training algorithm further ignore
the effects of blending as they suggested. Samples of the generated images
compared to other blending methods can be seen in Figure 4.

We can see that the inpainting model fills the missing pixels with existing
shapes of the background and combines them with the texture of the object. The
border of the object is slightly blurred and the transition is smooth. This
causes the detection system not to focus on the border of the object but on
its appearance. In contrast, the transition between the background and the
objects in the no blending and gaussian blur methods are more sharp and thus
more noticeable. In addition, we can see that the Poisson blending method
often causes large parts of the object to be occluded in order to create a
good blending.

We use a Faster R-CNN model [33] based on VGG-16 [36] pre-trained weights on
the MSCOCO [25] detection task to train the detector on the synthetic dataset
we created. For evaluation, we use the GMU Kitchen Dataset [10] which contains
9 kitchen scenes with 6,728 images. We evaluate on the 11 objects present in
the dataset overlapping with the BigBIRD [37] objects. We report Average
Precision (AP) at IOU of 0.5 in all our experiments for the task of instance
localization. Table 1 shows the evaluation results. From the results, we can
see that our approach beats almost all compared blending methods by a big
margin. In addition, we also made an ablation study in the All Blend + same
image setting. The model that was trained with data that was generated with
our blending method performed better then without it.

###  5.2 Training and Evaluation on Foreground Segmentation

To test our method in foreground segmentation we used the setup in Kassel et
al. [20]. The idea is to use an unsupervised segmenter to extract the
foreground objects from the training frames and when pasted it will be pasted
in its original location. This is well suited for the static camera setting
where the objects are placed in the right location, being of the right size,
color and shape, and in the right lighting conditions.

For this purpose we use four scene categories from the Change Detection 2014
(CDnet) dataset [44] baseline, night videos, low frame-rate, and thermal. The
background images are pixel-wise medians of a sequence of 50 frames. The
frames used to extract objects were the same 200 used by FgSegNet V2[24]. For
the proposal on extracting the foreground objects we use the state of the art
unsupervised method on the CDnet dataset [44], BSUV-Net v2 [40]. For every
scene, we trained a unique inpainting model that was trained on 200 training
images in random masking holes and also a fixed set of blending gap filling
that we created with the BSUV-Net v2 results. For each scene, we generated 500
training frames. Samples from the generated frames can be seen in Figure 3.

We can see that inpainting models fill the missing pixels with the right
context. e.g. shadow/light of the car, lines on the road, etc. In general, the
object is inserted in a good and consistent way to blend with the background
(Figure 1e). To test our generated data we use the FgSegNet V2 by Lim et al.
[24], which is a state-of-the-art method in the CDnet 2014 Challenge[44],
SBI2015[27], and UCSD Background Subtraction [1]. We ignored blending gap
pixels during the learning process since we didn’t know if the inpainting
model will generate foreground or background pixels. Since this approach is
totally unsupervised we compared it to the state of the art unsupervised
methods. Results are shown in Table 2. We can see that the model that trained
on data that was generated with our method performs better than all the state
of the art methods by a big margin, improving the existing segmentation in the
unsupervised setting.

##  6 Summary

We presented a novel approach to paste objects on background images when
synthesizing annotated training images for tasks like detection and
segmentation. Inspired by the notion that local realism is sufficient for
training convolutions based models we focused our efforts on the connection
between the pasted objects and the background image - we called this
connection the blending gap. Our approach uses the inpainting to learn the
task environment and fill the blending gap missing pixels successfully and
improves the generalization by a big margin.

## References

  * [1] Ucsd anomaly detection dataset.  In //www.svcl.ucsd.edu/projects/anomaly/ dataset.htm., 2010.
  * [2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [3] Olivier Barnich and Marc Van Droogenbroeck.  Vibe: A universal background subtraction algorithm for video sequences.  IEEE Transactions on Image processing, 20(6):1709–1724, 2010.
  * [4] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In European conference on computer vision, pages 404–417. Springer, 2006.
  * [5] Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen.  Synthesizing training images for boosting human 3d pose estimation.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 479–488. IEEE, 2016.
  * [6] Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa.  The moped framework: Object recognition and pose estimation for manipulation.  The international journal of robotics research, 30(10):1284–1306, 2011.
  * [7] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang.  Dovenet: Deep image harmonization via domain verification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8394–8403, 2020.
  * [8] E. Rosten; T. Drummond.  Machine learning for high-speed corner detection.  In European Conference on Computer Vision, pages 430––443, 2006\.
  * [9] Debidatta Dwibedi, Ishan Misra, and Martial Hebert.  Cut, paste and learn: Surprisingly easy synthesis for instance detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1301–1310, 2017.
  * [10] Georgios Georgakis, Md Alimoor Reza, Arsalan Mousavian, Phi-Hung Le, and Jana Košecká.  Multiview rgb-d dataset for object instance detection.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 426–434. IEEE, 2016.
  * [11] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2918–2928, 2021.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [13] James Hays and Alexei A Efros.  Scene completion using millions of photographs.  ACM Transactions on Graphics (ToG), 26(3):4–es, 2007.
  * [14] Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit.  Gradient response maps for real-time detection of textureless objects.  IEEE transactions on pattern analysis and machine intelligence, 34(5):876–888, 2011.
  * [15] Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, and Kurt Konolige.  On pre-trained image features and synthetic images for deep learning.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [16] Stefan Hinterstoisser, Olivier Pauly, Hauke Heibel, Marek Martina, and Martin Bokeloh.  An annotation saved is an annotation earned: Using fully synthetic training for object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [17] Edward Hsiao, Alvaro Collet, and Martial Hebert.  Making specific features less discriminative to improve point-based 3d object recognition.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2653–2660. IEEE, 2010.
  * [18] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [19] Suyog Jain, Bo Xiong, and Kristen Grauman.  Pixel objectness.  arXiv preprint arXiv:1701.05349, 2017.
  * [20] Levi Kassel and Michael Werman.  Using a supervised method without supervision for foreground segmentation.  arXiv preprint arXiv:2011.07954, 2020.
  * [21] Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox.  A large-scale hierarchical multi-view rgb-d object dataset.  In 2011 IEEE international conference on robotics and automation, pages 1817–1824. IEEE, 2011.
  * [22] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural computation, 1(4):541–551, 1989.
  * [23] Yijun Li, Sifei Liu, Jimei Yang, and Ming-Hsuan Yang.  Generative face completion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3911–3919, 2017.
  * [24] Long Ang Lim and Hacer Yalim Keles.  Learning multi-scale features for foreground segmentation.  Pattern Analysis and Applications, 23(3):1369–1380, 2020.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [27] Lucia Maddalena and Alfredo Petrosino.  Towards benchmarking scene background initialization.  In International conference on image analysis and processing, pages 469–476. Springer, 2015.
  * [28] Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [29] Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko.  Learning deep object detectors from 3d models.  In Proceedings of the IEEE international conference on computer vision, pages 1278–1286, 2015.
  * [30] Patrick Pérez, Michel Gangnet, and Andrew Blake.  Poisson image editing.  In ACM SIGGRAPH 2003 Papers, pages 313–318. 2003.
  * [31] Param S Rajpura, Hristo Bojinov, and Ravi S Hegde.  Object detection using deep cnns trained on synthetic images.  arXiv preprint arXiv:1706.06782, 2017.
  * [32] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [34] Artem Rozantsev, Vincent Lepetit, and Pascal Fua.  On rendering synthetic images for training an object detector.  Computer Vision and Image Understanding, 137:24–37, 2015.
  * [35] Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani.  Summarizing visual data using bidirectional similarity.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [36] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [37] Arjun Singh, James Sha, Karthik S Narayan, Tudor Achim, and Pieter Abbeel.  Bigbird: A large-scale 3d database of object instances.  In 2014 IEEE international conference on robotics and automation (ICRA), pages 509–516. IEEE, 2014.
  * [38] Pierre-Luc St-Charles, Guillaume-Alexandre Bilodeau, and Robert Bergevin.  Subsense: A universal change detection method with local adaptive sensitivity.  IEEE Transactions on Image Processing, 24(1):359–373, 2014.
  * [39] Chris Stauffer and W Eric L Grimson.  Adaptive background mixture models for real-time tracking.  In Proceedings. 1999 IEEE computer society conference on computer vision and pattern recognition (Cat. No PR00149), volume 2, pages 246–252. IEEE, 1999.
  * [40] M Ozan Tezcan, Prakash Ishwar, and Janusz Konrad.  Bsuv-net 2.0: Spatio-temporal data augmentations for video-agnostic supervised background subtraction.  IEEE Access, 9:53849–53860, 2021.
  * [41] Shashank Tripathi, Siddhartha Chandra, Amit Agrawal, Ambrish Tyagi, James M Rehg, and Visesh Chari.  Learning to generate synthetic data via compositing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 461–470, 2019.
  * [42] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang.  Deep image harmonization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3789–3797, 2017.
  * [43] Marc Van Droogenbroeck and Olivier Paquot.  Background subtraction: Experiments and improvements for vibe.  In 2012 IEEE computer society conference on computer vision and pattern recognition workshops, pages 32–37. IEEE, 2012.
  * [44] Yi Wang, Pierre-Marc Jodoin, Fatih Porikli, Janusz Konrad, Yannick Benezeth, and Prakash Ishwar.  Cdnet 2014: An expanded change detection benchmark dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 387–394, 2014.
  * [45] Yi Wang, Zhiming Luo, and Pierre-Marc Jodoin.  Interactive deep learning method for segmenting moving objects.  Pattern Recognition Letters, 96:66–75, 2017.
  * [46] Joe Watson, Josie Hughes, and Fumiya Iida.  Real-world, real-time robotic grasping with convolutional neural networks.  In Annual Conference Towards Autonomous Robotic Systems, pages 617–626. Springer, 2017.
  * [47] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang.  Gp-gan: Towards realistic high-resolution image blending.  In Proceedings of the 27th ACM international conference on multimedia, pages 2487–2495, 2019.
  * [48] Yue Wu, Wael Abd-Almageed, and Prem Natarajan.  Busternet: Detecting copy-move image forgery with source/target localization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 168–184, 2018.
  * [49] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6721–6729, 2017.
  * [50] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  * [51] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4471–4480, 2019.
  * [52] Lingzhi Zhang, Tarmily Wen, and Jianbo Shi.  Deep image blending.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 231–240, 2020.


# Open Chemistry: RESTful Web APIs, JSON, NWChem and the Modern Web
Application

MDHMarcus D Hanwell  BdJWibe A de Jong  CJHChristopher J Harris  Kitware,
Inc., 28 Corporate Drive, 12065 Clifton Park, NY, USA  LBNL, One Cyclotron
Road, 94720 Berkeley, CA, USA

###### Abstract

An end-to-end platform for chemical science research has been developed that
integrates data from computational and experimental approaches through a
modern web-based interface. The platform offers a highly interactive
visualization and analytics environment that functions well on mobile, laptop
and desktop devices. It offers pragmatic solutions to ensure that large and
complex data sets are more accessible. Existing desktop
applications/frameworks were extended to integrate with high-performance
computing (HPC) resources, and offer command-line tools to automate
interaction—connecting distributed teams to this software platform on their
own terms. The platform was developed openly, and all source code hosted on
the GitHub platform with automated deployment possible using Ansible coupled
with standard Ubuntu-based machine images deployed to cloud machines.

The platform is designed to enable teams to reap the benefits of the connected
web—going beyond what conventional search and analytics platforms offer in
this area. It also has the goal of offering federated instances, that can be
customized to the sites/research performed. Data gets stored using JSON,
extending upon previous approaches using XML, building structures that support
computational chemistry calculations. These structures were developed to make
it easy to process data across different languages, and send data to a
JavaScript web client.

Chemistry,

Web,

Data,

Semantic,

NWChem,

JSON,

###### keywords:

Research

[id=n1]Equal contributor

## Introduction

The in-silico determination of chemical and materials properties is a vital
capability that drives innovation across many market sectors. Its importance
is reflected in the number of codes that can perform simulations over a broad
range of levels of theory and length scale [1, 2, 3, 4, 5], and the enormous
investments in experimental facilities that can also produce large data that
is often difficult or impossible to reproduce. However, it is all too common
for experimental and computational studies to take place independently, with
large scale studies often involving heroic efforts developing one-off software
projects dedicated to the specific resource, such as the Protein Data Bank [6]
(over 100,000 experimental structures), Materials Project [7] (over 33,000
simulated materials), and the Clean Energy Project [8] (over 2.3 million
calculated structures).

Driven by the U.S. Materials Genome Initiative the development of new and
novel materials has become a multidisciplinary research endeavor where complex
simulation and experimental data get integrated, and analytics such as machine
learning techniques are utilized to aid in scientific discovery. This same
multidisciplinary approach is becoming essential in chemical and biological
research and development, in the design of new chemicals, biomolecules and
drugs, or new energy efficient chemical production processes.

There is a strong need to develop a collaborative scientific research software
platform that enables researchers to define concepts and hypotheses, add them,
and analyze integrated sets of experimental and computational data to offer
effective knowledge discovery more universally. This goes well beyond a web
portal to create a highly interactive platform integrating simulation,
experimental data, and analytics, while leveraging semantic web technologies
to support the federated storage of data across geographically dispersed
sites.

The web has evolved significantly in the last two decades, and should be
explored to assess what could be achieved in a platform that sought to use the
latest open source tools, technologies, standards, and approaches to deliver
an end-to-end platform for chemical/materials research. The basic approach
employed was to develop a server component written in the Python language that
exposes RESTful endpoints to interact with the data on the server. The Python
code can use a number of existing core functionality, Python modules, and
wrapped libraries developed in other programming languages.

Ideally no HTML, images, etc would be generated on the server, the server acts
as a data server primarily through RESTful endpoints that accept/return data,
along with user authentication (required by some endpoints). It can trigger
calculations, perform analyses, and batch jobs in order to make the data
discoverable. The RESTful server can then be consumed by a rich HTML5 web
client, more traditional desktop clients, and from command-line clients or
other servers to perform automated workflows.

In this work a rich HTML5 web interface was developed that made use of the
server’s RESTful API (application programming interface) briefly discussed
above. It consumes data from the server, uploading/editing data, and
maintaining local state in a one-page application reusing a popular open
source HTML5 framework. The application integrated other open source
frameworks for client side charting, and 3D rendering/visualization of
molecular structure.

The web application was developed using open source tools, a number of
frameworks, and was “built” as a static bundle of HTML5 web assets that are
downloaded by the web client. It dynamically constructs the page in response
to user interaction, server data, and other client events to provide a rich,
interactive experience. This also meant that many interactions take place
entirely on the client, requiring no interaction with or access to the server.
This offers highly interactive data visualization and analysis, even on
relatively low bandwidth links, once initial data about a molecule has been
downloaded.

## Methods

An open source prototype web platform was developed to demonstrate key
capabilities in addressing the needs outlined in the introduction, and
summarized in Figure 1. The application has a number of components developed
in several languages following modern development methodologies. It was
intentionally developed using some of the latest technology innovations, which
means that it requires a modern web browser that supports WebGL in order to
render 3D geometry, and it makes extensive use of HTML5. It is clear that not
all devices/web browsers have full support for these technologies at this
time, but that this support is already substantial and will grow in the coming
years.

Figure 1: Architecture of the chemical data platform. Overview of the high-
level architecture used for the chemical data platform.

The server-side components were written in Python 3, with wrapped C++ code
providing access to a number of existing chemistry/materials libraries such as
Avogadro [9] and Open Babel [10]. The basis for the server-side project was
the Girder project [11], which itself reuses a number of Python modules such
as CherryPy that provides an extensible RESTful web server, Swagger to
document the RESTful API, MongoDB to store data/metadata, and Virtuoso to
store triples. The chemistry specific functionality was developed as
additional API using the Girder project’s plugin mechanism to add additional
RESTful endpoints.

The client-side components were developed in modern HTML5, using open source
web frameworks/technologies such as AngularJS 1.6 and Materials Design to
provide a single page web application. 3DMol.js [12] was used to render
molecular geometry in 3D, D3 [13] to render charts, and responsive design
elements to accommodate devices of various sizes/aspect rations. The
capabilities developed have been demonstrated on desktop browsers, mobile
phones, and tablets on the major operating systems. This includes Windows,
macOS, Linux, iOS, and Android operating systems using browsers including
Chrome, Firefox and Safari.

In addition to the significant developments made in the web platform, the
reuse of the Avogadro 2 libraries enabled the rapid ingestion of chemical
data. Extensions to the Avogadro 2 libraries, and several other components,
were necessary. These changes have been merged into the main development
branch, and were made available in the 1.90 release of the software.
Significant capabilities added during the course of this project include
visualization/animation of vibrational data, and additional file formats
supporting the NWChem package. The JSON readers/writers built upon the JsonCpp
library [14], and the capabilities were exposed to the Python-based server
through Boost.Python wrapped calls to the C++ API.

The computational chemistry code used as a generator for the calculation
results is the open source NWChem software suite [1]. The JSON-Fortran library
[15] was integrated into the NWChem source to enable the coded to write out a
new JSON file in addition to the standard output or log file. APIs and
interfaces between the Fortran-90 routines of the JSON-Fortran library and the
Fortran-77 NWChem source were written to facilitate the transfer of the
computational chemistry data into the JSON format. The full JSON enabled
NWChem source code is available on Github [16]. To enable the end-user to
convert existing log files to the JSON format a Python 3 library was created.
The library and examples are available in a separate Github repository [17].

The collaborative nature of the development at Kitware and LBNL (Lawrence
Berkeley National Laboratory) has created the opportunity to rapidly prototype
new data structures, and look at the full end-to-end workflow from data
generation, through to ingestion, analysis, and visualization. Python scripts
were also developed to upload and add data files from the command-line,
enabling ingestion of both existing data sets, and new ones as they are
generated.

## Results

The software components were developed to serve the needs of data-centric
chemistry research using open source approaches that embrace the use of open
APIs, open data formats, and open components. The approach made extensive use
of client-side rendering/interaction wherever practical, and focused on a
server-side component that served data from RESTful endpoints using the web-
native JSON format where possible. The development spanned a number of
programming languages (Fortran, C++, Python, and JavaScript) in order to offer
structured data that can be stored, queried, edited, and visualized.

### MongoChemServer: Server Side Platform

The server code was developed using the Python 3 language as a Girder plugin.
The Girder framework is an open source project led by Kitware, and released
under the Apache 2.0 license. It has three main components:

  * •

Data organization and dissemination

  * •

User management and authentication

  * •

Authorization management

It is developed as an extensible data management platform, which itself reuses
a number of open source projects including CherryPy—“a pythonic, object-
oriented web framework”, providing a solid foundation for the Girder platform.
The code in the mongochemserver repository [18] extends the functionality
provided in a plugin that is loaded by the Girder process when it starts up.
The plugin adds RESTful API, and reuses Girder core functionality and its core
plugins for more generic features such as authentication, Gravatars, file
upload/download, access control, etc.

The Girder platform provides deep integration with MongoDB, using that to
store user credentials, access permissions, metadata, and other elements
exposed via its plugin system. Among the most useful abstractions provided in
the context of this project are the authentication, access permission, and
file storage systems. Almost all of these concepts must be exposed on both the
server and in the web client code to be used effectively.

The existing OAuth2 plugin was used, and coupled with Google’s OAuth2
implementation to offer single-sign on. This can be replaced with other
authentication schemes, or augmented with multiple options. For simplicity
this was the only authentication scheme exposed in the prototype described,
coupled with the use of encrypted SSL connections to provide secure
authenticated access. This choice enabled the deployment of a demonstration to
multiple locations, but was not always the most appropriate and will be
augmented in future development to include integration with site-wide systems
where appropriate.

The access permissions can be applied at several levels in the Girder code. A
RESTful API must be exposed as a resource which resolves various paths, which
refer to namespaces within the API prefix and are documented using a system
called “Swagger” [19]. This enables developers to document API as it is
written, provides an HTML5 web client that exposes this documentation, and
offers the ability to test API live on the web, shown in Figure 2. The API
exposed uses decorators to express whether a given piece of API is public, or
can only be accessed by authenticated users. API that requires authentication
can apply further restrictions based on user privileges, and provide filtered
results containing only data the authenticated user has the access privileges
for.

Figure 2: Swagger documentation for RESTful API. An example of Swagger being
used to test some of the RESTful API in the ‘molecules’ resource..

File upload/download sounds quite simple on the surface, but it involves a
number of distinct components in order to scale and integrate well in
different environments. The Girder environment uses asset stores to abstract
the storage backend, and the backend can then be mapped to file systems, S3
storage (as provided by Amazon EC2), and others. Large files must also be
uploaded/downloaded in “chunks”, something offered as part of the Girder file
API and exposed in the client application. File system storage proved
sufficient for the prototype, but future deployments would benefit from using
large file stores, with extension to archive servers at supercomputing
centers.

### MongoChemClient: Rich HTML5 Web Client

The Girder project has its own web interface, but this was not used—a custom
user interface was developed in the mongochemclient repository [20]. The web
interface developed in the mongochemclient repository is a modern HTML5 web
interface. This means that all HTML5 assets can be served as static files, and
the page is built up dynamically on the client-side using the RESTful API to
authenticate (if necessary), retrieve data, upload new date, and visualize
data. A number of technologies and projects were leveraged in order to create
a compelling, modern interface in a relatively short space of time.

Figure 3: HTML5 web client with molecular structure and vibrational modes. The
web client displaying a molecular structure in 3DMol.js, and selection of
vibrational modes in a D3 chart.

The main framework used to manage interaction, react to events, and coordinate
the single dynamic page approach was AngularJS. This framework is divided into
a number of modules that provide various services/extensions, such as easy
access to RESTful APIs, animations, routing (where the URL is updated to
reflect the current ‘location’ (state) despite being in a single-page web
application), and overall look and feel (Materials design in this instance).
AngularJS was chosen for the rich feature set, maturity, and encapsulation of
components along with its powerful web page layout framework. An example of
the single page application in action is shown in Figure 3, where a molecular
structure can be seen beside a plot of vibrational modes that can be animated
in the 3DMol.js based geometry viewer.

Instead of writing HTML code directly the Jade language was chosen for its
expressiveness, and conciseness. The translation to HTML is rather simple, and
it offers a number of conveniences over directly using HTML. The same
reasoning led the team to choose Stylus for CSS code, leading to much more
concise code in a more expressive form. The Jade and Stylus languages are both
well supported within webpack, which was used as both a development tool when
working on new code, and as a static asset compiler when deploying code to
production machines. Beyond these capabilities, Node.js also has a robust
package management system, offering seamless installation/deployment of
AngularJS, and even git repositories that contain other web-based components.

The static HTML5 content that serves as the frontend, and the dynamic RESTful
APIs offering access to the data must be presented to the web client. This is
where the NGINX web server [21] came in, offering an SSL-enabled endpoint for
encryption, serving the static content in the web root, and proxying requests
to the /api/v1 prefix to the Python-based backend. The Python-based backend
also had access to the MongoDB server where all metadata, access controls, and
links to files were stored. There is also an asset store, where a simple on-
disk asset store was used.

This describes the very basic layout of the prototype platform, and some of
the details of how this was extended will be detailed in the following text.
It is clear that even the early prototype involves a number of
build/deployment systems, and that the deployment repository was necessary to
coordinate the task of deploying everything to the right location, with
compatible software versions, and ensuring services are brought up/down in the
correct order.

### JSON Data Formats

A JSON format for chemical data from computational chemistry simulations has
been in development for a number of years, starting life within the Avogadro 2
project as a simple means of data storage and exchange. The JSON format was
developed based upon the requirements of storing data in BSON using MongoDB,
communicating chemical data over JSON-RPC 2.0 between desktop applications,
and later using web services. It was also motivated by the need for a format
to support an application being developed to edit and communicate chemical
structures.

The format was tested with small molecules through to molecules with millions
of atoms using a philosophy of being machine readable, efficient, and avoiding
repetition. It was written to represent a simple mapping of the in memory data
structures. It is based on many of the concepts developed in the CML format
[22] as part several ongoing collaborations under the broad umbrella of the
Blue Obelisk [23, 24]. We chose JSON because it is less cumbersome than XML,
with simpler parsers, and is a native format of the web.

This JSON format exposed the internal data model used in Avogadro 2, and was
used as the transport layer to the 3DMol.js client-side rendering. It was
extended to expose vibrational modes, and a primitive container for volumetric
data already present in the 3DMol.js project. The current state of the
Chemical JSON format is documented in a Github repository [25] and will
continue to evolve as it is extended to serve more use cases.

A second, more comprehensive ExtendedChem JSON format was developed with the
goal to encapsulate the data from computational chemistry software, such as
the open-source NWChem. The goals of the two formats were somewhat orthogonal.
While ExtendedChem JSON expresses the properties of a single molecule in
machine readable arrays with all data that might be visualized/displayed, the
Chemistry format is focused on storing all the essential input and output data
from the computational chemistry software simulation (here for NWChem) for the
(possibly multiple) jobs that were run as the calculation proceeded. This is
somewhat akin to the typical molecular file formats (XYZ, CML, SDF, etc)
versus a more structured quantum mechanical log file—a snapshot of definite
structure versus data discovery and recording.

This new format focuses on satisfying the need to replace a log file with
structured output, offering output of multiple calculations as a computational
chemistry code like NWChem is executed. The JSON file generated by the NWChem
application builds upon previous work done integrating CML into NWChem [26]
(an XML-based format), and is designed around the notion of using objects as
the primary representation. As a starting point the CML naming and conventions
were adopted and extended with new concepts. The format was developed and
designed to be portable to multiple computational chemistry codes but in this
work firmly targeted the NWChem code.

The relevant section representing the geometry/structure of JSON files
generated in the two formats, in this case for a water molecule, are compared
to clearly show the difference in design philosophies, i.e. arrays vs objects.
The listing below shows a molecule block within the Chemical JSON format:

⬇

{

"chemical json": 0,

"atoms": {

"coords": {

"3d": [ 0.00, 0.00, 0.14,

-0.76, 0.00,-0.46,

0.76, 0.00,-0.46 ]

},

"elements": {

"number": [8, 1, 1]

}

},

"bonds": {

"connections": {

"index": [0, 1, 0, 2]

},

"order": [1, 1]

}

}

The format focuses on using multiple arrays to store different aspects of
atoms, bonds, etc—such as the atomic number, bond order, etc. The 3D
coordinates are in a single array, and each 3D vector is offset by 3N where N
is the index of the atom. This offers simplicity, compactness, and a simple
representation with efficient storage of atomic data. It does not provide a
layout that is focused on human readability, and it is not intended to be used
as a directly editable set of objects in Python/JavaScript without some code
in front of it to aid in manipulation/keeping the representation consistent.

The listing below shows the same molecule block within the ExtendedChem JSON
format:

⬇

{

"molecule": {

"id": "Molecule.1",

"atoms": [

{

"id": "Atom.1.Mol.1",

"elementLabel": "o",

"elementSymbol": "O",

"elementNumber": 8,

"elementName": "Oxygen",

"cartesianCoordinates": {

"value": [

0.0E+0,

0.0E+0,

0.2106360400000002E+0

],

"units": "bohr"

}

},

{

"id": "Atom.2.Mol.1",

"elementLabel": "h",

"elementSymbol": "H",

"elementNumber": 1,

"elementName": "Hydrogen",

"cartesianCoordinates": {

"value": [

-0.1841188380000002E+1,

0.0E+0,

-0.8425441600000008E+0

],

"units": "bohr"

}

},

{

"id": "Atom.3.Mol.1",

"elementLabel": "h",

"elementSymbol": "H",

"elementNumber": 1,

"elementName": "Hydrogen",

"cartesianCoordinates": {

"value": [

0.1841188380000002E+1,

0.0E+0,

-0.8425441600000008E+0

],

"units": "bohr"

}

}

]

}

}

In contrast to the Chemical JSON listing, the information of each atom in
ExtendedChem JSON is collected in a single object. A key aspect of this format
is the explicit definition of units, a key aspect brought in for the CML
specification. It is relatively simple to map between the two formats, and
this is now possible in the Avogadro 2 libraries which feature readers for
both formats.

In developing the ExtendedChem JSON format, some additional referencing
features were introduced that are not natural to JSON. The listing below shows
an excerpt of a JSON data file containing the calculation setup and some of
the properties that can be calculated with quantum chemistry software:

⬇

{

"calculationType": "molecularProperties",

"molecularFormula": "H2O",

"id": "calculation.4",

"calculationSetup": {

"outputVectors": "./prop_h2o_run.movecs",

"molecularSpinMultiplicity": 1,

"molecule": "Molecule.2",

"charge": 0,

"waveFunctionType": "RHF",

"numberOfElectrons": 10,

"basisSet": "BasisSet.1",

"waveFunctionTheory": "Hartree-Fock",

"inputVectors": "./prop_h2o_run.movecs",

"id": "calculationSetup.4"

},

"calculationResults": {

"molecularProperties": [

{

"Molecule": "Molecule.2",

"dipoleMoment": {

"totalMoment": {

"units": "atomic units",

"value": 0.8052087008

},

},

"quadrupoleMoment": {

"diamagneticSusceptibility": {

"units": "atomic units",

"value": 18.596483

},

"momentXZ": {

"units": "atomic units",

"value": 0.0

}

}

},

{

"atom": "Atom.1.Mol.2",

"diamagneticShielding": {

"units": "atomic units",

"value": 23.457292

}

}

]

}

}

Input structures in computational chemistry codes are set up to allow a
sequence of calculations to be executed in one single run. To store this
sequence, the ExtendedChem JSON format has adopted the CML approach of storing
each calculation in a “calculations” array. The listing above shows an excerpt
of the fourth calculation in this series. Often, data from previous
calculation steps in the sequence is reused in the quantum chemistry code. To
avoid duplicating information, “id” tags were used to provide a pointer to the
first mention of the JSON object. An example of this in the listing above are
the key “molecule” to “Molecule.2”, which is defined as the id-tag in the
molecule block in the previous listing. Another example is the “basisSet”
keyword in the “calculationSetup” block. Even the “calculationSetup” could be
the same for multiple calculations, and could be pointed to in this fashion.
The same approach is used to identify the atom within the molecule to which
the calculated atomic properties belong. Here the “atom” keyword points to
atom “Atom.1.Mol.2” in the earlier ExtendedChem JSON listing for the molecular
geometry. Essentially, this referencing approach mimics the ability to link
different sets of data. The referencing feature is missing in the current JSON
specification but native to XML. The JSON pointer has been proposed in RFC
6091 [27] and a draft RFC is being developed for the JSON reference [28].
While JSON reference would provide the necessary flexibility, either could
readily be adopted in the format in the near future.

Examples of complete ExtendedChem JSON files generated by the NWChem quantum
chemistry code can be found on the Github repository [17].

The JSON file can be submitted to the web platform developed, and relevant
metadata will be extracted. The format is capable of encapsulating jobs that
have multiple steps, and it has been demonstrated with vibrational and
electronic structure data. The format breaks most elements of the output into
distinct objects, and it has been designed to be more semantically expressive
than the Chemical JSON format developed as part of the Avogadro 2 project.

The two formats were used as part of this project, with several extensions
made to the Chemical JSON format in order to support data needed by the
project. They are both supported by the Avogadro 2 libraries, and exposed in
the Python-wrapped API used on the server side. The Chemical JSON format is
designed in a pragmatic fashion, making extensive use of arrays, along with
key-value pairs for simple properties. The format closely matches in-memory
structures, it is optimized for storage in BSON (the binary form of JSON used
internally by MongoDB), is easy to visualize and store as a single document in
MongoDB. The new ExtendedChem JSON format developed tends to represent
everything as an object, can store multiple configurations/states of a
molecule, and uses links to reuse objects that have not changed. This makes it
more complex to parse, but more expressive, and it more closely reflects what
is currently stored in log files.

This collaborative development has highlighted extensions needed in the
internal models used in the Avogadro 2 projects, the 3Dmol.js project, and in
the model employed in the web framework. Ultimately, any given visualization
can only show a single state, and the web and desktop frameworks both
generally assume a file only has a single state with minimal links back to the
input file. This concept will need to be extended in order to visualize and
analyze data from more complex multi-step files, and some of this development
was started in the work described.

In the future we would like to extend this approach to use and draw upon new
developments in JSON-LD [29] that aims to bring linked data and meaning to
JSON using the same approaches used in semantic data structures.

## Discussion

Research in chemistry is becoming more data intensive, and as a result it is
vital that we act to create platforms that can be easily deployed, used, and
shared with a focus on data. This requires standard formats that support a
multitude of facets of chemical data that leverage industry standard
technologies such as XML, JSON, and the semantic web.

As we move closer to a landscape where the web is an essential part of any
workflow we must embrace web-native formats, generally using JSON and similar
containers, to enable the free exchange, analysis and visualization of data.
This can be further augmented through the use of JSON-LD, and is something the
authors would like to do in order to offer semantic meaning and enable the
simple transformation of JSON-LD documents to triples.

Once these components are available the open source code referenced here, and
available as a demonstration capability, show what can be built around it. The
RESTful APIs, databases, and HTML5 frontends can be readily coupled with
command line and powerful desktop applications to offer a compelling ecosystem
of data-centric applications that serve the chemical research enterprise.

The definition of suitable schema, codification of that into JSON-LD, and
community agreement still represent significant barriers to wider adoption.
The development of permissively licensed open source code, which can be
readily inspected, reused, and modified is one approach to improving the
situation. Coupled with the use of community platforms such as GitHub for
collaborative development, and workshops gathering interested experts, these
approaches can be iterated upon and standardized.

## Conclusions

A “full-stack” open source web application was developed, making heavy use of
Python and open source community tools on the backend, and HTML5/JavaScript on
the frontend. Years of experience in using, converting, and developing data
formats was drawn upon to create some new formats for the output of structured
data from the NWChem code, ingestion into databases, and the subsequent
visualization/analysis of data. This was demonstrated in a new web client, and
added to the Avogadro 2 desktop application. The primary data types were 3D
chemical structure, electronic structure, and vibrational modes on the web and
desktop.

If the field is to progress it is important to move away from disparate
formats developed using non-standard line printer style formats. The ingestion
and normalization of data still represents a serious challenge to reaping the
benefits powerful data-centric platforms promise. Offering simple, well
documented formats, and a powerful, open, reference platform is one path to
show the benefits, and begin working with the wider community to share data
more widely while encouraging greater reuse.

## Declarations

### Ethics approval and consent to participate

Not applicable.

### Consent for publication

Not applicable.

#### List of abbreviations

3D—Three dimensional
API—Application programming interface
CSS—Cascading style sheets
CML—Chemical markup language
HTML—Hypertext markup language
HPC—high-performance computing
JSON—JavaScript object notation
LBNL—Lawrence Berkeley National Laboratory
REST—Representational state transfer
SSL—Secure sockets layer
XML—Extensible markup language

#### Availability of data and materials

All code and data discussed in this manuscript is available under open
licenses on GitHub. The following repositories contain all the data and
materials:

  * •

Chemical data RESTful server [11, 18]

  * •

Web frontend single-page application [20]

  * •

Chemical JSON specification and examples [25]

  * •

NWChem with JSON output [16]

  * •

Conversion tool from NWChem log files to JSON [17]

### Competing interests

The authors declare that they have no competing interests.

### Funding

The authors would like to acknowledge funding from the Department of Energy’s
Office of Science, under contract DE-SC0013250.

### Author’s contributions

MDH and CJH developed the client and server code, new file format support in
Avogadro 2, and the Chemical JSON file format. WDJ developed the CompChem JSON
data format, the JSON writer in the NWChem application and the Python NWChem
output converter, and generated the simulation data used in testing. MDH and
WDJ wrote the manuscript. All authors read and approved the final manuscript.

### Acknowledgements

Not applicable.

## References

  * [1] Valiev, M., Bylaska, E., Govind, N., Kowalski, K., Straatsma, T., van Dam, H., Wang, D., Nieplocha, J., Apra, E., Windus, T., de Jong, W.: NWChem: a comprehensive and scalable open-source solution for large scale molecular simulations. Comput. Phys. Commun. 181, 1477 (2010)
  * [2] Kong, J., White, C.A., Krylov, A.I., Sherrill, D., Adamson, R.D., Furlani, T.R., Lee, M.S., Lee, A.M., Gwaltney, S.R., Adams, T.R., Ochsenfeld, C., Gilbert, A.T.B., Kedziora, G.S., Rassolov, V.A., Maurice, D.R., Nair, N., Shao, Y., Besley, N.A., Maslen, P.E., Dombroski, J.P., Daschel, H., Zhang, W., Korambath, P.P., Baker, J., Byrd, E.F.C., Van Voorhis, T., Oumi, M., Hirata, S., Hsu, C.-P., Ishikawa, N., Florian, J., Warshel, A., Johnson, B.G., Gill, P.M.W., Head–Gordon, M., Pople, J.A.: Q-chem 2.0: a high-performance ab initio electronic structure program package. Journal of Computational Chemistry 21(16), 1532–1548 (2000)
  * [3] Hutter, J., Iannuzzi, M., Schiffmann, F., VandeVondele, J.: cp2k: atomistic simulations of condensed matter systems. Wiley Interdisciplinary Reviews: Computational Molecular Science 4(1), 15–25 (2014)
  * [4] Gonze, X., Amadon, B., Anglade, P.-M., Beuken, J.-M., Bottin, F., Boulanger, P., Bruneval, F., Caliste, D., Caracas, R., Côt;é, M., Deutsch, T., Genovese, L., Ghosez, P., Giantomassi, M., Goedecker, S., Hamann, D.R., Hermet, P., Jollet, F., Jomard, G., Leroux, S., Mancini, M., Mazevet, S., Oliveira, M.J.T., Onida, G., Pouillon, Y., Rangel, T., Rignanese, G.-M., Sangalli, D., Shaltaf, R., Torrent, M., Verstraete, M.J., Zerah, G., Zwanziger, J.W.: Abinit: First-principles approach to material and nanosystem properties. Computer Physics Communications 180(12), 2582–2615 (2009). 40 {YEARS} {OF} CPC: A celebratory issue focused on quality software for high performance, grid and novel computing architectures
  * [5] List of Quantum Chemistry and Solid-state Physics Software. https://en.wikipedia.org/wiki/List_of_quantum_chemistry_and_solid-state_physics_software
  * [6] Berman, H.M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T.N., Weissig, H., Shindyalov, I.N., Bourne, P.E.: The protein data bank. Nucleic Acids Research 28, 235–242 (2000)
  * [7] Jain, A., Ong, S.P., Hautier, G., Chen, W., Richards, W.D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., Persson, K.a.: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials 1(1), 011002 (2013)
  * [8] Hachmann, J., Olivares-Amaya, R., Atahan-Evrenk, S., Amador-Bedolla, C., Sánchez-Carrera, R.S., Gold-Parker, A., Vogt, L., Brockway, A.M., Aspuru-Guzik, A.: The Harvard Clean Energy Project: large-scale computational screening and design of organic photovoltaics on the World Community Grid. The Journal of Physical Chemistry Letters 2(17), 2241–2251 (2011)
  * [9] Hanwell, M.D., Curtis, D.E., Lonie, D.C., Vandermeersch, T., Zurek, E., Hutchison, G.R.: Avogadro: an advanced semantic chemical editor, visualization, and analysis platform. J. Chem. Inf. 4(17) (2012)
  * [10] OBoyle, N., Banck, M., James, C., Morley, C., Vandermeersch, T., Hutchison, G.: Open Babel: An open chemical toolbox. J. Chem. Inf. 3, 1–14 (2011). 10.1186/1758-2946-3-33
  * [11] Girder. https://github.com/girder/girder
  * [12] Rego, N., Koes, D.: 3dmol.js: molecular visualization with webgl. Bioinformatics 31(8), 1322 (2015)
  * [13] Bostock, M., Ogievetsky, V., Heer, J.: D3: Data-driven documents. IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis) (2011)
  * [14] JsonCpp. https://github.com/open-source-parsers/jsoncpp
  * [15] JSON-Fortran Version 2.0.0. https://github.com/jacobwilliams/json-fortran/wiki
  * [16] NWChem-JSON Writer Github Repository. https://github.com/wadejong/NWChem-Json
  * [17] Python JSON-NWChem Github Repository. https://github.com/wadejong/NWChemOutputToJson
  * [18] mongochemserver. https://github.com/openchemistry/mongochemserver
  * [19] Swagger. http://swagger.io
  * [20] mongochemclient. https://github.com/openchemistry/mongochemclient
  * [21] NGINX. https://www.nginx.com/
  * [22] Murray-Rust, P., Rzepa, H.S.: Markup languages- how to structure chemistry related documents. Chemistry Intl. 4(24), 9–13 (2002)
  * [23] Guha, R., Howard, M.T., Hutchison, G.R., Murray-Rust, P., Rzepa, H., Steinbeck, C., Wegner, J., Willighagen, E.L.: The blue obelisk—interoperability in chemical informatics. Journal of chemical information and modeling 46(3), 991 (2006)
  * [24] O’Boyle, N.M., Guha, R., Willighagen, E.L., Adams, S.E., Alvarsson, J., Bradley, J.-C., Filippov, I.V., Hanson, R.M., Hanwell, M.D., Hutchison, G.R., et al.: Open Data, Open Source and Open Standards in chemistry: The Blue Obelisk five years on. J. Chem. Inf. 3(37) (2011)
  * [25] Chemical JSON Github Repository. https://github.com/OpenChemistry/chemicaljson/blob/master/chemicaljson.md
  * [26] de Jong, W.A., Walker, A.M., Hanwell, M.D.: From data to analysis: linking NWChem and Avogadro with the syntax and semantics of Chemical Markup Language. J. Chem. Inf. 5(25) (2013)
  * [27] RFC 6091 for JSON Pointer. https://tools.ietf.org/html/rfc6901
  * [28] Draft RFC for JSON Reference. https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03
  * [29] JSON-LD. http://json-ld.org/