# MMChat: Multi-Modal Chat Dataset on Social Media

###### Abstract

Incorporating multi-modal contexts in conversation is important for developing
more engaging dialogue systems. In this work, we explore this direction by
introducing MMChat: a large-scale Chinese multi-modal dialogue corpus (32.4M
raw dialogues and 120.84K filtered dialogues). Unlike previous corpora that
are crowd-sourced or collected from fictitious movies, MMChat contains image-
grounded dialogues collected from real conversations on social media, in which
the _sparsity_ issue is observed. Specifically, image-initiated dialogues in
common communications may deviate to some non-image-grounded topics as the
conversation proceeds. To better investigate this issue, we manually annotate
100K dialogues from MMChat and further filter the corpus accordingly, which
yields MMChat-hf. We develop a benchmark model to address the sparsity issue
in dialogue generation tasks by adapting the attention routing mechanism on
image features. Experiments demonstrate the usefulness of incorporating image
features and the effectiveness of handling the sparsity of image features.

Keywords: Dialogue Systems, Multi-Modality, Social Media

languageresourceLanguage Resources

MMChat: Multi-Modal Chat Dataset on Social Media

Yinhe Zheng$\spadesuit$††thanks:  Equal Contribution, Guanyi
Chen$\clubsuit$11footnotemark: 1, Xin Liu$\heartsuit$, Jian
Sun$\spadesuit$††thanks:  Corresponding Author
---
$\spadesuit$ Alibaba Group, $\clubsuit$ Utrecht University,
$\heartsuit$Samsung Research China - Beijing (SRC-B)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The ability to converse like a human is one of the desiderata for building
open-domain dialogue systems [Adiwardana et al. (2020]. Current attempts to
build human-like open-domain dialogue systems generally follow two angles: 1)
Enriching the dialogue system with textual or structural contexts such as
knowledge [Madotto et al. (2018] or personalities [Zhang et al. (2018a, Zheng
et al. (2019]; 2) Enabling the dialogue systems to perceive multi-modality
contexts beyond text such as vision, voice, or even gesture [Shuster et al.
(2020b, Shuster et al. (2020c, Liao et al. (2018, Ju et al. (2019]. Systems
built following the second angle are also known as Multi-Modal Dialogue
Systems (MMDSs).

To facilitate the development of data-driven MMDSs, a few dialogue datasets
containing visual information have been constructed [Mostafazadeh et al.
(2017, Mogadala et al. (2019, AlAmri et al. (2019, Kottur et al. (2019,
Pasunuru and Bansal (2018, Shuster et al. (2020a, Meng et al. (2020]. For
instance, ?) introduced a crowd-sourced image grounded dialogue corpus Image-
Chat, in which annotators are employed to chat in accordance with given
images. ?) proposed OpenViDial by directly extracting dialogues and their
visual contexts from movies and TV series. There are also works on visual
question answering [Das et al. (2017] that focus on the question answering
tasks involving image inputs.

Figure 1: Example dialogues from MMChat (translated from Chinese).

A significant drawback of existing datasets is that they postulated every
utterance in a dialogue to be grounded on the given image. Nevertheless, this
is not always true in our daily communications. Concretely, the topic
triggered by the image may drift in the conversation flow so that not every
utterance in a dialogue session is image-grounded. Taking the right dialogue
in Figure 1 as an example, it is initialized by objects shown in the
associated image (i.e., “ _paintbox_ ” and “ _dried paint_ ”) but the focus of
the following dialogue move to the speaker’s own experience as a painter,
which is not image-related anymore. A similar pattern is also observed in the
left example of Figure 1. We refer to this phenomenon as the issue of
_sparsity_ and dialogues that exhibit this phenomenon as the _sparse image-
grounded dialogues_.

To tackle the above issue, we introduce MMChat: a large-scale dataset that
contains sparse image-grounded dialogues in Chinese. We first collected 32.4M
sessions of raw dialogues and 8.41M associated images from social media. Based
on these raw dialogues, we design an elaborate data filtering process and
construct MMChat, which contains 120.84K sessions of filtered high-quality
dialogues and 204.32K images. Dialogues that are incoherent or involve
offensive content are filtered. Two example dialogue sessions are shown in
Figure 1. Unlike previous multi-modal dialogue datasets that only provide a
single raw image per dialogue session, each filtered dialogue session in
MMChat corresponds to one or multiple images. The semantic information of each
image is further revealed in our dataset using a pre-trained image caption
model. Specifically, a set of detected object labels and a generated
descriptive caption are released for each image in MMChat.

To further improve the quality of dialogues in MMChat, we sample 100K dialogue
sessions from MMChat, and manually check the quality of images and whether
these dialogues are strongly correlated with the associated images. This
yields a “human filtered” dataset MMChat-hf that contains 19.90K dialogue
sessions and 52.24K images.

Building on both MMChat and MMChat-hf, We provide a strong benchmark model to
tackle the image-sparsity issue in open-domain dialogue generation tasks based
on the attention routing mechanism [Zheng et al. (2020]. Evaluation results on
both datasets suggest that incorporating visual contexts contributes
positively to dialogue modeling, and the approach used in our benchmark model
helps alleviate the sparsity issue.

Besides enlightening advanced models for realistic multi-modal conversations,
MMChat is also built to help understand how Chinese multi-modal communications
are conducted from the aspect of social science [Jovanovic and Van Leeuwen
(2018]. The vast amount of dialogues and images in MMChat can also benefit the
study of multi-modal pretraining models.

In what follows, we summarize our main contributions:

  * •

We construct a large multi-modal dialogue dataset MMChat, addressing the issue
of “sparsity”. A dedicated automatic filtering process is proposed to clean
the dataset.

  * •

We offer a human filtered dataset MMChat-hf based on 100K dialogue sessions
sampled from MMChat.

  * •

We build benchmark models on MMChat. The results indicate that incorporating
visual contexts contribute positively to dialogue modeling, and our benchmark
model can better tackle the sparsity issue.

Our dataset and code are available in https://github.com/silverriver/MMChat

## 2\. Dataset Construction: MMChat

MMChat originates from a Chinese social media on which users can share their
daily lives through images and texts. This section starts by introducing how
MMChat is constructed (Section 2.1) and cleaned (Section 2.2). We then sketch
how to manually annotate MMChat to produce MMChat-hf (Section 2.3). At length,
we report an analysis of the MMChat dataset.

### 2.1. Data Collection

A two-phase pipeline is used to construct raw dialogues in MMChat: the first
phase aims to collect seed users who are active on social media. We start this
phase with a few hand-collected mass media accounts. Professionals maintain
these accounts and are committed to posting daily news on broad topics. The
users who comment under this news are collected as our _seed users_. The
second phase starts from the seed users collected above. Specifically, the
images posted by these seed users are obtained, and the comments under these
images are collected. Dialogues along these images are constructed by
restoring the reply relationship between these comments.

The two-phase data collection approach used in our study effectively avoids
spammers’ noises since most spammers will not bother to follow and reply to
daily news. Moreover, we also filter out seed users that are not active to
make the data collection process more effective. Finally, we collected a
corpus containing about 32.4M sessions of raw dialogues.

### 2.2. Data Filtering and Post-processing

#(Dialogues) | 120.84K
---|---
#(Total Images) | 204.32K
#(Total Utterances) | 314.13K
#(Dialogue Sessions) Longer than 4 | 17.32K
#(Image) per Dialogue | 2.91
#(Utterance) per Dialogue | 2.59
#(Character) per Utterance | 8.52
#(Raw Dialogues) | 32.4M
Table 1: Statistics of MMChat. |  Caption: A store on the street during night. Detected Objects: building, light, red signature, red calligraphy, plants, flowers, lamps, table |  |  Caption: A brick house with guidepost. Detected Objects: building, wall, black signature, red signature, calligraphy, lamp
---|---|---|---
|  Caption: Flags on the street. Detected Objects: building, blue sky, red flags, roof, people, trees |  |
A: 去王家大院看看。 (Will you have a visit to the Wang Family Courtyard?) B: 跟古城差不多吧。
(Isn’t there having the same scene with the ancient city?) A: 比古城好，有时间可以去。
(Much better than the ancient city. You can have a visit if you have spare
time.) B: 好的 古城过度开发了都成商业街了。 (Sure. The ancient city has been over-
commercialized.) A: 嗯嗯，王家大院很震撼我。 (Uh-huh. The Wang Family Courtyard impressed
me a lot.)
Table 2: An example dialogue (translated from Chinese) and its associated
images from MMChat. Each image’s semantic information (including object
labels, attributes, bounding boxes and image captions) is provided.

To improve the quality of MMChat, a set of rules is carefully designed to
filter out low-quality images and dialogues from the raw corpus collected in
Section 2.1. Specifically, images with extremely low resolution (fewer than
$500$ pixels) or high aspect ratios (larger than $10$) are abandoned, and
dialogues that contain extremely long utterances (longer than 200 tokens) are
filtered. Moreover, we only retain dialogues that contain more than $3$
utterances. The offensive contents are also filtered using an offensive word
list and a pre-trained offensive content classifier [Wang et al. (2020].

To ensure the dialogue contents in MMChat are related to the corresponding
images in the first few turns of the conversation, we only retain images that
are uploaded through the direct-share mode. This mode allows users to share
images without providing textual content. We argue that the initial few turns
of the dialogues following these image-only posts are usually triggered by the
visual information because there are no previous textual contexts except for
the uploaded images.

Note that eliminating posts that are not uploaded through the direct-share
mode filters out a vast majority of collected raw dialogues. However, this
rule is adopted not because these filtered dialogues are of low quality but
because we only have limited computation resources. We want our model to focus
on dialogues that are more closely related to its multi-modal contexts. We
believe these filtered raw dialogues are useful in building large-scale multi-
modal dialogue models or multi-modal pre-trained models. We will release all
the collected raw dialogues to facilitate further studies in this direction.

The statistics of the resulting MMChat dataset are shown in Table 1. Each
dialogue session is associated with at least one image (9 images maximum), and
a considerable amount of sessions (more than 17.32K) in MMChat contain at
least 4 utterances (i.e., 2 turns). Note that different dialogues may share
the same post (i.e., the same set of images). To protect data privacy, MMChat
is released under strict terms for academic users only. More details for the
data release protocol can be found in the Broader Consideration section.

### 2.3. Human Filtering

#(Dialogues) | 19.90K
---|---
#(Total Images) | 52.66K
#(Total Utterances) | 81.06K
#(Dialogue Sessions) Longer than 4 | 8.91K
#(Image) per Dialogue | 2.70
#(Utterance) per Dialogue | 4.07
#(Character) per Utterance | 11.93
Table 3: Statistics of MMChat-hf.

To facilitate further studies on MMChat, we recruit annotators to manually
filter MMChat, and construct a dataset MMChat-hf with higher quality.
Concretely, we random sample 100k dialogue sessions from MMChat and ask
annotators to annotate each session from the following three aspects: 1)
_Whether the associated images are qualified_. The associated images of a
dialogue session are not qualified if any of the images is overlong/flat
(i.e., depth-width ratio $>$ 10 or $<$ 0.1) or is a screenshot of texts (e.g.,
email, news, etc.). We also identify selfies and offensive images as
disqualified; 2) _Whether the dialogue contents are non-offensive_. Though we
have filtered offensive contents automatically, we ask annotators to check
them further manually; 3) _Whether the dialogue content is strongly correlated
with the associated images_. A dialogue session is annotated as “true” in this
aspect if its content contains mentions of any object/person/background of its
associated images.

Figure 2: Overview of multi-modal dialogue generation model.

In MMChat-hf, we only retain dialogue sessions that are annotated as “true” in
all the above three aspects. This results in 19.90K dialogue sessions as well
as 52.66K images. More statistics are shown in Table 3. In MMChat-hf,
dialogues contains 4.07 utterances on average and utterances contains 11.93
characters on average, which are longer than dialogues in MMChat (see Table
1). All these annotations will be released under our data release protocol.

### 2.4. Data Analysis

We try to reveal the semantics of images contained in MMChat. Specifically, a
Faster R-CNN model [Ren et al. (2015, Lu et al. (2019] trained with attribute
head on the Visual Genome [Krishna et al. (2017] dataset is used to detect
objects in each image 111We use the pre-trained model provided by
https://github.com/peteanderson80/bottom-up-attention. The regions where any
class detection confidence exceeds a threshold (0.2 in our case following
[Anderson et al. (2018]) are selected to further detect the specific object
labels. We follow the work of ?) to use an object and attribute vocabulary
with the size of 1600 and 400, respectively. An average count of 11.42 objects
is detected in each image. This indicates that images in our dataset contain
rich semantic information and thus are informative. An example dialogue
session, together with its associated images, is shown in Table 2.

## 3\. Dialogue Generation on MMChat

This section starts by formally defining the task of MMDS (Section 3.1).
Moreover, we propose to use the attention routing mechanism [Zheng et al.
(2020] to capture the issue of sparsity (Section 3.2).

Model | BLEU-2 | BLEU-3 | BLEU-4 | Dist-1 | Dist-2 | Ent-1 | Ent-2
---|---|---|---|---|---|---|---
Seq2Seq | 2.830 | 1.376 | 0.805 | 2.63 | 33.92 | 6.00 | 9.47
Seq2Seq+PIMG | 2.928 | 1.469 | 0.888 | 2.73 | 34.34 | 6.01 | 9.45
| (+3.46%) | (+6.76%) | (+10.31%) | (+3.80%) | (+1.24%) | (+0.17%) | (-0.21%)
Seq2Seq+IMG | 3.001 | 1.588 | 1.006 | 2.82 | 35.38 | 6.07 | 9.52
| (+6.04%) | (+15.41%) | (+24.97%) | (+7.22%) | (+4.30%) | (+1.17%) | (+0.53%)
Human Reference | N/A | N/A | N/A | 9.09 | 48.77 | 6.69 | 9.64

Table 4: Evaluation Results on MMChat. Relative improvements compared to the Seq2Seq baseline is shown in parentheses. Model | BLEU-2 | BLEU-3 | BLEU-4 | Dist-1 | Dist-2 | Ent-1 | Ent-2
---|---|---|---|---|---|---|---
Seq2Seq | 3.779 | 2.405 | 1.641 | 5.35 | 45.62 | 6.11 | 9.26
Seq2Seq+PIMG | 4.576 | 3.094 | 2.230 | 5.04 | 42.61 | 6.01 | 9.14
| (+21.09%) | (+28.65%) | (+35.89%) | (-5.79%) | (-6.60%) | (-1.64%) | (-0.21%)
Seq2Seq+IMG | 4.818 | 3.381 | 2.541 | 5.75 | 45.35 | 6.05 | 9.15
| (+27.49%) | (+40.58%) | (+54.84%) | (+7.48%) | (-0.59%) | (-0.98%) | (+1.19%)
Human Reference | N/A | N/A | N/A | 6.17 | 47.43 | 5.98 | 9.00
Table 5: Evaluation Results on MMChat-hf. Relative improvements compared to
the Seq2Seq baseline is shown in parentheses.

### 3.1. Task Definition

The task of MMDS is to learn a function $f$ that can map textual contexts
$\mathcal{C}$ (e.g., dialogue histories) and multi-modal contexts
$\mathcal{I}$ (e.g., images, audio or video) into dialogue responses $Y$,
i.e., learn $f:\\{\mathcal{C,I}\\}\mapsto Y$. In this study, we focus on the
image modality in $\mathcal{I}$, i.e., $\mathcal{I}$ is composed of a set of
images $\\{I_{n}\\}_{n=1}^{N}$.

### 3.2. Dialogue Generation Model

The Seq2Seq architecture is used as our backbone to build a multi-modal
dialogue generation model. As shown in Figure 2, two encoders are used to
respectively encode the textual context $\mathcal{C}$ and image context
$\mathcal{I}$ into encoded representations $E_{\mathcal{C}}$ and
$E_{\mathcal{I}}$. An attention routing module is utilized to merge
$E_{\mathcal{C}}$ and $E_{\mathcal{I}}$ in the decoder, and the response $Y$
is decoded auto-regressively.

#### 3.2.1. Encoder

The encoder for the textual context $\mathcal{C}$ is parameterized with the
Transformer architecture [Vaswani et al. (2017] (12 layers, 12 attention
heads, and 768 hidden states). To improve the generation quality, we
initialize its weights using a pre-trained GPT model [Radford et al. (2018].
Utterances in the dialogue history are concatenated using a special token
“[SEP]”, and $E_{\mathcal{C}}$ is obtained by feeding the concatenated token
sequence into the textual encoder.

The encoder for the image context $\mathcal{I}$ is implemented as the Faster
R-CNN model with ResNet-101 backbone. The weights of this encoder are pre-
trained on the Visual Genome dataset and fixed in the training process 222We
have released the pre-trained weights. Specifically, a feature vector with the
size of 2048 is extracted from each image region. The top-50 high confidence
regions are used to produce $E_{\mathcal{I}}$ with a linear layer to adjust
the feature-length, i.e., the resulting $E_{\mathcal{I}}$ contains 50
features, each has a length of 768.

#### 3.2.2. Decoder

We implement the dialogue decoder with the Transformer architecture and share
its weights with our textual encoder. To tackle the sparsity issue, we equip
the dialogue decoder with the attention routing mechanism [Zheng et al. (2020]
to balance the contribution of each region feature. Specifically, given the
encoding of the dialogue context $E_{\mathcal{C}}$, image context
$E_{\mathcal{I}}$, and previous decoded tokens $E_{\mathcal{\mbox{pre}}}$,
three attention routes are computed as:

$\displaystyle O_{\mathcal{C}}$
$\displaystyle=\mbox{MHA}(E_{\mathcal{\mbox{pre}}},E_{\mathcal{C}},E_{\mathcal{C}}),$
(1) $\displaystyle O_{\mathcal{I}}$
$\displaystyle=\mbox{MHA}(E_{\mathcal{\mbox{pre}}},\gamma
E_{\mathcal{I}},\gamma E_{\mathcal{I}}),$ (2) $\displaystyle
O_{\mathcal{\mbox{pre}}}$
$\displaystyle=\mbox{MMHA}(E_{\mathcal{\mbox{pre}}},E_{\mathcal{\mbox{pre}}},E_{\mathcal{\mbox{pre}}}),$
(3)

where $\gamma\in[0,1]$ is a hyper-parameter to re-scale $E_{\mathcal{I}}$. MHA
and MMHA represent masked and unmasked multi-head attention, respectively, in
which $E_{\mathcal{\mbox{pre}}}$ serves as the query. The results of each
attention operation are averaged before proceeding to the next sub-module:

$O_{\mbox{merge}}=\frac{O_{\mathcal{C}}+O_{\mathcal{I}}+O_{\mathcal{\mbox{pre}}}}{3}.$
(4)

Note that the attention route on image features (i.e., Eq. 2) assigns
different weights to different image regions. This facilitates more flexible
control over image features in the decoding process and thus helps ease the
sparsity issue.

Input Image | Dialogue Contexts and Generated Responses
---|---
|  Context: “在拜托啦学长那里看到你，欢迎来我们福建玩， 坐标福建漳州。不过现在在外上学” (“I saw you on the channel
”Please! Seniors.”. Welcome to Fujian! I am from Zhangzhou, Fujian, but
studying out of Fujian.”) Human: 厦门。 (Xiamen.) Seq2Seq: 啊，那还真的好啊，我已经回来了。 (Aha!
That’s great! I have been back.) Seq2Seq+PIMG: 我是漳州的学妹。 (I am your younger
female schoolmate from Zhangzhou.) Seq2Seq+IMG: 原来是这样，下次有机再来 福建找你玩哈。 (I see.
I’ll look you up, if I ever have a chance to visit Fujian again.)
|  Context: “你又睡那么晚”, “美甲！” “你有空了就来撒” (“Why you go to sleep so late again?”,
“Just back from a nail salon.”, “Shall we meet if you have spare time?”)
Human: 好。 (Okey.) Seq2Seq: 我就在四小时以左右开始睡不着觉！其他时候都是两三点。 (I found it is hard to
fall asleep at 4 o’clock this morning. This used to be 2 or 3 o’clock.)
Seq2Seq+PIMG: 上完课就去上，没时间了。 (I’ll go there after class. There is no time.)
Seq2Seq+IMG: 好滴！国庆之后就能见到了。 (Okey. I suppose to meet you soon after the
national day.)
Table 6: Example outputs (with English translations) generated by our multi-
modal dialogue systems.

## 4\. Experiments

Experiments are performed to assess both our model and datasets. Specifically,
we train our model on both MMChat and MMChat-hf. For MMChat, we sample 4.0K
and 2.0K dialogue sessions for testing and validation, respectively, and for
MMChat-hf, we sample 1.0K and 1.0K dialogue sessions for testing and
validation, respectively. Two baselines are also implemented in our study.

### 4.1. Implementation Details

In our proposed dialogue model (referred to as Seq2Seq+IMG), the encoder and
decoder are 12-layer transformers with 768-dimensional hidden states and 12
attention heads. For the position-wise feed-forward networks,
3,072-dimensional inner states are used. The Adam optimizer is used to train
our model with $\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=10^{-9}$. The
maximum learning rate is set to 1.0e-4. The training starts with a warmup step
of 1,000, and the learning rate is annealed proportionally to the inverse
square root of the step number. The batch size is set to 360, and the training
iterators 60 epochs. A character-level vocabulary of size 13,084 is used.
Other settings of our Transformer model follow the work of ?).

We share the weights of the encoder and decoder in the dialogue model and
initialize these weights using a pre-trained GPT model [Wang et al. (2020].
The pre-training corpus contains about 0.5 billion tokens, and the pre-
training process lasts for a week on 8 GTX1080Ti GPUs. We use the top-K
($K=20$) decoding scheme with beam search for the dialogue model in the
inference phase. The beam size and length penalty are 4 and 2.0, respectively.
The training of our model lasted for about 24 hours on 2 NVIDIA Tesla V100
GPUs. The number of parameters for our model (textual encoder and decoder, not
including the image encoder) is 191.01M. Note that due to the large
computation loads to utilize the pre-training based method in our model and
our baselines, we inherit most of the hyper-parameter settings from the
previous studies, such as ?), and skip the hyper-parameter tuning process.
Moreover, for fair comparisons, we use a fixed set of hyper-parameters in all
our experiments (including all the baselines).

### 4.2. Baselines

We also implement two baselines to validate our dataset and model: Seq2Seq: a
transformer-based Seq2Seq model is built with only textual inputs;
Seq2Seq+PIMG: an image-grounded dialogue model is built with a single pooled
image representation. Specifically, an max-pooling operation is applied to
$E_{\mathcal{I}}$, and the pooled vector is added to each representation
vector in $E_{\mathcal{C}}$. The attention route to $E_{\mathcal{I}}$ (i.e.,
Eq. 2) is not applied. Note that the first baseline does not use image
contexts, and the second baseline does not model the sparsity phenomenon.

For fair comparisons, all baselines employ the same architecture, hyper-
parameter setting and initialization scheme with our model Seq2Seq+IMG.

### 4.3. Metrics

We use the following metrics: BLEU [Papineni et al. (2002] measures the n-gram
(n=2,3,4) overlaps between generated and reference responses; _Distinct_
(Dist) [Li et al. (2016] measures the proportion of unique n-gram in the
generated response (n=1,2); _Entropy_ (Ent) [Zhang et al. (2018b] measures how
evenly the empirical n-gram (n=1,2) distribution is:

$\mbox{Ent}=\frac{1}{\sum_{w}F(w)}\sum_{w\in
V}F(w)\log\frac{F(w)}{\sum_{w}F(w)},$ (5)

where $V$ is the set of all n-grams and $F(w)$ is the frequency of n-gram $w$.
Note that both distinct and the entropy measure the diversity of the generated
responses.

### 4.4. Results

Table 4 and Table 5 shows the results on MMChat and MMChat-hf, respectively.
Table 6 lists some example outputs of our dialogue generation models and
baselines. Generally speaking, our Seq2Seq+IMG outperforms both baselines on
most metrics. The exception is on the entropy metric: the difference between
Seq2Seq and Seq2Seq+IMG on both datasets is marginal (approximately 1%).

Based on the above results, we can observe that: 1) Incorporating image
contexts in dialogue models helps to produce better responses. Specifically,
our model obtains 24.97% and 54.84% relative improvements on the BLEU-4 score
compared to the text-only baseline Seq2Seq on MMChat and MMChat-hf,
respectively. Meanwhile, similar improvements are also identified when
comparing Seq2Seq+PIMG and Seq2Seq. These results validate our motivation to
incorporate multi-modal features in the dialogue generation model and prove
that MMChat (as well as MMChat-hf) can be used to build image-grounded
dialogue models. 2) Our model, Seq2Seq+IMG, obtains greater relative
improvement on BLEU than Seq2Seq+PIMG. This indicates that explicitly modeling
the sparsity phenomenon helps to further improve the dialogue generation
performance, and MMChat/MMChat-hf facilitates the study of such a phenomenon.

Moreover, by comparing results on MMChat and MMChat-hf, we find that: 1)
Models trained on MMChat-hf generally receive higher BLEU and distinct scores
than those trained on MMChat-hf; 2) Incorporating image information on MMChat-
hf introduce higher improvement on the BLEU score comparing to the improvement
observed on MMChat (e.g., BLEU-4 is improved 54.84% on MMChat-hf while
improved 24.97% on MMChat). These results indicate that filtering out low-
quality images and dialogues that are irrelative to their associated images
(see Section 2.3) do help build a better dataset.

Note that the diversity improvement of our model Seq2Seq+IMG is not
significant compared to the baseline Seq2Seq, particularly on the MMChat-hf
dataset. This may be because the generated responses are bounded by more
contexts (i.e., images). 333Also note that models receive higher distinct
scores on MMChat-hf than MMChat, which could, to a large extent, because the
test set of MMChat-hf is smaller than that of MMChat..

## 5\. Conclusion

We introduce MMChat, a large-scale multi-modal dialogue corpus that reveals
the image-sparsity phenomenon in real conversations. Our dataset contains
120.84K dialogue sessions filtered from 32.4M sessions of raw multi-modal
dialogues. Building on 100K dialogues from MMChat, we further conduct human
filtering, yielding MMChat-hf, in which there are 19.9K high-quality multi-
modal dialogue sessions. A dialogue model is proposed to tackle the image-
sparsity issue utilizing MMChat. Experiment results indicate that both MMChat
and MMChat-hf help to develop image-grounded dialogue systems and facilitate
further study of the image-sparsity issue. Besides the filtered dialogues in
MMChat, we will also release all the raw dialogues obtained in the data
collection process to facilitate further studies.

## 6\. Broader Considerations

Our dataset MMChat originates from a Chinese social media. The dataset
collection and release protocols are carefully designed to avoid violating the
privacy of each user on that social media. Specifically, each user’s
permission setting is strictly respected so that only publicly visible
contents are collected. Rules are designed to filter out dialogues that may
potentially expose users’ private information, such as phone numbers or
emails. Moreover, we will not host these images in MMChat on our own server.
Only the URLs to these images will be released along with the download
scripts.

To further enforce the data privacy, MMChat is released under strict terms for
academic uses only, in which they promise no abuse of MMChat besides academic
purposes.

In addition to the privacy issues, there might also be toxic or biased texts
in MMChat or be generated by MMDSs trained on MMChat. Although we take the
responsibility to remove toxic texts (using an offensive word list, an
offensive content classifier, and human filtering), we cannot guarantee that
there are no offensive contents left. However, as offensive and abusive
content recognition is a rapidly developing area [Vidgen et al. (2019], we
would deploy more advanced filters once the new state-of-the-art offensive and
abusive classifiers are proposed in the future.

Regarding the potential biases, except those from the dataset itself
[Henderson et al. (2018] (which always exists in dialogue datasets), biases
might be introduced by the pre-trained language model [Bender et al. (2021]
and the pre-trained image encoder [Steed and Caliskan (2020] used in this
work. In the future, we head to apply and develop corresponding mitigation
techniques (following works such as ?) and ?)).

During annotation, we pay each annotator 0.5 CNY per item. This results in
approximately 60 CNY per hour, which is 4.5 times the minimum wage standard in
China.

Besides, we also note that the goal of our work is to facilitate further work
on multi-modal dialogue systems. Although the model used in this work is still
far from realistic, our dataset can be regarded as an initial step toward the
sparsity issue in real-world multi-modal conversations.

## 7\. Bibliographical References

## References

  * Adiwardana et al. (2020 Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al.  (2020).  Towards a human-like open-domain chatbot.  arXiv preprint arXiv:2001.09977.
  * AlAmri et al. (2019 AlAmri, H., Cartillier, V., Das, A., Wang, J., Cherian, A., Essa, I., Batra, D., Marks, T. K., Hori, C., Anderson, P., Lee, S., and Parikh, D.  (2019).  Audio visual scene-aware dialog.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 7558–7567. Computer Vision Foundation / IEEE.
  * Anderson et al. (2018 Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L.  (2018).  Bottom-up and top-down attention for image captioning and visual question answering.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6077–6086. IEEE Computer Society.
  * Bender et al. (2021 Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S.  (2021).  On the dangers of stochastic parrots: Can language models be too big.  Proceedings of FAccT.
  * Das et al. (2017 Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., and Batra, D.  (2017).  Visual dialog.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1080–1089. IEEE Computer Society.
  * Dinan et al. (2020 Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J.  (2020).  Queens are powerful too: Mitigating gender bias in dialogue generation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online, November. Association for Computational Linguistics.
  * Henderson et al. (2018 Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J.  (2018).  Ethical challenges in data-driven dialogue systems.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129.
  * Jovanovic and Van Leeuwen (2018 Jovanovic, D. and Van Leeuwen, T.  (2018).  Multimodal dialogue on social media.  Social Semiotics, 28(5):683–699.
  * Ju et al. (2019 Ju, D., Shuster, K., Boureau, Y.-L., and Weston, J.  (2019).  All-in-one image-grounded conversational agents.  arXiv preprint arXiv:1912.12394.
  * Kottur et al. (2019 Kottur, S., Moura, J. M. F., Parikh, D., Batra, D., and Rohrbach, M.  (2019).  CLEVR-dialog: A diagnostic dataset for multi-round reasoning in visual dialog.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 582–595, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Krishna et al. (2017 Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al.  (2017).  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73.
  * Li et al. (2016 Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B.  (2016).  A diversity-promoting objective function for neural conversation models.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California, June. Association for Computational Linguistics.
  * Liao et al. (2018 Liao, L., Ma, Y., He, X., Hong, R., and Chua, T.-s.  (2018).  Knowledge-aware multimodal dialogue systems.  In Proceedings of the 26th ACM international conference on Multimedia, pages 801–809.
  * Liu et al. (2020 Liu, H., Wang, W., Wang, Y., Liu, H., Liu, Z., and Tang, J.  (2020).  Mitigating gender bias for neural dialogue generation with adversarial learning.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 893–903, Online, November. Association for Computational Linguistics.
  * Lu et al. (2019 Lu, J., Batra, D., Parikh, D., and Lee, S.  (2019).  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Hanna M. Wallach, et al., editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13–23.
  * Madotto et al. (2018 Madotto, A., Wu, C.-S., and Fung, P.  (2018).  Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468–1478, Melbourne, Australia, July. Association for Computational Linguistics.
  * Meng et al. (2020 Meng, Y., Wang, S., Han, Q., Sun, X., Wu, F., Yan, R., and Li, J.  (2020).  Openvidial: A large-scale open-domain dialogue dataset with visual contexts.  arXiv preprint arXiv:2012.15015.
  * Mogadala et al. (2019 Mogadala, A., Kalimuthu, M., and Klakow, D.  (2019).  Trends in integration of vision and language research: A survey of tasks, datasets, and methods.  arXiv preprint arXiv:1907.09358.
  * Mostafazadeh et al. (2017 Mostafazadeh, N., Brockett, C., Dolan, B., Galley, M., Gao, J., Spithourakis, G., and Vanderwende, L.  (2017).  Image-grounded conversations: Multimodal context for natural question and response generation.  In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 462–472, Taipei, Taiwan, November. Asian Federation of Natural Language Processing.
  * Papineni et al. (2002 Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  (2002).  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics.
  * Pasunuru and Bansal (2018 Pasunuru, R. and Bansal, M.  (2018).  Game-based video-context dialogue.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 125–136, Brussels, Belgium, October-November. Association for Computational Linguistics.
  * Radford et al. (2018 Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  (2018).  Improving language understanding by generative pre-training.
  * Ren et al. (2015 Ren, S., He, K., Girshick, R. B., and Sun, J.  (2015).  Faster R-CNN: towards real-time object detection with region proposal networks.  In Corinna Cortes, et al., editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91–99.
  * Shuster et al. (2020a Shuster, K., Humeau, S., Bordes, A., and Weston, J.  (2020a).  Image-chat: Engaging grounded conversations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414–2429, Online, July. Association for Computational Linguistics.
  * Shuster et al. (2020b Shuster, K., Ju, D., Roller, S., Dinan, E., Boureau, Y.-L., and Weston, J.  (2020b).  The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2453–2470, Online, July. Association for Computational Linguistics.
  * Shuster et al. (2020c Shuster, K., Smith, E. M., Ju, D., and Weston, J.  (2020c).  Multi-modal open-domain dialogue.  arXiv preprint arXiv:2010.01082.
  * Steed and Caliskan (2020 Steed, R. and Caliskan, A.  (2020).  Image representations learned with unsupervised pre-training contain human-like biases.  arXiv preprint arXiv:2010.15052.
  * Vaswani et al. (2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  (2017).  Attention is all you need.  In Isabelle Guyon, et al., editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.
  * Vidgen et al. (2019 Vidgen, B., Harris, A., Nguyen, D., Tromble, R., Hale, S., and Margetts, H.  (2019).  Challenges and frontiers in abusive content detection.  In Proceedings of the Third Workshop on Abusive Language Online, pages 80–93, Florence, Italy, August. Association for Computational Linguistics.
  * Wang et al. (2020 Wang, Y., Ke, P., Zheng, Y., Huang, K., Jiang, Y., Zhu, X., and Huang, M.  (2020).  A large-scale chinese short-text conversation dataset.  In Natural Language Processing and Chinese Computing.
  * Zhang et al. (2018a Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J.  (2018a).  Personalizing dialogue agents: I have a dog, do you have pets too?  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia, July. Association for Computational Linguistics.
  * Zhang et al. (2018b Zhang, Y., Galley, M., Gao, J., Gan, Z., Li, X., Brockett, C., and Dolan, B.  (2018b).  Generating informative and diverse conversational responses via adversarial information maximization.  In Samy Bengio, et al., editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 1815–1825.
  * Zheng et al. (2019 Zheng, Y., Chen, G., Huang, M., Liu, S., and Zhu, X.  (2019).  Personalized dialogue generation with diversified traits.  arXiv preprint arXiv:1901.09672.
  * Zheng et al. (2020 Zheng, Y., Zhang, R., Huang, M., and Mao, X.  (2020).  A pre-training based personalized dialogue generation model with persona-sparse data.  In AAAI, pages 9693–9700.


# Million.js: A Fast Compiler-Augmented Virtual DOM for the Web

Aiden Bai  Camas High SchoolCamasWashingtonUSA 0000-0002-3676-3726
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Interactive web applications created with declarative JavaScript User
Interface (UI) libraries have increasingly dominated the modern internet.
However, existing libraries are primarily made for run-time execution, and
rely on the user to load and render web applications. This led us to create
Million.js, a fast compiler-augmented virtual Document Object Model (DOM) for
the web. Million.js reduces load time and time-to-interactive by creating a
compiler to compute interactive regions of a web application before the user
visits the page. The virtual DOM run-time optimizes interactive content
through compiler flags, compute batching, scheduling, and reactive data
primitives to achieve optimal performance. When benchmarked against the most
popular virtual DOM libraries, Million.js resulted in 133% to 300% faster
rendering and 2347% faster load. In a real-world web application with both
comparative benchmarks and an informal user study, Million.js loaded 35.11%
faster after migrating from React. The findings show that web applications
have the potential to be orders of magnitude faster through JavaScript UI
libraries that use Million.js.

Web technologies, Web Frameworks, JavaScript, User interface (UI), Virtual
DOM, Compilers, Optimization

††journalyear: 2023††copyright: acmlicensed††conference: The 37th ACM/SIGAPP
Symposium on Applied Computing; March 27-March 31, 2023; Tallinn,
Estonia††booktitle: The 37th ACM/SIGAPP Symposium on Applied Computing (SAC
’23), March 27-March 31, 2023, Tallinn, Estonia††price: 15.00††doi:
10.1145/3555776.3577683††isbn: 978-1-4503-9517-5/23/03††article: 4††ccs:
Human-centered computing Web-based interaction††ccs: Information systems Web
applications

##  1\. Introduction

> My advice for anyone who wants to make a dent in the future of web
> development: time to learn how compilers work.
>
>  \- Tom Dale (Dale, 2017)

Websites were initially written in Hypertext Markup Language (HTML) and
Cascading Style Sheets (CSS) and designed to be static, which limited user
interaction (Wieruch, 2017). Eventually, browsers introduced JavaScript and
the DOM to provide basic interactivity (MDN, 2022; Garrett et al., 2005),
however, web applications that required significant user interaction to
operate suffered scaling and maintenance issues (Zakas, 2010). To support
interactive web applications, single-page applications (SPA) were created to
allow state-driven development (SDD) through the virtual DOM and its variants
(Grov, 2015). However, due to inefficient rendering strategies, SPA and
virtual DOM-based web applications have become slower to load and render on
user devices (Zakas, 2010), which can lead to worse user retention and
experience (Pavic et al., 2020; Glynn, 2020).

A wide variety of meta-frameworks (such as Next.js (Vercel, 2016) and Astro
(Company, 2022)) are currently used as a solution to improve the load time of
static content in SPA (Dinku, 2022). However, these frameworks do not address
the virtual DOM’s fundamental problem of slow rendering speed on interactive
content.

As a result, when web developers want to improve the rendering speed of
interactive content, they have to choose between refactoring to a DOM-based
implementation or fine-tuning their virtual DOM code with manual optimizations
(e.g. memoization (Suresh et al., 2017), conditional rendering, state
management). Choosing the DOM will yield the most efficient results, but
worsens the developer experience (Zakas, 2010). Increasing the rendering speed
of the virtual DOM requires significant knowledge of internals and may be
difficult to optimize to a meaningful degree (Kishore and Mahendra, 2020).

This paper presents Million.js, a virtual DOM that optimizes both load time
and rendering speed. Million.js has a compiler that can be integrated into
JavaScript UI libraries and meta-frameworks to optimize interactive content.
When benchmarked, Million.js yields 133% to 300% faster rendering with the JS
Framework Benchmark (Krausest, 2015) and 2347% faster load time than React
with the Chrome Devtools Performance benchmark (Saverimoutou et al., 2018).

Million.js is implemented as an open-source JavaScript library, so web
developers can use Million.js wherever they use JavaScript, primarily within
browser environments. As such, Million.js is used in 85 open-source projects.
In a real-world application using the Million.js React compatibility layer,
there was a 35% improvement in rendering speed after migrating from React.

The paper makes the following contributions:

  * •

Million.js—a compiler-augmented virtual DOM—released as open-source software
that can be downloaded and used today111https://github.com/aidenybai/million.

  * •

the Million.js React compatibility layer, which allows web developers and
existing projects to use React’s API.

  * •

a series of benchmarks and a real-world test that measure the load time and
rendering speed of Million.js.

##  2\. Related Work

Million.js extends prior work on SDD, SPA, and virtual DOM approaches in
JavaScript UI library development.

###  2.1. Event-Driven Development

JavaScript UI library authors have created tools and libraries to simplify the
process of creating web applications (Zou et al., 2014; Ladan, 2015). One of
these libraries is jQuery, a widely used DOM utility library. jQuery’s
ergonomic API helped developers solve inconsistencies with the verbose DOM
syntax across browsers (Muyldermans, 2019). As a result, jQuery flourished in
usage in the JavaScript ecosystem (Persson, 2020).

Although jQuery had a monumental impact on web development, many web
developers faced problems with the imperative event-driven API of jQuery and
the DOM (Domański et al., 2014). While jQuery could reasonably handle
interactivity for static websites, it became unscalable as web applications
became more complex (Delcev and Draskovic, 2018; Ladan, 2015), leading
developers to create custom solutions (Meyerovich et al., 2009).

###  2.2. SDD and Virtual DOM

To address the limitations of the DOM and jQuery, software engineers at
Facebook created React, a JavaScript UI library that enabled SDD through the
virtual DOM.

Figure 1. Example of virtual DOM diff and patch steps. The reconciliation
process includes the diff and patch steps. The diff step calculates the
changes to the UI when given the current and new UI. The patch step takes the
calculated UI changes from the diff step and uses the DOM to update the UI.

The creation of React and the virtual DOM subsequently enabled state-driven
web applications that minimized DOM manipulations, reflows, and repaints
(Levlin, 2020) without needing to describe control flow. The virtual DOM is a
tree diffing algorithm that finds differences between the current and new
virtual DOM tree to update UI (see Figure 1). This led to the implementation
of the virtual DOM in other JavaScript UI libraries like Angular, Ember, and
Vue (Gizas et al., 2012).

Unlike other JavaScript UI libraries, React has its own domain-specific
language (DSL) called JSX (Gackenheimer, 2015), which “provides an extension
to JavaScript by adding XML-like syntax that is similar to HTML” (Muyldermans,
2019). React uses the virtual DOM through JSX inside functional components, a
superset of JavaScript that compiles down to the virtual DOM, while Vue and
Angular use directives and Single-File Components (SFC) to set boundaries and
compile to the virtual DOM (Kumar and Singh, 2016). Thus, different API
abstraction decisions in libraries allow developers to choose a JavaScript UI
library that fits their project (Gizas et al., 2012) through the virtual DOM
at the cost of load time and rendering speed (Levlin, 2020).

###  2.3. Meta-frameworks

Figure 2. Yahoo Finance web application with interactive content highlighted
in pink. Static content is loaded before the user visits the page and
interactive content will be hydrated upon page visit.

To improve load time, techniques like Server Side Rendering (SSR) and Static
Site Generation (SSG) (Petersen, 2016) have led to the creation of meta-
frameworks like Next.js (Vercel, 2016), which augment and improve the load
time of JavaScript UI libraries like React and Vue, using pre-rendering and
hydration (Iskandar et al., 2020). Pre-rendering turns the web application
static for initial load, and hydration turns the static page into an
interactive page. Meta-frameworks like Astro (Company, 2022) are exploring how
SSG and SSR can be optimized with partial hydration, a subset of hydration
that only parts of the page interactive (Figure 2). These techniques can
significantly improve the performance of web applications that require less
interactivity, such as blogs or eCommerce websites (Iskandar et al., 2020).

###  2.4. Compiled JavaScript UI Libraries

While meta-frameworks such as Next.js have helped optimize static content, new
JavaScript UI libraries, including Svelte, leverage an optimizing compiler
(Kennedy and Allen, 2001) to address the performance constraints of the
interactive content of a web application (Harris, 2018). Svelte uses reactive
assignments to generate DOM manipulations instead of an intermediate
representation of the DOM (Muyldermans, 2019). With this technique, Svelte
significantly improves the performance of value-based UI updates but suffers
from large structural UI updates where the virtual DOM excels (Zou et al.,
2014).

With explorations into compiled frameworks, reactive assignments, and
efficient hydration, comparative benchmark results of virtual DOM libraries
(such as snabbdom (Vindum, 2015), virtual-dom (Esch, 2014), tiny-vdom (Bai,
2022), simple-virtual-dom (Livoras, 2015)) are slower (depending on
architectural decisions, see Table 1) in contrast to JavaScript UI libraries
that utilize these new techniques (Mariano, 2017).

Library | Keyed Algorithm | Virtual Nodes | Render Strategy | Composability | Size
---|---|---|---|---|---
snabbdom | Longest Common Subsequence | Mutable | One pass |  node, props, and children separation | 4.93kB
virtual-dom | N/A | Immutable | Two pass |  diff and patch separation | 6.03kB
tiny-vdom | Naive | Immutable | One pass | N/A | 0.58kB
simple-virtual-dom | Naive | Immutable | Two pass |  diff and patch separation | 2.14kB

Table 1. Differences between methodology and algorithmic differences of the
virtual DOM libraries: snabbdom, virtual-dom, tiny-vdom, simple-virtual-dom.
The longest common subsequence algorithm is the most efficient keyed
algorithm. Mutable virtual nodes are less memory efficient compared to
immutable virtual nodes unless used with a one-pass render strategy. More
separation means more composability and easier adaptation of the API for
different needs.

We extend this prior work by providing the first virtual DOM that can
integrate with a compiler.

##  3\. Overview of Million.js

Million.js is a compiler-augmented virtual DOM for the web. The software is an
open-source JavaScript library and is published on Node Package Manager (NPM)
and various Content Delivery Networks (CDN). The tool is built using the
TypeScript language and can be used to build new JavaScript UI libraries or
modify user interfaces directly (Greenberg and Krishnamurthi, 2007). As such,
Million.js is compatible with JavaScript UI libraries that use a virtual DOM,
like React.

Million.js contains two key components: the virtual DOM run-time and the
optimizing compiler. The virtual DOM run-time updates the UI, similar to
existing virtual DOM libraries. However, unlike these libraries, Million.js
uses an optimizing compiler to analyze JavaScript and compute unnecessary work
during compile time to reduce load time and speed up rendering. Together, the
virtual DOM and the optimizing compiler allow for declarative UI development;
in particular, it gives authors access to the underlying rendering process to
construct customized JavaScript UI libraries (Voutilainen et al., 2016).

Principle | DOM | Virtual DOM | Million.js
---|---|---|---
Load time | 1x | <27.1x slower | <1.5x slower
Render speed | 1x | 1.7x slower | <1.1x slower
Developer friendly | N/A |  $\sim$560 stars/year | 5.4k stars/year

Table 2. Million.js design principles and Objective and Key Results (OKR)
table. All objectives were achieved within one year. Million.js achieved under
1.2x load time and within ¡1.1x render speed. Load time is measured using
Chrome Devtools and render speed with the JS Framework Benchmark.

To optimize for performance and developer usage, Million.js establishes three
main design principles in Figure 2: fast load time, fast render speed, and
ease of use. The virtual DOM suffers from slow load time and rendering speeds,
so Million.js aims to improve them by increasing parity with DOM rendering
speeds. Each principle is measured, and key results are established to measure
the success of Million.js.

###  3.1. Implementation

The virtual DOM render process has three main steps: compile, diff, and patch
as seen in Table 2. Each phase is completed in different stages and has
different purposes. The compile phase is executed once before the user visits
the page, and the diff and patch phases are executed at run-time upon
rendering.

Figure 3. Flowchart of Million.js virtual DOM render process separated into
three steps. The compiler step optimizes the interactive content of a website.
The diff step is a composable API that is used to calculate the changes to the
UI when given the current and new UI. The patch step applies diff changes to
the UI.

####  3.1.1. Compile Step

The compile step can occur in conjunction with a meta-framework such as
Next.js, or during build time with an SSG tool. Within the compile step, an
optimizing compiler will statically analyze the DSL of the source code to find
redundant or unnecessary code. While the meta-framework or SSG tool will pre-
render the user interface (UI) and hydrate the HTML page, the compiler will
optimize the interactive content as seen in Figure 3.

The optimizing compiler implements virtual node flattening (Keller and
Chakravarty, 1998) and static tree hoisting (Pawlik and Augsten, 2016) to
improve the load time and rendering speed of interactive content (see Figure
4. It targets the output of DSL compilation: hyperscript, a set of function
calls that construct a virtual node tree at run-time (Islam Naim, 2017).
Hyperscript is used because it is an easy target for DSL compilation but comes
at the cost of unnecessary memory allocation during run-time. The virtual node
flattening approach addresses this issue by flattening each function call
within the hyperscript during the compile step, removing the need to allocate
memory during run-time. Within virtual node flattening, compiler flags (Ballal
et al., 2015) are assigned to reduce run-time computation (similar to Inferno
(Inferno, 2020)). Along with virtual node flattening, the static tree hoisting
approach further reduces unnecessary memory allocation by moving flattened
hyperscript that does not depend on the state to the global JavaScript scope.

    h(’div’, null, ’Hello World’);
    // Turns into:
    {
        tag: ’div’,
        children: [’Hello World’],
        flag: Flags.ONLY_TEXT_CHILDREN
    };

Figure 4. Code samples of hyperscript and flattened and hoisted virtual node,
ordered by less to more optimizations by the compiler. The latter code sample
is a compiler optimization exclusive to Million.js.

####  3.1.2. Diff Step

The diff step occurs on the user’s device during run-time upon each render to
calculate the changes to the UI when given the current and new UI. In Figure
1., the virtual DOM calculates the specific changes during the diff step and
applies those changes to the UI during the patch step.

Diff and patch steps exist in some capacity within all virtual DOM
implementations. Million.js uses a two-pass, immutable virtual node run-time,
with separation at the node, props, and children level (see Table 1 for
comparison). The main focus, however, is improvement in the performance of the
diff and patch steps by integrating fast paths for the compile step to reduce
extraneous diffing. When fast paths are used during run-time, the virtual DOM
allocates less memory and performs less computation, leading to faster load
times and rendering speeds (Golestani et al., 2019). These fast paths are
keyed nodes (Antonio, 2015), compiler flags, and delta. The methodology can be
visualized in Figure 5.

Figure 5. Example of fast path optimization represented as a to-do list. Step
1 occurs during the compile step, step 2 occurs during the diff step, and step
3 occurs during the patch step. Every step works together to optimize the
rendering speed and load time of UI updates.

The keyed nodes approach is a run-time optimization that reduces the amount of
computation needed by identifying virtual children nodes with a key. Each key
represents a unique virtual node, meaning nodes can be moved instead of basic
update and delete operations. Moving keyed nodes is a technique used in a
special algorithm to reduce computation based on an O(ND) algorithm
(Apostolico, 1986; Myers, 1986). By removing the need to diff head and tail
nodes, a key map can be generated to efficiently move nodes to reduce
computation when changes between virtual children nodes are minimal.

Compiler flags are generated during the compile step to change the behavior of
the diff step. They are used to infer the shape of virtual nodes during run-
time to skip conditions of the diff step or invoke more efficient algorithms.
By default, virtual nodes have the same flag, but when virtual children node
edge cases are met, such as no children, only text children, or only keyed
children, less computation is used to diff the virtual node.

Delta is special data that can be attached to virtual nodes that provide an
escape hatch for imperative UI changes. Delta can be used by the compile step
to skip the diff step entirely, or in run-time by the web developer. This is
useful for feed-like UI, such as post feeds or to-do lists.

Along with the aforementioned fast paths, Million.js amortizes computation
with strategies like scheduling and batching. Traditionally, virtual DOM
implementations use a single, uninterrupted, synchronous render. Once an
update starts rendering, nothing can interrupt it until the user can see the
result on the screen. This includes user interactions, meaning that the user
cannot interact with the page until rendering is complete. Million.js
introduces scheduling, which prioritizes more urgent CPU-bound updates (such
as creating DOM nodes and running component code (Degenbaev et al., 2016)) and
defers less important updates to user interactions. Batching is also used to
cancel intermediate renders when similar updates are executed in succession
and should be used in conjunction with scheduling (Wloka, 2003).

####  3.1.3. Patch step

The patch step takes the calculated UI changes from the diff step and applies
those changes to the UI. The diff and patch steps run in a two-pass format to
separate virtual DOM change calculation with browser reflow and repaint
calculations (Ast, 2016). Separating these two categories allows the browser
to optimize calculations for both run-time steps (Kaul, 2020).

###  3.2. React Compatibility Layer

Million.js provides a fast underlying virtual DOM to update user interfaces.
However, the virtual DOM is not intended to be directly used by web
developers, as it does not provide concepts like state or components that
define JavaScript UI libraries. Thus, we created a React compatibility layer
that allows web developers to start existing or migrate React projects to
Million.js, without having to re-learn a new API.

The React compatibility layer integrates with existing React projects like
Vite (You, 2022) and meta-frameworks like Next.js through a plugin. The plugin
injects a run-time that imitates the React API implemented with Million.js
internals. Some limitations of the API include new React version 18 concurrent
mode features like Suspense, HTML streaming, and React Server Components
(React, 2022). As such, the React compatibility layer lacks the ability to
prioritize renders out of order, a bonus feature for increasing frame rate.

One notable feature Million.js provides through the React compatibility layer
a reactive data primitive that performs at optimal speeds with constant time
complexity by bypassing the diff step altogether. Feeds or list-like UI are
prevalent in modern web applications and have the potential for optimization.
With traditional virtual DOM, lists have linear time complexity, meaning
rendering speed and memory allocation gets worse as more entries are loaded
(Dermawan et al., 2020). Through the Delta optimization, direct imperative UI
changes can be made using the JavaScript Proxy API (Keil et al., 2015),
greatly reducing the computation and increasing rendering speed.

##  4\. Evaluation

Three benchmarks are used to determine the performance of Million.js: a re-
implementation of a subset of the JS Framework Benchmark (Krausest, 2015)
measuring rendering speed, a Devtools performance benchmark measuring load
time, and real-world application performance. These benchmarks measure render
speed in operations per second and load time in milliseconds to quantify the
performance of rendering methods (Ratanaworabhan et al., 2010).

###  4.1. JS Framework Benchmark

We tested ten different measurement benchmark suites on seven different
implementations of web page rendering using a subset of the JS Framework
Benchmark, as seen in Table 3. The first type of method is the raw rendering
method: DOM, innerHTML. The second type of method is the virtual DOM library:
Million.js, snabbdom (Vindum, 2015), virtual-dom (Esch, 2014), tiny-vdom (Bai,
2022), simple-virtual-dom (Livoras, 2015).

The virtual DOM libraries, except for Million.js and snabbdom, had fewer
operations per second when compared with raw DOM rendering implementations,
with the worst performing implementations being tiny-vdom and simple-virtual-
dom.

Raw rendering implementations, like DOM and innerHTML had relatively higher
operations per second in aggregate insertions or deletions, but relatively
lower operations per second in partial or complex updates when compared to
virtual DOM implementations. Additionally, implementations with keyed diffing,
specifically Million.js and snabbdom, had higher operations per second when
compared to the other virtual DOM libraries.

Method (ops/s) | DOM | million | innerHTML | snabbdom | virtual-dom | tiny-vdom | simple-virtual-dom
---|---|---|---|---|---|---|---
Append 1,000 rows to a table of 10,000 rows | 21.7 ±0.9% | 21.8 ±3.3% | 9.4 ±2.9% | 6.6 ±5.9% | 5.9 ±3.5% | 5.1 ±3.5% | 4.6 ±3.1%
Clear a table with 1,000 rows | 207 ±2.4% | 190 ±2.4% | 172 ±2.9% | 128 ±4.3% | 97.7 ±3.8% | 93.7 ±3.6% | 64.1 ±2.3%
Create 10,000 rows | 36.8 ±4.6% | 30.5 ±5% | 36.1 ±4.8% | 22.2 ±1.9% | 22.3 ±1.2% | 12.9 ±3.4% | 9.3 ±4.6%
Create 1,000 rows | 36 ±5.2% | 30.3 ±4.7% | 35.8 ±4.6% | 22.2 ±2.1% | 22.4 ±1.6% | 12.7 ±3.6% | 9.0 ±4.9%
Update every 10th row for 1,000 rows | 220 ±3.6% | 221 ±3.3% | 135 ±3.6% | 76.4 ±5.4% | 69.8 ±3.4% | 58.7 ±4% | 51.2 ±3.7%
Remove a random row from 1,000 rows | 225 ±5.2% | 202 ±3.8% | 138 ±3.7% | 74.2 ±8.9% | 57.6 ±6.7% | 41.7 ±7.8% | 31.4 ±5.1%
Update 1000 rows | 119 ±1.9% | 127 ±3.3% | 123 ±2.8% | 50.7 ±0.4% | 55.5 ±6.7% | 42.3 ±5.2% | 33.2 ±6.1%
Select a random row from 1,000 rows | 210 ±5.9% | 208 ±5.7% | 123 ±5.3% | 72.5 ±7.5% | 66.6 ±7.2% | 46.8 ±6.3% | 45.4 ±4.9%
Swap 2 rows for table with 1,000 rows | 224 ±3.5% | 198 ±4.4% | 118 ±5.6% | 73.5 ±8.2% | 57.9 ±8.9% | 46.8 ±6.3% | 31.6 ±5.8%
Geometric mean | 105.7 ±3.5% | 98.8 ±3.9% | 74.3 ±3.9% | 44.1 ±3.7% | 39.8 ±4% | 29.7 ±4.7% | 23.1 ±3.4%

Table 3. JS Framework Benchmark compatible benchmarks tested on
Chrome/94.0.4606.71. Green colors signify more operations per second, and red
colors signify fewer operations per second. Million.js ranked the second-best
scores overall in the benchmark of seven implementations with the geometric
mean of benchmarks at 98.8 operations per second.

The geometric mean of benchmark results is visualized in Figured 6.
Additionally, due to uncertainties with benchmark measurements, an average
deviation of 1.856 operations per second is also visualized. The DOM method
has the highest operations per second of the implementations tested, with a
geometric mean of 105.7 operations per second. Million.js is relatively
comparable to the DOM method and significantly outperforms the latter
implementations by 133% to 300%.

Additionally, Million.js has a much smaller bundle size at 1.1 kilobytes
compared to snabbdom and virtual-dom, but slightly larger than tiny-vdom at
0.58 kilobytes (see Figure 1). Since Million.js has a composable API, the
bundle size can become even lower through tree shaking and dead code
elimination (Obbink et al., 2018).

Figure 6. Geometric means of render speeds for benchmarks measured in
operations per second. Green colors signify more operations per second, and
red colors signify fewer operations per second (higher is better). Million.js
ranked the second best scores overall in the benchmark of seven
implementations, ranging from 133% to 300% faster-rendering speed.

###  4.2. Devtools Performance Flamegraph

We used the Chrome Devtools Performance benchmark (Saverimoutou et al., 2018)
to measure the load time of common interactive UI-like feeds
222https://github.com/aidenybai/million-delta-react-test. The benchmark tests
the total scripting time of different implementations to add 5000 DOM nodes to
the UI consecutively with one iteration after a warm-up round. We tested four
different implementations: Direct DOM updates, Delta rendering, Keyed
rendering, Virtual DOM rendering. The latter three implementations use the
React compatibility layer for Million.js, with Delta rendering using reactive
data primitives, keyed rendering using the keyed nodes optimization, and
virtual DOM being the default method of rendering Million.js provides.

As seen in Figure 7, direct DOM updates yielded the lowest scripting time at
149 milliseconds, with a small and consistent call stack. Delta rendering
yielded a scripting time of 172 milliseconds and is 15% longer compared to
direct DOM updates. Delta rendering showed characteristics of batching, with
some scripting occurring before and after delta calculation shown in green and
pink, which is what allows for such low overhead compared to direct DOM. The
keyed rendering yielded a scripting time of 4022 milliseconds, with a large
call stack due to diff computations, and is 2238% longer than direct DOM
updates. The virtual DOM rendering yielded 4036 milliseconds, with slightly
more memory allocation than keyed rendering, and is 2247% longer than direct
DOM updates.

While direct DOM updates yielded the lowest scripting time of all
implementations tested, Delta rendering was a fast implementation that used
the React compatibility layer, being 24 times faster than virtual DOM
rendering.

Figure 7. Devtools performance results of node append benchmark of DOM update,
Delta render, keyed render, and virtual DOM (VDOM) render tested on
Chrome/94.0.4606.71. Benchmark for DOM update finished in 149 milliseconds,
Delta render in 172 milliseconds, 2997 milliseconds, VDOM render in 4036
milliseconds.

###  4.3. Real World Application Performance

To assess real-world performance, we integrated the React compatibility layer
into the Wyze Web View (Wyze, 2022) web application. The Wyze Web View runs on
Next.js and displays a live video feed of smart home cameras. The Wyze Web
View regularly receives thousands of unique sessions daily, and as a result,
is a critical component in the Wyze smart home ecosystem. One of the main
issues reported by many Wyze users was that the camera feed would take upwards
of several minutes to load cameras completely.

By migrating the Next.js project from React to Million.js, the Wyze Web View
experienced a 35.11% decrease in compute time taken to render all five cameras
as seen in Figure 8. The React-based implementation experienced a total
compute time of 10126 milliseconds and a scripting time of 7497 milliseconds,
and the new Million.js implementation experienced a total compute time of 6571
milliseconds, with a scripting time of 1650 milliseconds.

After migrating to the new Million.js implementation, an informal user study
was conducted on ten active Wyze Web View users. All reported that they
experienced a reduction in load time and less perceived lag with the new
improvements.

Figure 8. Wyze Web View compute time breakdown, left is Million.js’ React
compatibility layer and right is React. The React-based implementation
experienced a total compute time of 10126 milliseconds and the new Million.js
implementation experienced a total compute time of 6571 milliseconds.
Switching to Million.js resulted in a 35.11% faster load time.

##  5\. Discussion & Future Work

While the JS Framework Benchmark and the Devtools performance benchmark both
are good indicators of rendering speed and load time for large page
navigations (Mohammadi, 2017), the real-world application performance
benchmark is important for assessing the viability of Million.js in a product
application scenario.

In real-world web applications, it is the user code that often is the
bottleneck to rendering (Maragatham et al., 2018; Hardy, 2021; Souders, 2008).
For instance, the Wyze Web View uses much of the load time to establish a
connection to create video feeds. Furthermore, the real-world application
benchmark and the Devtools performance benchmark primarily measure load time
and don’t necessarily account for subsequent renderings, such as page
navigations. Despite this, well-maintained and standardized benchmarks, such
as the ones presented in this paper, can lead researchers to learn a lot about
how Million.js behaves.

There is still much work to be done in this area. Million.js is open source
and other researchers may use Million.js as a starting point for further
extensions like new JavaScript UI libraries or high-performance web
applications. In the future, more effort will go towards exploring more
complex optimization techniques during compilation like static analysis to
further improve load time and render speed for UI, increase compatibility with
the React ecosystem, and integrate Million.js into more real-world web
applications.

##  6\. Conclusion

We presented Million.js, a fast compiler-augmented virtual DOM for the web.
The initial objectives (see Table 2) of Million.js were to create a novel
virtual DOM implementation with quick load time, fast render speed, and to
make it easy for developers to use.

By leveraging an optimizing compiler and reactive data primitives to improve
the load time and render speed of UI, we were successfully able to achieve
fast load time and fast render speed. We conducted two benchmarks to measure
load time and render speed, showing that Million.js loads 23.47 times faster
compared to virtual DOM and renders 133% to 300% faster than existing virtual
DOM implementations.

We were also successfully able to increase developer use. Within one year,
Million.js received over 5.1 thousand stars, becoming the third most
starred333https://seladb.github.io/StarTrack-js/ virtual DOM library on GitHub
(see Figure 9), and was featured in major developer publications like
JavaScript Weekly, React Status, and React Newsletter. Furthermore, Million.js
is actively used in 80 open-source projects on GitHub. The GitHub repository
culminated in 1.5 thousand contributions, 166 pull requests, and 70 feature
requests.

Figure 9. Star history graph of Million.js, snabbdom, simple-virtual-dom,
hundred (previously tiny-vdom), virtual-dom. Million.js ranked as the third
most-starred virtual DOM library and is expected to be the most-starred
library in one year.

In the next decade, web users will continue to see improvement in load and
rendering speeds for better user retention and experience (Pavic et al., 2020;
Developers, 2008; Glynn, 2020). With internet usage growing rapidly in
developing nations (Wang et al., 2011) and through the Internet of Things (Sin
and Shin, 2016) market, fast UI performance has never been more important.

##  7\. Using Million.js

Million.js is an open-source JavaScript package that users can download using
NPM, a Node.js package manager. Million.js can be used in any JavaScript run-
time with the JavaScript DOM API. Information on how to download Million.js
and the source code can be accessed at https://millionjs.org.

##  8\. Acknowledgments

The work was supported in part by CHS MST Magnet program and Wyze Labs, Inc.
We thank Matthew Conlen, the Million.js open-source contributors, and user
study participants for their help and valuable input.

## References

  * (1)
  * Antonio (2015) Cássio de Sousa Antonio. 2015\.  Inside the DOM Abstraction.  In _Pro React_. Springer, 25–49.
  * Apostolico (1986) Alberto Apostolico. 1986\.  Improving the worst-case performance of the Hunt-Szymanski strategy for the longest common subsequence of two strings.  _Inform. Process. Lett._ 23, 2 (1986), 63–69.
  * Ast (2016) Markus Ast. 2016\.  Incremental DOM for Web Components.  _Studierendensymposium Informatik 2016 der TU Chemnitz_ (2016).
  * Bai (2022) Aiden Bai. 2022\.  tiny-vdom.  https://github.com/aidenybai/hundred
  * Ballal et al. (2015) Prathibha A Ballal, H Sarojadevi, and PS Harsha. 2015\.  Compiler optimization: A genetic algorithm approach.  _International Journal of Computer Applications_ 112, 10 (2015).
  * Company (2022) Astro Technology Company. 2022\.  Astro.  https://astro.build
  * Dale (2017) Tom Dale. 2017.  Compilers are the New Frameworks.  https://tomdale.net/2017/09/compilers-are-the-new-frameworks/
  * Degenbaev et al. (2016) Ulan Degenbaev, Jochen Eisinger, Manfred Ernst, Ross McIlroy, and Hannes Payer. 2016.  Idle time garbage collection scheduling.  _ACM SIGPLAN Notices_ 51, 6 (2016), 570–583.
  * Delcev and Draskovic (2018) Sanja Delcev and Drazen Draskovic. 2018.  Modern JavaScript frameworks: A survey study. In _2018 Zooming Innovation in Consumer Technologies Conference (ZINC)_. IEEE, 106–109.
  * Dermawan et al. (2020) KT Dermawan, IMD Maysanjaya, and IM Putrama. 2020\.  Encrypted real-time communication web (WEBRTC) using the principle of virtual DOM rendering cycles. In _Journal of Physics: Conference Series_ , Vol. 1516. IOP Publishing, 012004.
  * Developers (2008) Google Developers. 2008\.  Google I/O ’08 Keynote by Marissa Mayer.  https://www.youtube.com/watch?v=6x0cAzQ7PVs
  * Dinku (2022) Zerihun Dinku. 2022\.  React.js vs. Next.js.  (2022).
  * Domański et al. (2014) Adam Domański, Joanna Domańska, and Sebastian Chmiel. 2014\.  JavaScript frameworks and Ajax applications. In _International Conference on Computer Networks_. Springer, 57–68.
  * Esch (2014) Matt Esch. 2014\.  virtual-dom.  https://github.com/Matt-Esch/virtual-dom
  * Gackenheimer (2015) Cory Gackenheimer. 2015\.  Jsx fundamentals.  In _Introduction to React_. Springer, 43–64.
  * Garrett et al. (2005) Jesse James Garrett et al. 2005\.  Ajax: A new approach to web applications.  (2005).
  * Gizas et al. (2012) Andreas Gizas, Sotiris Christodoulou, and Theodore Papatheodorou. 2012\.  Comparative evaluation of javascript frameworks. In _Proceedings of the 21st International Conference on World Wide Web_. 513–514.
  * Glynn (2020) Peter Glynn. 2020\.  Milliseconds Make Millions.  https://www2.deloitte.com/ie/en/pages/consulting/articles/milliseconds-make-millions.html
  * Golestani et al. (2019) Hossein Golestani, Scott Mahlke, and Satish Narayanasamy. 2019\.  Characterization of Unnecessary Computations in Web Applications. In _2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 11–21.
  * Greenberg and Krishnamurthi (2007) Michael Greenberg and Shriram Krishnamurthi. 2007.  Declarative, composable views.
  * Grov (2015) Marianne Grov. 2015\.  Building User Interfaces Using Virtual DOM.  (May 2015).
  * Hardy (2021) Paul David Hardy. 2021\.  Performance: The Third Pillar of Code Quality.  In _Improving the Quality of ABAP Code_. Springer, 257–343.
  * Harris (2018) Rich Harris. 2018\.  Virtual DOM is pure overhead.  https://svelte.dev/blog/virtual-dom-is-pure-overhead
  * Inferno (2020) Inferno. 2020.  Inferno Optimizations.  https://www.infernojs.org/docs/guides/optimizations
  * Iskandar et al. (2020) Taufan Fadhilah Iskandar, Muharman Lubis, Tien Fabrianti Kusumasari, and Arif Ridho Lubis. 2020\.  Comparison between client-side and server-side rendering in the web development. In _IOP Conference Series: Materials Science and Engineering_ , Vol. 801. IOP Publishing, 012136.
  * Islam Naim (2017) Naimul Islam Naim. 2017\.  ReactJS: An Open Source JavaScript Library for Front-end Development.  (2017).
  * Kaul (2020) Boris Kaul. 2020\.  ivi.  https://github.com/localvoid/ivi/tree/master/packages/ivi
  * Keil et al. (2015) Matthias Keil, Sankha Narayan Guria, Andreas Schlegel, Manuel Geffken, and Peter Thiemann. 2015\.  Transparent object proxies for JavaScript.  _arXiv preprint arXiv:1504.08100_ (2015).
  * Keller and Chakravarty (1998) Gabriele Keller and Manuel MT Chakravarty. 1998.  Flattening trees. In _European Conference on Parallel Processing_. Springer, 709–719.
  * Kennedy and Allen (2001) Ken Kennedy and John R Allen. 2001.  _Optimizing compilers for modern architectures: a dependence-based approach_.  Morgan Kaufmann Publishers Inc.
  * Kishore and Mahendra (2020) P Kishore and BM Mahendra. 2020.  Evolution of Client-Side Rendering over Server-Side Rendering.  _Recent Trends in Information Technology and its Application_ 3, 2 (2020).
  * Krausest (2015) Krausest. 2015.  JS Framework Benchmark.  https://github.com/krausest/js-framework-benchmark
  * Kumar and Singh (2016) Anurag Kumar and Ravi Kumar Singh. 2016.  Comparative analysis of angularjs and reactjs.  _International Journal of Latest Trends in Engineering and Technology_ 7, 4 (2016), 225–227.
  * Ladan (2015) Zlatko Ladan. 2015\.  Comparing performance between plain JavaScript and popular JavaScript frameworks.
  * Levlin (2020) Mattias Levlin. 2020\.  DOM benchmark comparison of the front-end JavaScript frameworks React, Angular, Vue, and Svelte.  (2020).
  * Livoras (2015) Livoras. 2015.  simple-virtual-dom.  https://github.com/livoras/simple-virtual-dom
  * Maragatham et al. (2018) G Maragatham, SN Aravind Balaji, K SaiKarthikeyan, V Gokulakrishnan, and M Siddharth. 2018\.  A Study on Performance Analysis for Different Wordpress and Hand Code Webpages. In _2018 International Conference on Smart Systems and Inventive Technology (ICSSIT)_. IEEE, 191–204.
  * Mariano (2017) Carl Lawrence Mariano. 2017\.  Benchmarking javascript frameworks.  _Benchmarking_ (2017).
  * MDN (2022) MDN. 2022.  Introduction to the DOM - Web APIs.  https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction
  * Meyerovich et al. (2009) Leo A Meyerovich, Arjun Guha, Jacob Baskin, Gregory H Cooper, Michael Greenberg, Aleks Bromfield, and Shriram Krishnamurthi. 2009.  Flapjax: a programming language for Ajax applications. In _Proceedings of the 24th ACM SIGPLAN conference on Object oriented programming systems languages and applications_. 1–20.
  * Mohammadi (2017) Rajab Mohammadi. 2017\.  Performance, and Efficiency based Comparison of Angular and React in a case study of Single page application (SPA).  (2017).
  * Muyldermans (2019) Daisy Muyldermans. 2019\.  How does the virtual DOM compare to other DOM update mechanisms in JavaScript frameworks?  (May 2019).
  * Myers (1986) Eugene W Myers. 1986\.  An O(ND) difference algorithm and its variations.  _Algorithmica_ 1, 1 (1986), 251–266.
  * Obbink et al. (2018) Niels Groot Obbink, Ivano Malavolta, Gian Luca Scoccia, and Patricia Lago. 2018.  An extensible approach for taming the challenges of JavaScript dead code elimination. In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 291–401.
  * Pavic et al. (2020) Bojan Pavic, Chris Anstey, and Jeremy Wagner. 2020\.  Why does speed matter?  https://web.dev/why-speed-matters/
  * Pawlik and Augsten (2016) Mateusz Pawlik and Nikolaus Augsten. 2016.  Tree edit distance: Robust and memory-efficient.  _Information Systems_ 56 (2016), 157–173.
  * Persson (2020) Morgan Persson. 2020\.  JavaScript DOM Manipulation Performance: Comparing Vanilla JavaScript and Leading JavaScript Front-end Frameworks.
  * Petersen (2016) Hillar Petersen. 2016\.  From Static and Dynamic Websites to Static Site Generators.  _university of TARTU, Institute of Computer Science_ (2016).
  * Ratanaworabhan et al. (2010) Paruj Ratanaworabhan, Benjamin Livshits, and Benjamin G Zorn. 2010.  $\\{$JSMeter$\\}$: Comparing the Behavior of $\\{$JavaScript$\\}$ Benchmarks with Real Web Applications. In _USENIX Conference on Web Application Development (WebApps 10)_.
  * React (2022) React. 2022.  React v18.0 – React Blog.  https://reactjs.org/blog/2022/03/29/react-v18.html
  * Saverimoutou et al. (2018) Antoine Saverimoutou, Bertrand Mathieu, and Sandrine Vaton. 2018\.  Web browsing measurements: An above-the-fold browser-based technique. In _2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 1630–1635.
  * Sin and Shin (2016) Dongig Sin and Dongkun Shin. 2016.  Performance and resource analysis on the javascript runtime for iot devices. In _International Conference on Computational Science and Its Applications_. Springer, 602–609.
  * Souders (2008) Steve Souders. 2008\.  High-performance web sites.  _Commun. ACM_ 51, 12 (2008), 36–41.
  * Suresh et al. (2017) Arjun Suresh, Erven Rohou, and André Seznec. 2017\.  Compile-time function memoization. In _Proceedings of the 26th International Conference on Compiler Construction_. 45–54.
  * Vercel (2016) Vercel. 2016.  Next.js.  https://nextjs.org
  * Vindum (2015) Simon Vindum. 2015\.  snabbdom.  https://github.com/snabbdom/snabbdom
  * Voutilainen et al. (2016) Jari-Pekka Voutilainen, Tommi Mikkonen, and Kari Systä. 2016\.  Synchronizing application state using virtual DOM trees. In _International Conference on Web Engineering_. Springer, 142–154.
  * Wang et al. (2011) Zhen Wang, Felix Xiaozhu Lin, Lin Zhong, and Mansoor Chishtie. 2011\.  Why are web browsers slow on smartphones?. In _Proceedings of the 12th Workshop on Mobile Computing Systems and Applications_. 91–96.
  * Wieruch (2017) Robin Wieruch. 2017\.  _The Road to React: Your journey to master plain yet pragmatic React.js_.  Robin Wieruch.
  * Wloka (2003) Matthias Wloka. 2003\.  Batch, batch, batch: What does it really mean. In _Presentation at game developers conference_.
  * Wyze (2022) Wyze. 2022.  https://view.wyze.com
  * You (2022) Evan You. 2022.  Vite.  https://vitejs.dev
  * Zakas (2010) Nicholas C. Zakas. 2010\.  _High Performance JavaScript_.  “O’Reilly Media, Inc.”.
  * Zou et al. (2014) Yunxiao Zou, Zhenyu Chen, Yunhui Zheng, Xiangyu Zhang, and Zebao Gao. 2014.  Virtual DOM coverage for effective testing of dynamic web applications. In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_. 60–70.