1035 Research theory/model A. Wu, F. Cheng, and H. Qu are with the Hong Kong
University of Science and Technology. E-mail: {awuac, fchengaa,
<EMAIL_ADDRESS>D. Deng and Y. Wu are with the State Key Lab of
CAD&CG, Zhejiang University. E-mail: {ycwu<EMAIL_ADDRESS>S. Liu is
with School of Software, Tsinghua University, China. E-mail:
<EMAIL_ADDRESS>Biv et al.: Global Illumination for Fun and Profit

# In Defence of Visual Analytics Systems: Replies to Critics

Aoyu Wu  Dazhen Deng  Furui Cheng  Yingcai Wu  Shixia Liu  and Huamin Qu

###### Abstract

The last decade has witnessed many visual analytics (VA) systems that make
successful applications to wide-ranging domains such as urban analytics and
explainable AI. However, those systems are often designed, developed, and
evaluated on an ad-hoc basis, provoking and spotlighting criticisms about the
research rigor and contributions within the visualization community. We come
in defence of VA systems by contributing two interview studies with VA
researchers to gather critics and replies to those critics. First, we
interview 24 researchers about criticisms for VA systems they have received
from peers. Through an iterative coding and refinement process, we summarize
the interview data into a list of 36 common criticisms. Second, we interview
17 researchers to validate our list and collect replies to those criticisms.
We conclude by discussing eight important problems and future research
opportunities to advance the theoretical and practical underpinnings of VA
systems. We highlight that the presented knowledge is deep, extensive, but
also imperfect, provocative, and controversial, and thus recommend reading
with an inclusive and critical eye. We hope our work can provide solid
foundations and spark discussions to promote the research field forward more
rigorously and vibrantly.

###### keywords:

Visual Analytics, Theory, Qualitative Study, Design Study, Application,
Theoretical and Empirical Research

K.6.1Management of Computing and Information SystemsProject and People
ManagementLife Cycle; K.7.mThe Computing ProfessionMiscellaneousEthics
Introduction The last decades have witnessed extensive and ever-growing
research interests on visual analytics. By combining automated data analysis
techniques with interactive visualizations, visual analytics facilitates an
effective understanding, reasoning, and decision-making on large and complex
datasets [27]. Since the IEEE Conference on Visual Analytics Science and
Technology (VAST) was founded in 2006, researchers have contributed many
systems for solving complex problems in wide-ranging applications, to which we
refer as VA systems. Many of those systems are outcomes of problem-driven
research, whose goal is to help real users solve real-world problems in
certain application domains [62]. The values of VA systems have been
demonstrated through successful applications and stories in high-impact
domains such as social media [83], sports [84, 1], urban analytics [19, 39]
and explainable AI [41, 56]. VA systems have also become an important and
impactful research field in visualization (Figure 1).

Despite the success of those application-oriented VA systems, their research
contributions and rigor have been extensively challenged, discussed, and
debated [45, 62, 79, 8]. Underlying those debates is the tension between the
impetus to create specific software artifacts and the drive of academic
research to produce general knowledge [54]. This tension raises the frequently
asked question of “what our visualization community can learn from the VA
system beyond solving the domain-specific problem”. Furthermore, the design
and validation methodology of VA systems is human-centered and thus
qualitative and subjective in nature. This nature might contrast with the
tropism in science and computer science that embraces quantification and
objectivity [45, 26].

The above discrepancies point out some practical problems in the research
field. For contributors, their process of planning, developing, validating,
and reporting VA systems is prone to diverse mistakes or pitfalls at different
stages [62]. The process is challenging because conducting research on VA
systems require a wide range of skills such as leveraging HCI approaches to
understand target users, implementing automated algorithms, and designing
effective visual designs [71]. For reviewers, assessing the quality of VA
system research requires judging and weighing the above aspects. This
assessing process tends to be subjective due to the lack of a shared ground
among reviewers regarding objective criteria for evaluating VA systems [79].

Figure 1: Research on VA systems accounts for 27% (30/111) of the full paper
published in IEEE Visualization Conference (VIS) 2021. Award-winning works
include (A) IRVINE [15], (B) Loon [33], (C) VBridge [12], (D) M2Lens [77], and
(E) [30].

We, therefore, seek to identify a common space of critics for VA systems and
corresponding replies to critics. We note that existing instructions on
assessing the broad scope of visualization research are not specific to VA
systems [36]. Besides, the current reflections and discussions on the vigor of
visualization design studies (e.g., [45, 62, 44]) are mainly constructed from
authors’ own engagement and experiences. While those discussions are
undoubtedly critical and insightful, we argue that surveying a broader scope
of VA researchers will lead to a more diverse, representative, and convincing
understanding.

We conduct two interview studies to reflect on VA systems in terms of their
common criticisms and corresponding replies. As shown in Figure 2, we start by
interviewing 24 researchers in charge of 47 VA systems by asking them, “what
are the criticisms you have received during peer-reviewing?” The interview
allows us to collect 257 instances of criticisms, which are further classified
into 36 common types through iterative categorization. In response to those
criticisms, we conduct another interview study with 17 researchers to validate
our classification and gather replies to each criticism. In doing so, we
observe insightful but occasionally conflicting opinions. We conclude with a
discussion of eight important problems and the implications of our interviews
for the future development of research on VA systems.

We position our work as a preliminary probe into assessment criteria for VA
systems. Thus, we expect that our work will benefit both VA contributors and
reviewers by offering an evidence-based reference for assessing VA systems.
We, however, emphasize that criticisms are intrinsically biased, changing,
controversial, and fallible, so is our result. It is not our intention to
advocate a golden template, but to construct a preliminary and debatable set
of criteria for assessing VA systems. We hope our work can provide foundations
and spark discussions to make the research field more rigorous and vibrant. We
make our interview data and other supplemental material available at
`missing`.

##  1 Related Work

We focus on research that primarily contributes a VA system for an
application. Thus, we review existing literature regarding theoretical
advances of visualization application and design study, visual analytics, and
empirical methods for understanding the field of visualization.

###  1.1 Visualization Application and Design Study

Sedlmair et al [62] formally introduced a visualization design study as ‘‘a
project in which visualization researchers analyze a specific real-world
problem faced by domain experts, design a visualization system that supports
solving this problem, validate the design, and reflect about lessons learned
in order to refine visualization design guidelines’’. Following this
definition, they further proposed a nine-stage methodology framework for
visualization design studies. Their well-cited framework has become a common
method of developing visualization systems for solving a domain-specific
application and inspired many alternative design methodologies such as design
activity framework [43], design by immersion [22], design study lite [68], and
data-first design study [52].

However, concerns and debates have raised about the research contributions and
rigor of design studies and applications. Sedlmair [61] characterized 7 types
of research contributions resulting from design studies. Meyer and Dykes [44]
questioned how a specific design study might generalize to and benefit other
visualization contexts, advocating the need for developing standards to
reflect on applied design studies to generate general knowledge. Weber et al
[79] argued for the benefits and contributions of visualization application
papers, but called actions to develop criteria for how application papers can
make clear and assessable visualization.

In response to those debates, Meyer and Dykes [45] developed a set of six
criteria for rigor in design studies: informed, reflexive, abundant,
plausible, resonant, and transparent. They took a deductive, top-down
perspective by drawing conclusions from established criteria and principles in
social science, resulting in a set of criteria that is high-level. In
response, we adopt an inductive, bottom-up approach by observing criticisms in
peer-reviewing to identify generality. Our resulting set of criteria is thus
low-level and specific.

###  1.2 Visual Analytics

A VA system is a software artifact applying visual analytics techniques. It is
common to decompose a VA system into two components, namely data processing
(data mining) and interactive visualizations [27], while the latter is often
further unfolded into visualization and interactions [8]. In this sense, VA
systems extend information visualization systems by highlighting the usage of
advanced data analysis algorithms to accomplish data analysis tasks [27].
Thus, VA systems are considered to be complex [57, 8].

This complexity poses unique challenges in evaluating VA systems. Scholtz [58]
analyzed reviews for entries to the 2009 VAST Symposium Challenge to develop
an initial set of guidelines for evaluating VA systems, which were further
expanded into metrics including accuracy, the analytics process, the
visualizations, and interactions [59]. Chen et al [8] proposed an ontological
framework to evaluate VA systems by analyzing their symptoms, causes,
remedies, and side effects. In addition to understanding metrics, multiple
studies aimed to survey evaluation methods. Drawing upon surveys about
evaluation methods in information visualization [31] and the whole
visualization field [26], Khayat et al [29] summarized seven common evaluation
methods for VA systems and discussed their validity, generalizability, and
feasibility.

We extend those discussions on evaluating VA systems to the assessment of
research on VA systems. Assessing the research values and rigor of VA systems
requires not only judging the resulting VA systems, but also the overall
scientific process of planning, designing, validating, and reporting VA
systems. We note that evaluating VA systems remains a challenging and
controversial issue, and present our findings of common criticisms on the
evaluation.

Figure 2: Our study consists of two phrases including (A) identifying common
criticisms and (B) gathering replies to each type of criticisms.

###  1.3 Understanding the Field of Visualization

It has been common to survey visualization literature to understand the field
of visualization. For example, Isenberg et al [25] analyzes the keywords in
visualization papers to identify research topics and trends, which is later
integrated into a meta collection of visualization literature [24]. Lee et al
[35] summarized 25 common contributions of visualization research. Besides,
researchers have contributed many surveys about specific issues of
visualizations such as evaluation methods [31, 26], design spaces (e.g.,
visualization tasks[60]), and trending topics (e.g., AI4VIS [80] and ML4VIS
[75]).

Another line of research focuses on surveying visualization researchers to
gather insights on trending topics such as immersive analytics [17] and big
data visual analytics [2]. Furthermore, some visualization approaches have
been developed to understand the research profiles [34] and career paths [78]
of visualization researchers.

Our work contributes a new practice of surveying visualization researchers in
the context of peer reviews. We leverage the dual roles of researchers as both
contributors (authors) and reviewers to ask the probably tasteless and
consequentialism-oriented questions ‘‘what are the criticisms you received
from peer reviews’’ and ‘‘how to react to those criticisms.’’ This practice
allows us to gather a rich and diverse corpus of professional opinions, that
is deep, extensive, but also nuanced and contradictory. We hope that our
practice could stimulate dialogue and debate around peer reviews that are
vital to our research community.

##  2 Methodology

Our work consists of two interview studies with researchers on VA systems. As
shown in Figure 2, we first interviewed researchers to collect instances of
criticisms that they received in their experience of contributing VA systems,
which were further classified into 36 categories through iterative open
coding. In the second study, we interviewed 17 researchers to understand their
replies to those criticisms.

###  2.1 Study 1: Gathering Criticisms

We conducted structured interviews with researchers on sample criticisms of VA
systems. The word ‘‘researcher’’ refers to researchers that have both
contributed and reviewed at least one paper that falls into our scope (i.e.,
work where the primary contribution is a VA system) in the IEEE Visualization
Conference (VIS) or IEEE Transactions on Visualization and Computer Graphics
(TVCG).

Participant. We interviewed 24 participants that were reported to have
authored a total number of 47 research papers as the first author. We
recruited them from personal and professional connections as well as emails.
We acknowledge that this recruitment method might introduce bias in our
results and harm external validity [37] (i.e., whether the results can
generalize to other situations and researcher groups). To make it clear, our
goal is to acquire an initial understanding of common criticisms by
interviewing a sufficient number of researchers, but not to develop exhaustive
understandings or conduct quantitative studies. Those studies might require
actions from the community and we hope that our initial results would propel
such actions.

Interview. We conducted structured interviews with individual participants,
asking them about their experience of receiving criticisms for their VA
systems. We were aware of concerns over the copyrights and anonymity of peer
reviews, and thus purposeful requested participants not to quote the reviews
but instead reported on aggregated understandings based on all their research
instances. We acknowledge that this setting might threaten internal validity
[37] (i.e., whether the results are trustworthy). For example, participants
might not precisely interpret and relay all criticisms. There were also
potential opportunities to conduct quantitative analysis to gain deeper
insights, such as analyzing occurrences and dependencies among criticisms, and
correlating criticisms to other factors like the acceptance result. However,
such quantitative analysis would require a reasonably large number of original
peer-review texts that are beyond our capacity.

Analysis. We iteratively open-coded interview data and applied an informative
structure to organize and report the findings, i.e., grouping the criticisms
by their corresponding components or sections in the manuscript (Figure 3).
The major consideration of choosing this grouping scheme is the intuitiveness
and the strong connections between writing components and design stages [62].
To overcome the phraseology inconsistencies of writing components in different
interviews (e.g., someone say ‘‘design requirements’’ and others say ‘‘design
goals’’), we summarized the common writing components from existing VA
research papers. We claim that grouping by writing components or sections may
not be the best way to organize findings. Still, it is the most practical and
objective way compared with other alternative grouping schemes (see the
supplemental material).

Three authors labeled and categorised those criticisms iteratively, and
discussed conflicts until reaching a consensus. This iterative process yielded
several local adjustments about splitting, merging, and removing items in the
list. For example, "not working with real experts" was split from "unclear
evaluation methods" and merged with "not using real data" into "lacking
realism". We provide logs of changes in the supplemental material to encourage
future discussions.

###  2.2 Study 2: Gathering Replies

With the above list of criticisms, our next goal was to validate the list and
provide replies in response to each criticism. Different from previous work
that offered suggestions on writing or reviewing visualization papers based on
authors’ experience (e.g., by Munzer [47], Stasko [65], and Elmqvist [16]), we
sought to be more objective and evidence-based by interviewing active
researchers.

Participant. We adopted a similar method to recruit participants as in the
previous study. We recruited 17 participants (5 females), including 6 senior
Ph.D. students, 3 postdoc researchers, 4 research scientists, and 4
professors. They had published a sufficient number of papers in IEEE VIS/TVCG
(Mean: 6.6, SD: 3.8). We additionally collected their career ages [72], which
were the number of years since the author published their first paper in our
scope (Mean: 5.3, SD: 1.9).

Interview. We conducted semi-structured interviews with individual
researchers, except for two group interview sessions with three interviewees.
Each interview session lasted for 1 to 2 hours. We asked participants pre-
defined questions but also encouraged them to share any opinions. Our
interview questions were designed surrounding two research questions
($\mathcal{Q}$1-2) and were classified into two types: one-off (OO) and
repeated for each type of criticisms (RE).

$\mathcal{Q}1$ \- Is our list mutually exclusive and collectively exhaustive?

  * •

Did you encounter this criticism when serving as VA system contributors or
reviewers, respectively? (RE)

  * •

Do you think this criticism is related to or can be merged with others? (RE)

  * •

Did you encounter any other criticisms that are not listed? (OO)

$\mathcal{Q}2$ \- How can VA researchers respond to those criticisms?

  * •

How important is it? What are the recurring pain points? How to avoid or
address it? (RE)

  * •

What do you think are the most important criticisms for the field? How to
improve? (OO)

Analysis and Results. We first analyzed the results for our first research
question. We found that every type of criticism was encountered by at least 2
contributors (Mean: 8.2, SD: 3.8) and at least 2 reviewers (Mean: 10.7, SD:
3.9). No participant reported new types of criticisms not covered in our list.
Therefore, we conclude that our list is reasonably collectively exhaustive
based on our sample.

However, we note many relationships among different types of criticisms. For
example, many participants linked usability with over-complex designs. Thus,
our list may not be mutually exclusive. Future work should study the topology
among the complex network of criticisms, e.g., by Streeb et al [66]’s
approach.

We analyze the remaining interview data and discuss them for each type of
criticisms in section 3. We use representative quotes from interviewees in the
second interview study (denoted P1-17) throughout the section to support our
claims.

##  3 Criticisms and Replies

In this section, we discuss each of the criticism and corresponding replies
according to our interviews. To guide our discussion, we follow the order as
in Figure 3.

Figure 3: Eight components of VA systems manuscripts. While the overall
structure is sequential, each component is linked with others.

C-1: Lacking links of coherence and consistency.

As shown in Figure 3, the eight components are ordered sequentially but
interlinked. Missing the links of coherence and consistency is the first
issue. There are diverse cases of inconsistency: claims about novelties are
not substantiated by comparisons with related work; participants in the
evaluation do not conform to proposed user groups in problem abstraction and
task analysis.

Tackling this problem would require ‘‘following a systematic approach to
design, develop, and validate VA systems’’ (P3) and ‘‘logical writing’’ (P6).
P4 advocated that researchers clearly illustrate the connections between
domain problems, analysis tasks, VA designs, and evaluations. To this end, he
suggested using ‘‘layered graph models’’ to guide and report research
processes, where nodes in each layer represent different components.
Correspondingly, one can draw edges to denote relationships, e.g., a view
accomplishes some tasks, enabling graph-based analysis such as conducting
coverage tests and finding isolated nodes. We suggest future research to
propose formalisms and evaluate them to progress the theorization of VA
systems.

###  3.1 Introduction

The introduction component results from careful consideration and organization
of the overall research during iterative paper writing. Thus, issues in this
component are high-level and linked with other sections. As such, they need to
be considered throughout the whole research. We hope not to mislead our
readers that they are pure writing issues. Instead, we suggest pondering them
iteratively throughout the design, development, and validation of VA systems.

C-2: Unclear relevance to visual analytics.

The first step of reporting research results is to provide the background
information that motivates the research. In our context, it is important to
articulate why this is a visual analytics problem. A common motivation
scenario is that ‘‘automated methods do not solve the problem’’ (P10),
bringing the need to ‘‘integrate computational methods with humans through
interactive visualizations’’ (P6) to make decisions. That said, visual
analytics problems are not about ‘‘visualizations of computed results’’ (P6).
Furthermore, it could be a pitfall by simply stating that there is no prior VA
system for the domain problem. No previous work does not necessarily lead to
the necessity of VA systems, since it can also because the domain problem is
not challenging. Thus, it is essential to articulate the challenges of domain
problems and the limitations of existing solutions.

C-3: Vague/over-claimed contributions.

The ending part of an introduction is often dedicated to a brief overview of
the contributions. This has been a common practice according to our survey,
i.e., only one paper [73] does not list the contribution. A clear statement of
contributions is particularly vital for VA systems research, whose
contribution types can be very diverse. Besides, the stated contributions need
to accord with the targeted problems. Because VA systems are often designed
for specific problems, one should not ‘‘over-claim the scope of application’’
(P12).

Table 1: Classifications and statistics of contribution types found in 30
surveyed papers in IEEE VIS 2021.

Contribution Type | Count | Example (Quoted from the Text)
---|---|---
Evaluation | 27 |  The evaluation of IRVINE together with six automotive engineers [15].
VA system/tool/prototype | 23 |  ThreadStates, an interactive web-based tool for state-based visual analytics of disease progression [76].
VA design/workflow/framework | 17 |  We introduce a risk-aware framework, namely, VideoModerator, to facilitate the efficient moderation of e-commerce videos [70].
Visual representation | 10 |  New visualization methods were developed to assist users to further understand critical factor data [42].
Design study (Problem abstraction) | 9 |  We characterize the problem domain of visual analytics of time-varying effects of multiple factors on academic career success [78].
Data mining algorithm | 8 |  We propose a dynamic clustering algorithm to enable the efficient clustering of fast-paced incoming streaming data [30].
Open-sourced implementation | 3 |  An open-sourced, web-based implementation ... (omitted) [53].
Reflection | 2 |  A start-to-end description of the lessons learned from this successful, multi-site remote collaboration [21].
New domain and problem | 2 |  The first visual analytics framework for diagnosing and improving deep semantic segmentation models ... (omitted) [23].
Data model | 2 |  A technique to model users’ analytic behavior from interactions with the data [49].
Dataset | 1 |  A dataset of scraped metadata from 59, 232 academic articles [50].

  * •

Remark: One research paper [73] does not explicitly claim the contribution and
is excluded from this analysis.

C-4: Unclear contributions to the VA community.

We observe concerns regarding how the contributions are relevant to the VA
community, e.g., ‘‘what our community can learn from this research beyond
solving the domain-specific problem’’? This criticism sparked extensive
discussions during our interviews, promoting us to survey what contributions
had been claimed (see the supplemental material for details). As shown in
Table 1, the most frequent claimed ones are the evaluation (27), followed by
VA systems/tools/prototypes (23), and VA design/flow/framework (17). Those
contributions are intrinsically specific to the domain problem. Our
interviewees generally agreed that solving the domain problem is the primary
contribution, but they also expressed a clear need for ‘‘generating new
knowledge for visualization researchers’’ (P6).

What might generalize to other problems are novel visual representation (10)
and data mining algorithms (8) that are capable for certain data types and
analytical tasks. Characterizing the domain problem (9) might also inspire
future research in the domain, especially when applying to application domains
that not previously appeared in the visualization literature (2), i.e., ‘‘It
is a significant contribution to firstly apply VA systems to a kind of
application domains’’ (P2,4).

There are other opportunities for providing general knowledge for
visualization researchers, including open-sourced tools (3), reflection (2),
data model (1), and dataset (1). Our interviewees also outlined other
possibilities, including generalized ‘‘advice for the visualization design’’
(P6) and demonstration of the generalizability to other datasets or
application domains (P12,14). We recommend our readers think creatively and
broadly about other possible valuable contributions [35].

C-5: Unclear novelty of VA systems.

The novelty of the VA system plays an important role, i.e., ‘‘a grab-bag of
existing techniques is hard to be appreciated by visualization researchers’’
(P1). The novelty might not necessarily be novel data mining techniques or
visual designs, but instead ‘‘lies in the workflow, in particular how the VA
workflow is integrated into the domain workflow’’ (P5). Similarly P16
commented, ‘‘I usually consider the paper has sufficient novelties if it has
novelty in any part of the VA pipeline, e.g., data processing, user
interaction, visual design, user study experiment design and etc.’’ To
demonstrate the novelty, it is often necessary to ‘‘qualitatively compare with
existing VA systems’’ (P12). We will discuss quantitative comparisons in
subsection 3.7.

###  3.2 Related Work

Composing the related work section prompts authors to identify relevant
research topics, survey and summarize publications to shed light on gaps and
articulate novelties.

C-6: Missing related work.

Missing related work is a common criticism. Although it is often considered as
a minor issue, it requires thorough understandings of the core contributions
and relevant research topics. This can be difficult since a VA system concerns
a wide range of topics, such as the domain problem, automated and interactive
VA solutions to the domain problem, and visualizations for the abstract data
(e.g., text visualization) and analytical tasks (e.g., visual cluster
analysis). P15 suggested building a database to organize existing VA systems
‘‘from different perspectives such as application domains and data types’’ to
help communicate the differences and commonalities.

C-7: Inadequate discussion about related work.

Writing the literature review is not an enumeration of relevant publications,
but instead a critical organized account of the current state of research and
knowledge. The results need to be synthesized into a summary of what is known
and unknown in the research field, and how the new VA system advances the
field. Lacking the depth of critical discussions would lead to an issue that
fails to inform readers of the current research frontier. Besides, it is
crucial to provide ‘‘a qualitative comparison with closely related systems’’
(P10).

###  3.3 Problem Abstraction

This section formally signifies the entrance to the design study, i.e.,
working with domain experts to understand the domain problem, collect user
requirements, and abstract the data and analytical tasks to inform the design
of VA systems.

C-8: Unclear domain experts or target users.

An immediate question arises regarding the profiling of domain experts and
target users. We differentiate end-users from domain experts. For example,
when designing a learning tool for students, teachers play the role of domain
experts, helping researchers characterize domain problems. It should be made
clear whether they are real users or fictional personas, what are their
working fields and domains, and what are their required knowledge to interact
with the system. Regarding this criticism, P4 suggested that it could help to
clarify ‘‘their knowledge about visualization’’ since VA systems can need
visualization expertise. While this criticism seems a writing problem, P11
stressed that ‘‘without a clear definition of users, the design requirements
and evaluation can be doubtful.’’

C-9: Unclear methodology and methods for problem abstraction.

In addition to domain experts, it is important to adopt a systematic method
for problem abstraction and provide sufficient details. P5 emphasized that
this is a common problem, saying ‘‘many submissions do not detail this step.
Understanding the domain problem is difficult and requires iterative
collaboration with experts.’’

According to Sedlmair et al [62], methodology is like a recipe describing
‘‘strategy, plan of action, process, or design lying behind the choice and use
of particular methods‘‘ and methods are like ingredients. The common
methodology in our surveyed papers includes design study (e.g., [15, 12, 69]),
user-centered design and its extension (e.g., [55, 76, 21]), and the nested
model of visualization design [78]. Methods for understanding the domain
problems are primarily interviews (discussions and meetings) and literature
review. Other surveyed methods include contextualized design (e.g., one author
embedded himself in the domain experts’ research group for one year [33]),
workshops [33], formative and pilot studies (e.g., [12, 70]). We note that
those methods are not exhaustive and encourage readers to explore other
approaches such as ‘‘fly-on-the-wall’’ (P6).

In addition to methodology and methods, it is important to provide a
structured discussion about the domain problems. P11 commented, ‘‘writing this
part can take much time, especially for a new domain.’’ For instance, what are
the current workflows and practices of domain problems? What are the
challenges encountered by domain experts? Providing such information could
facilitate understandings by readers who are not familiar with the specific
domain, i.e., ‘‘why do experts have those problems’’ (P10,12). Furthermore,
because the design study is often iterative, domain problems can be revisited
and revised throughout the whole design process. In those circumstances, the
problems need to be linked with the design iterations.

C-10: Insufficient abstraction from domain to VA problems.

With the derived domain problems and requirements, it is essential to perform
data abstraction and task abstraction to inform the design of the VA system.
While domain problems are often expressed in domain-specific language, data
and visualization tasks need to be described in visualization language to
communicate the relevance to visualization researchers. Such abstraction could
make the visual design potentially applicable to other domain problems.

Many interviewees stressed the challenges for abstraction for many reasons:
different domain problems have varying levels of granularity which might be
difficult to abstract (P1); domain problems are even not well-defined (P4,8);
it is hard to differentiate between domain-specific and visualization-specific
terms (P2,6); and different papers describe similar visualization problems in
different languages (P11,15). Their comments call for actions to organize
exiting practice of problem abstraction to clarify the terminology, identify
common patterns and topology, and establish common vocabularies to guide
future research.

###  3.4 Data and Data Processing

Developing VA systems typically starts with processing the raw data and
developing models (i.e., data analysis or mining models) for automated
analysis [27]. We collectively refer to them as data and data processing. Data
processing is not standalone in VA systems but is tightly integrated with
interactive visualization. For this reason, we find it is difficult to
completely seclude criticisms in this component from those in visual design
(subsection 3.5) and system (subsection 3.6). In addition, we observe a large
number of technical problems that are closely related and specific to the
domain problems. Therefore, we exclude those specific technical problems from
our discussions.

C-11: Unclear definitions and explanations of data.

This section usually starts with explaining the data and data-related issues
such as definitions, metrics, and features. This is particularly important
when the datasets are not public and ‘‘the systems are not designed for common
type data’’ (P17). In addition to plain text description, a useful approach is
to ‘‘provide an example of the dataset in a table or figure’’ (P2). It is
suggested to describe characteristics of the abstract data, such as the data
type (e.g., relational or graph) and the number of dimensions. Such
information might help readers understand whether the VA system could
generalize to other datasets.

C-12: Missing technical details.

Following the data, one should describe technical details for the subsequent
data processing and mining process. The goal is to enable readers to
comprehend and replicate the methods by following the description. However, we
find this criticism to be common in our surveyed sample. Part of the reason is
the ‘‘curse of expertise’’ [20] that experts tend to overestimate their
ability to explain their areas of formal expertise. Another reason might be
the ‘‘limited page lengths that make it hard to provide all details’’ (P4). An
alternative approach is to describe details in the supplemental material or
provide source codes with documentation (P15).

C-13: Novel data processing algorithms.

As shown in Table 1, contributions of VA systems could include novel data
processing algorithms. However, we observe diverse opinions about such
contributions. On the one hand, some interviewees considered them to be an
‘‘addition’’ that were not necessarily well-targeted for the visualization
community and difficult to assess (P11,14), i.e., ‘‘if they are standalone
contributions, they should be split into another paper to other venues’’ (P6).
On the other hand, P5 appreciated such contributions, saying ‘‘model is a part
of visual analytics.’’ A helpful opinion to reconcile this conflict may be
‘‘it depends on whether the algorithm is closely integrated into the VA
systems, for example, to support real-time interaction for streaming data’’
(P6). We report on those controversies in an attempt to spark constructive
discussions. From our perspective, we echo the calls for ‘‘broadening
intellectual diversity in visualization research papers’’ [35] and argue that
the visualization community should stay inclusive and open to computational
analytics techniques. However, we note that such contributions might be
challenging to assess, thus advocating involving data processing professionals
during peer reviews.

C-14: Unclear choices of algorithms/models.

It is certainly legitimate to adopt existing methods for data processing and
mining. However, the choices should be justified, i.e., ‘‘the WHY question is
more important than the HOW question for researchers’’ (P13). The purpose is
to convince that the selected methods are solid and adequately applied to
solve the particular problem, i.e., ‘‘the decision should not be arbitrary’’
(P4). Suggestions include ‘‘abstracting the problem, for example, clustering
or frequent pattern mining’’ (P7), ‘‘clarifying the input and output’’ (P6),
followed by explaining candidates and decisions.

###  3.5 Visual Design

The core component of VA systems is the interactive visualization that
integrates automated data analysis with human analysis [27]. Because both
computational data analysis and interactive visualization can be regarded as
techniques, we find criticisms in visual design to be similar to that in data
and data processing, i.e., missing details, unclear novelties, and unclear
design choices. However, as visual designs are often the core interest to the
visualization community, we also find several specific criticisms.

C-15: Missing details of visual designs.

A clear description of the visual designs makes the complex design, sometimes
with multiple views, more comprehensible to readers, especially visualization
novices. To this end, a brief overview of the whole visualization is often
required. For each view, the description includes the input data, data
transformation, and visual mappings. This description can be followed by
explaining the interactions and coordination among views.

Nevertheless, it remains an open challenge to explain visual encodings of
common data visualizations effectively [85]. It is, therefore, far more
challenging to explain visual encodings of VA systems that are considerably
more complicated. Multiple interviewees emphasized that the figure should
provide legends (P2,4,7,8), and the demonstration videos should explain the
encodings (P12,13). We encourage empirical research to investigate guidelines
for explaining complex visualization design and making them more
understandable (e.g., [63]).

C-16: Unclear novelty of visual designs.

Although novel designs are not a necessary product from design studies, our
interviewees confirmed that the criticism of ‘‘lacking novelty of visual
designs’’ was still commonly raised in peer reviews. As a reaction, some
interviewees argued for de-emphasizing novel designs. P13 said, ‘‘in the early
years, I used to raise this concern. But now I think it is not critical.’’ P17
further challenged, ‘‘the goal of VA system is solve the problem. Novel
visualizations might not work in real-world scenarios.’’ P4 challenged on the
death of novel visualizations, commenting ‘‘our field has explored the design
space extensively. There does not leave much space for novel designs.’’ P10
further raised that ‘‘the judgment (of novelties) can be subjective. There
lacks a database to assure the novelty.’’ Those opinions underscore the
importance of continuing discussions on the assessment criteria for VA systems
and developing a database to communicate the state-of-the-art visualization
designs.

C-17: Unclear rationale and justifications of visual designs.

We find this criticism to be the most common one in our survey. This finding
is not surprising since visual design is of core interest to the visualization
community, i.e., ‘‘VA is a design problem. Without justifications, it is hard
to convince’’ (P12). However, justifying visual designs requires solid
argumentation, spanning diverse issues such as whether the visual channels are
effectively encoded, whether the design is mostly approximate for the task,
whether designs are consistent among views, and what are the potential
pitfalls (e.g., visual clutter). These multiple-criteria decision-makings
often lead to trade-offs that require careful consideration and strong
justification.

Despite the importance as seen in peer reviews, our interviewees suggested a
lack of theoretical research on how to justify visual designs and judge the
rigor. They had adopted and seen a variety of methods, such as referring to
visualization design guidelines and well-established principles, providing
evidence that the design was chosen and proven by experts, and discussing
alternative designs. However, they also expressed concerns that existing
practices are mainly based on argumentation or anecdotal evidence. Therefore,
it is important to develop systematic methods to improve the scientific rigor.
To that end, P15 said, ‘‘I wonder whether we can derive general knowledge from
existing justifications to find common design patterns’’.

C-18: Problematic visual designs.

An extreme case of the above criticism is that the visual design is considered
to be problematic. While concerns on C-17 are mostly ‘‘soft’’ (i.e., designs
require further justifications), this criticism is ‘‘hard’’ (i.e., designs
have to be revised and improved). For example, the design violates well-
established guidelines without necessary explanations, e.g., using rainbow
color maps. Such problems can be surfaced by consulting experience
visualization researchers.

C-19: Missing discussions about alternative designs.

As discussed in C-17, interviewees thought of discussing alternative designs
as one method for justifying design choices. Thus, it is sometimes criticized
that there lack justifications against alternative designs. The underlying
problem is whether the chosen visual design is the most approximate one among
many possibilities. Successful designs typically require starting with a broad
consideration space of possible solutions and subsequently narrowing proposal
space [62]. Thus, comparing with alternatives provides more evidence on the
validity of design decisions [40], i.e., ‘‘if we do not follow this practice,
the design progress might be unsystematic and weak’’ (P7). However, some
interviewees commented that this design progress was often neglected,
expressing the demands for better tools for facilitating the exploration,
comparison, and management of alternative designs.

C-20: Over-complicated visual designs.

We find a considerable amount of issues challenging that the visual design is
over-complicated. We observe mixed opinions from our interviewees. On the one
hand, some argued that the complexity was unavoidable due to the complex
domain problems (P1,2,4,10). The study should be treated as a success if
users’ workflows are improved with the complex VA systems; even the solution
looks not ‘‘elegant’’. Thus, they linked this criticism to insufficient
justifications (C-17).

On the other hand, more interviewees stressed that complexity could pose
threats to usability, i.e., VA systems may be too difficult for analysts to
learn and use (P3,5,6,11-15). They further imputed over-complexity to the
pursuit of novel designs, e.g., ‘‘novel designs are being more and more
complex’’ (P11). P12 commented, ‘‘our field has undergone the evolution from
simple to complex designs and I think now it is the time to reverse - to
reflect on existing designs and pursue simplification.’’ Therefore, it is
promising to study systematic methods for simplifying VA systems, e.g., by
finding unnecessary components through coverage testing. This perspective also
raises new questions about measuring the complexity of visualization and VA
systems. It might be helpful to refer to research on measuring the
memorability [4], understandability [63], and aesthetics [82] of
visualization.

###  3.6 System and Implementation

We refer to the system as the compilation of the data, algorithms, and visual
design. In this section, we discuss system-level criticisms. However, it
should be noted that system-level criticisms could root from component-level
criticisms (e.g., the choices of algorithms).

C-21: Lacking workflow overviews or system demonstration.

Because the system can be complicated, readers often need an overview of the
overall system and the workflow to understand how all components work
together. There are many methods, such as providing an illustrative figure and
describing usage scenarios as a walk-through for workflow. P17 said, ‘‘without
clear introductions to the connections between views, it is hard to understand
how the whole system work.’’ Besides, it could help to provide a system
demonstration in interactive software or videos to guide readers of the
system.

C-22: Uncertainty/stability/sensibility.

We observe the emergence of issues regarding uncertainty, stability, and
sensibility, especially to the choice of algorithm parameters. Those issues
can root in the chosen algorithms. For instance, different initial conditions
in heuristics such as t-SNE [74] could result in a significant difference,
posing threats to the reliability of the overall VA workflow. Drawing insights
on those problems could be instructive since they inform readers about
potential pitfalls and promote reproducibility.

Table 2: Evaluation strategies and collected data in 30 surveyed paper in IEEE
VIS 2021. Numbers in brackets indicate the counts and numbers in braces denote
the mean and standard deviation of the number of participants. References are
example instead of full instances.

Strategy |  Definition |  Collected Data
---|---|---
Observational studies (22) $\\{N:4.7\pm 3.5\\}$ |  By observing how real users work with the VA system in the field or in the laboratory, e.g., conducting case studies and expert interviews [12]. This strategy also includes longitudinal field studies [73]. |  qualitative subjective opinion (interview) (21), analytic workflow and insights* (15), quantitative subjective opinion (questionnaire) (6), quantitative objective data (3) (logs (2), task performance (1))
Usage scenarios (12) |  By describing how the VA system could be used by hypothetical users [55]. |  analytic workflow and insights* (12)
Demonstration (5) $\\{N:5.6\pm 4.4\\}$ |  By demonstrating the VA system to the users, i.e., users do not work with the VA system but might ‘‘explore it freely’’ [64]. |  qualitative subjective opinion (interview) (5), quantitative subjective opinion (questionnaire) (1)
Model experiment (5) |  By analyzing the model (i.e., data processing and data mining algorithms) performances [30]. |  algorithm performance (e.g., accuracy and running time) (4), insight quality (1)
Experimental studies (3) $\\{N:18.0\pm 8.5\\}$ |  By conducting controlled experiments where users work with the VA system versus baseline systems [70]. |  qualitative subjective opinion (interview) (2), quantitative subjective opinion (questionnaire) (2), quantitative objective data (2) (logs (1), task performance (1))

  * •

*: Analytic workflow and insights are considered to be qualitative and either subjective or objective [29].

C-23: Scalability.

Scalability is a commonly raised concern, questioning how the system,
including data processing algorithms and visualization, scales to the
increasing amount of data. This concern is mainly because visual analytics is
motivated by ‘‘the rapidly increasing amount of data’’ [27], which is further
promoted during the big data era [28]. It is, therefore, often expected that
VA systems could handle massive volumes of data. However, interviewees noted
that it depends on the typical data size in the application domains. Some
commented that it was not yet a very common practice to conduct quantitative
experiments for system scalability, advocating the use of software testing
strategies to surface the scalability problem (P5,9).

C-24: Generalizability.

Generalizability appears as a common concern, and many interviewees considered
it vital, i.e., ‘‘design study papers usually are more tailored to a specific
application. I generally would prefer systems that provide a good abstraction
of the data to make it generalizable to other applications.’’ (P16). On the
one hand, interviewees agreed that many VA systems were designed for a
specific domain problem, and ‘‘generalization is not their goals’’ (P5). On
the other hand, interviewees found poor generalizability a universal concern
of VA systems. Thus, it is important to study how to promote the
generalizability of VA systems so that ‘‘our field can continue building up
and accumulating knowledge’’ (P6). Many participants acknowledge that this
question was challenging and warrants future research.

C-25: Usability.

Usability assesses how easy the visual interface is to use. Criticisms on
usability are tightly interwoven with those on over-complicated visual designs
(C-20), doubting that the design is complex enough to raise questions on
learning curves and real-world usability. However, P6 noted that ‘‘few VA
systems pay enough attention to this problem and evaluate the usability in the
field through longitudinal studies’’, which is worthy of more research
attention and effort.

###  3.7 Evaluation

Evaluating visualization systems has been traditionally difficult [26], and it
is arguably even harder to evaluate complex VA systems. To guide our
discussions, we start with surveying existing evaluation methods for VA
systems (see the supplemental material for details). The survey serves as an
extension of existing literature from InfoVis [31], VAST [29], and VIS [26],
more focusing on evaluations of VA systems.

As shown in Table 2, observational studies in the field or in the laboratory
(e.g., case studies) are the most common evaluation strategy. In observational
studies, a variety of data can be collected and analyzed. We classify the data
according to two axes - qualitative versus quantitative and subjective versus
objective [29]. The second common approach is usage scenarios. Despite
previous calls to distinguish usage scenarios from the more formal case study
method [26, 60], we find several self-claimed case studies to be actual usage
scenarios. Thirdly, different from previous surveys, we find the use of
‘‘interview studies’’ where researchers demonstrate the system and cases to
the experts and collect feedback through interviews and questionnaires. In
those interview studies, experts do not work with the system, which is
different from observational studies. Fourthly, there are model experiments to
evaluate automated analysis algorithms. Finally, we note three controlled
experimental studies where the VA system is compared with baselines. A notable
difference is that such studies involve a larger number of participants than
observational studies.

C-26: Evaluations are incomplete and insufficient.

An insight from Table 2 is that evaluating VA systems could require multiple
strategies and collect various data for analysis. For instance, the surveyed
30 papers, on average, adopt 1.57 strategies and collect 2.47 types of
collected data. Critically, the diverse aspects of VA systems often can hardly
be tackled with a single evaluation strategy. For this reason, evaluations are
often criticized for being incomplete and insufficient. Therefore, it needs to
choose appropriate combinations of evaluation methods to validate the argued
contributions.

C-27: Unclear evaluation methods and protocols.

It is important to carry out evaluations using well-established protocols and
report the details, for both without or with human subjects. For the former,
one should clarify the benchmark datasets and measurements. For the latter,
the authors should clarify the participants’ demographics, user study
procedures, data collection, and coding scheme. Those details enable readers
to judge the methodological validity, promoting reproducibility and
comparability.

C-28: Lacking realism (e.g., real users and datasets).

Because VA systems are typically driven by real-world problems, they are
expected to be validated under real-world scenarios, e.g., with real end-users
and datasets. It is a legitimate form of evaluation to present usage scenarios
with hypothetical users to describe the workflow and present insights derived
from the VA systems. However, it is considered far more solid and convincing
to evaluate with real domain experts [26]. Some interviewees even argued that
‘‘usage scenarios are not evaluation’’ (P2,8). Besides, P15 advocated in-the-
field evaluations such as longitudinal studies to ‘‘gain more insight into
actual use’’ and understand ‘‘how the VA systems change the experts’
workflow’’.

C-29: Lacking comparison with baseline approaches.

The evaluation sometimes is criticized for not being comparative or
controlled. On the one hand, interviewees acknowledged the difficulties of
carrying out controlled experiments for evaluating VA systems. Those
difficulties have multiple reasons, such as the difficulties to identify fair
baselines in many domains (P1,3,6,12-15), the lack of benchmark datasets and
tasks (P5,6), ‘‘many systems are not open-sourced and thus hard to reproduce’’
(P4), and ‘‘it is unclear how to set users tasks for comparing VA systems’’
(P11). On the other hand, controlled experiments are a solid and convincing
method to convince the benefits of new VA systems. Thus, it is worthy to
develop, debate, test, and validate comparative evaluation methods, for
example, to break complex VA systems into components and conduct ablation
studies (P14), and to compare the final systems with early prototypes (P4).

C-30: Lacking quantitative evaluation and feedback.

Many evaluation methods such as case studies and expert interviews are
qualitative. Their goal is to maximize the realism of findings, thereby
understanding how the complex systems behave in the field [6]. However,
qualitative feedback is subject to experimenter bias, expectancy effects, and
other biases in human opinions. As such, they are insufficient and thus
‘‘become a weakness in many submissions’’. Quantitative evaluation methods
build on measurable variables to interpret the evaluated criteria. Integrating
qualitative and quantitative evaluation could make the evaluation more solid
and convincing.

A common method for quantitative feedback is questionnaires. However,
questionnaires could be less meaningful due to the limited sample size and
subjective bias. P5 and P11 suggested using objective quantitative data such
as logs, mouse movement, and eye-tracking data.

C-31: Analysis insights are suspicious or not new.

It is argued that the purpose of visual analytics is insights [9]. Thus, some
evaluation methods such as case studies seek to report on analysis workflow
and insights, that is, to demonstrate that users could derive insights from
the VA system. Correspondingly, it degrades the system if the discovered
insights are not new or doubtful. The interviewees suggested evaluating with
domain experts to gather feedback about the quality of insights and
demonstrate how the insights advance the understandings of the domain problem
(P4,7,11,13). We also note the use of experimental studies to evaluate the
quality of insights [53], which might inspire more rigorous approaches for
validating the insights.

C-32: Unclear interpretation of expert feedback.

Feedback from the end-users provides a valuable resource to critically reflect
on the VA system. However, it is not considered a rigorous evaluation by only
reporting on the positive feedback, that ‘‘might be cherry-picked’’ (P14).
Instead, the feedback should be analyzed systematically to provide insightful
discussions, especially how the design can be further improved and potentially
inform design in related domains.

###  3.8 Discussion and Conclusion

C-33: Insufficient discussions on limitations and implications.

The discussion should provide critical thoughts on the limitations and
implications to inform future work. However, it remains challenging to reflect
on visualization application research due to the lack of standards [44]. Thus,
many participants found it very challenging to compose reflections (P2,4),
since it requires ‘‘abstracting the experience for the particular VA system to
knowledge that generalizes to other domains’’ (P12). Thus, we encourage
thinking broadly about what implications can be beneficial to the
visualization community.

###  3.9 Others

We identify several issues that not specific to any of the eight components in
VA system manuscript (Figure 3).

C-34: Various presentation issues.

The most frequent one is the presentation issues, covering varying aspects
such as language, grammar, writing organization, and figures (I-35). In
particular, P6 emphasized that ‘‘figures are particularly important to show
professionalism in visualization.’’

C-35: Ethics.

Ethics appears as an emerging concern. It often relates to data privacy,
especially in sensitive domains such as videos, medicine, and social media
(e.g., [88, 87]). However, P10 commented that ethics seemed not to gain enough
attention in VA system research, in contrast with empirical research. She
advised researchers to seek approval from the human research ethics committee
in prior whenever applicable.

C-36: Open-source.

We observe multiple cases criticizing the research for not making the data and
codes public. During the interviews, we find this issue to be controversial.
Many interviewees agreed that there has been a tendency towards open-source
codes in computer science, and that open-source codes can improve independent
validation of the system, promote trust, and accelerate scientific progress.
However, they pointed out hindrances such as private data and ‘‘many VA
systems are prototypes’’ (P17). Those controversies underscore the importance
of continuing discussions on the reproducibility of visualization research
[18].

##  4 Reflection

From the interviews, we identify eight critical problems of research on VA
systems (Figure 4), which have sparked considerable interest, discussions, and
critical reflections. We summarize them as follows:

Usability. Many VA systems are complex and might be difficult for VA novices
to use. It could be helpful to explore systematic usability engineering
methods [51] to improve usability.

Evaluation. Evaluating VA systems mainly bases on case studies and expert
interviews, which are often considered insufficient and informal. There needs
to propose and test additional methods that can be feasible and more rigorous,
e.g., promoting a new paper type, namely deployment paper, to encourage
continuing longitudinal studies for VA systems that had been published as
application papers before.

Deployment. Many VA systems are developed in the laboratory and not deployed
in the field. Only a few systems claim actual deployment. More empirical
studies are needed to understand the challenges of field deployment,
especially in the industry, to increase the impacts.

Replicability. Replicability has been an increasingly important principle in
computer science. There are many open-sourced libraries for creating data
visualizations that significantly boost visualizations’ influence. However,
only a few VA systems publish the programs (e.g., [13, 33, 46, 67]). There
need more discussions and actions to promote the replicability of VA systems
[18].

Generalization. VA systems are often an ad-hoc solution. As a research field,
it is important to study how to derive general knowledge from many ad-hoc
systems, e.g., by reverse-engineering VA systems to identify design
(anti-)patterns and formalize ‘‘best’’ practices [5].

Theorization. The accumulation of research on VA systems creates new
opportunities to reflect on existing practices. Such reflection could lead to
refinements of methodologies, strengthening the theoretical underpinnings such
as new taxonomies, guidelines, formalisms, conceptual models, and theoretic
frameworks [10].

Intelligence. Designing, developing, and validating VA systems is challenging
and labor-intensive. The development of AI approaches offers new potential for
solving the above problems (e.g., mining design knowledge [38]) and automating
certain manual efforts (e.g., developing recommenders [81, 86]).

Education. Research on VA systems requires a wide range of professional
skills, such as leveraging HCI approaches to understanding target users,
implementing automated algorithms, and designing effective visual designs.
Those skills are broad and challenging to master. However, there exists little
educational material in training students to conduct and assess VA system
research [36].

In the following text, we discuss implications and future research
opportunities to promote the research field forward.

###  4.1 Assessment Criteria of VA System Research

Our study surfaces a common ground of assessment criteria for VA system
research. Those results provide a timely and evidence-based response to the
ongoing discussion about standards for rigor. Compared with Meyer and Dykes
[45]’s six criteria for rigor in visualization design study, our results are
low-level and multifaceted, concerning the designing, development, evaluation,
and reporting of VA systems. Therefore, assessing the overall research values
and rigor of VA systems would require a weighted consideration of those low-
level factors.

We see different weights adopted by researchers, and we are far from and
conceivably impossible to reach a consensus. Some argue for changing the
current favor of complex visualization designs to embrace actual usability,
which needs to be supported and evidenced by longitudinal deployment in the
field. Some other researchers advocate actionable measures from the community
to encourage replicable VA systems. Although many of those opinions seem
controversial and provocative, such discussions are undoubtedly constructive
and helpful. We hope our study will inspire and engage researchers to think
critically, express opinions, and continue discussions.

Figure 4: Eight challenging problems for research on VA systems.

###  4.2 Knowledge Base of VA Systems

Recent years have witnessed increasing efforts in building knowledge bases and
datasets for visualization research, such as VisPubData [24], VIS30K [7], and
VisImages [14]. They offer valuable resources to reflect on visualization
research, mine common patterns, and inform future research, e.g., to guide
layout designs in multiple-view systems [11] and to summarize frameworks for
problem abstraction [32].

Inspired by those efforts, we argue for the need for developing knowledge
bases of VA systems. Historically, Tycho’s records of the motion of Mars had
empowered Kepler to induce the laws of planetary motion. Decades of VA
research have contributed an ever-growing number of VA systems, which,
however, are often ad-hoc and specific. Developing a meta-collection of VA
systems will enable an inductive approach, that is, to summarize current
practices and identify common patterns and anti-patterns to theorize about
general knowledge.

This perspective brings out research challenges. Technically speaking, VA
systems are complex and diverse, making it challenging to describe and
represent them in a standard form. Besides, in contrast with the
aforementioned datasets, datasets of VA systems are unlikely to be in sole
text or image forms, but involve multimodal information such as image, text
description, and codes. This multimodal characteristic will require intensive
efforts on data cleaning and fusion.

Addressing those challenges will likely lead to valuable research
opportunities. For instance, there exist different practices and use of terms
in problem abstraction, such as design requirements, design rationales,
analytical tasks, and visualization tasks. This issue could prompt researchers
to summarize and develop a taxonomy of tasks or requirements in VA systems.
Moreover, the complex visualization designs offer opportunities for us to
revisit taxonomies of visualizations, which could inspire down-streaming
applications such as designing declarative languages for VA techniques.

###  4.3 A Software Perspective to VA Systems

The planning, execution, and reporting of VA system research has traditionally
been rooted in Munzner’s nested model [48] and design study methodology [62],
which are greatly informed and influenced by HCI research. Due to the
interdisciplinary nature of visualization research, we argue that an inclusive
vision to explore alternatives will likely enhance our theoretical and
practical underpinnings. In particular, we argue for a software engineering
perspective, given the strong focus on the production of VA system software.

First, software engineering research has proposed many languages such as UML
[3] to standardize the disparate systems and represent information such as
system structure, behavior, and interaction. In line with the ever-growing
number of VA systems, we envision that a formal language of VA systems will
provide a standard way to document VA systems and conceptual ideas. Such
standards could help researchers to identify common patterns in VA system
designs, making the VA system design and development progress more efficient.

Second, software testing is an objective and quantitative method for
validating and verifying software systems. As evaluating VA systems has been a
longstanding challenge, we ideate that software testing can potentially
provide an alternative approach for evaluating VA systems. For example, we
might run coverage testing [89] to identify components that are rarely used in
VA systems, providing potential opportunities to remove unnecessary components
and simplify complex designs. We hope this perspective could inspire
researchers to design and develop other rigorous and feasible approaches to
evaluate VA systems.

##  5 Conclusion and Future Work

Decades of research have produced a large number of VA systems. Those VA
systems are valuable and successful stories to specific application domains,
yet sometimes ad-hoc. Being ad-hoc does not mean unclear research values and
rigor, however. We must remember that Johannes Kepler’s laws of planetary
motion was induced from Tycho Brahe’s ad-hoc observations of Mars. We thus
come in defence of VA systems by arguing the need for reflecting on existing
VA system research to identify criticisms, problems, replies, and
opportunities. Such reflection could advance the theoretical and practical
underpinnings.

As one step towards this vision, we present two interview studies with VA
researchers. Our results are a list of 36 low-level criticisms and valuable
suggestions for responding to those criticisms. Such knowledge is evidence-
based and reliable but also imperfect and controversial. We adopt a linear
model to summarize and present the criticisms, which might be insufficient. A
next step towards deeper understandings is to develop a topological model for
untangling the complex network of criticisms and arguments.

Besides, our interviewees are not representative of the whole community, so is
the generated knowledge. We also plan to reach out to a broader range of
interviewees, such as researchers out of our field, industrial practitioners,
and the end ‘‘clients’’ of VA systems. We hope that our work can provide solid
foundations and spark constructive discussions, e.g., at panels, keynotes, and
workshops, to move the field forward more rigorously and vibrantly.

###### Acknowledgements.

We sincerely thank the participants in our interviews for their kind patience
and insightful viewpoints.

## References

  * [1] G. Andrienko, N. Andrienko, G. Anzer, P. Bauer, G. Budziak, G. Fuchs, D. Hecker, H. Weber, and S. Wrobel.  Constructing spaces and times for tactical analysis in football.  IEEE transactions on visualization and computer graphics, 2019.
  * [2] G. Andrienko, N. Andrienko, S. Drucker, J.-D. Fekete, D. Fisher, S. Idreos, T. Kraska, G. Li, K.-L. Ma, J. Mackinlay, et al.  Big data visualization and analytics: Future research challenges and emerging applications.  In BigVis 2020-3rd International Workshop on Big Data Visual Exploration and Analytics, 2020.
  * [3] G. Booch.  The unified modeling language user guide.  Pearson Education India, 2005.
  * [4] M. A. Borkin, A. A. Vo, Z. Bylinskii, P. Isola, S. Sunkavalli, A. Oliva, and H. Pfister.  What makes a visualization memorable?  IEEE Transactions on Visualization and Computer Graphics, 19(12):2306--2315, 2013.
  * [5] G. Canfora, M. Di Penta, and L. Cerulo.  Achievements and challenges in software reverse engineering.  Communications of the ACM, 54(4):142--151, 2011.
  * [6] S. Carpendale.  Evaluating information visualizations.  In Information visualization, pp. 19--45. Springer, 2008.
  * [7] J. Chen, M. Ling, R. Li, P. Isenberg, T. Isenberg, M. Sedlmair, T. Moller, R. S. Laramee, H.-W. Shen, K. Wunsche, et al.  Vis30k: A collection of figures and tables from ieee visualization conference publications.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [8] M. Chen and D. S. Ebert.  An ontological framework for supporting the design and evaluation of visual analytics systems.  In Computer Graphics Forum, vol. 38, pp. 131--144. Wiley Online Library, 2019.
  * [9] M. Chen, L. Floridi, and R. Borgo.  What is visualization really for?  In The Philosophy of Information Quality, pp. 75--93. Springer, 2014\.
  * [10] M. Chen, G. Grinstein, C. R. Johnson, J. Kennedy, and M. Tory.  Pathways for theoretical advances in visualization.  IEEE Computer Graphics and Applications, 37(4):103--112, 2017.
  * [11] X. Chen, W. Zeng, Y. Lin, H. M. Ai-Maneea, J. Roberts, and R. Chang.  Composition and configuration patterns in multiple-view visualizations.  IEEE Transactions on Visualization and Computer Graphics, 27(2):1514--1524, 2020.
  * [12] F. Cheng, D. Liu, F. Du, Y. Lin, A. Zytek, H. Li, H. Qu, and K. Veeramachaneni.  Vbridge: Connecting the dots between features and data to explain healthcare models.  IEEE Transactions on Visualization and Computer Graphics, 28(1):378--388, 2021.
  * [13] F. Cheng, Y. Ming, and H. Qu.  Dece: Decision explorer with counterfactual explanations for machine learning models.  IEEE Transactions on Visualization and Computer Graphics, 27(2):1438--1447, 2020.
  * [14] D. Deng, Y. Wu, X. Shu, J. Wu, M. Xu, S. Fu, W. Cui, and Y. Wu.  Visimages: a corpus of visualizations in the images of visualization publications.  arXiv preprint arXiv:2007.04584, 2020.
  * [15] J. Eirich, J. Bonart, D. Jäckle, M. Sedlmair, U. Schmid, K. Fischbach, T. Schreck, and J. Bernard.  Irvine: A design study on analyzing correlation patterns of electrical engines.  IEEE Transactions on Visualization and Computer Graphics, 28(1):11--21, 2021.
  * [16] N. Elmqvist.  Mistakes reviewers make.  https://sites.umiacs.umd.edu/elm/2016/02/01/mistakes-reviewers-make/, Feb 2016.
  * [17] B. Ens, B. Bach, M. Cordeil, U. Engelke, M. Serrano, W. Willett, A. Prouzeau, C. Anthes, W. Büschel, C. Dunne, et al.  Grand challenges in immersive analytics.  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), pp. 1--17, 2021.
  * [18] J.-D. Fekete and J. Freire.  Exploring reproducibility in visualization.  IEEE Computer Graphics and Applications, 40(5):108--119, 2020.
  * [19] N. Ferreira, J. Poco, H. T. Vo, J. Freire, and C. T. Silva.  Visual exploration of big spatio-temporal urban data: A study of new york city taxi trips.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2149--2158, 2013.
  * [20] M. Fisher and F. C. Keil.  The curse of expertise: When more knowledge leads to miscalibrated explanatory insight.  Cognitive science, 40(5):1251--1269, 2016.
  * [21] C. Floricel, N. Nipu, M. Biggs, A. Wentzel, G. Canahuate, L. Van Dijk, A. Mohamed, C. D. Fuller, and G. E. Marai.  Thalis: Human-machine analysis of longitudinal symptoms in cancer therapy.  IEEE Transactions on Visualization and Computer Graphics, 28(1):151--161, 2021.
  * [22] K. W. Hall, A. J. Bradley, U. Hinrichs, S. Huron, J. Wood, C. Collins, and S. Carpendale.  Design by immersion: A transdisciplinary approach to problem-driven visualizations.  IEEE Transactions on Visualization and Computer Graphics, 26(1):109--118, 2019.
  * [23] W. He, L. Zou, A. K. Shekar, L. Gou, and L. Ren.  Where can we help? a visual analytics approach to diagnosing and improving semantic segmentation of movable objects.  IEEE Transactions on Visualization and Computer Graphics, 28(1):1040--1050, 2021.
  * [24] P. Isenberg, F. Heimerl, S. Koch, T. Isenberg, P. Xu, C. D. Stolper, M. Sedlmair, J. Chen, T. Möller, and J. Stasko.  vispubdata.org: A metadata collection about ieee visualization (vis) publications.  IEEE Transactions on Visualization and Computer Graphics, 23(9):2199--2206, 2016.
  * [25] P. Isenberg, T. Isenberg, M. Sedlmair, J. Chen, and T. Möller.  Visualization as seen through its research paper keywords.  IEEE Transactions on Visualization and Computer Graphics, 23(1):771--780, 2016.
  * [26] T. Isenberg, P. Isenberg, J. Chen, M. Sedlmair, and T. Möller.  A systematic review on the practice of evaluating visualization.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2818--2827, 2013.
  * [27] D. Keim, G. Andrienko, J.-D. Fekete, C. Görg, J. Kohlhammer, and G. Melançon.  Visual analytics: Definition, process, and challenges.  In Information Visualization, pp. 154--175. Springer, 2008.
  * [28] D. Keim, H. Qu, and K.-L. Ma.  Big-data visualization.  IEEE Computer Graphics and Applications, 33(4):20--21, 2013.
  * [29] M. Khayat, M. Karimzadeh, D. S. Ebert, and A. Ghafoor.  The validity, generalizability and feasibility of summative evaluation methods in visual analytics.  IEEE Transactions on Visualization and Computer Graphics, 26(1):353--363, 2019.
  * [30] J. Knittel, S. Koch, T. Tang, W. Chen, Y. Wu, S. Liu, and T. Ertl.  Real-time visual analysis of high-volume social media posts.  IEEE Transactions on Visualization and Computer Graphics, 28(1):879--889, 2021.
  * [31] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale.  Empirical studies in information visualization: Seven scenarios.  IEEE Transactions on Visualization and Computer Graphics, 18(9):1520--1536, 2011.
  * [32] H. Lam, M. Tory, and T. Munzner.  Bridging from goals to tasks with design study analysis reports.  IEEE Transactions on Visualization and Computer Graphics, 24(1):435--445, 2017.
  * [33] D. Lange, E. Polanco, R. Judson-Torres, T. Zangle, and A. Lex.  Loon: Using exemplars to visualize large scale microscopy data.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [34] S. Latif and F. Beck.  Vis author profiles: Interactive descriptions of publication records combining text and visualization.  IEEE Transactions on Visualization and Computer Graphics, 25(1):152--161, 2018.
  * [35] B. Lee, K. Isaacs, D. A. Szafir, G. E. Marai, C. Turkay, M. Tory, S. Carpendale, and A. Endert.  Broadening intellectual diversity in visualization research papers.  IEEE Computer Graphics and Applications, 39(4):78--85, 2019.
  * [36] B. Lee, P. Isenberg, J. Stasko, D. Weiskopf, and R. Maciejewski.  Beyond paper types: How to evaluate and communicate vis research contributions, Oct 2019.
  * [37] L. Leung.  Validity, reliability, and generalizability in qualitative research.  Journal of Family Medicine and Primary Care, 4(3):324, 2015.
  * [38] H. Li, Y. Wang, S. Zhang, Y. Song, and H. Qu.  Kg4vis: A knowledge graph-based approach for visualization recommendation.  IEEE Transactions on Visualization and Computer Graphics, 28(1):195--205, 2021.
  * [39] D. Liu, D. Weng, Y. Li, J. Bao, Y. Zheng, H. Qu, and Y. Wu.  Smartadp: Visual analytics of large-scale taxi trajectories for selecting billboard locations.  IEEE Transactions on Visualization and Computer Graphics, 23(1):1--10, 2016.
  * [40] J. Liu, N. Boukhelifa, and J. R. Eagan.  Understanding the role of alternatives in data analysis practices.  IEEE Transactions on Visualization and Computer Graphics, 26(1):66--76, 2019.
  * [41] M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu.  Towards better analysis of deep convolutional neural networks.  IEEE Transactions on Visualization and Computer Graphics, 23(1):91--100, 2016.
  * [42] K. Maher, Z. Huang, J. Song, X. Deng, Y.-K. Lai, C. Ma, H. Wang, Y.-J. Liu, and H. Wang.  E-ffective: A visual analytic system for exploring the emotion and effectiveness of inspirational speeches.  IEEE Transactions on Visualization and Computer Graphics, 28(1):508--517, 2021.
  * [43] S. McKenna, D. Mazur, J. Agutter, and M. Meyer.  Design activity framework for visualization design.  IEEE Transactions on Visualization and Computer Graphics, 20(12):2191--2200, 2014.
  * [44] M. Meyer and J. Dykes.  Reflection on reflection in applied visualization research.  IEEE Computer Graphics and Applications, 38(6):9--16, 2018.
  * [45] M. Meyer and J. Dykes.  Criteria for rigor in visualization design study.  IEEE Transactions on Visualization and Computer Graphics, 26(1):87--97, 2019.
  * [46] Y. Ming, H. Qu, and E. Bertini.  Rulematrix: Visualizing and understanding classifiers with rules.  IEEE transactions on visualization and computer graphics, 25(1):342--352, 2018.
  * [47] T. Munzner.  Process and pitfalls in writing information visualization research papers.  In Information Visualization, pp. 134--153. Springer, 2008.
  * [48] T. Munzner.  A nested model for visualization design and validation.  IEEE Transactions on Visualization and Computer Graphics, 15(6):921--928, 2009.
  * [49] A. Narechania, A. Coscia, E. Wall, and A. Endert.  Lumos: Increasing awareness of analytic behavior during visual data analysis.  IEEE Transactions on Visualization and Computer Graphics, 28(1):1009--1018, 2021.
  * [50] A. Narechania, A. Karduni, R. Wesslen, and E. Wall.  Vitality: Promoting serendipitous discovery of academic literature with transformers & visual analytics.  IEEE Transactions on Visualization and Computer Graphics, 28(1):486--496, 2021.
  * [51] J. Nielsen.  Usability engineering.  Morgan Kaufmann, 1994.
  * [52] M. Oppermann and T. Munzner.  Data-first visualization design studies.  In 2020 IEEE Workshop on Evaluation and Beyond-Methodological Approaches to Visualization (BELIV), pp. 74--80. IEEE, 2020.
  * [53] H. Park, N. Das, R. Duggal, A. P. Wright, O. Shaikh, F. Hohman, and D. H. P. Chau.  Neurocartography: Scalable automatic visual summarization of concepts in deep neural networks.  IEEE Transactions on Visualization and Computer Graphics, 28(1):813--823, 2021.
  * [54] P. Parsons.  Understanding data visualization design practice.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [55] J. Pu, H. Shao, B. Gao, Z. Zhu, Y. Zhu, Y. Rao, and Y. Xiang.  matexplorer: Visual exploration on predicting ionic conductivity for solid-state electrolytes.  IEEE Transactions on Visualization and Computer Graphics, 28(1):65--75, 2021.
  * [56] P. E. Rauber, S. G. Fadel, A. X. Falcao, and A. C. Telea.  Visualizing the hidden activity of artificial neural networks.  IEEE Transactions on Visualization and Computer Graphics, 23(1):101--110, 2016.
  * [57] G. Robertson, D. Ebert, S. Eick, D. Keim, and K. Joy.  Scale and complexity in visual analytics.  Information Visualization, 8(4):247--253, 2009.
  * [58] J. Scholtz.  Developing guidelines for assessing visual analytics environments.  Information Visualization, 10(3):212--231, 2011.
  * [59] J. Scholtz, C. Plaisant, M. Whiting, and G. Grinstein.  Evaluation of visual analytics environments: The road to the visual analytics science and technology challenge evaluation methodology.  Information Visualization, 13(4):326--335, 2014.
  * [60] H.-J. Schulz, T. Nocke, M. Heitzler, and H. Schumann.  A design space of visualization tasks.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2366--2375, 2013.
  * [61] M. Sedlmair.  Design study contributions come in different guises: Seven guiding scenarios.  In Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp. 152--161, 2016.
  * [62] M. Sedlmair, M. Meyer, and T. Munzner.  Design study methodology: Reflections from the trenches and the stacks.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2431--2440, 2012.
  * [63] X. Shu, A. Wu, J. Tang, B. Bach, Y. Wu, and H. Qu.  What makes a data-gif understandable?  IEEE Transactions on Visualization and Computer Graphics, 27(2):1492--1502, 2020.
  * [64] H. Song, Z. Dai, P. Xu, and L. Ren.  Interactive visual pattern search on graph data via graph representation learning.  IEEE Transactions on Visualization and Computer Graphics, 28(1):335--345, 2021.
  * [65] J. Stasko.  Tips for being a good visualization paper reviewer.  https://jts3blog.wordpress.com/2016/12/23/tips-for-being-a-good-visualization-paper-reviewer/, 12 2016.
  * [66] D. Streeb, M. El-Assady, D. Keim, and M. Chen.  Why visualize? untangling a large network of arguments.  IEEE Transactions on Visualization and Computer Graphics, 27(3):2220--2236, 2021.
  * [67] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush.  Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks.  IEEE transactions on visualization and computer graphics, 24(1):667--676, 2017.
  * [68] U. H. Syeda, P. Murali, L. Roe, B. Berkey, and M. A. Borkin.  Design study lite methodology: Expediting design studies and enabling the synergy of visualization pedagogy and social good.  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), pp. 1--13, 2020.
  * [69] J. Tang, Y. Zhou, T. Tang, D. Weng, B. Xie, L. Yu, H. Zhang, and Y. Wu.  A visualization approach for monitoring order processing in e-commerce warehouse.  IEEE Transactions on Visualization and Computer Graphics, 28(1):857--867, 2021.
  * [70] T. Tang, Y. Wu, Y. Wu, L. Yu, and Y. Li.  Videomoderator: A risk-aware framework for multimodal video moderation in e-commerce.  IEEE Transactions on Visualization and Computer Graphics, 28(1):846--856, 2021.
  * [71] J. Thomas and J. Kielman.  Challenges for visual analytics.  Information Visualization, 8(4):309--314, 2009.
  * [72] N. Tovanich, P. Dragicevic, and P. Isenberg.  Gender in 30 years of ieee visualization.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [73] N. Tovanich, N. Soulié, N. Heulot, and P. Isenberg.  Miningvis: Visual analytics of the bitcoin mining economy.  IEEE Transactions on Visualization and Computer Graphics, 28(1):868--878, 2021.
  * [74] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(11), 2008.
  * [75] Q. Wang, Z. Chen, Y. Wang, and H. Qu.  A survey on ml4vis: Applying machine learning advances to data visualization.  arXiv preprint arXiv:2012.00467, 2020.
  * [76] Q. Wang, T. Mazor, T. A. Harbig, E. Cerami, and N. Gehlenborg.  Threadstates: State-based visual analysis of disease progression.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [77] X. Wang, J. He, Z. Jin, M. Yang, Y. Wang, and H. Qu.  M2lens: Visualizing and explaining multimodal models for sentiment analysis.  IEEE Transactions on Visualization and Computer Graphics, 28(1):802--812, 2021.
  * [78] Y. Wang, T.-Q. Peng, H. Lu, H. Wang, X. Xie, H. Qu, and Y. Wu.  Seek for success: a visualization approach for understanding the dynamics of academic careers.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [79] G. H. Weber, S. Carpendale, D. Ebert, B. Fisher, H. Hagen, B. Shneiderman, and A. Ynnerman.  Apply or die: On the role and assessment of application papers in visualization.  IEEE Computer Graphics and Applications, 37(3):96--104, 2017.
  * [80] A. Wu, Y. Wang, X. Shu, D. Moritz, W. Cui, H. Zhang, D. Zhang, and H. Qu.  Ai4vis: Survey on artificial intelligence approaches for data visualization.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [81] A. Wu, Y. Wang, M. Zhou, X. He, H. Zhang, H. Qu, and D. Zhang.  Multivision: Designing analytical dashboards with deep learning based recommendation.  IEEE Transactions on Visualization and Computer Graphics, 28(1):162--172, 2021.
  * [82] A. Wu, L. Xie, B. Lee, Y. Wang, W. Cui, and H. Qu.  Learning to automate chart layout configurations using crowdsourced paired comparison.  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), pp. 1--13, 2021.
  * [83] Y. Wu, S. Liu, K. Yan, M. Liu, and F. Wu.  Opinionflow: Visual analysis of opinion diffusion on social media.  IEEE Transactions on Visualization and Computer Graphics, 20(12):1763--1772, 2014.
  * [84] Y. Wu, X. Xie, J. Wang, D. Deng, H. Liang, H. Zhang, S. Cheng, and W. Chen.  Forvizor: Visualizing spatio-temporal team formations in soccer.  IEEE transactions on visualization and computer graphics, 25(1):65--75, 2018.
  * [85] L. Yang, C. Xiong, J. K. Wong, A. Wu, and H. Qu.  Explaining with examples lessons learned from crowdsourced introductory description of information visualizations.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [86] L. Yuan, Z. Zhou, J. Zhao, Y. Guo, F. Du, and H. Qu.  Infocolorizer: Interactive recommendation of color palettes for infographics.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [87] H. Zeng, X. Shu, Y. Wang, Y. Wang, L. Zhang, T.-C. Pong, and H. Qu.  Emotioncues: Emotion-oriented visual summarization of classroom videos.  IEEE Transactions on Visualization and Computer Graphics, 27(7):3168--3181, 2020.
  * [88] H. Zeng, X. Wang, A. Wu, Y. Wang, Q. Li, A. Endert, and H. Qu.  Emoco: Visual analysis of emotion coherence in presentation videos.  IEEE Transactions on Visualization and Computer Graphics, 26(1):927--937, 2019.
  * [89] H. Zhu, P. A. Hall, and J. H. May.  Software unit test coverage and adequacy.  ACM Computing Surveys, 29(4):366--427, 1997.