# Towards Value-Sensitive Learning Analytics Design

Bodong Chen  0000-0003-4616-4353 University of Minnesota159 Pillsbury Drive
SEMinneapolisMN, USA55455<EMAIL_ADDRESS>and  Haiyi Zhu  University of
Minnesota200 Union St SEMinneapolisMN, USA55455<EMAIL_ADDRESS>
(2019)

###### Abstract.

To support ethical considerations and system integrity in learning analytics,
this paper introduces two cases of applying the Value Sensitive Design
methodology to learning analytics design. The first study applied two methods
of Value Sensitive Design, namely stakeholder analysis and value analysis, to
a conceptual investigation of an existing learning analytics tool. This
investigation uncovered a number of values and value tensions, leading to
design trade-offs to be considered in future tool refinements. The second
study holistically applied Value Sensitive Design to the design of a
recommendation system for the Wikipedia WikiProjects. To proactively consider
values among stakeholders, we derived a multi-stage design process that
included literature analysis, empirical investigations, prototype development,
community engagement, iterative testing and refinement, and continuous
evaluation. By reporting on these two cases, this paper responds to a need of
practical means to support ethical considerations and human values in learning
analytics systems. These two cases demonstrate that Value Sensitive Design
could be a viable approach for balancing a wide range of human values, which
tend to encompass and surpass ethical issues, in learning analytics design.

learning analytics, values, value sensitive design, data ethics, social media

††journalyear: 2019††copyright: acmlicensed††conference: The 9th International
Learning Analytics & Knowledge Conference; March 4–8, 2019; Tempe, AZ,
USA††booktitle: The 9th International Learning Analytics & Knowledge
Conference (LAK19), March 4–8, 2019, Tempe, AZ, USA††price: 15.00††doi:
10.1145/3303772.3303798††isbn: 978-1-4503-6256-6/19/03††ccs: Information
systems Data analytics††ccs: Information systems Social networking sites††ccs:
Human-centered computing Visual analytics††ccs: Applied computing
Collaborative learning

##  1\. Introduction

Over the past few years, considerations of the ethical implications of
learning analytics have become a central topic in the field (Prinsloo and
Slade, 2017b). The increasing emphasis on ethical considerations in learning
analytics is situated within growing public concerns with algorithmic opacity
and bias, as well as awakening advocacy of algorithmic transparency and
accountability (ACM US Public Policy Council, 2017). In the field of learning
analytics, a rich body of theoretical, practical, and policy work has emerged
to promote the ethical collection and use of learning data. In 2016, the
_Journal of Learning Analytics_ published a special section featuring work on
ethics and privacy issues (Ferguson et al., 2016). Ethical frameworks, codes
of ethical practice, and ethical consideration checklists have been developed
(Sclater, 2016; Prinsloo and Slade, 2016; Drachsler and Greller, 2016). New
policy documents are also put forward by higher education institutions to
enhance transparency with regard to data collection and data usage.111See the
SHEILA project for examples: http://sheilaproject.eu/la-policies/

In contrast with the private sector, education as a social function also has a
moral responsibility to promote student success. Therefore, ethical
considerations in learning analytics need to go beyond issues that are
foregrounding the current public discourse surrounding surveillance and
privacy (Ferguson et al., 2016). Educational institutions have “an obligation
to act”—to effectively allocate resources to facilitate effective teaching and
learning through ethical and timely interventions (Prinsloo and Slade, 2017a).
Ethical practice in learning analytics entails considerations of not only
typical ethical issues but also factors and values pertinent to the larger
education system. Recently, Buckingham Shum (Buckingham Shum, 2017) proposes a
Learning Analytics System Integrity framework that goes beyond algorithmic
transparency and accountability to consider multiple pillars of a learning
analytics system including learning theory, algorithm, human–computer
interaction, pedagogy, and human-centered design. While learning analytics as
a field is rightfully giving more attention to ethical challenges with data
analytics, to foster ethical practice we also need effective methods to
consider ethical challenges in a manner that would engage perspectives from
different stakeholders while allowing education—as a complex system—to fulfill
its societal function.

To this end, this paper proposes to apply _Value Sensitive Design_ —a theory
and methodology initially developed in the field of Human–Computer Interaction
(HCI)—to support the design of value-sensitive learning analytics systems. We
posit that Value Sensitive Design can (1) guide researchers and practitioners
in the process of evaluating and refining existing learning analytics
applications, and (2) provide systematic guidance on developing new learning
analytics applications to more holistically address values pertinent to
stakeholders, educators, and the society.

The remainder of this paper is structured as follows. We first briefly review
work on ethics frameworks, algorithmic transparency, and accountability in the
field of learning analytics. Then we turn to the Value Sensitive Design
literature, with a focus on its application to the development of information
systems. After outlining our research goals, we introduce two studies in
Sections 3 and 4. We conclude this paper by discussing the importance of
considering values and value tensions in learning analytics design and
development.

##  2\. Background

###  2.1. Learning Analytics Systems: Ethics and Values

Learning analytics is defined as “the measurement, collection, analysis and
reporting of data about learners and their contexts, for purposes of
understanding and optimising learning and the environments in which it occurs”
(Long and Siemens, 2011, p. 34). By harnessing increasingly abundant
educational data and data mining algorithms, learning analytics aspires to
build applications to personalize learning based on individual progress,
predict student success for timely intervention, provide real-time feedback on
academic writing, and so forth (Siemens, 2013). As learning analytics
applications are increasingly integrated in the existing academic technology
infrastructure (e.g. student information systems, learning management systems,
student support systems), more educational institutions are becoming equipped
with strong, connected analytics capacity to better understand students and
hopefully to also better support student success. Empirical evidence
accumulated so far has indeed demonstrated promise of learning analytics in
supporting student learning (Huberth et al., 2015).

While learning analytics are garnering international interests across
different levels of education, ethical and privacy concerns are increasingly
mentioned in both public and scholarly discourse. Slade and Prinsloo (Slade
and Prinsloo, 2013) listed a number of ethical challenges in learning
analytics, including the location and interpretation of data; informed
consent, privacy, and de-identification of data; and the classification and
management of data. Scholars have organized a series of workshops at the
Learning Analytics and Knowledge (LAK) conference to collectively tackle
ethical challenges (e.g. Drachsler et al., 2016), leading to the development
of ethical principles, frameworks, and checklists to guide ethical research
and practice (Willis et al., 2016; Drachsler and Greller, 2016; Ferguson et
al., 2016; Rodríguez-Triana et al., 2016). Empirical work has also been done,
for example, to examine student perceptions of privacy issues with regard to
learning analytics applications (Ifenthaler and Tracey, 2016). Design work
following the Privacy by Design principles has shown promise in identifying
solutions to addressing privacy concerns (Verhagen et al., 2016). While
institution-level code of ethics are being created, efforts are also made to
help individual practitioners apply abstract principles and code of ethics in
real-world scenarios (Lang et al., 2018).

Undergirding these discussions of ethics and privacy are _values_ in learning
analytics systems. Generally speaking, value represents either “something
important, worthwhile, or useful” or “one’s judgment of what is important in
life” (Oxford English Dictionaries, 2018). For instance, privacy is a
fundamental human right, which holds not only value for a person but also
social and public values for democratic societies (Regan, 1995). So
safeguarding privacy has naturally become an important ethical consideration.
There are other values that are also important in learning analytics but often
go unstated. For instance, as educators we value student success, learner
agency, and equitable access to education (Ferguson et al., 2016), but these
values, despite their educational significance, are often neglected in the
discussion of data ethics. In other words, various values are not equally
recognized in current discourse, leaving some values and tensions among values
neglected or under-explored. To promote ethical practice in learning
analytics, we need means to interrogate the values that are often in tension
with each other. The learning analytics accountability analysis put forward by
Buckingham Shum (Buckingham Shum, 2017) considers system integrity from
multiple angles and holds promise for eliciting values beyond current thinking
on ethical issues. Take automated feedback on academic writing for example.
Values supported by an academic writing analytics tool include the accurate
detection of rhetorical moves based on linguistic features and usable feedback
interface that can facilitate student sense-making. However, the accurate
detection of rhetorical moves may demand more learner data and is hereby in
tension with the value of privacy foregrounded in ethical considerations. To
better support learning analytics design, the field needs strategies for
weighing competing values and deriving design trade-offs to mitigate value
tensions.

###  2.2. Value Sensitive Design

Stemming from an orientation towards human values, _Value Sensitive Design_
“represents a pioneering endeavor to proactively consider human values
throughout the process of technology design” (Davis and Nathan, 2013, p. 1).
In Value Sensitive Design, value is operationally defined as “what is
important to people in their lives, with a focus on ethics and morality”
(Friedman et al., 2017, p. 68). Value Sensitive Design offers a systemic
approach with specific strategies and methods to help researchers and
designers explicitly incorporate the consideration of human values into design
(Davis and Nathan, 2013; Friedman et al., 2017). It recognizes the complexity
of social life and attempts to illuminate _value tensions_ among
_stakeholders_.

Value Sensitive Design is a theory, a methodology, and an established set of
methods (Davis and Nathan, 2013). As a methodology, Value Sensitive Design
involves three types of investigations: conceptual, empirical, and technical
(Davis and Nathan, 2013). Together they could form an iterative process of
identifying and interrogating values in technology design. According to
(Friedman et al., 2013; Davis and Nathan, 2013), conceptual investigations are
primarily concerned with identifying direct and indirect stakeholders and
eliciting values held by different stakeholders. A conceptual investigation
can draw from philosophical and theoretical texts in order to conceptualize a
value at hand (e.g. trust, autonomy). Empirical investigations venture further
to collect empirical data from stakeholders about their perceptions of a value
and value tensions (e.g. privacy vs. utility) in a specific context. Both
qualitative and quantitative methods can be used in empirical investigations.
Finally, technical investigations focus on the designed technology itself and
seek to either proactively design new technological features to support values
or examine how a current design would support or undermine human values. These
three types of investigations can be integrated in an iterative design process
to holistically address human values.

Fourteen different methods can be drawn to support Value Sensitive Design
(Friedman et al., 2017). For example, direct and indirect stakeholder analysis
can lead to the identification of indirect stakeholders of transit information
tools (Watkins et al., 2013); fictional value scenarios are useful for
revealing values and tensions when developing mobile apps (Czeskis et al.,
2010); value dams and value flows could be a useful analytic method for
resolving value tensions among design choices—by removing a design choice
based on strong objection from a small percentage of stakeholders (the value
dams) or accepting a design option based on favorable reaction from a certain
portion of stakeholders (the value flow) (Czeskis et al., 2010). These methods
suit varied purposes of design and a project adopting Value Sensitive Design
can draw from multiple methods.

###  2.3. Overview of the Paper

To examine how Value Sensitive Design—the theory, methodology, and
methods—could be applied to learning analytics design, this paper reports on
two studies.

The first study is a conceptual investigation of a learning analytics
application that has already been developed. Applying two Value Sensitive
Design methods—stakeholder analysis and value analysis (Friedman et al.,
2017)—this study aimed to (1) identify key stakeholders of the developed
application, (2) elicit values and value tensions, and (3) derive design
trade-offs to be considered in future design iterations.

Moving beyond conceptually investigating an existing system, the second study
introduces a case of applying Value Sensitive Design to proactively consider
human values throughout a design process. The introduced case is about
developing a value-sensitive recommendation system for the Wikipedia
WikiProjects. Applying ideas from Value Sensitive Design, we and colleagues
(1) engaged stakeholders in the early stages of the algorithm design and used
stakeholders’ insights to guide the algorithm development, (2) iteratively
improved, adapted, and refined the algorithmic system based on the
stakeholders’ feedback, and (3) evaluated the recommendation system not only
based on accuracy but also stakeholders’ acceptance and their perceived
impacts of the system.

##  3\. Study I: A Conceptual Investigation of a Discussion Visualization Tool

###  3.1. Study Context

The first study was a conceptual investigation of a learning analytics tool
developed to visualize online discussions in a learning platform named
Yellowdig.222See https://yellowdig.com/ This study and the creation of
Yellowdig were contextualized within higher education’s interests in fostering
social learning and peer interactions among students. Branded as a social
media platform designed for education use, Yellowdig resembles many of social
network sites such as Facebook and Twitter. For example, it features a news
feed similar to that of Facebook; learners can contribute text or multimedia
posts to the news feed, where other learners can “like” or comment on each
other’s posts . By providing social-media features students are already
familiar with, this tool aims to facilitate social learning in online and
blended classes without intruding into students’ personal social networks.
Since its inception in 2014, Yellowdig has been adopted by a number of higher
education institutions.

As one of the early adopters of Yellowdig, Northwestern University developed a
Yellowdig Visualization Tool to visualize students’ peer interactions and
conceptual engagement on Yellowdig.333See
https://www.teachx.northwestern.edu/projects2016/gruber/ The design of this
tool was similar to many existing social learning analytics tools such as
SNAPP (Dawson et al., 2010), Threadz,444https://threadz.ewu.edu/ and CavanNet
(Chen et al., 2018). The tool was chosen for this study because it represented
a more recent design effort, has a fairly polished design, and has been
informed by participatory design workshops attended by students at the
university. In other words, this tool could well represent the state-of-the-
art of learning analytics applications designed for this particular context.

As described by its project website, the development of the Yellowdig
Visualization Tool was motivated by an interest in helping students and
faculty learn more about their online discussions by accessing Yellowdig data.
Prior research on educational use of Web 2.0 tools and online discussion
environments has demonstrated the need to support learner participation and
engagement from pedagogical, technological, and sociocultural angles (Greenhow
and Askari, 2017; Czerkawski, 2016). The Yellowdig Visualization Tool was one
example of supporting student use of Yellowdig through designing a new
learning analytics application, which is expected to be coupled with
pedagogical interventions. During the design process, the design team came up
with an initial prototype and engaged students in participatory design to
elicit their ideas about visualizations that could be useful for their
learning. The version of the tool analyzed in this study contained the
following key features (see Figure 1):

  * •

Multi-mode, multi-plex network visualization. The visualization tool
represents one class’s Yellowdig discussion as a multi-mode, multi-plex
network that contains multiple types of nodes (students, posts, and comments)
and multiple types of relations (posting, commenting). In this network, each
Yellowdig post (also called “pin”) is represented by a blue square filled
either by solid blue color (if the post contains pure text) or a thumbnail (if
the post contains a rich media object); comments on posts are shown as light
green circles; users are represented by grey user icons. Following a typical
network visualization heuristic, the size of a post nodes is correlated with
the number of connections it has. The whole network is plotted using a force-
directed layout, which keeps well connected nodes in the center and pushes
less connected nodes to the edge.

  * •

Node highlighting. When a node is chosen by the user, the visualization would
highlight the node’s “2.0 level ego network,” which comprises the chosen node
(i.e. the ego) and all other nodes that can be reached within two steps. The
content of the chosen node is also displayed in the top panel of the tool (see
Figure 1).

  * •

Visualization of temporal growth. A movie can be played to inspect both daily
and weekly growth of the discussion network. The placement of all nodes and
edges are fixed in advance. The temporal visualization adds nodes and edges
into the network in a step-wise manner.

  * •

Filtering mechanisms for instructor use. The instructor view of this
visualization tool provides additional mechanisms for filtering the network by
student names, post tags, and named entities (such as Uber) recognized by text
mining algorithms. These filters are displayed on the right panel of the
visualization for the instructor to explore the discussion network.

Figure 1. Yellowdig Visualization Tool. (_Note_ : Used with permission of the
Yellowdig Visualization development team at Northwestern University.)

In comparison with earlier social learning analytics applications, the
Yellowdig Visualization Tool is novel in several ways. First of all, in
comparison with one-mode networks adopted in most social network analytics,
the Yellowdig Visualization Tool turns discussion data into a multi-mode,
multiplex, temporal network that provides a more holistic picture of student
discussions. Its node highlighting feature allows students and the instructor
to inspect local network structures, and its filtering mechanisms support the
instructor to quickly identify discussion trends. To a great extent, the
visualization tool provides a unique view of the Yellowdig discussion, which
can facilitate sense-making activities by its users and serve assessment needs
of the instructor.

###  3.2. Methods

To study values in such a nascent learning analytics tool, we conducted a
conceptual investigation (Davis and Nathan, 2013) to identify its stakeholders
and elicit values and value tensions among identified stakeholders. This
conceptual investigation had two components:

  1. (1)

Stakeholder analysis. The first component was to identify the direct and
indirect stakeholders of this tool. According to (Friedman et al., 2017), this
analysis includes individuals who are in direct contact with this tool (direct
stakeholders) and those who may not interact with the tool directly but are
likely to be impacted by tool usage (indirect stakeholders).

  2. (2)

Value analysis. The second component was to identify and define the benefits,
harms, and values implicated by this learning analytics tool and to illuminate
value tensions embedded in the tool and its larger education systems. Values
at stake in this analysis include generic human values that represent “what is
important to people in their lives” (Friedman et al., 2017, p. 68), as well as
values endorsed by the educational literature.

This conceptual investigation was grounded in our knowledge of higher-
education classes in similar settings and aided by technical inspection of
this Yellowdig Visualization Tool. It is expected that through this study we
could come up with suggestions for considering identified values and design
trade-offs in future iterations of this particular tool as well as other
similar applications.

###  3.3. Findings

####  3.3.1. Stakeholders

In this context, the direct stakeholders included students and instructors
accessing the Yellowdig Visualization Tool. It was expected that the tool
would assist instructors in analyzing the data generated on Yellowdig to
inform their instructional decisions. The tool design also aimed to enhance
students’ mastery of the course content and provide them with another way of
engaging with the course material and with each other (see project website).
However, the adoption of such a learning analytics tool is often decided by
the instructor and its acceptance among students may vary. Therefore, students
may be further divided into those who choose to use the tool and those who
reject the tool for various reasons.

Because Yellowdig is a private social media platform, the visualization tool
is based on data generated by the class and is not accessible beyond the
class. The indirect stakeholders of this tool may include future cohorts of
the class whose discussion performance may be compared (explicitly or
implicitly) with the present cohort. Academic technology staff members are
also indirect stakeholders as they are charged with supporting technology use
and may be asked to provide additional technical support for this tool.
Student support services, such as academic advisers and academic analytics
teams, are also indirect stakeholders. For example, an academic adviser may be
contacted by the instructor when she finds from the tool that a student has
been inactive for a few weeks.

####  3.3.2. Values and tensions

The Yellowdig Visualization Tool aimed at providing another way of engaging
with the class discussion, enhancing student learning and social interaction,
and assisting the instructor to grasp discussion content in order to make
informed instructional decisions. However, through the conceptual
investigation we found these values supported by the tool can be in tension
with other values.

For such a learning analytics tool that can be used for assessment purposes,
student autonomy is an important value that is in tension with the tool’s
utility for assessment. The network visualization and node highlighting
features of the tool make it easy to identify posts and comments made by each
individual student. Even though such information is also accessible via the
original Yellowdig discussion interface, the visualization tool makes it more
readily accessible to all members of the class. These design features are
meant to support the values of student success, accountability, discussion
engagement, and usability, but they can also press students to participate
more actively and hereby hinder their autonomy as discussion participants.

In a same vein, because of the aggregation of student participation data,
student privacy as another important value is also at tension with these
intended benefits. Similarly, even though the visualization tool does not
collect new data from students, it reveals a bird-eye view of each student’s
participation patterns and could appear intruding to some students who are
less comfortable with such aggregated reporting.

Social well-being, a sense of belonging and social inclusion, is another
important value to be considered in this context. Students in the class—using
the tool or not—can reasonably expect to be free from embarrassment caused by
less active participation or less central status in the network. The force-
directed layout adopted to generate the network visualization pushes less
active participants to the edge and make them more identifiable by members of
the class. This design is intuitively understandable, is broadly applied in
network visualization, and supports pedagogical decisions by revealing to the
instructor students “at risk” of dropping out of Yellowdig discussion.
Nonetheless, students on the edge may feel embarrassed among peers should
their identities be revealed. This layout can potentially contribute to the
“rich-club” phenomenon in online discussions (Vaquero and Cebrian, 2013),
potentially channeling more attention to students in the center and leading to
the exclusion of less active students in future interactions.

There are also values that are implicated by algorithmic decisions embedded in
the visualization tool, including freedom from bias and self-image. For
instance, the force-directed layout algorithm inherently boost the status of
students who are more connected in terms of connection volumes and do not
consider the quality of their posts. Displaying the thumbnails of discussion
posts renders posts with multimedia objects more attractive in comparison with
posts with pure text, and can hereby direct varied attention to posts with and
without multimedia objects. Also notable is the choice of scaling posts based
on the number of comments, which directs more attention to posts that have
already attracted more attention. These differentiated treatments of
discussion posts embedded in the algorithms are linked to students’ self-
image. When a student’s node gets highlighted in the network visualization,
the highlighted ego network essentially represents what the student has
produced and how his/her participation is linked with peers in the community.
Being able to see all connections of a student at once can benefit the sense
of community especially when a student see herself well connected. However, it
may also pose questions to a student’s self-image if she is less connected
with peers in the community.

The visualization tool was designed to provide another way to make sense of
and engage in online discussions on Yellowdig. Values facilitated by the tool
include utility, usability, and ease of information seeking in Yellowdig. In
particular, the tool provides views of both the overall discussion structure
and micro, local interaction patterns that are not revealed by the Yellowdig
interface. The instructor view provides rich filtering mechanisms that are
useful for filtering discussion posts by students, tags, and/or named
entities. Being able to look at entity names, such as brands in a business
class, allows the instructor to quickly identify popular topics in student
discussions and make pedagogical decisions on, for example, whether to address
a specific topic in class. However, the fact that these advanced features are
reserved for the instructor is at tension with the values of epistemic agency
and fairness. One can argue that the advanced features can equally benefit
students, who also need to complete information seeking tasks to effectively
participate in Yellowdig discussions. Nevertheless, one may also argue that
providing these additional features places unnecessary cognitive overload and
burden on students.

###  3.4. Discussion and Implications

With aid from technical analysis of the Yellowdig Visualization Tool, the
conceptual investigation has revealed a number of values and value tensions
that have implications for future refinements of the tool. One premise of
discussing the implications is that the accommodation of these values should
not be solely achieved through technology design, but also the design of
pedagogical interventions (Wise and Vytasek, 2017). Nonetheless, based on
findings from the value analysis, we suggest the following possible design
trade-offs that could be considered in future technology design:

  * •

Considering the tension between utility (e.g. revealing a student’s
participation) and the values of students’ autonomy, privacy, social well-
being, and self-image, one design idea is to give each student the option of
not revealing his/her name to their peers in the network visualization. In
this way, students can choose to become less identifiable in the tool. Another
possible idea is to make student nodes non-clickable so that the node
highlighting feature is only limited to post and comment nodes. Both design
ideas will sacrifice utility of the visualization tool to support other values
such as privacy and social well-being.

  * •

To support the values of freedom from bias and self-image, future
implementation of the network visualization could consider adjusting
parameters of the force-directed layout or consider other graph drawing
algorithms (such as the circular, arc diagram, hive plot, and hierarchical
graph layouts). Here, design trade-offs are likely to involve interpretability
of a network visualization and its complications concerning student status in
the given sociocultural context. Empirical investigations of student
perceptions of different layouts could be necessary.

  * •

Considering the values of fairness, epistemic agency, and ease of information
seeking, the filtering mechanisms provided to the instructor could be also
considered for student access. Given the tension between privacy and ease of
information seeking, filtering by student names could still be reserved to the
instructor, while the other two filtering mechanisms (i.e. by tags or named
entities) can be made available for students.

The design ideas mentioned above are presented here to illustrate the promise
of Value Sensitive Design for the particular context, instead of disproving
the current visualization tool that is quite well-designed. These design ideas
are tentative and by no means comprehensive. Further empirical and technical
investigations are necessary in order to advance these design ideas.

##  4\. Study II: A Holistic Application of Value Sensitive Design to an
Algorithmic System of WikiProjects

###  4.1. Study Context

While the first study focuses on social learning in formal classes, the second
study was situated in the informal context of Wikipedia editing. Specifically,
this study was concerned with designing recommendation algorithms for group
formation in Wikipedia’s WikiProjects. A WikiProject organizes “a group of
participants… to achieve specific editing goals, or to achieve goals relating
to a specific field of knowledge.”555See
https://en.wikipedia.org/wiki/WikiProject Editing a Wikipedia article is
itself a learning and knowledge-creation process. Learning scientists have
recognized mass collaboration, on Wikipedia for instance, as an emerging
paradigm of education (Cress et al., 2016). Computer-supported collaborative
learning (CSCL) researchers have studied the social process of building
collective knowledge on Wikipedia, leading to findings about the intricate
relations between contributors and Wikipedia articles (Halatchliyski et al.,
2014). Indeed, coordinated editing in a WikiProject involves a great deal of
sense-making, negotiation, synthesis, and knowledge transformation. For this
reason, since its inception Wiki as a Web 2.0 technology has been widely
adopted in formal classrooms to support sophisticated collaborative learning
experiences (Greenhow and Askari, 2017). Therefore, facilitating knowledge
processes on the Wiki technology—and on Wikipedia in particular—is of interest
to learning analytics because Wikipedia editing entails learning, and
intelligent supports built for Wikipedia are transferable to Wiki-based
learning scenarios.

Prior work has illuminated that Wikipedia, like other social media
environments, now heavily depends on data-driven algorithmic systems to
support its massive-scale collaboration and governance (Geiger, 2017). On
Wikipedia, algorithmic systems have been used to support a variety of critical
tasks such as counter-vandalism, task routing, and the Wikipedia education
program. Many of these existing algorithmic systems are driven by “Big Data”
and supervised machine learning algorithms. Take predicting vandalism for
example. The first step of the task is to define a prediction target, i.e. a
malicious edit. The second step in the process is to use historical data,
often in large volumes, to train and validate machine learning models.
Finally, the validated models are applied to new edits in order to generate
predictive scores of being malicious edits. Such algorithmic systems play
important roles in the Wikipedia ecosystem.

However, the data-driven approach can lead to unintended biases in algorithms
and potentially negative impacts on the editor community. It is recognized
that the data-driven approach relies largely on historical human judgments,
which are subject to historical stereotypes, discrimination, and prejudices.
Using historical data to inform the future runs the risk of reinforcing and
repeating historical mistakes and thus fails to capture and incorporate human
insights on how the system can be improved for the future. Recent ethnographic
work of Wikipedia has revealed critical issues with current algorithmic
systems and calls for the study of sociocultural processes, such as newcomer
socialization, that are strongly impacted by the automated systems (Geiger,
2017).

Researchers have called for the development of systematic methods to detect
and mitigate biases in existing algorithmic systems (Sandvig et al., 2014). In
contrast with a reactive approach that attempts to address bias when it gets
exposed, we argue for an approach that would proactively consider human values
throughout the process of algorithm design in a principled and comprehensive
manner. One possible way of achieving this goal is to apply principles of
Value Sensitive Design (Friedman et al., 2017) to the creation of each
algorithmic system. This holistic approach, explicated in great detail
elsewhere (Zhu et al., 2018), engages stakeholders in the early stages of
algorithm development and incorporates stakeholders’ tacit knowledge and
insights into the creation of an algorithm system. By doing this, it is hoped
we could mitigate biases embedded in design choices and hence avoid
undermining important stakeholder values.

In this section, we briefly introduce a case of applying Value Sensitive
Design to designing a recommendation engine for group formation in Wikipedia’s
WikiProjects. Similar recommendation tasks are also explored in the MOOC
(massive open online course) and collaborative problem-solving contexts (Rosé
et al., 2015; Dowell et al., 2018). In contrast with Study I, this study was
focused on the recommendation algorithm, a key component of a learning
analytics system (Buckingham Shum, 2017). By doing so, we hope to illustrate
the possibility of attending values when building an algorithmic system’s sub-
component that is crucial but not observable from the user interface. Below,
we explain the process of applying Value Sensitive Design to the development
of the recommendation engine, and discuss the implications for the design of
learning analytics systems.

Figure 2. The interface of the prototype newcomer recommendations for
WikiProject organizers.

###  4.2. Methods

####  4.2.1. Study Context

Retaining and socializing newcomers is a crucial challenge for many volunteer-
based knowledge creation community such as Wikipedia. The number of active
contributors in Wikipedia has been declining steadily since 2007, due at least
in part to a sharp decline in the retention of new volunteer editors (Halfaker
et al., 2013). WikiProjects, which are groups of contributors who work
together to improve Wikipedia, serve as important socialization hubs within
the Wikipedia community. Prior work (Forte et al., 2012; Zhu et al., 2012)
suggests that WikiProjects provide three valuable support mechanisms for new
members: (1) enabling them to locate suitable assistance and expert
collaborators; (2) guiding and structuring their participation by organizing
tasks, to-do lists, and initiatives such as “Collaborations of the Week;” and
(3) offering new editors “protection” for their work, by shielding it from
unwarranted reverts and “edit wars.”

However, recommending new editors to WikiProjects is not a trivial task. In
the English version of Wikipedia alone there are currently about 2,000
WikiProjects, with on average 38,628 new users/editors registered to
participate each month. The goal of the case study was to create algorithmic
tools to recommend new Wikipedia editors to existing WikiProjects.

####  4.2.2. Value Sensitive Design Process

Drawing on the principles of Value Sensitive Design (Friedman et al., 2017),
we derived the following five-stage process to proactively accommodate values
in the WikiProjects recommendation system.

  1. (1)

Review literature and conduct empirical studies to understand relevant
stakeholders’ values, motivations and goals, and identify potential trade-
offs.

  2. (2)

Based on the results of the first step, identify algorithmic approaches and
create prototype implementations.

  3. (3)

Engage and work closely with the stakeholders to identify viable means to
deploy the developed algorithms, recruit participants, and gather user
feedback.

  4. (4)

Deploy the algorithms, gather stakeholder feedback, and refine and iterate as
necessary.

  5. (5)

Evaluate stakeholders’ acceptance, algorithm accuracy, and impacts of the
algorithms on relevant stakeholders.

In contrast with Study I that was focused on a conceptual investigation, this
five-stage process incorporates all three types of Value Sensitive Design
methods—conceptual, empirical, and technical—to synergistically address human
values. In the proposed process, each component itself should be familiar to
HCI scholars. However, the derived process represents an attempt to put
multiple Values Sensitive Design methods into action to collectively consider
values.

###  4.3. Results

Following the described five-stage process, we have developed the algorithmic
system to recommend new Wikipedia contributors to existing Wikipedia projects.

####  4.3.1. Step 1: Review literature and conduct empirical studies to
understand stakeholders’ values and goals of importance to them, and potential
trade-offs.

To achieve the design goal, we reviewed prior research and conducted a survey
study with 59 members from 17 WikiProjects in order to answer the following
questions: (1) Which factors would motivate new editors to participate in a
WikiProject? (2) Which factors would motivate WikiProjects organizers to
recruit new members? (3) What collective outcomes are important for
WikiProjects and for Wikipedia in general?

Findings from the survey study included:

  * •

Both newcomers and WikiProject organizers valued the interest match (i.e. the
match between the editor’s interests and the project topic) and personal
connections (i.e. the relationships between newcomers and current members of
the community).

  * •

WikiProject organizers valued productivity and wanted to recruit newcomers who
could produce high-quality work;

  * •

Although WikiProjects organizers valued editors’ prior experience and were
more interested in recruiting newcomers who already had some editing
experience in Wikipedia, the Wikipeida community as a whole was more concerned
with socializing “brand-new” editors.

  * •

WikiProject organizers hoped to remain control of the recruitment process.
Specifically, they objected to the idea of complete automation, i.e. to
automatically identify and invite newcomers deemed relevant by a
recommendation system. They wanted to have the final say on whom to be invited
to their projects.

####  4.3.2. Step 2: Identify algorithmic approaches and develop system
prototypes.

Drawing on the results of the first step, the team developed recommendation
algorithms that would meet the following criteria that were informed by
identified stakeholders’ values.

First, the recommendation algorithms needed to satisfy the goals of new
editors and WikiProjects by considering the match between the editor’s
interests and the project topic. To this end, we created four different
recommendation algorithms—two interest-based algorithms and two relationship-
based algorithms.

  * •

Interest-based algorithms rank candidate editors based on how closely their
editing history matches the topic of a WikiProject. Two types of interest-
based algorithms were considered. A _rule-based algorithm_ ranks the match of
an editor to a WikiProject by counting the number of edits by that editor to
articles within the scope of the project. A _category-based algorithm_ ranks
editors by computing a similarity score between an editor’s edit history and
the topic of a WikiProject.

  * •

Relationship-based algorithms rank candidate editors based on relationships
with current members of a WikiProject. Two types of relationship-based
algorithms were developed. A _bonds-based algorithm_ ranks editors by the
strength of “social connections” the editor has to current members of a
WikiProject. A _co-edit-based algorithm_ is a version of collaborative
filtering and is inspired by the design of SuggestBot (Cosley et al., 2007).
Candidate editors are ranked by the similarity of their edit histories to the
edit histories of current members of a WikiProject.

Second, the recommendation algorithms needed to balance the interests of
WikiProjects and the collective goal of Wikipedia community by targeting both
experienced editors and “brand-new” editors. Specifically, the candidate
editors included both the editors who were new to Wikipedia and editors who
were moderately experienced (but not highly experienced). This design
instantiated the tension we identified previously. Moreover, we also ranked
the two types of newcomers separately, so that experienced ones did not
overshadow the inexperienced ones.

Finally, the recommendation algorithms should satisfy the goals of
WikiProjects and their organizers by excluding editors likely to produce low-
quality work.

With these developed algorithms, we further designed a user interface to
present top recommendations from each of the four algorithms to WikiProject
organizers (see Figure 2). The decision of presenting multiple results was
based on a finding from Step 1 that some project organizers strongly objected
to the idea of complete automation and wished to “remain in the loop” of
inviting newcomers. With the prototype interface, WikiProject organizers could
review the recommendations and decide who they would like to invite to their
projects (Zhu et al., 2018). As such, their desire to remain in the loop was
accommodated.

####  4.3.3. Step 3: Engage and work closely with the community.

Online communities like Wikipedia constitute a rich laboratory for research.
Digital traces of collaboration, social interaction, and production are made
visible, offering rich opportunities for empirical studies and
experimentation. However, there is an unfortunate tradition of some
researchers treating such communities merely as data sources for research
instead of “living organisms” with their own values, goals, and norms.
Research studies of Wikipedia often encounter resistance from Wikipedia
editors because community values are sometimes violated by research
activities.

To avoid these problems, and to ensure community values are well attended to,
the team worked with stakeholders to develop a research protocol666See
https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation that was
acceptable to the community. Essentially, we developed our research plan on
the Wikipedia platform. In other words, we were developing and deploying our
algorithms not just for, but with, the Wikipedia community.

####  4.3.4. Step 4: Deploy the algorithms and iteratively refine them.

We iteratively tested and refined our algorithms. Over a duration of six
months we sent weekly batches of recommendations to our pilot participants,
conducted short surveys to seek their views on these recommendations, and also
interviewed project organizers and newcomers. We used their feedback to make
significant changes to the algorithms. Example revisions included refining
algorithm explanations and boosting the threshold of the matching algorithms.
In particular, candidate editors will only be recommended by the rule-based
matching algorithm if they have made minimally five edits on project-related
articles in the previous month.

####  4.3.5. Step 5: Evaluate algorithms based on stakeholder acceptance,
algorithm accuracy, and impacts of the algorithm on the community.

We conducted qualitative and quantitative studies to systematically examine
the acceptance, accuracy, and impacts of the algorithms. The goal was to fully
understand the influence of the algorithmic tool on the community and
stakeholders, and to monitor any unintended consequences.

Results from our initial evaluation included:

  1. (1)

Assessing acceptance among stakeholders. We conducted interviews with
community stakeholders, including newcomers and WikiProject organizers who
used our recommendation systems. The feedback was positive. This gave us
confidence that our system was acceptable to the stakeholders. For example,
one organizer wrote: “This puts some science behind recommendations, and will
be a great supplement to the current processes.” Editors who were invited to
join projects also reacted positively. For example, an editor who was invited
to join WikiProject Africa wrote: “Thank you for reaching out to me and thank
you for informing me about the WikiProject Africa … I appreciate it.”

  2. (2)

Assessing accuracy. Organizers rated the rule-based algorithm higher compared
to the other three types of algorithms that were category-based, bonds-based
and co-edit based. The average rating for the rule-based algorithm was 3.24 in
a 5-point scale, significantly higher than the other three types of algorithms
(t = 3.51, p ¡.001). The invitation rate (which is analogous to the click-
through rate) for the rule-based algorithm was 47%, which was also higher than
category-based (16%), bonds-based (22%), and co-edit based (28%). There was no
significant difference among the other three algorithms. We also found that
inexperienced editors and experienced editors were equally invited by the
project organizers through the recommendation system.

  3. (3)

Assessing impacts. Our evaluation of the impacts on the community sought to
understand what happened to newcomers if WikiProject organizers invited them
to the projects. Initial results suggested that: (1) Only experienced editors
who received invitations from project organizers had a significant increase in
their within-project activities over the baseline group composed of equally
competent editors who did not receive invitations; and (2) the increase in the
invited experienced editors’ contributions did not come at the cost of fewer
edits of Wikipedia articles beyond the joined WikiProject (see Zhu et al.,
2018).

###  4.4. Discussion and Implications

####  4.4.1. Challenges and Future Directions

While this WikiProjects case demonstrated the promise of Value Sensitive
Design and the five-stage holistic approach, it also helped reveal several
challenges. First, it is challenging to explain the algorithms in ways that
enable stakeholders to assess them and to provide sensible feedback to improve
them. This constitutes a barrier to full stakeholder engagement in the design
process. One future direction is to explore the effectiveness and trade-offs
of different strategies and interfaces to help non-expert stakeholders
understand algorithmic decision-making.

Second, there is a lack of holistic understanding of the mapping between a
wide range of human values and a variety of algorithmic design options. One
promising direction for future work is to design ways to translate human
values into different algorithmic choices, covering areas such as data pre-
processing, model regularization during the learning process, model post-
processing, and model selection.

####  4.4.2. Implications for Learning Analytics System Design

This study has a number of implications for learning analytics design. First,
the designed algorithm products can be transferred to other learning contexts.
For instance, given the increasing popularity of virtual collaboration in
MOOCs (Wen et al., 2015), we could design similar recommendation algorithms to
support the formation of productive virtual learning groups in MOOCs. We have
not seen work in this area attending to learner values yet, and the Value
Sensitive Design methodology applied to the Wikipedia context could contribute
to settings like MOOCs.

Second, the five-stage design process is directly applicable to a learning
analytics design project. Even though the study was primarily focused on
algorithms, the Value Sensitive Design process introduced in this study can be
applied to the design of a learning analytics system, of which algorithm is an
important component (Buckingham Shum, 2017). While we could embrace the
process when choosing or developing network visualization layout algorithms
for a social network analytics tool, we can also apply the process to other
parts of the analytics system. For example, as demonstrated in this case
study, the consideration of human values is critical for the process of
communicating analytics results to stakeholders. Other aspects of the system,
such as the technological infrastructure used to store and transit learning
data can also benefit from the Value Sensitive Design approach.

##  5\. General Discussion and Conclusions

To support ethical considerations and system integrity in learning analytics,
this paper introduces the application of Value Sensitive Design in learning
analytics design processes. The first study was a conceptual investigation of
an existing learning analytics tool. Applying Value Sensitive Design
methods—specifically stakeholder analysis and value analysis—we uncovered a
number of values (e.g. student autonomy, self-image) and value tensions with
the studied tool. Based on these analyses, we proposed design trade-offs that
are subject to further empirical, technical, and conceptual investigations.

The second study went a step further. It proposed a holistic process of
conducting Value Sensitive Design and demonstrated its application to building
recommendation systems for WikiProjects—an informal, mass collaboration
scenario. The design process includes literature analysis, empirical
investigations, prototype development, community engagement, iterative testing
and refinement, and continuous evaluation. Even though this study was focused
on algorithms, the process can be readily applied to the design of a whole
learning analytics system.

Overall, this paper is motivated by a need of practical means to support
ethical considerations and human values in learning analytics system. Learning
analytics operates in the “middle space” between learning and analytics and
needs to consider a wide range of factors including learning theory, data
integrity, human–computer interaction, and sociocultural factors (Suthers and
Verbert, 2013). As a result, a wide range of values, which go beyond ethical
concerns and are often competing with each other, need to be considered.

Value Sensitive Design is a promising approach to promoting ethical practice
in learning analytics. Preliminary work in this area has already been reported
by colleagues (Verhagen et al., 2016), and more systematic push for the
application of Value Sensitive Design in learning analytics is needed. Two
studies reported in this paper mitigate the gap. They demonstrate two ways of
adopting Value Sensitive Design in learning analytics projects: one was a
conceptual investigation (a “lite” version), whereas the other was a detailed
design process comprising multiple Value Sensitive Design methods (a
“holistic” version). It is hoped both versions of using Value Sensitive Design
can strengthen future design of learning analytics systems.

To improve learning analytics systems (esp. its ethical dimension),
algorithmic transparency and accountability are not enough; rather, we should
consider the system’s integrity holistically (Buckingham Shum, 2017). This
paper is only an initial step towards this direction. Much work remains to be
done to infuse Value Sensitive Design more broadly in the field of learning
analytics. Interestingly, several projects have emerged to advance this work.
Besides the Privacy by Design approach mentioned above (Verhagen et al.,
2016), the Connected Intelligence Centre at the University of Technology
Sydney is working on an Ethical Design Critique (EDC) approach to involve
stakeholders to consider ethical and risk factors involved in new learning
analytics tools. Even though this approach does not directly apply Value
Sensitive Design, it considers competing values identified by different
stakeholders and aims to reach resolutions. One future direction can be
purposefully exploring synergies among algorithm auditing, Privacy by Design,
Ethical Design Critique, and Value Sensitive Design to comprehensively
consider human values in learning analytics.

## References

  * (1)
  * ACM US Public Policy Council (2017) ACM US Public Policy Council. 2017\.  Statement on Algorithmic Transparency and Accountability.  Online.
  * Buckingham Shum (2017) Simon Buckingham Shum. 2017\.  Black box learning analytics? Beyond algorithmic transparency. Keynote presented at the Learning Analytics Summer Institute, Ann Arbor, MI.
  * Chen et al. (2018) Bodong Chen, Yu-Hui Chang, Fan Ouyang, and Wanying Zhou. 2018\.  Fostering student engagement in online discussion through social learning analytics.  _The Internet and Higher Education_ 37 (2018), 21–30.  https://doi.org/10.1016/j.iheduc.2017.12.002
  * Cosley et al. (2007) Dan Cosley, Dan Frankowski, Loren Terveen, and John Riedl. 2007.  SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia. In _Proceedings of the 12th International Conference on Intelligent User Interfaces_ _(IUI ’07)_. ACM, New York, NY, USA, 32–41.  https://doi.org/10.1145/1216295.1216309
  * Cress et al. (2016) Ulrike Cress, Johannes Moskaliuk, and Heisawn Jeong. 2016\.  _Mass Collaboration and Education_.  Springer.  https://doi.org/10.1007/978-3-319-13536-6
  * Czerkawski (2016) Betul C Czerkawski. 2016\.  Networked learning: design considerations for online instructors.  _Interactive Learning Environments_ 24, 8 (Nov. 2016), 1850–1863.  https://doi.org/10.1080/10494820.2015.1057744
  * Czeskis et al. (2010) Alexei Czeskis, Ivayla Dermendjieva, Hussein Yapit, Alan Borning, Batya Friedman, Brian Gill, and Tadayoshi Kohno. 2010.  Parenting from the pocket: value tensions and technical directions for secure and private parent-teen mobile safety. In _Proceedings of the Sixth Symposium on Usable Privacy and Security_. ACM, New York, NY, USA, 15.  https://doi.org/10.1145/1837110.1837130
  * Davis and Nathan (2013) Janet Davis and Lisa P Nathan. 2013.  Value Sensitive Design: Applications, Adaptations, and Critiques.  In _Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains_ , Jeroen van den Hoven, Pieter E Vermaas, and Ibo van de Poel (Eds.). Springer Netherlands, Dordrecht, 1–26.  https://doi.org/10.1007/978-94-007-6994-6_3-1
  * Dawson et al. (2010) Shane Dawson, Aneesha Bakharia, and Elizabeth Heathcote. 2010\.  SNAPP: Realising the affordances of real-time SNA within networked learning environments. In _Proceedings of the 7th International Conference on Networked Learning_ , Lone Dirckinck-Holmfeld, Vivien Hodgson, Chris Jones, Maarten de Laat, David McConnell, and Thomas Ryberg (Eds.). Lancaster University, Lancaster, UK, 125–133.
  * Dowell et al. (2018) Nia M M Dowell, Tristan M Nixon, and Arthur C Graesser. 2018\.  Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions.  _Behavior research methods_ (Aug. 2018).  https://doi.org/10.3758/s13428-018-1102-z
  * Drachsler and Greller (2016) Hendrik Drachsler and Wolfgang Greller. 2016.  Privacy and analytics: It’s a DELICATE issue a checklist for trusted learning analytics. In _Proceedings of the Sixth International Conference on Learning Analytics & Knowledge_. ACM, New York, NY, USA, 89–98.  https://doi.org/10.1145/2883851.2883893
  * Drachsler et al. (2016) Hendrik Drachsler, Tore Hoel, Adam Cooper, Gábor Kismihók, Alan Berg, Maren Scheffel, Weiqin Chen, and Rebecca Ferguson. 2016\.  Ethical and Privacy Issues in the Design of Learning Analytics Applications. In _Proceedings of the Sixth International Conference on Learning Analytics & Knowledge_ _(LAK ’16)_. ACM, New York, NY, USA, 492–493.  https://doi.org/10.1145/2883851.2883933
  * Ferguson et al. (2016) Rebecca Ferguson, Tore Hoel, Maren Scheffel, and Hendrik Drachsler. 2016.  Guest Editorial: Ethics and Privacy in Learning Analytics.  _Journal of Learning Analytics_ 3, 1 (2016), 5–15.  https://doi.org/10.18608/jla.2016.31.2
  * Forte et al. (2012) Andrea Forte, Niki Kittur, Vanessa Larco, Haiyi Zhu, Amy Bruckman, and Robert E Kraut. 2012\.  Coordination and beyond: social functions of groups in open content production. In _Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work_. ACM, 417–426.
  * Friedman et al. (2017) Batya Friedman, David G Hendry, and Alan Borning. 2017\.  A Survey of Value Sensitive Design Methods.  _Foundations and Trends® in Human–Computer Interaction_ 11, 2 (2017), 63–125.  https://doi.org/10.1561/1100000015
  * Friedman et al. (2013) Batya Friedman, Peter H Kahn, Alan Borning, and Alina Huldtgren. 2013.  Value Sensitive Design and Information Systems.  In _Early engagement and new technologies: Opening up the laboratory_ , Neelke Doorn, Daan Schuurbiers, Ibo van de Poel, and Michael E Gorman (Eds.). Springer Netherlands, Dordrecht, 55–95.  https://doi.org/10.1007/978-94-007-7844-3_4
  * Geiger (2017) R Stuart Geiger. 2017\.  Beyond opening up the black box: Investigating the role of algorithmic systems in Wikipedian organizational culture.  _Big Data & Society_ 4, 2 (Sept. 2017), 2053951717730735\.  https://doi.org/10.1177/2053951717730735
  * Greenhow and Askari (2017) Christine Greenhow and Emilia Askari. 2017.  Learning and teaching with social network sites: A decade of research in K-12 related education.  _Education and Information Technologies_ 22, 2 (March 2017), 623–645.  https://doi.org/10.1007/s10639-015-9446-9
  * Halatchliyski et al. (2014) Iassen Halatchliyski, Johannes Moskaliuk, Joachim Kimmerle, and Ulrike Cress. 2014.  Explaining authors’ contribution to pivotal artifacts during mass collaboration in the Wikipedia’s knowledge base.  _International Journal of Computer-Supported Collaborative Learning_ 9, 1 (2014), 97–115.
  * Halfaker et al. (2013) Aaron Halfaker, R Stuart Geiger, Jonathan T Morgan, and John Riedl. 2013.  The Rise and Decline of an Open Collaboration System: How Wikipedia’s Reaction to Popularity Is Causing Its Decline.  _The American behavioral scientist_ 57, 5 (May 2013), 664–688.  https://doi.org/10.1177/0002764212469365
  * Huberth et al. (2015) Madeline Huberth, Patricia Chen, Jared Tritz, and Timothy A McKay. 2015.  Computer-tailored student support in introductory physics.  _PloS one_ 10, 9 (Sept. 2015), e0137001.  https://doi.org/10.1371/journal.pone.0137001
  * Ifenthaler and Tracey (2016) Dirk Ifenthaler and Monica W Tracey. 2016.  Exploring the relationship of ethics and privacy in learning analytics and design: implications for the field of educational technology.  _Educational technology research and development: ETR & D_ 64, 5 (Oct. 2016), 877–880.  https://doi.org/10.1007/s11423-016-9480-3
  * Lang et al. (2018) Charles Lang, Leah P Macfadyen, Sharon Slade, Paul Prinsloo, and Niall Sclater. 2018.  The Complexities of Developing a Personal Code of Ethics for Learning Analytics Practitioners: Implications for Institutions and the Field. In _Proceedings of the 8th International Conference on Learning Analytics and Knowledge_ _(LAK ’18)_. ACM, New York, NY, USA, 436–440.  https://doi.org/10.1145/3170358.3170396
  * Long and Siemens (2011) P Long and G Siemens. 2011\.  Penetrating the Fog: Analytics in Learning and Education.  _Educause Review_ 46, 5 (2011), 30–32.
  * Prinsloo and Slade (2016) Paul Prinsloo and Sharon Slade. 2016.  Student Vulnerability, Agency and Learning Analytics: An Exploration.  _Journal of Learning Analytics_ 3, 1 (2016), 159–182.  https://doi.org/10.18608/jla.2016.31.10
  * Prinsloo and Slade (2017a) Paul Prinsloo and Sharon Slade. 2017a.  An Elephant in the Learning Analytics Room: The Obligation to Act. In _Proceedings of the Seventh International Learning Analytics & Knowledge Conference_ _(LAK ’17)_. ACM, New York, NY, USA, 46–55.  https://doi.org/10.1145/3027385.3027406
  * Prinsloo and Slade (2017b) Paul Prinsloo and Sharon Slade. 2017b.  Ethics and Learning Analytics: Charting the (Un)Charted.  In _Handbook of Learning Analytics_ (first ed.), Charles Lang, George Siemens, Alyssa Wise, and Dragan Gasevic (Eds.). Society for Learning Analytics Research (SoLAR), 49–57.  https://doi.org/10.18608/hla17.004
  * Regan (1995) Priscilla M Regan. 1995\.  _Legislating Privacy: Technology, Social Values, and Public Policy_.  University of North Carolina Press, Chapel Hill, NC.
  * Rodríguez-Triana et al. (2016) María Jesús Rodríguez-Triana, Alejandra Martínez-Monés, and Sara Villagrá-Sobrino. 2016\.  Learning Analytics in Small-Scale Teacher-Led Innovations: Ethical and Data Privacy Issues.  _Journal of Learning Analytics_ 3, 1 (2016), 43–65.
  * Rosé et al. (2015) Carolyn Penstein Rosé, Oliver Ferschke, Gaurav Tomar, Diyi Yang, Iris Howley, Vincent Aleven, George Siemens, Matthew Crosslin, Dragan Gasevic, and Ryan Baker. 2015.  Challenges and opportunities of dual-layer MOOCs: Reflections from an edX deployment study. In _Proceedings of the 11th International Conference on Computer Supported Collaborative Learning. (CSCL 2015). Vol. 2_ , Vol. 15. International Society of the Learning Sciences, Gothenburg, Sweden, 848–851.
  * Sandvig et al. (2014) Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014.  Auditing algorithms: Research methods for detecting discrimination on internet platforms.  Data and discrimination: converting critical concerns into productive inquiry, a pre-conference at the 64th Annual Meeting of the International Communication Association.
  * Sclater (2016) Niall Sclater. 2016\.  Developing a Code of Practice for Learning Analytics.  _Journal of Learning Analytics_ 3, 1 (2016), 16–42.  https://doi.org/10.18608/jla.2016.31.3
  * Siemens (2013) George Siemens. 2013\.  Learning analytics: The emergence of a discipline.  _The American behavioral scientist_ 57, 10 (2013), 1380–1400.
  * Slade and Prinsloo (2013) Sharon Slade and Paul Prinsloo. 2013.  Learning analytics: Ethical issues and dilemmas.  _The American behavioral scientist_ 57, 10 (Oct. 2013), 1510–1529.  https://doi.org/10.1177/0002764213479366
  * Suthers and Verbert (2013) Dan Suthers and Katrien Verbert. 2013.  Learning analytics as a “middle space”. In _Proceedings of the Third International Conference on Learning Analytics and Knowledge - LAK ’13_. ACM Press, New York, New York, USA, 1–4.  https://doi.org/10.1145/2460296.2460298
  * Vaquero and Cebrian (2013) Luis M Vaquero and Manuel Cebrian. 2013.  The rich club phenomenon in the classroom.  _Scientific reports_ 3 (2013), 1–8.  https://doi.org/10.1038/srep01174
  * Verhagen et al. (2016) Josine Verhagen, Lucie Dalibert, Federica Lucivero, Tjerk Timan, and Larissa Co. 2016.  Designing values in an adaptive learning platform. In _2nd workshop on Ethics & Privacy in Learning Analytic at the 6th International Learning Analytics & Knowledge Conference_. CEUR Proceedings, Edinburgh, UK, 1–5.
  * Watkins et al. (2013) Kari Edison Watkins, Brian Ferris, Yegor Malinovskiy, and Alan Borning. 2013.  Beyond Context-Sensitive Solutions: Using Value-Sensitive Design to Identify Needed Transit Information Tools. In _Urban Public Transportation Systems 2013_. American Society of Civil Engineers, Reston, VA, 296–308.  https://doi.org/10.1061/9780784413210.026
  * Wen et al. (2015) Miaomiao Wen, Diyi Yang, and Carolyn Penstein Rosé. 2015\.  Virtual Teams in Massive Open Online Courses. In _Artificial Intelligence in Education_ _(Lecture Notes in Computer Science)_. Springer, Cham, 820–824.  https://doi.org/10.1007/978-3-319-19773-9_124
  * Willis et al. (2016) James E Willis, Sharon Slade, and Paul Prinsloo. 2016\.  Ethical oversight of student data in learning analytics: A typology derived from a cross-continental, cross-institutional perspective.  _Educational Technology Research and Development_ 64, 5 (Oct. 2016), 881–901.  https://doi.org/10.1007/s11423-016-9463-4
  * Wise and Vytasek (2017) Alyssa Friend Wise and J Vytasek. 2017.  Learning analytics implementation design.  In _Handbook of learning analytics_ , Charles Lang, George Siemens, Alyssa Friend Wise, and Dragan Gašević (Eds.). Society for Learning Analytics Research, 151–160.
  * Zhu et al. (2012) Haiyi Zhu, Robert Kraut, and Aniket Kittur. 2012.  Organizing without formal organization: group identification, goal setting and social modeling in directing online production. In _Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work_. ACM, 935–944.
  * Zhu et al. (2018) Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018\.  Value-Sensitive Algorithm Design: Method, Case Study, and Lessons.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (2018), 194.  https://doi.org/10.1145/3274463


* Zollner, Gopalan, and Cardona (1991) S. Zollner, S. Gopalan, and M. Cardona, “The temperature dependence of the band gaps in InP, InAs, InSb, and GaSb,” Sol. State Comm. 77, 485–488 (1991).
  * Marini (2008) A. Marini, “Ab-initio finite temperature excitons,” Phys. Rev. Lett. 101, 106405 (2008).
  * Giustino, Louie, and Cohen (2010) F. Giustino, S. Louie, and M. Cohen, “Electron-phonon renormalization of the direct band gap of diamond,” Phys. Rev. Lett. 105, 265501 (2010).
  * Cannuccia and Marini (2011) E. Cannuccia and A. Marini, “Effect of the quantum zero-point atomic motion on the optical and electronic properties of diamond and trans-polyacetylene,” Phys. Rev. Lett. 107, 255501 (2011).
  * Gonze, Boulanger, and Côté (2011) X. Gonze, P. Boulanger, and M. Côté, “Theoretical approaches to the temperature and zero-point motion effects on the electronic band structure,” Annalen der Physik (Leipzig) 523, 168–178 (2011).
  * Cannuccia and Marini (2012) E. Cannuccia and A. Marini, “Zero point motion effect on the electronic properties of diamond, trans-polyacetylene and polyethylene,” Eur. Phys. J. B 85, 320 (2012).
  * Kawai  _et al._ (2014) H. Kawai, K. Yamashita, E. Cannuccia, and A. Marini, “Electron-electron and electron-phonon correlation effects on the finite-temperature electronic and optical properties of zinc-blende GaN,” Phys. Rev. B 89, 085202 (2014).
  * Poncé et al. (2014a) S. Poncé et al., “Verification of first-principles codes: Comparison of total energies, phonon frequencies, electron–phonon coupling and zero-point motion correction to the gap between ABINIT and QE/Yambo,” Comp. Mat. Sci. 83, 341–348 (2014a).
  * Poncé et al. (2014b) S. Poncé et al., “Temperature dependence of electronic eigenenergies in the adiabatic harmonic approximation,” Phys. Rev. B 90, 214304 (2014b).
  * Antonius et al. (2015) G. Antonius et al., “Dynamical and anharmonic effects on the electron-phonon coupling and the zero-point renormalization of the electronic structure,” Phys. Rev. B 92, 085137 (2015).
  * Poncé et al. (2015) S. Poncé et al., “Temperature dependence of the electronic structure of semiconductors and insulators,” J. Chem. Phys. 143, 102813 (2015).
  * Nery et al. (2018) J. Nery et al., “Quasiparticles and phonon satellites in spectral functions of semiconductors and insulators: Cumulants applied to the full first-principles theory and the Fröhlich polaron,” Phys. Rev. B 97, 115145 (2018).
  * not (a) For more references, see the Supporting Information, Sec. II.D. Note also that not all mentioned references present computations of equal quality. In particular, the difficult sampling of the Brillouin Zone for AHC has rarely been tackled correctly before 2015. Similarly the supercell size convergence for ASC is also a concern.
  * Capaz  _et al._ (2005) R. B. Capaz, C. D. Spataru, P. Tangney, M. L. Cohen, and S. G. Louie, “Temperature dependence of the band gap of semiconducting carbon nanotubes,” Phys. Rev. Lett. 94, 036801 (2005).
  * Patrick and Giustino (2013) C. E. Patrick and F. Giustino, “Quantum nuclear dynamics in the photophysics of diamondoids,” Nature Communications 4, 2006 (2013).
  * Monserrat, Drummond, and Needs (2013) B. Monserrat, N. Drummond, and R. Needs, “Anharmonic vibrational properties in periodic systems: energy, electron-phonon coupling, and stress,” Phys. Rev. B 87, 144302 (2013).
  * Antonius  _et al._ (2014) G. Antonius, S. Poncé, P. Boulanger, M. Côté, and X. Gonze, “Many-body effects on the zero-point renormalization of the band structure,” Phys. Rev. Lett. 112, 215501 (2014).
  * Monserrat  _et al._ (2014) B. Monserrat, N. Drummond, C. Pickard, and R. Needs, “Electron-phonon Coupling and the metallization of solid helium at terapascal pressures,” Phys. Rev. Lett. 112, 055504 (2014).
  * Monserrat and Needs (2014) B. Monserrat and R. Needs, “Comparing electron-phonon coupling strength in diamond, silicon, and silicon carbide: First-principles study,” Phys. Rev. B 89, 214304 (2014).
  * Monserrat, Engel, and Needs (2015) B. Monserrat, E. Engel, and R. Needs, “Giant electron-phonon interactions in molecular crystals and the importance of nonquadratic coupling,” Phys. Rev. B 92, 140302(R) (2015).
  * Engel, Monserrat, and Needs (2015) E. A. Engel, B. Monserrat, and R. J. Needs, “Vibrational renormalisation of the electronic band gap in hexagonal and cubic ice,” J. Chem. Phys. 143, 244708 (2015).
  * Karsai  _et al._ (2018) F. Karsai, M. Engel, G. Kresse, and E. Flage-Larsen, “Electron–phonon coupling in semiconductors within the GW approximation,” New Journal of Physics 20, 123008 (2018).
  * Giustino (2017) F. Giustino, “Electron-phonon interactions from first principles,” Rev. Mod. Phys. 89, 015003 (2017).
  * Fröhlich (1954) H. Fröhlich, “Electrons in lattice fields,” Adv. Phys. 3, 325–361 (1954).
  * Feynman (1955) R. P. Feynman, “Slow electrons in a polar crystal,” Phys. Rev. 97, 660–665 (1955).
  * Mishchenko  _et al._ (2000) A. Mishchenko, N. Prokof’ev, A. Sakamoto, and B. Svistunov, “Diagrammatic quantum Monte Carlo study of the Fröhlich polaron,” Phys. Rev. B 62, 6317–6336 (2000).
  * Mishchenko  _et al._ (2003) A. S. Mishchenko, N. Nagaosa, N. Prokof’ev, A. Sakamoto, and B. Svistunov, “Optical conductivity of the Fröhlich polaron,” Phys. Rev. Lett. 91, 236401 (2003).
  * Devreese and Alexandrov (2009) J. Devreese and A. Alexandrov, “Fröhlich polaron and bipolaron: recent developments,” Rep. Prog. Phys. 72, 066501 (2009).
  * Emin (2012) D. Emin,  _Polarons_ (Cambrige University Press, Cambridge, 2012).
  * Hahn  _et al._ (2018) T. Hahn, S. Klimin, J. Tempere, J. T. Devreese, and C. Franchini, “Diagrammatic Monte Carlo study of Fröhlich polaron dispersion in two and three dimensions,” Phys. Rev. B 97, 134305 (2018).
  * not (b) KS-DFT ASC Calculations in Ref. Karsai  _et al._ , 2018 are based on GGA-PBE, except for Si, Ge, GaAs and CdSe, where the PBE0 hybrid functional has been used. Thus the better score of Ge and GaAs for ASC calculations than for AHC calculations might be partly explained by this different KS-DFT functional.
  * Rangel et al. (2020) T. Rangel et al., “Reproducibility in G0W0 calculations for solids,” Comput. Phys. Commun. 255, 107242 (2020).
  * Cardona and Thewalt (2005) M. Cardona and M. Thewalt, “Isotope effects on the optical spectra of semiconductors,” Rev. Mod. Phys. 77, 1173–1224 (2005).
  * Mahan (1965) G. D. Mahan, “Temperature dependence of the band gap in CdTe,” J. Phys. Chem. Solids 26, 751–756 (1965).
  * Trebin and Rössler (1975) H.-R. Trebin and U. Rössler, “Polarons in the degenerate-band case,” Phys. Status Solidi B 70, 717–726 (1975).
  * Devreese  _et al._ (2010) J. T. Devreese, S. N. Klimin, J. L. M. van Mechelen, and D. van der Marel, “Many-body large polaron optical conductivity in SrTi(1-x)NbxO3,” Phys. Rev. B 81, 125119 (2010).
  * Schlipf, Poncé, and Giustino (2018) M. Schlipf, S. Poncé, and F. Giustino, “Carrier lifetimes and polaronic mass enhancement in the hybrid halide perovskite CH3NH3PbI3 from multiphonon Fröhlich coupling,” Phys. Rev. Lett. 121, 086402 (2018).
  * Vogl (1976) P. Vogl, “Microscopic theory of electron-phonon interaction in insulators or semiconductors,” Phys. Rev. B 13, 694–704 (1976).
  * Verdi and Giustino (2015) C. Verdi and F. Giustino, “Fröhlich electron-phonon vertex from first principles,” Phys. Rev. Lett. 115, 176401 (2015).
  * Gonze et al. (2016) X. Gonze et al., “Recent developments in the ABINIT software package,” Comput. Phys. Commun. 205, 106–131 (2016).
  * Perdew, Burke, and Ernzerhof (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, “Generalized gradient approximation made simple,” Phys. Rev. Lett. 77, 3865–3868 (1996).
  * van Setten et al. (2018) M. van Setten et al., “The pseudodojo: Training and grading a 85 element optimized norm-conserving pseudopotential table,” Comput. Phys. Commun. 226, 39–54 (2018).
  * Montanari and Harrison (2002) B. Montanari and N. M. Harrison, “Lattice dynamics of TiO2 rutile: influence of gradient corrections in density functional calculations,” Chem. Phys. Lett. 364, 528–534 (2002).
  * Baroni  _et al._ (2001) S. Baroni, S. de Gironcoli, A. D. Corso, and P. Giannozzi, “Phonons and related crystal properties from density-functional perturbation theory,” Rev. Mod. Phys. 73, 515–562 (2001).
  * Gonze and Lee (1997) X. Gonze and C. Lee, “Dynamical matrices, Born effective charges, dielectric permittivity tensors, and interatomic force constants from density-functional perturbation theory,” Phys. Rev. B 55, 10355–10368 (1997).
  * Laflamme Janssen et al. (2016) J. Laflamme Janssen et al., “Precise effective masses from density functional perturbation theory,” Phys. Rev. B 93, 205147 (2016).
  * Guennou  _et al._ (2010) M. Guennou, P. Bouvier, J. Kreisel, and D. Machon, “Pressure-temperature phase diagram of SrTiO3 up to 53 GPa,” Phys. Rev. B 81, 054115 (2010).
  * Tadano and Tsuneyuki (2015) T. Tadano and S. Tsuneyuki, “Self-consistent phonon calculations of lattice dynamical properties in cubic SrTiO3 with first-principles anharmonic force constants,” Phys. Rev. B 92, 054301 (2015).
  * Müller and Burkard (1979) K. A. Müller and H. Burkard, “SrTi${\mathrm{O}}_{3}$: An intrinsic quantum paraelectric below 4 K,” Phys. Rev. B 19, 3593–3602 (1979).
  * Zhong and Vanderbilt (1996) W. Zhong and D. Vanderbilt, “Effect of quantum fluctuations on structural phase transitions in ${\mathrm{SrTiO}}_{3}$ and ${\mathrm{BaTiO}}_{3}$,” Phys. Rev. B 53, 5047–5050 (1996).
  * Hellman  _et al._ (2013) O. Hellman, P. Steneteg, I. A. Abrikosov, and S. I. Simak, “Temperature dependent effective potential method for accurate free energy calculations of solids,” Phys. Rev. B 87, 104111 (2013).
  * Tadano, Gohda, and Tsuneyuki (2014) T. Tadano, Y. Gohda, and S. Tsuneyuki, “Anharmonic force constants extracted from first-principles molecular dynamics: applications to heat transfer simulations,” J. Phys. Condens. Matter 26, 225402 (2014).
  * Sio  _et al._ (2019) W. H. Sio, C. Verdi, S. Poncé, and F. Giustino, “Polarons from first principles, without supercells,” Phys. Rev. Lett. 122, 246403 (2019).
  * Persson et al. (2018) K. Persson et al., “Materials project,”  (2018), URL https://materialsproject.org.
  * Shishkin and Kresse (2007) M. Shishkin and G. Kresse, “Self-consistent GW calculations for semiconductors and insulators,” Phys. Rev. B 75, 235102 (2007).
  * not (c) We refer to Fig. 8 in the Supplementary Material of Ref. Poncé et al., 2015 for AlN-w (note that in the figure caption, the unextrapolated values are mentioned), and to Fig. 15 in the Supplementary Material of Ref. Poncé et al., 2015 for BN-c (note that in the figure caption, the extrapolated values are mentioned). In these captions, there are errors concerning the name of the material corresponding to each figure.
  * Monkhorst and Pack (1976) H. Monkhorst and J. Pack, “Special points for Brillouin-zone integrations,” Phys. Rev. B 13, 5188–5192 (1976).
  * Nery and Allen (2016) J. P. Nery and P. B. Allen, “Influence of Fröhlich polaron coupling on renormalized electron bands in polar semiconductors: Results for zinc-blende GaN,” Phys. Rev. B 94, 115135 (2016).
  * Lytle (1964) F. W. Lytle, “X-Ray Diffractometry of Low-Temperature Phase Transformations in Strontium Titanate,” J. Appl. Phys. 35, 2212–2215 (1964).
  * Zacharias, Scheffler, and Carbogno (2020) M. Zacharias, M. Scheffler, and C. Carbogno, “Fully anharmonic, non-perturbative theory of vibronically renormalized electronic band structures,” arXiv: cond-mat.mtrl-sci 2003, 10417 (2020).
  * Pässler (2002) R. Pässler, “Dispersion-related description of temperature dependencies of band gaps in semiconductors,” Phys. Rev. B 66, 085201 (2002).
  * Pässler (2003) R. Pässler, “Semi-empirical descriptions of temperature dependences of band gaps in semiconductors,” Phys. Status Solidi B 3, 710–728 (2003).
  * not (d) Experimental gaps come from Refs. [Shishkin and Kresse, 2007], [Cardona and Thewalt, 2005], [van Setten  _et al._ , 2017], [Benrekia  _et al._ , 2012], [Kawai  _et al._ , 2014], [Yang  _et al._ , 2016], and [Singh  _et al._ , 2011].
  * Parks  _et al._ (1994) C. Parks, A. K. Ramdas, S. Rodriguez, K. M. Itoh, and E. E. Hailer, “Electronic band structure of isotopically pure germanium: Modulated transmission and reflectivity study,” Phys. Rev. B 49, 14244–14250 (1994).
  * Karaiskaj  _et al._ (2002) D. Karaiskaj, M. Thewalt, T. Ruf, M. Cardona, and M. Konuma, “Photoluminescence studies of isotopically enriched silicon: isotopic effects on the indirect electronic band gap and phonon energies,” Solid State Communications 123, 87–92 (2002).
  * Zhang  _et al._ (1998) J. M. Zhang, T. Ruf, R. Lauck, and M. Cardona, “Isotope effects on exciton energies in CdS,” Phys. Rev. B 57, 9716–9722 (1998).
  * Manjon  _et al._ (2005) F. Manjon, M. Mollar, B. Mari, N. Garro, A. Cantarero, R. Lauck, and M. Cardona, “Effect of isotopic mass on the photoluminescence spectra of $\beta$ zinc sulfide,” Solid State Communications 133, 253–258 (2005).
  * Collins  _et al._ (1990) A. T. Collins, S. C. Lawson, G. Davies, and H. Kanda, “Indirect energy gap of 13C diamond,” Phys. Rev. Lett. 65, 891–894 (1990).
  * Benrekia  _et al._ (2012) A. Benrekia, N. Benkhettou, A. Nassour, M. Driz, M. Sahnoun, and S. Lebegue, “Structural, electronic and optical properties of cubic SrTiO3 and KTaO3: Ab initio and GW calculations,” Physica B 407, 2632–2636 (2012).
  * Kok  _et al._ (2015) D. J. Kok, K. Irmscher, M. Naumann, C. Guguschev, Z. Galazka, and R. Uecker, “Temperature-dependent optical absorption of SrTiO3,” Phys. Stat. Sol. (a) 212, 1880–1887 (2015).
  * Manjon  _et al._ (2003) F. Manjon, M. Mollar, M. Hernandez-Fenollosa, B. Mari, R. Lauck, and M. Cardona, “Effect of isotopic mass on the photoluminescence spectra of zinc oxide,” Sol. State Comm. 128, 35–39 (2003).
  * van Setten  _et al._ (2017) M. J. van Setten, M. Giantomassi, X. Gonze, G.-M. Rignanese, and G. Hautier, “Automation methodologies and large-scale validation for GW: Towards high-throughput GW calculations,” Phys. Rev. B 96, 155207 (2017).
  * Pässler (1999) R. Pässler, “Parameter sets due to fittings of the temperature dependencies of fundamental bandgaps in semiconductors,” Phys. Status Solidi B 216, 975–1007 (1999).
  * Garro  _et al._ (1996) N. Garro, A. Cantarero, M. Cardona, A. Göbel, T. Ruf, and K. Eberl, “Dependence of the lattice parameters and the energy gap of zinc-blende-type semiconductors on isotopic masses,” Phys. Rev. B 54, 4732–4740 (1996).
  * Villegas, Rocha, and Marini (2016a) C. Villegas, A. Rocha, and A. Marini, “Anomalous thermal dependence of the band gap in black phosphorus,” Nanoletters 16, 5095 (2016a).
  * Rignanese, Michenaud, and Gonze (1996) G. M. Rignanese, J. P. Michenaud, and X. Gonze, “Ab initio study of the volume dependence of dynamical and thermodynamical properties of silicon,” Phys. Rev. B 53, 4488–4497 (1996).
  * Carrier, Wentzcovitch, and Tsuchiya (2007) P. Carrier, R. Wentzcovitch, and J. Tsuchiya, “First-principles prediction of crystal structures at high temperatures using the quasiharmonic approximation,” Phys. Rev. B 76, 064116 (2007).
  * Friedrich  _et al._ (2015) M. Friedrich, A. Riefer, S. Sanna, W. Schmidt, and A. Schindlmayr, “Phonon dispersion and zero-point renormalization of LiNbO3 from density-functional perturbation theory,” J. Phys. Condens. Matter 27, 385402 (2015).
  * Saidi, Poncé, and Monserrat (2016) W. Saidi, S. Poncé, and B. Monserrat, “Temperature dependence of the energy levels of methylammonium lead iodide perovskite from first principles,” J. Phys. Chem Lett. 7, 5247 (2016).
  * Molina-Sanchez  _et al._ (2016) A. Molina-Sanchez, M. Palummo, A. Marini, and L. Wirtz, “Temperature-dependent excitonic effects in the optical properties of single-layer MoS2,” Phys. Rev. B 93, 155435 (2016).
  * Antonius and Louie (2016) G. Antonius and S. G. Louie, “Temperature-induced topological phase transitions: Promoted versus suppressed nontrivial topology,” Phys. Rev. Lett. 117, 246401 (2016).
  * Villegas, Rocha, and Marini (2016b) C. E. P. Villegas, A. R. Rocha, and A. Marini, “Electron-phonon scattering effects on electronic and optical properties of orthorhombic GeS,” Phys. Rev. B 94, 134306 (2016b).
  * Ziaei and Bredow (2017) V. Ziaei and T. Bredow, “Dynamical electron-phonon coupling, GW self-consistency, and vertex effect on the electronic band gap of ice and liquid water,” Phys. Rev. B 95, 235105 (2017).
  * Tutchton, Marchbanks, and Wu (2018) R. Tutchton, C. Marchbanks, and Z. Wu, “Structural impact on the eigenenergy renormalization for carbon and silicon allotropes and boron nitride polymorphs,” Phys. Rev. B 97, 205104 (2018).
  * Cao  _et al._ (2019) J. Cao, J. D. Querales-Flores, S. Fahy, and I. Savic, “Thermally induced band gap increase and high thermoelectric figure of merit of n-type pbte,” arXiv: cond-mat.mtrl-sci 1906, 05322 (2019).
  * Querales-Flores  _et al._ (2019) J. Querales-Flores, J. Cao, S. Fahy, and I. Savic, “Temperature effects on the electronic band structure of pbte from first principles,” Phys. Rev. Mat. 3, 055405 (2019).
  * Querales-Flores  _et al._ (2020) J. D. Querales-Flores, P. Aguado-Puente, Ðorđe Dangić, J. Cao, P. Chudzinski, T. N. Todorov, and M. Grüć1, “Towards temperature-induced topological phase transition in snte: A first principles study,” arXiv: cond-mat.mtrl-sci 2004, 11959 (2020).
  * Lihm and Park (2020) J.-M. Lihm and C.-H. Park, “Phonon-induced renormalization of electron wave functions,” Phys. Rev. B 101, 121102 (2020).
  * Han and Bester (2013) P. Han and G. Bester, “Large nuclear zero-point motion effect in semiconductor nanoclusters,” Phys. Rev. B 88, 165311 (2013).
  * Patrick, Jacobsen, and Thygesen (2015) C. E. Patrick, K. W. Jacobsen, and K. S. Thygesen, “Anharmonic stabilization and band gap renormalization in the perovskite CsSnI3,” arXiv: cond-mat.mtrl-sci , 6152 (2015).
  * Zacharias, Patrick, and Giustino (2015) M. Zacharias, C. E. Patrick, and F. Giustino, “Stochastic approach to phonon-assisted optical absorption,” Phys. Rev. Lett. 115, 177401 (2015).
  * Monserrat (2016a) B. Monserrat, “Correlation effects on electron-phonon coupling in semiconductors: Many-body theory along thermal lines,” Phys. Rev. B 93, 100301(R) (2016a).
  * Zacharias and Giustino (2016) M. Zacharias and F. Giustino, “One-shot calculation of temperature-dependent optical spectra and phonon-induced band-gap renormalization,” Phys. Rev. B 94, 075125 (2016).
  * Monserrat (2018) B. Monserrat, “Electron-phonon coupling from finite differences,” J. Phys. Condens. Matter 30, 083001 (2018).
  * Bravic and Monserrat (2019) I. Bravic and B. Monserrat, “Finite temperature optoelectronic properties of bas from first principles,” Phys. Rev. Materials 3, 065402 (2019).
  * Engel  _et al._ (2020) M. Engel, M. Marsman, C. Franchini, , and G. Kresse, “Electron-phonon interactions using the projector augmented-wave method and wannier functions,” Phys. Rev. B 101, 184302 (2020).
  * Botti and Marques (2013) S. Botti and M. A. L. Marques, “Strong renormalization of the electronic band gap due to lattice polarization in the GW formalism,” Phys. Rev. Lett. 110, 226404 (2013).
  * Lambrecht, Bhandari, and van Schilfgaarde (2017) W. R. L. Lambrecht, C. Bhandari, and M. van Schilfgaarde, “Lattice polarization effects on the screened coulomb interaction W of the GW approximation,” Phys. Rev. Mat. 1, 043802 (2017).
  * Bhandari  _et al._ (2018) C. Bhandari, M. van Schilgaarde, T. Kotani, and W. R. L. Lambrecht, “All-electron quasi-particle self-consistent GW band structures for SrTiO3 including lattice polarization corrections in different phases,” Phys. Rev. Mat. 2, 013807 (2018).
  * Zhang  _et al._ (2020) Y. Zhang, Z. Wang, J. Xi, and J. Yang, “Temperature-dependent band gaps in several semiconductors: from the role of electron-phonon renormalization,” J. Phys. Condens. Matt. YYYY, in press, https://doi.org/10.1088/1361–648X/aba45d (2020).
  * Fröhlich, Pelzer, and Zienau (1950) H. Fröhlich, H. Pelzer, and S. Zienau, “Properties of slow electrons in polar materials,” Phil. Mag. 41, 221–242 (1950).
  * Fröhlich (1952) H. Fröhlich, “Interaction of electrons with lattice vibrations,” Proc. R. Soc. Lond. A 215, 291–298 (1952).
  * Mahan (2000) G. D. Mahan,  _Many-Particle Physics_ (Kluwer, New York, 2000).
  * Mecholsky  _et al._ (2014) N. Mecholsky, L. Resca, I. Pegg, and M. Fornari, “Theory of band warping and its effects on thermoelectronic transport properties,” Phys. Rev. B 89, 155131 (2014).
  * Veithen, Gonze, and Ghosez (2005) M. Veithen, X. Gonze, and P. Ghosez, “Nonlinear optical susceptibilities, raman efficiencies, and electro-optic tensors from first-principles density functional perturbation theory,” Phys. Rev. B 71, 125107 (2005).
  * Luttinger and Kohn (1955) J. M. Luttinger and W. Kohn, “Motion of electrons and holes in perturbed periodic fields,” Phys. Rev. 97, 869–883 (1955).
  * Monserrat (2016b) B. Monserrat, “Vibrational averages along thermal lines,” Phys. Rev. B 93, 014302 (2016b).
  * Yang  _et al._ (2016) X. Yang, Y. Wang, H. Yan, and Y. Chen, “Effects of epitaxial strains on spontaneous polarizations and band gaps of alkaline-earth-metal oxides MO (M = Mg, Ca, Sr, Ba),” Computational Materials Science 121, 61–66 (2016).
  * Singh  _et al._ (2011) H. Singh, M. Singh, S. Kumar, and M. K. Kashyap, “Full potential calculation of electronic properties of rutile RO2 (R=Si, Ge, Sn and Pb) compounds via modified Becke Johnson potential,” Physica B 406, 3825–3830 (2011).