# Multi-View MOOC Quality Evaluation via Information-Aware
Graph Representation Learning

Lu Jiang1, Yibin Wang1, Jianan Wang1, Pengyang Wang2∗, Minghao Yin1,3∗
Corresponding author

###### Abstract

In this paper, we study the problem of MOOC quality evaluation which is
essential for improving the course materials, promoting students’ learning
efficiency, and benefiting user services. While achieving promising
performances, current works still suffer from the complicated interactions and
relationships of entities in MOOC platforms. To tackle the challenges, we
formulate the problem as a course representation learning task-based and
develop an Information-aware Graph Representation Learning(IaGRL) for multi-
view MOOC quality evaluation. Specifically, We first build a MOOC
Heterogeneous Network (HIN) to represent the interactions and relationships
among entities in MOOC platforms. And then we decompose the MOOC HIN into
multiple single-relation graphs based on meta-paths to depict the multi-view
semantics of courses. The course representation learning can be further
converted to a multi-view graph representation task. Different from
traditional graph representation learning, the learned course representations
are expected to match the following three types of validity: (1) the agreement
on expressiveness between the raw course portfolio and the learned course
representations; (2) the consistency between the representations in each view
and the unified representations; (3) the alignment between the course and MOOC
platform representations. Therefore, we propose to exploit mutual information
for preserving the validity of course representations. We conduct extensive
experiments over real-world MOOC datasets to demonstrate the effectiveness of
our proposed method.

## Introduction

Massive open online course (MOOC) has been prevalent for online tutoring and
self-studying in recent decades by providing numerous course materials,
intermediate feedback, and interactions between student and instructors. Among
which, MOOC course quality evaluation is one of the vital tasks in MOOC
platform management for helping improve the course materials, promote
students’ learning efficiency (Jiang et al. 2021), and benefit user services
(e.g., course recommendation (Wang et al. 2022; Jiang et al. 2022a), cognitive
diagnosis (Jiang et al. 2022b), etc).

Current studies in MOOC quality evaluation lie in two aspects: (1) manual
evaluation (Wang et al. 2021), which evaluates the course quality by domain
experts based on a pre-defined rubric; and (2) automated evaluation (Pérez-
Martín, Rodríguez-Ascaso, and Molanes-Lopez 2021; Betanzos, Costa-jussà, and
Belanche 2017), which judges the course quality automatically based on
historical records in the platform. While achieving promising results, current
works still exhibit limitations: First of all, manual-based methods are time-
consuming and labor-intensive. And, in most of cases, the experts do not have
the complete domain knowledge to evaluate every course on the MOOC platform.
Second, most of the automated methods utilize students’ reviews as the
criteria for evaluating course quality. However, students’ evaluation of the
quality of MOOCs is biased and subjective, and cannot yield unified
evaluations. In fact, course quality evaluation in the MOOC platform is a
complicated process involving by multiple parties, which can be examined from
different views. Therefore, integrating semantics and opinions from different
views to inform unified representations of the course becomes the key to
reasonable MOOC quality evaluation.

However, two unique challenges arise in achieving this goal. First, how to
design an appropriate data structure for capturing complex interactions among
different types of entities in the MOOCs platform? Second, how should we
guarantee the validity of the multi-view representations? Next, we will
outline how we tackle these challenges.

First, in a MOOCs platform, we observe that in addition to the student and
course, there exist multiple types of entities and multiple types of
relationships between pairs of different entities. The complex MOOC data
structures are always represented in heterogeneous information networks (MOOC
HIN) (Shi et al. 2017). Considering the participation of multiple entities on
the MOOC platform and the impact of interactions between entities and courses
on MOOC quality evaluation. It is difficult to obtain a comprehensive
evaluation of the MOOC quality if we merely depend on a single view. Only
utilizing a single type of interaction may overlook important relationships
between courses and other entities. For example, ”student click course” and
”teacher upload course” have dissimilar semantics even though they are
included in the same course. These heterogeneous relationships provide rich
information from multi-view. Thus, it requires incorporating these
heterogeneous relationships into the representation learning of the course
entities. To address the above issues, we use meta-paths (Sun et al. 2017) as
the guidance to capture multi-view representations of courses in MOOC
heterogeneous information network.

Second, although multi-view node embedding can be obtained by performing
representation learning on MOOC HIN, how guaranteeing the validity of the
learned course representations remains a challenge. Specifically, the validity
of course representations lies in three aspects: (1) the course
representations should preserve the same semantics as the raw course
portfolio; (2) the representations in each view should be consistent with the
unified representations of the course; (3) the course representations should
be aligned with the overall representations of the MOOC platform. The three
types of validity indicate strong correlations between the three pairs of
representations. Therefore, to ensure validity, we aim to maximize the three
correlations between the pair of course representations and the raw course
portfolio, the pair of unified course representations and each view, and the
pair of course representations and platform representations. In this paper, we
exploit mutual information (MI), a powerful correlation measure, to quantify
the correlations in each pair of representations.

In summary, we propose an Information-aware Graph Representation
Learning(IaGRL) for multi-view MOOC quality evaluation. The main contributions
are as follows:

  * •

We formulate the problem of MOOC quality evaluation as a multi-view graph
representation learning task.

  * •

We construct MOOC HIN and propose to exploit meta-paths to extract the
semantics of MOOC relationships in different views.

  * •

We identify three types of validity of course representations, and provide an
information-aware course representation learning framework.

  * •

We conduct extensive experiments over real-world MOOC datasets to validate the
effectiveness of our proposed method.

## Definitions and Problem Statement

We introduce the key definitions and the problem statement. Then we present
the overview of the proposed method. Some important notations are summarized
in Table 1.

Table 1: Symbol and Definitions Symbol | Definition
---|---
$\mathcal{G}$ | A given MOOC heterogeneous graph
$\mathcal{V,E}$ | Set of nodes, edges
$v,e$ | MOOC HIN node, edge
$\mathcal{MP}$ | A set of meta-paths
$\mathbf{X}$ | The features matrix of courses
$\mathbf{A}$ | The adjacency matrix base different meta-paths
$\tilde{\textbf{D}}$ | The diagonal matrix base different meta-paths
$\tilde{\textbf{h}}$ | The multi-view course representation
W | The weights of GCN layer
h | The unified course representation
$\alpha^{MP_{i}}$ | The importance of each meta-path
M | The platform representation
$\mathcal{D}$ | Mutual information based discriminator
$\lambda_{q},\lambda_{j},\lambda_{s},\lambda_{y}$ | The weight of different losses

### Definitions and Problem Statement

###### Definition 1

MOOC Heterogeneous Information Network(HIN) A MOOC HIN is defined as
$\mathcal{G}=(\mathcal{V},\mathcal{E})$ with a node type mapping function
$\phi:\mathcal{V}\rightarrow\mathcal{N}$ and an edge type mapping function
$\psi:\mathcal{E}\rightarrow\mathcal{R}$. Specifically, in MOOC HIN, there are
four types of nodes: students(denoted as U), teachers(denoted as T),
courses(denoted as C), and subjects(denoted as S). And there are three types
of links: ”click” which is to demonstrate the relation between students and
courses, ”upload” which is to demonstrate the relation between teachers and
courses, and ”include” which is to demonstrate the relation between subjects
and courses. The MOOC HIN is defined as the following groups of triplet
facts:(1) $<$student, $"$click$"$, course$>$, (2) $<$teacher, $"$upload$"$,
course$>$, and (3) $<$subject, $"$include$"$, course$>$.

###### Definition 2

Meta-path based on MOOC HIN A meta-path $MP$ based on MOOC HIN is defined as a
path in the form of
$\mathcal{V}_{1}\stackrel{{\scriptstyle\mathcal{E}_{1}}}{{\to}}\mathcal{V}_{2}\stackrel{{\scriptstyle\mathcal{E}_{2}}}{{\to}}\cdots\stackrel{{\scriptstyle\mathcal{E}_{l}}}{{\to}}\mathcal{V}_{l+1}$(abbreviated
as $\mathcal{V}_{1},\mathcal{V}_{2},\cdots,\mathcal{V}_{l+1}$), which
describes a composite relation
$\mathcal{E}=\mathcal{E}_{1}\circ\mathcal{E}_{2}\circ\cdots\circ\mathcal{E}_{l}$
between object $\mathcal{V}_{1},\mathcal{V}_{2},\cdots,\mathcal{V}_{l+1}$,
where $\circ$ denotes the composition operator on relations. In the MOOC HIN,
two courses can be connected via multiple paths, e.g.,
$C\stackrel{{\scriptstyle upload}}{{\longrightarrow}}T\stackrel{{\scriptstyle
upload^{-1}}}{{\longrightarrow}}C$, $C\stackrel{{\scriptstyle
click}}{{\longrightarrow}}U\stackrel{{\scriptstyle
click^{-1}}}{{\longrightarrow}}C$, $C\stackrel{{\scriptstyle
include}}{{\longrightarrow}}S\stackrel{{\scriptstyle
include^{-1}}}{{\longrightarrow}}C$. A set of meta-paths are defined as
$\mathcal{MP}=\\{MP_{1},MP_{2},\cdots MP_{||MP||}\\}$.

###### Definition 3

Problem Statement In this paper, we study the problem of MOOC quality
evaluation. We formulate the problem as an information-aware graph
representation learning task. Formally, we aim to find a mapping function
$f:\mathcal{G}\rightarrow\textbf{h}$ that takes the MOOC HIN $\mathcal{G}$ as
input, and outputs information-aware representations
$\textbf{h}=\\{\textbf{h}_{1},\textbf{h}_{2},...,\textbf{h}_{N}\\}$, for
evaluating the course quality on a MOOC platform.

Figure 1: Framework Overview.

## Method

In this section, we first present an overview of our proposed framework. Then,
we introduce the multi-view course representation learning with validity
guarantee in detail.

### Framework Overview

Figure 1 shows an overview of the proposed two-stage framework: (1) Stage 1:
multi-view course representation of learning; and (2) Stage 2: course
representation validity guarantee. Specifically, in Stage 1, we extracted our
three meta-paths from the MOOC HIN and constructed the course adjacency matrix
under three different views through the meta-path. Then, we proposed to learn
the representation of course under different views by using the graph
convolutional network(GCN) (Kipf and Welling 2016a) through the encoder-
decoder paradigm. By minimizing the reconstruction loss that follows the
convention of contrastive learning styles. In Stage 2, we exploit MI
maximization to ensure the three types of course representation validity, with
(i) the agreement on expressiveness between the raw course portfolio and the
learned course representations; (ii) the consistency between the
representations in each view and the unified representations; and (iii) the
alignment between the course and MOOC platform representations.

### Course Representation Learning

#### Multi-View Representations.

Given the MOOC HIN $\mathcal{G=(V,E)}$ with a set of meta-paths
$\mathcal{MP}=\\{MP_{1},MP_{2},\cdots MP_{||MP||}\\}$ and the corresponding
adjacency matrix $\mathbf{A}=\\{A_{1},A_{2},\cdots A_{|MP|}\\}$, and $|MP|$
denotes the number of meta-paths. Let $\mathbf{X}=\\{X_{1},X_{2},\cdots
X_{n}\\}$ denoted as the course attribute matrix of the MOOC HIN. In this
paper, the course attributes are represented by a $d$-dimensional vector that
describes the contents of the course, including the headline, abstract, etc.
The course attribute matrix is generated through the Doc2Vec (Le and Mikolov
2014) model.

For better generality, we learn multi-view representation with GCN in an
unsupervised fashion. We use generalized advantage estimation(GAE) (Schulman
et al. 2016) to learn representation in an encode-decode paradigm.
Specifically, the encoder is a GCN with the following layer-wise propagation
rule, the multi-view course representation can be denoted as:

$\tilde{\textbf{h}}=(\tilde{\textbf{D}}^{-\frac{1}{2}}{\tilde{\textbf{A}}}\tilde{\textbf{D}}^{-\frac{1}{2}})\textbf{X}\textbf{W},$
(1)

where $\tilde{\textbf{A}}=\textbf{A}+\textbf{I}$ is the adjacency matrix
corresponds to a single meta-path with self-connections and I is the identity
matrix. $\tilde{\textbf{D}}_{ii}=\sum_{i}\tilde{\textbf{A}}_{ij}$ is the
diagonal matrix, and W is the weight. The decoder is an inner product of the
learned representation to recover the adjacency matrix:

$\hat{\textbf{A}}=sigmoid(\tilde{\textbf{h}}{\tilde{\textbf{h}}^{T}})$ (2)

The objective is to minimize the reconstruction loss between the original
adjacency matrix $\tilde{\textbf{A}}$ and reconstructed adjacency matrix
$\hat{\textbf{A}}$. We follow the implementation of VGAE (Kipf and Welling
2016b) to do the sampling and loss optimization: we take connected neighbors
as positive nodes, and disconnected nodes as negative nodes, and sample a few
of them to construct the data samples. We expect positive samples to be
connected, and negative samples to be disconnected after reconstruction, thus
the reconstruction is converted to a classification task, which can be
optimized using cross-entropy loss.

$\mathcal{L}_{q}=-log\hat{\textbf{A}}_{pos}-log(1-\hat{\textbf{A}}_{neg}),$
(3)

where $\hat{\textbf{A}}_{pos}$ and $\hat{\textbf{A}}_{neg}$ are derived from
the positive course nodes pairs and the negative course node pairs
respectively, based on Equation 3.

#### Unified Course Representation.

Going through the GAE, we learn the representations for each meta-path.
However, different meta-paths should not be considered equally. To address
this problem, we adopt the idea of heterogeneous deep graph infomax(HDGI) (Ren
et al. 2019), and utilize the self-attention mechanism to fuse the embedding
of courses learned under the guide of different meta-paths and generate the
unified course embedding. Specifically, we learn the self-attention weights
for different meta-paths as follows:

$\textbf{h}=\sum_{i}^{|MP|}att(\tilde{\textbf{h}}_{i}),$ (4)

where att($\cdot$) indicates the self-attention function, and h indicates the
unified course representation, which has integrated the self-attention weights
of different meta-paths. In this paper, we mainly focus on the course from a
multi-view. In order to make representations from different meta-paths
comparable, we transform each course’s representation from a different view
with a linear transformation. The parameters are shared weight matrix
$\textbf{W}^{\prime}$ and shared bias vector $\vec{b}$. Based on the
distinguishing ability of views, we introduce the shared attention vector
$\vec{q}$ of different views to calculate the importance of each view. The
importance of the meta-paths can be calculated as follows:

$a^{MP_{i}}=\frac{1}{|MP|}\sum^{|MP|}_{j=1}tanh(\vec{q}^{T}\cdot[\textbf{W}^{\prime}\cdot\tilde{\textbf{h}}^{{MP}_{i}}_{j}+\vec{b}])$
(5)

Then, we use the softmax function to normalize the importance of meta-paths,
the normalized weight of each meta-path can be calculated as follows:

$\alpha^{MP_{i}}=\frac{exp(a^{MP_{i}})}{\sum_{j=1}^{|MP|}exp(a^{MP_{j}})}$ (6)

The self-attention unified course representation h can be represented as
follows:

$\textbf{h}=\sum_{i=1}^{|MP|}\alpha^{MP_{i}}\tilde{\textbf{h}}_{i}$ (7)

### Validity Guarantee

#### Raw Portfolio-Representation Agreement.

The learned course representations are expected to achieve agreement with the
raw course portfolio in describing courses. We refer to this as Raw Portfolio-
Representation Agreement. We use mutual information (MI) to quantify the
agreement between the representation h learned by the course node in the
unified view and the representation X of the raw features of the MOOC.
Following the idea of DIM and DGI (Velickovic et al. 2019), a Jensen Shannon
MI estimator is defined to estimate and maximize the MI between X and h:

$\displaystyle MI(\textbf{X},\textbf{h}):=$
$\displaystyle\mathbb{E}_{\textbf{X}}[-sp(-\mathcal{D}(\textbf{X},\textbf{h}^{(MP)}))]+$
(8)
$\displaystyle\mathbb{E}_{\bar{\textbf{X}}}[sp(\mathcal{D}(\textbf{X},\textbf{h}^{(MP)}))],$

where sp is the softplus function that $sp(c)\times log(1+e^{c})$, X is the
positive sample set and $\bar{\textbf{X}}$ is negative sample set. We will
present how we generate positive and negative samples later. Since the noise-
contrastive type objective with a standard binary cross-entropy (BCE) can
effectively maximize mutual information, we define the loss function as:

$\displaystyle\mathcal{L}_{j}=-\frac{1}{|MP|}\sum_{j=1}^{|MP|}\sum_{i}$
$\displaystyle\mathbb{E}_{\textbf{X}}[log\mathcal{D}_{j}(\textbf{X}^{({MP}_{j})}_{i},{\textbf{h}}_{i})]-$
(9)
$\displaystyle\mathbb{E}_{\bar{\textbf{X}}}[log(1-\mathcal{D}_{j}(\tilde{\textbf{X}}_{i}^{({MP}_{j})},{\textbf{h}}_{i}))],$

where $\mathcal{D}_{j}$ denotes a discriminator to justify the given pairs as
positive or negative. For the $k$-th view, we regard the positive sample as
the pair of $(X^{({MP}_{j})}_{i},{\textbf{h}}_{i})$ and the negative samples
as the pairs of $(\tilde{X}^{({MP}_{j})}_{i},{\textbf{h}}_{i})$.

#### Multi-View Consistency.

Although we deconstruct heterogeneous graphs into different views, the course
representations in each view are expected to be consistent with the unified
course representations in semantics, which is defined as multi-view
consistency. We propose to exploit MI to measure the multi-view consistency.
First, we get the multi-view representation $\tilde{\textbf{h}}$ and unified
course representation h. Specifically, h is obtained from self-attention on
the one hand and $\tilde{\textbf{h}}$ is obtained from the GCN encoder. Then,
we use neural network estimation MI $({\textbf{h}};\tilde{\textbf{h}})$ to
maximize the mutual information between the unified course representation h
and multi-view representation $\tilde{\textbf{h}}$. We have a similar noise
contrastive loss function:

$\displaystyle\mathcal{L}_{s}=-\frac{1}{|MP|}\sum_{j=1}^{|MP|}\sum_{i}$
$\displaystyle\mathbb{E}_{\textbf{X}}[log\mathcal{D}_{s}({\textbf{h}}_{i},\tilde{\textbf{h}}_{i}^{(MP_{j})})]-$
(10)
$\displaystyle\mathbb{E}_{\bar{\textbf{X}}}[log(1-\mathcal{D}_{s}({\textbf{h}}_{i},\acute{\textbf{h}}_{i}^{({MP}_{j})}))],$

where $\mathcal{D}_{s}$ denotes a discriminator for discriminating positive
consistent pairs, ${\textbf{h}}_{i}$ is unified course
representation.$\acute{\textbf{h}_{i}}$ is the result of $\tilde{\textbf{h}}$
after random shuffling. For the $k$-th view, we design the positive samples as
the pairs of $({\textbf{h}}_{i},\tilde{\textbf{h}}_{i}^{({MP}_{j})})$, and the
negative samples as the pairs of
$({\textbf{h}}_{i},\acute{\textbf{h}}_{i}^{({MP}_{j})})$. The objective is to
minimize $\mathcal{L}_{s}$, which is equivalent to maximize
$MI({\textbf{h}},\tilde{\textbf{h}})$.

#### Course-Platform Alignment.

While there is no doubt that the courses are different from each other, the
course representations are required to align with the MOOC platform
representations within the same semantic scope. To accomplish the course-
platform alignment, we first obtain the platform representation by considering
it as the graph-level representation of MOOC HIN for courses. Along this line,
we take the platform summary vector M by averaging over all course
representations:

$\textbf{M}=\sigma(\frac{1}{N}\sum^{N}_{i=1}{\textbf{h}}_{i}),$ (11)

where $\sigma$ is the sigmoid function and $N$ is the number of course nodes.
We continue to leverage MI to capture the course-platform alignment, by
maximizing the MI between In order to maximize course-platform alignment, we
introduce MI and then based on the relationship between the Jensen-Shanno
degree and mutual information. We can maximize the mutual information between
platform representation and unified course representation using the binary
cross-entropy loss of the discriminator as follows:

$\displaystyle\mathcal{L}_{y}=-\sum_{i}$
$\displaystyle\mathbb{E}_{\textbf{X}}[log\mathcal{D}_{y}({\textbf{h}}_{i},\textbf{M})]-$
(12)
$\displaystyle\mathbb{E}_{\bar{\textbf{X}}}[log(1-\mathcal{D}_{y}(\bar{\textbf{h}}_{i},\textbf{M}))],$

where $\mathcal{D}_{y}$ denotes a discriminator to provide probability scores
for sampled course-platform pairs, h is unified course
representation.$\bar{\textbf{h}}$ is the result of h after random shuffling.
We design the positive samples as the pairs of
$({\textbf{h}}_{i},\textbf{M})$, and the negative samples as the pairs of
$(\bar{\textbf{h}}_{i},\textbf{M})$. The objective is to minimize
$\mathcal{L}_{y}$, which is equivalent to maximize
$MI(\textbf{h},\textbf{M})$.

### Optimization

The loss of the model includes: (i) the contrastive learning loss for graph
reconstruction $\lambda_{q}$ (Equation 3); (ii) the raw portfolio-
representation agreement learning loss $\lambda_{j}$ (Equation 9); (iii) the
multi-view consistency learning loss $\lambda_{s}$ (Equation 10); and (iv) the
course-platform alignment learning loss $\lambda_{y}$ (Equation 11). The
objective is to minimize the overall loss $\mathcal{L}$ as follows:

$\displaystyle\mathcal{L}=\lambda_{q}\mathcal{L}_{q}+\lambda_{j}\mathcal{L}_{j}+\lambda_{s}\mathcal{L}_{s}+\lambda_{y}\mathcal{L}_{y}$
(13)

where $\lambda_{q}$, $\lambda_{j}$, $\lambda_{s}$ and $\lambda_{y}$ are the
weights for $\mathcal{L}_{q}$, $\mathcal{L}_{j}$, $\mathcal{L}_{s}$ and
$\mathcal{L}_{y}$, respectively. The above loss can be optimized through
gradient descent, and the representations of nodes can be learned when the
optimization is completed.

## Experiment

In the experiment, we aim to answer the following three research questions:

  * •

Q1. How is the performance of our proposed IaGRL in the MOOC quality
evaluation task?

  * •

Q2. How do the meta-paths affect the course quality evaluation performance?

  * •

Q3. How do the different learning losses affect the course quality evaluation
performance?

Then, we will introduce statistical information about real-world MOOC data,
and experiment settings and compare IaGRL with several baselines on this data.

### Data Description

We evaluate the performance over real-world MOOC data. The data constitute a
MOOC heterogeneous information network containing 4 types of entities and 3
types of relations. The course scores range from 0 to 5. In the data
preprocessing step, we filtered out users that have fewer than 3 links. After
data preprocessing, we split the datasets into two non-overlapping sets: 20%
of the datasets as the testing set and the rest 80% as the training set. Table
2 shows the detailed statistics of the dataset.

Table 2: Statistics and descriptions of dataset Properties | Descriptions | Statistics
---|---|---
Student | Users who studied course | 4931
Course | Learning materials | 10919
Teacher | Users who uploaded course | 1213
Subject | An area of knowledge | 35
Time | Time period | 2015-2018

### Baselines and Evaluation Metrics

We compare the performances of our method with the following baselines:

(1) MLP. The multilayer perceptron(MLP) is a feedforward supervised artificial
neural network structure. The MLP can contain multiple hidden layers to
realize the classification modeling.

(2) DeepWalk. The DeepWalk model is a recently proposed network embedding
method that extends the word2vec model (Mikolov et al. 2013) by truncated
random walks (Perozzi, Al-Rfou, and Skiena 2014).

(3) DeepWalk+F. The DeepWalk+F model not only adds the neighbor features after
the random walk but also adds the attribute features of practice.

(4) GCN. The graph convolutional network(GCN) performs information aggregation
based on the Laplacian or adjacent matrix of the complete graph (Kipf and
Welling 2016a).

(5) GAE. The graph autoencoder(GAE) learns node representation, we use an
embedding layer to encode the node with GCN (Schulman et al. 2016).

(6) GraphSAGE. The GraphSAGE proposes neighborhood batch sampling to enable
scalable training with max, min, and LSTM aggregation functions (Hamilton,
Ying, and Leskovec 2017).

(7) GAT. The graph attention network(GAT) introduces a multi-head attention
mechanism into the aggregation function, which learns the importance of the
neighborhood of each node for information aggregation (Velickovic et al.
2017).

(8) GATv2. GATv2 solves the simple graph problem of GAT using a static
attention mechanism, a dynamic graph attention variant that is more expressive
than GAT (Brody, Alon, and Yahav 2021).

The last five are graph representation learning methods. To evaluate the
performance of the models for MOOC quality, we adopt two widely used
evaluation metrics for multi-classification performance, e.g., Accuracy and
Macro-F1. Specifically, Accuracy measures the evaluation accuracy that the
user scores successfully rated. And macro f1 help to consider performance
comprehensively in case of imbalanced data. For both metrics, the larger the
value, the better the performance.

### Parameter Setting

For Deepwalk and Deepwalk+F, we set the number of walks = 80, the size of
representation = 128, the walk length = 20, and the window size = 10. For GCN,
GAE and GraphSAGE, we set the number of layers = 2, the input feature
size=128, the output feature size = 128, and the learning rate = 0.001. For
GAT and GATv2, we set the number of layers = 2, the layer heads = [2,1], the
input feature size = 128, the output feature size = 128, and the learning rate
= 0.003. For my model we set the learning rate = 0.001, the l2 = 0.001, the
dropout = 0.1, the input feature = 128, and the out feature = 128. The device
we used was two RTX 6000 with 24GiB memory and CUDA=11.2.

### Overall Performance(Q1)

In this section, we compare the overall performance of all models on the real-
world dataset. In general, Figure 2 shows our model outperforms other baseline
methods for both Accuracy and Macro-F1 metric. Compared to MLP, which is the
representative of node attribute, random walk-based methods (DeepWalk,
Deepwalk+F), graph convolution network-based methods (GCN, GAE, GraphSAGE,
GAT, GATv2), and information-based methods (our proposed method) perform
better in modeling MOOC heterogeneous network. Compared to DeepWalk and
DeepWalk+F, which is the random walk-based methods, our proposed framework
additionally considers heterogeneous graph embedding and information-aware of
the learned representation. Compared to graph convolution network-based
methods, the information guarantee provided by our proposed method further
elevates the reasonability of the learned representations. In summary, the
results validate that incorporating multi-view and information-aware can
improve the quality of representation learning.

(a) Accuracy.

(b) Macro-F1.

Figure 2: Overall comparisons of evaluation Accuracy and Macro-F1 of our
methods with other baselines.

### Influence of Meta-paths(Q2)

In this part of the experiments, we analyze how meta-paths affect the
performance of methods. We consider both single meta-path and their
combinations in our method. Specifically, we select three types of meta-paths
to represent the relatedness between pair of courses, including MP1:
$C\rightarrow T\stackrel{{\scriptstyle-1}}{{\rightarrow}}C$, MP2:
$C\rightarrow U\stackrel{{\scriptstyle-1}}{{\rightarrow}}C$ and MP3:
$C\rightarrow S\stackrel{{\scriptstyle-1}}{{\rightarrow}}C$. To analyze the
impact of meta-paths, we study the performance in three aspects:(1)with single
view attention weights; (2)with single course-related meta-path and their
combinations on our method; and (3)with meta-path on baselines.

#### Compared with attention weights.

We calculate the importance of the attention weights for each meta-path in our
method, the results are shown in table 3. From the table, we can find the most
important meta-path is MP1, from the teacher view, follow by MP3(from the
subject view) and MP2(from the student view), respectively. It is easy to
understand that when evaluating a course quality from multiple views, the
teacher’s influence on course quality is more important. One interesting
observation is that the student’s view has the least impact on course quality,
even less than the subject view, which is an objective perspective. A possible
explanation is that our data came from a MOOC platform based on primary
education. On the one hand, the student’s cognitive level is in the primary
state, on the other hand, the students’ behavior of clicking courses is guided
by the teacher, which is less subjective.

Table 3: The importance of each meta-path. Weights | MP1 | MP2 | MP3
---|---|---|---
$\alpha^{MP_{i}}$ | 0.4655 | 0.2242 | 0.3103

#### Compared with the different meta-paths combination.

We compare our method with both single meta-path and their combinations. The
results are shown in Table 4, we can find that every single meta-path exhibits
different performance, where the performance ranking is MP1$>$MP3$>$MP2, and
the combinations of single meta-paths follow the same tendency. This
illustrates that different meta-paths indicate different relations and the
combination including more meta-paths will exhibit better performance, and the
best performance is achieved by combining all three meta-paths.

Table 4: The results from different combinations of meta-paths on our method. Meta-path | Accuracy | Macro-F1
---|---|---
MP1 | 0.3596 | 0.1529
MP2 | 0.3407 | 0.1397
MP3 | 0.3543 | 0.1707
MP1 & MP2 | 0.3697 | 0.1536
MP1 & MP3 | 0.3864 | 0.1783
MP2 & MP3 | 0.3656 | 0.1742
MP1 & MP2 & MP3 | 0.4101 | 0.2011

#### Compared with meta-paths on baselines.

And in order to further verify the effect of meta-paths, we study the meta-
path on the baselines based on graph convolutional network methods. The
results are shown in Table 5, from Table 5, we can find that compared with the
original algorithms, including meta-path combinations will show better
performance. Especially in the GCN, and GraphSAGE methods, the growth of
performance is quite obvious.

Table 5: The results from different combinations of meta-paths on baselines. Methods | Accuracy | Macro-F1
---|---|---
GCN | 0.2368 | 0.0724
GCN & MPi=1,2,3 | 0.3466 | 0.0857
GAE | 0.3501 | 0.1307
GAE & MPi=1,2,3 | 0.3555 | 0.1686
GraphSAGE | 0.2770 | 0.1130
GraphSAGE & MPi=1,2,3 | 0.3620 | 0.1092
GAT | 0.2914 | 0.0987
GAT & MPi=1,2,3 | 0.3187 | 0.1076
GATv2 | 0.3252 | 0.1026
GATv2 & MPi=1,2,3 | 0.3258 | 0.1220

### Analysis of $\mathcal{L}_{j}$, $\mathcal{L}_{s}$, $\mathcal{L}_{y}$ (Q3)

In order to analyze the contribution of representation raw portfolio
agreement, multi-view consistency and course-platform alignment, we define six
variants of our proposed model: (1) MI-IaGRL-J, which adds $\mathcal{L}_{j}$
to the base model; (2) MI-IaGRL-S, which adds $\mathcal{L}_{s}$ to the base
model; (3) MI-IaGRL-Y, which adds $\mathcal{L}_{y}$ to the base model; (4) MI-
IaGRL-J,S, which adds $\mathcal{L}_{j}$ and $\mathcal{L}_{s}$ to the base
model; (5) MI-IaGRL-J,Y, which adds $\mathcal{L}_{j}$ and $\mathcal{L}_{y}$ to
the base model; and (6) MI-IaGRL-S,Y, which adds $\mathcal{L}_{s}$ and
$\mathcal{L}_{y}$ to the base model.

As shown in Figure 3, we compare the MI-IaGRL-J, MI-IaGRL-S, MI-IaGRL-Y, MI-
IaGRL-J,S, MI-IaGRL-J,Y, MI-IaGRL-S,Y and MI-IaGRL in the experiment. When the
combination of loss functions increases from single to two to three, the
overall trend of metric values are increasing. The results indicate that the
integrated raw portfolio-representation agreement, the multi-view consistency,
and the course-platform alignment significantly improve the performance of
MOOC quality evaluation.

(a) Accuracy.

(b) Macro-F1.

Figure 3: Analysis of $\mathcal{L}_{j}$, $\mathcal{L}_{s}$, $\mathcal{L}_{y}$.

## Related Work

Our work is related to the following two domains of prior work, including MOOC
quality evaluation and graph representation learning.

### MOOC Quality Evaluation

Our work has a connection with MOOC quality evaluation. Prior literature on
MOOC quality evaluation lies in two aspects: (1) manual evaluation (Wang et
al. 2021), and (2) automated evaluation (Pérez-Martín, Rodríguez-Ascaso, and
Molanes-Lopez 2021; Betanzos, Costa-jussà, and Belanche 2017). Manual-based
methods evaluate the course quality by domain experts based on a pre-defined
rubric. For example, Wang et al. discussed the quality analysis of
instructional design based on the ten-principle framework (Wang et al. 2021).
The manual-based method can evaluate course quality accurately, but they are
not suitable for large-scale applications. More and more automatic methods
appear, such as, Zhuo et al. designed a teaching quality assessment model on
the MOOC platform based on comprehensive fuzzy evaluation (Zhuo and Dong
2017).

### Graph Representation Learning

Different from the traditional graph optimization method (Chen et al. 2023;
wang2020sccwalk) focusing on efficiency, the graph embedding method focuses on
information extraction. Graph embedding aims to project nodes in a graph into
a $d$-dimensional vector space, in which the representation of nodes can
reflect the relationship between nodes, and retain the semantic information of
nodes. Graph embedding methods can be categorized into homogeneous graph
embedding(node2vec (Grover and Leskovec 2016), struct2vec (Ribeiro, Saverese,
and Figueiredo 2017) and Deepwalk (Perozzi, Al-Rfou, and Skiena 2014)),
heterogeneous graph embedding(metapath2vec (Dong, Chawla, and Swami 2017),
HHNE (Wang, Zhang, and Shi 2019), SHNE (Zhang, Swami, and Chawla 2019)).
However, the above methods ignore the mutual information. In order to handle
the information-aware of graphs, there are several methods have been proposed,
including DGI (Velickovic et al. 2019), HDGI (Ren et al. 2019).

## Conclusion

In this paper, we study the problem of MOOC course quality evaluation with
MOOC heterogeneous information networks and propose an information-aware graph
representation learning framework for multi-view MOOC quality evaluation.
Specifically, we first formulate the problem of MOOC quality evaluation as a
multi-view graph representation learning task. Second, we construct MOOC HIN
and propose to exploit meta-paths to extract the semantics of MOOC
relationships from different views. Third, we identify three types of validity
of course representations, with (i) the agreement on expressiveness between
the raw course portfolio and the learned course representations; (ii) the
consistency between the representations in each view and the unified
representations; and (iii) the alignment between the course and MOOC platform
representations. Finally, we conduct extensive experiments over real-world
MOOC data to validate the effectiveness of our method.

## Acknowledgments

This work is supported by the Fundamental Research Funds for the Central
Universities 2412019ZD013, NSFC (under Grant No.61976050, 61972384, 61806050,
62106040), Jilin Science and Technology Department 20200201280JC, the Science
and Technology Development Fund, Macau SAR (File no. SKL-IOTSC-2021-2023 to
Pengyang Wang), the Start-up Research Grant of University of Macau (File no.
SRG2021-00017-IOTSC to Pengyang Wang).

## References

  * Betanzos, Costa-jussà, and Belanche (2017) Betanzos, M.; Costa-jussà, M. R.; and Belanche, L. 2017.  Tradares: A Tool for the Automatic Evaluation of Human Translation Quality within a MOOC Environment.  _Appl. Artif. Intell._ , 31(3): 288–297.
  * Brody, Alon, and Yahav (2021) Brody, S.; Alon, U.; and Yahav, E. 2021.  How Attentive are Graph Attention Networks?  _CoRR_ , abs/2105.14491.
  * Chen et al. (2023) Chen, J.; Cai, S.; Wang, Y.; Xu, W.; Ji, J.; and Yin, M. 2023.  Improved local search for the minimum weight dominating set problem in massive graphs by using a deep optimization mechanism.  _Artificial Intelligence_ , 314: 103819.
  * Dong, Chawla, and Swami (2017) Dong, Y.; Chawla, N. V.; and Swami, A. 2017.  metapath2vec: Scalable Representation Learning for Heterogeneous Networks.  In _KDD_ , 135–144. ACM.
  * Grover and Leskovec (2016) Grover, A.; and Leskovec, J. 2016.  node2vec: Scalable Feature Learning for Networks.  In _KDD_ , 855–864. ACM.
  * Hamilton, Ying, and Leskovec (2017) Hamilton, W. L.; Ying, Z.; and Leskovec, J. 2017.  Inductive Representation Learning on Large Graphs.  In _NIPS_ , 1024–1034.
  * Jiang et al. (2021) Jiang, L.; Wang, P.; Cheng, K.; Liu, K.; Yin, M.; Jin, B.; and Fu, Y. 2021.  EduHawkes: A Neural Hawkes Process Approach for Online Study Behavior Modeling.  In _SDM_ , 567–575. SIAM.
  * Jiang et al. (2022a) Jiang, L.; Wang, Y.; Xie, S.; Wu, J.; Yin, M.; and Wang, J. 2022a.  Which courses to choose? recommending courses to groups of students in online tutoring platforms.  _Applied Intelligence_.
  * Jiang et al. (2022b) Jiang, L.; Zhang, W.; Wang, Y.; Luo, N.; and Yue, L. 2022b.  Augmenting Personalized Question Recommendation with Hierarchical Information for Online Test Platform.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 13087: 103–117.
  * Kipf and Welling (2016a) Kipf, T. N.; and Welling, M. 2016a.  Semi-Supervised Classification with Graph Convolutional Networks.  _CoRR_ , abs/1609.02907.
  * Kipf and Welling (2016b) Kipf, T. N.; and Welling, M. 2016b.  Variational Graph Auto-Encoders.  _CoRR_ , abs/1611.07308.
  * Le and Mikolov (2014) Le, Q. V.; and Mikolov, T. 2014.  Distributed Representations of Sentences and Documents.  In _ICML_ , volume 32 of _JMLR Workshop and Conference Proceedings_ , 1188–1196. JMLR.org.
  * Mikolov et al. (2013) Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013.  Distributed Representations of Words and Phrases and their Compositionality.  In _NIPS_ , 3111–3119.
  * Pérez-Martín, Rodríguez-Ascaso, and Molanes-Lopez (2021) Pérez-Martín, J.; Rodríguez-Ascaso, A.; and Molanes-Lopez, E. M. 2021.  Quality of the captions produced by students of an accessibility MOOC using a semi-automatic tool.  _Univers. Access Inf. Soc._ , 20(4): 677–690.
  * Perozzi, Al-Rfou, and Skiena (2014) Perozzi, B.; Al-Rfou, R.; and Skiena, S. 2014.  DeepWalk: online learning of social representations.  In _KDD_ , 701–710. ACM.
  * Ren et al. (2019) Ren, Y.; Liu, B.; Huang, C.; Dai, P.; Bo, L.; and Zhang, J. 2019.  Heterogeneous Deep Graph Infomax.  _CoRR_ , abs/1911.08538.
  * Ribeiro, Saverese, and Figueiredo (2017) Ribeiro, L. F. R.; Saverese, P. H. P.; and Figueiredo, D. R. 2017.  _struc2vec_ : Learning Node Representations from Structural Identity.  In _KDD_ , 385–394. ACM.
  * Schulman et al. (2016) Schulman, J.; Moritz, P.; Levine, S.; Jordan, M. I.; and Abbeel, P. 2016.  High-Dimensional Continuous Control Using Generalized Advantage Estimation.  In Bengio, Y.; and LeCun, Y., eds., _ICLR_.
  * Shi et al. (2017) Shi, C.; Li, Y.; Zhang, J.; Sun, Y.; and Yu, P. S. 2017.  A Survey of Heterogeneous Information Network Analysis.  _IEEE Trans. Knowl. Data Eng._ , 29(1): 17–37.
  * Sun et al. (2017) Sun, Y.; Yuan, N. J.; Xie, X.; McDonald, K.; and Zhang, R. 2017.  Collaborative Intent Prediction with Real-Time Contextual Data.  _ACM Trans. Inf. Syst._ , 35(4): 30:1–30:33.
  * Velickovic et al. (2017) Velickovic, P.; Cucurull, G.; Casanova, A.; Romero, A.; Liò, P.; and Bengio, Y. 2017.  Graph Attention Networks.  _CoRR_ , abs/1710.10903.
  * Velickovic et al. (2019) Velickovic, P.; Fedus, W.; Hamilton, W. L.; Liò, P.; Bengio, Y.; and Hjelm, R. D. 2019.  Deep Graph Infomax.  In _ICLR_. OpenReview.net.
  * Wang et al. (2022) Wang, C.; Zhu, H.; Wang, P.; Zhu, C.; Zhang, X.; Chen, E.; and Xiong, H. 2022.  Personalized and Explainable Employee Training Course Recommendations: A Bayesian Variational Approach.  _ACM Trans. Inf. Syst._ , 40(4): 70:1–70:32.
  * Wang et al. (2021) Wang, X.; Lee, Y.; Lin, L.; Mi, Y.; and Yang, T. 2021.  Analyzing instructional design quality and students’ reviews of 18 courses out of the Class Central Top 20 MOOCs through systematic and sentiment analyses.  _Internet High. Educ._ , 50: 100810.
  * Wang, Zhang, and Shi (2019) Wang, X.; Zhang, Y.; and Shi, C. 2019.  Hyperbolic Heterogeneous Information Network Embedding.  In _AAAI_ , 5337–5344. AAAI Press.
  * Zhang, Swami, and Chawla (2019) Zhang, C.; Swami, A.; and Chawla, N. V. 2019.  SHNE: Representation Learning for Semantic-Associated Heterogeneous Networks.  In _WSDM_ , 690–698. ACM.
  * Zhuo and Dong (2017) Zhuo, C.; and Dong, X. 2017.  An Applicable Way of Teaching Quality Evaluation Based on MOOC Platform.  _Int. J. Emerg. Technol. Learn._ , 12(3): 57–67.