11institutetext: Pattern Recognition and Bioinformatics group, Delft
University of Technology, Delft, Netherlands

# Self-Supervised PPG Representation Learning Shows High Inter-Subject
Variability

Ramin Ghorbani Corresponding author ()11<EMAIL_ADDRESS>Marcel J.T.
Reinders 11 David M.J. Tax 11

###### Abstract

With the progress of sensor technology in wearables, the collection and
analysis of PPG signals are gaining more interest. Using Machine Learning, the
cardiac rhythm corresponding to PPG signals can be used to predict different
tasks such as activity recognition, sleep stage detection, or more general
health status. However, supervised learning is often limited by the amount of
available labeled data, which is typically expensive to obtain. To address
this problem, we propose a Self-Supervised Learning (SSL) method with a
pretext task of signal reconstruction to learn an informative generalized PPG
representation. The performance of the proposed SSL framework is compared with
two fully supervised baselines. The results show that in a very limited label
data setting (10 samples per class or less), using SSL is beneficial, and a
simple classifier trained on SSL-learned representations outperforms fully
supervised deep neural networks. However, the results reveal that the SSL-
learned representations are too focused on encoding the subjects.
Unfortunately, there is high inter-subject variability in the SSL-learned
representations, which makes working with this data more challenging when
labeled data is scarce. The high inter-subject variability suggests that there
is still room for improvements in learning representations. In general, the
results suggest that SSL may pave the way for the broader use of machine
learning models on PPG data in label-scarce regimes.

###### Keywords:

Self-Supervised Learning, Representation Learning, Autoencoder, PPG, Human
Activity Recognition, Inter-Subject Variability

##  1 Introduction

In recent years, wearables such as smartwatches and health trackers, equipped
with a photoplethysmography (PPG) sensor, are becoming increasingly popular
[1]. PPG is a non-invasive, low-cost optical measurement that can measure
tissue blood flow over time following each pulse wave ejected from the heart.
PPG works on the principle of pulse oximetry, wherein a sensor emits light to
the skin and measures the intensity of light that is reflected or transmitted
through the skin. Changes in arterial blood volume cause PPG signal variations
[2, 3]. The cardiac rhythm corresponding to the PPG signal’s periodicity can
be used to obtain additional useful information from the users and predict
various tasks. Some examples of research on PPG signals are related to
Activity Recognition [4], Heart Rate Estimation [5], Blood Pressure Prediction
[6], Biometric Identification [7], Sleep Staging Detection [8], and Atrial
Fibrillation Detection [9].

In existing research, analyzing the PPG signals can be broadly categorized
into signal processing and machine learning methods. The majority of machine
learning solutions for PPG-based tasks utilize fully-supervised learning
methods, which can be associated with several limitations. A fully-supervised
learning setup usually requires considerable computational resources and time.
Additionally, this setup requires large human-annotated datasets for high
performance. Typically, obtaining labeled data is very costly and time-
intensive, and the amount of labeled data is therefore insufficient in real-
world applications, for instance, in the case of heart failure detection or
fall detection. When automated detectors have to be trained on this type of
problem, a good representation of the data with few numbers of informative
features is essential [10]. Therefore, it is necessary to address the label-
scarcity problem.

One approach to obtain a good informative representation is ‘Self-Supervised
Learning’ (SSL). In SSL, two tasks are defined: a ‘pretext’ task and a
‘downstream’ task. The pretext task is the task of learning informative
representations by itself. For instance, an auto-encoder tries to precisely
reconstruct the input, squeezing the information through a bottleneck layer.
It thereby learns a condensed, low-dimensional representation containing all
necessary information to reconstruct the input exactly [11]. It is assumed
that this learned low-dimensional representation reduces the complexity of the
data by reducing anomalies and noise and, at the same time, improves the
ability to detect patterns in the data simpler and better. Hence, learned
representations from the pretext task should be helpful for learning a second-
stage classifier on the downstream task, which is the actual task of interest
that we want to solve.

The latest research in the field of machine learning shows the potential of
SSL for finding generalized and robust representations [12, 13, 14, 15]. The
existing works in representation learning are generally concentrated on image-
based applications where variations in the data could be visually observed.
However, SSL is rarely applied to the field of time series data, especially
biosignals. In recent years, some have applied SSL to time series data to show
that this method can improve the representation, and they could confirm the
potential of self-supervision in capturing important information even in the
absence of labeled data. For instance, Lan et al. [16] introduced an Intra-
inter Subject self-supervised Learning (ISL) model customized for ECG signals.
Their model integrates medical knowledge into self-supervision to effectively
learn from intra-inter subject differences. Their results over different
evaluation scenarios showed that the learned representations are information-
rich and more generalizable than other state-of-the-art methods for diagnosing
cardiac arrhythmias in label-scarce regimes. As another example, Banville et
al. [15] investigated SSL to learn representations from EEG signals. They
explored two pretext tasks based on temporal context prediction and
contrastive predictive coding on two clinically EEG-relevant downstream tasks.
The results show that linear classifiers trained on SSL-learned
representations consistently outperform purely supervised deep neural networks
in label-scarce regimes while reaching competitive performance when all labels
are available. These findings are, however, not yet shown on noisy PPG
signals, so it still remains to be shown whether self-supervision can bring
improvements over standard supervised approaches on PPG signals.

In this paper, we focus on Human Activity Recognition (HAR) from PPG data.
This is gaining interest since PPG data can be easily acquired from any of the
widely available wearable devices [17]. Researchers have been exploring how
SSL techniques can be either extended or explicitly designed for HAR tasks on
accelerometer and gyroscope data. However, they have not yet looked into the
PPG data specifically. In one of the early pioneering works, Saeed et al. [18]
used the task of identifying which signal transformation has been applied to a
particular data sample as a pretext task using accelerometer and gyroscope
data. The results show that SSL drastically reduces the requirement of labeled
activity data, narrowing the gap between supervised and unsupervised
techniques for learning meaningful representations.

Concluding, to the best of our knowledge, there are currently no studies using
SSL on PPG data in label-scarce regimes. Therefore, we present the first
detailed analysis of SSL tasks on PPG signals with attention to Activity
Recognition as a downstream task. Our main contributions are:

  1. 1.

Proposing a SSL framework for PPG data in label-scarce regimes

  2. 2.

Evaluating whether human activity recognition task can be done better when
using SSL representations

  3. 3.

Investigating the Inter-subject variability in PPG data and exploring how this
is captured by the SSL representation

##  2 Proposed Framework

An overview of the proposed SSL framework is shown in Figure 1. We use an
Autoencoder (AE) to learn a representation of the (unlabeled) data
(unsupervised learning). Given an unlabeled dataset
$D_{U}=\left\\{\textbf{x}_{i}\right\\}_{i=1}^{N_{u}}$ where
$\textbf{x}_{i}\in\mathbb{R}^{1\times T}$ is a vector of length $T$ and
$N_{u}$ is the number of vectors (samples). The encoder maps each input vector
into a latent space representation $\textbf{h}_{i}=E_{\phi}(\textbf{x}_{i})$
where $\textbf{h}_{i}\in\mathbb{R}^{1\times d}$ where $d<T$. After that,
$\textbf{h}_{i}$ is fed into the decoder component of the model, which follows
the same approach to map $\textbf{h}_{i}$ to the output values
$\hat{\textbf{x}}_{i}=D_{\theta}(\textbf{h}_{i})$ where
$\hat{\textbf{x}}_{i}\in\mathbb{R}^{1\times T}$. The encoder and decoder are
parametrized by $\phi$ and $\theta$, respectively. The AE is trained to
minimize the mean squared error between $\textbf{x}_{i}$ and
$\hat{\textbf{x}}_{i}$ [19]:

$\textbf{L}_{Total}=1/N_{u}\sum_{i=1}^{N_{u}}(1/T\parallel\textbf{x}_{i}-\hat{\textbf{x}}_{i}\parallel^{2})$
(1)

We have used a combination of Convolutional Neural Network layers (CNN) with
the AE architecture, a Convolutional Neural Network AutoEncoder (CNN-AE). The
main advantage of using a CNN-AE is a better reconstruction for the PPG signal
as it exploits the correlations between time measurements in the PPG signal
and thus captures the time-dependent information better.

For the downstream task, we use the original preprocessed PPG as the input to
$E_{\phi}$ with frozen trained weights from the reconstruction task to get the
related latent representation $\textbf{h}_{i}$. This $\textbf{h}_{i}$ is used
as input to train a simple classifier such as Logistic Regression (LR) or
k-Nearest Neighbors (kNN). We do this because when the number of training
samples is small, these simple classifiers often outperform more flexible and
complex models. The classifier is trained on $\textbf{h}_{i}$’s from several
subjects and tested on a completely new subject.

Figure 1: The overall proposed Self-Supervised framework

##  3 Experimental Setup

###  3.1 Dataset

We use the PPG-Dalia public dataset, which was collected by Reiss et al. [20]
for the PPG-based heart rate estimation task. This dataset contains recordings
of 15 subjects performing daily activities such as _sitting,
ascending/descending stairs, playing table soccer, cycling, driving a car,
having lunch, walking,_ and _working_. The measurements are obtained from
wrist and chest-worn devices. Besides the activities, the transient periods
between the activities are also recorded. We removed data from subject number
6 due to hardware issues during data recording. Note that _having lunch,
driving a car_ , and _working_ activities are categorized as concurrent or
inter-leaved human activities, where actions of multiple activities are
carried out simultaneously or where activities contain various activities
while their actions can be interleaved in a shuffled manner [21, 22].
Therefore, we only considered the remaining five human activities for further
study. Detailed information about the dataset is available in Appendix A.

###  3.2 Data Preprocessing

For the pretext task, a band-pass $2^{th}$ order Butterworth filter with low
and high frequencies of 0.1 - 6 Hz is applied to the whole PPG signal of each
subject individually. The filtered signal is normalized to zero mean and unit
variance per subject. The final normalized filtered signals are segmented into
a fixed window size of 8 seconds while two successive windows overlap by 6
seconds in the test dataset (this setting is common for PPG data). To increase
the training set size, the two successive windows overlap by 7 seconds in the
training dataset. For the downstream task, the PPG signals are split into 8
seconds windows while two successive windows overlap by 6 seconds in both
training and test datasets. Activity labels are assigned to the corresponding
PPG windows based on the available annotations. Detailed information about the
labeling process of PPG windows is available in Appendix A.1.

###  3.3 Implementation

####  3.3.1 Proposed SSL

The hyperparameters and the architecture of the proposed CNN-AE are
systematically determined by searching through all possible combinations to
obtain the best performance. Eventually, we used a CNN-AE architecture deep
learning model consisting of three convolution layers, followed by the
Exponential Linear Unit (ELU) activation function, Batch Normalization, and
MaxPooling layers. The decoder consists of the hidden layers in the reverse
order of the encoder section. The Adam optimizer with a learning rate of 0.01,
a decay rate of 0.001, and a clip-norm value of 0.9 are used. The batch size
is 128, and training runs for 200 epochs. Finally, the parameters of all
layers are randomly initialized. To assess the randomness of the deep learning
framework, each training process for each test subject is repeated five times.
Leave-One-Subject-Out cross-validation (LOSO) is used to evaluate the
reconstruction performance. The final implemented CNN-AE model details are
available in Appendix B.1.

For the downstream task, two simple classifiers are trained on the SSL-learned
representations separately: a Logistic Regression and a kNN classifier (SSL-LR
and SSL-kNN, respectively). The SSL-LR is regularized with the L2 penalty term
and is solved using LIBLINEAR [23]. The SSL-kNN is trained with reweighted
neighbors [24], where points are weighted by the inverse of their distance.
Therefore, closer neighbors of a query point will have a larger influence than
far away neighbors. Due to the different number of training samples which are
2, 5, 10, 50, and 1000 per class, the number of neighbors is selected as 8,
19, 39, 115, and 350, respectively. The LOSO is used to evaluate the AUC
performance.

####  3.3.2 Comparative Baselines

The performance of the SSL method is compared with two other baseline models:
a simple and a more complex one. The simple baseline model (a typical baseline
in SSL research) is trained directly on the original preprocessed PPG
representations and consists of the encoder part of the CNN-AE from the
pretext task, extended with one classification layer at the end. The encoder
part is thus trained on the classification task immediately and not in a self-
supervised setting. The Adam optimizer with a learning rate of 0.001 and a
clip-norm value of 0.6 are used. The batch size is 128, and training runs for
200 epochs. The more complex baseline is a CNN-LSTM model also trained
directly on the original preprocessed PPG representations. The architecture of
the complex baseline consists of a convolution layer followed by a hyperbolic
tangent function, Batch Normalization, MaxPooling layers, and then a LSTM
layer with a hyperbolic tangent activation function, followed by a
classification layer at the end. The Adam optimizer with a learning rate of
0.001 and a clip-norm value of 0.6 are used. The batch size is 128, and
training runs for 200 epochs. To assess the randomness of these Deep Learning
frameworks, each training process for each test subject is repeated five
times. Both baseline models use the LOSO to evaluate the AUC performance. The
details of the implemented models are available in Appendix B.2 and B.3.

####  3.3.3 Biometric Identification (BI) for Exploring Inter-Subject
Variability

If there is a large inter-subject variability, the subjects should be easily
discriminated in the representation. To check if a subject can indeed be
easily discriminated, we train and evaluate a kNN classifier with reweighted
neighbors ($k=20$) per activity. Note that this classifier is not optimized at
all on the SSL-learned representations; the kNN fully relies on the metric
that is induced by the CNN-AE latent representation $\textbf{h}_{i}$. A good
performance of the kNN for the BI task suggests that the learned
representation from CNN-AE is heavily biased towards encoding different
subjects and not so many other tasks like activities. In this experiment, PPG
data is preprocessed with the same steps as the downstream task preprocessing.
Afterward, the PPG windows of each activity are selected to sample a separate
balanced training set over the subjects. The 4-fold cross-validation (75% for
the training and 25% for the test set) is used to evaluate the AUC
performance.

##  4 Results

###  4.1 Pretext Task

To determine a suitable dimensionality $d$ of the learned representation
$\textbf{h}_{i}$, we compute the relative MSE (i.e., MSE in Eq. (1) normalized
by the total variance across test subjects’ data) by varying $d$ using the
CNN-AE. The results are shown in Table 1. It can be seen that the
reconstruction error decreases with increasing dimensionality $d$. As the
representation with a lower dimension is more suited for learning with limited
labels, we chose $d=64$ when proceeding with the downstream task. Also, later
experiments show that $d=64$ leads to better performance on the downstream
task compared to other dimensionalities. Details on the effect of SSL-learned
representation dimensionality on the downstream task performance is shown in
Appendix C.

Table 1: Mean Relative MSE results of test subjects (LOSO) for signal reconstruction task by varying $d$ using the CNN-AE  Dimension of the $\textbf{h}_{i}$ |  Relative MSE Results
---|---
$d=2$ |  $0.83\pm 0.02$
$d=8$ |  $0.59\pm 0.03$
$d=32$ |  $0.14\pm 0.03$
$\bm{d=64}$ |  $\bm{0.02\pm 0.00}$
$d=128$ |  $0.00\pm 0.00$

###  4.2 Downstream Task

In Figure 2a, we show the AUC performances on the downstream task of
predicting activity type for a varying number of training samples per class.
The performance of the proposed SSL method is compared with the simple and
complex baseline methods. As the number of training samples per class
decreases, the performances of all methods drop, confirming the negative
influence of when less and less samples with labels are available. The linear
SSL-LR model fails to improve the performance compared to the baseline models
when a few ($<10$) training samples per class are available. However, SSL-kNN,
as a non-linear solution, outperforms the baselines and the SSL-LR in the
label-scarce regimes. This suggests that the SSL-learned representation is
still too complex for a simple linear solution like LR. One reason for such a
behavior could be the high Inter-Subject variability in the SSL-learned
representations. Figure 2b shows the SSL-kNN performance of each of the
individual test subjects, for a varying number of training samples per class.
It can be observed that the AUC performance can vary between 0.5 and 0.7 for a
small training size of $N=2$, and even for very large training sizes of
$N=1000$, the AUC still varies between 0.55 and 0.8. This indicates that the
data distributions of different subjects vary significantly, thus indicating
large inter-subject variability.

To explore the inter-subject variability more deeply, we also investigated the
possibility of classifying subjects during each activity using a kNN model.
Note that this is now different than our initial domain task; here, we are
interested in whether subjects are still separable in the latent
representations $\textbf{h}_{i}$ (which, in principle, is undesired when
generalizing over subjects). The results of the BI task in Table 2 show that
subjects can be discriminated perfectly for some activities such as _sitting_
or _walking_. It can be seen that there are enough differences among subjects
in the original as well as the SSL-learned representations. Moreover, the SSL-
learned representation seems to highlight the inter-subject variation as its
mean performance is consistently higher than on the original data. This
suggests that the AE learned representation is more focused on encoding the
different subjects and not so much on the domain tasks of interest, that is,
predicting the different activities.

(a)

(b)

Figure 2: a) Mean AUC performance of test subjects (using LOSO) for Activity Recognition downstream task over the different number of training samples per class. b) The variability among different test subjects in AUC performance for Activity Recognition downstream task using SSL-kNN Table 2: Mean AUC Performance of test sets (4-fold cross-validation) for Biometric Identification task  Activities | Input
---|---
Original Representation |  SSL-Learned Representation
Sitting |  $0.84\pm 0.12$ |  $0.84\pm 0.01$
Ascending/Descending Stairs |  $0.63\pm 0.02$ |  $0.64\pm 0.03$
Playing Table Soccer |  $0.53\pm 0.01$ |  $0.61\pm 0.02$
Cycling |  $0.59\pm 0.05$ |  $0.61\pm 0.06$
Walking |  $0.72\pm 0.02$ |  $0.75\pm 0.02$

##  5 Discussion and Conclusion

We have evaluated the usefulness of self-supervised representation for the
activity recognition task when suffering from a label-scarcity in PPG data.
The representation is not optimized on the downstream classification task (for
which just a few labeled training samples may be available), but it is first
optimized to perform a data reconstruction pretext task (for which no
supervised information is needed). The results reveal that the SSL method can
compete and outperform fully supervised baselines when a kNN model is trained
on the SSL-learned representations in label-scarce regimes (with less than 50
samples per class). However, training a simple linear classifier like LR
(instead of kNN) is not helpful since the inter-subject variability introduces
too much non-linearities in the decision boundaries.

One should note that in the current study setup, fixed hyperparameters are
used across the data regimes for all baseline models. When copious amounts of
(unlabeled) data from all subjects would be available, all hyperparameters
could be optimized for every different task.

The poor performance of the LR classifier on the SSL-learned representations
shows that there is high inter-subject variability. High inter-subject
variability makes the generalization more challenging. In this case, a
subject-specific model could be a solution for improving the performance over
the learned representations. However, training a subject-specific model can be
expensive because a large amount of (labeled) data that has to be obtained
from each subject. Here, the SSL representation can come to the rescue, as we
have shown that this representation can improve performance with respect to
the original representation. However, there should be more focus on
disentangling the inter- and intra-subject variability.

This matter opens the door for future research to learn more generalized
informative PPG representations while addressing the inter-subject variability
problem. For instance, removing the subject-specific factors in order to
disentangle the inter-subject variations using a factor disentangling
sequential autoencoder [25], or performing contrastive learning among subjects
to learn distinctive representations [16] can be promising directions in
learning informative PPG representations.

## References

  * Castaneda et al. [2018] Denisse Castaneda, Aibhlin Esparza, Mohammad Ghamari, Cinna Soltanpur, and Homer Nazeran.  A review on wearable photoplethysmography sensors and their potential future applications in health care.  _International journal of biosensors & bioelectronics_, 4(4):195, 2018.
  * Kamal et al. [1989] AAR Kamal, JB Harness, G Irving, and AJ Mearns.  Skin photoplethysmography—a review.  _Computer methods and programs in biomedicine_ , 28(4):257–269, 1989.
  * Aoyagi and Miyasaka [2002] Takuo Aoyagi and Katsuyuki Miyasaka.  Pulse oximetry: its invention, contribution to medicine, and future tasks.  _Anesthesia and analgesia_ , 94(1):S1–S3, 2002\.
  * Boukhechba et al. [2019] Mehdi Boukhechba, Lihua Cai, Congyu Wu, and Laura E Barnes.  Actippg: using deep neural networks for activity recognition from wrist-worn photoplethysmography (ppg) sensors.  _Smart Health_ , 14:100082, 2019.
  * Zhang et al. [2014] Zhilin Zhang, Zhouyue Pi, and Benyuan Liu.  Troika: A general framework for heart rate monitoring using wrist-type photoplethysmographic signals during intensive physical exercise.  _IEEE Transactions on biomedical engineering_ , 62(2):522–531, 2014.
  * Ghosh et al. [2016] Shrimanti Ghosh, Ankur Banerjee, Nilanjan Ray, Peter W Wood, Pierre Boulanger, and Raj Padwal.  Continuous blood pressure prediction from pulse transit time using ecg and ppg signals.  In _2016 IEEE Healthcare Innovation Point-Of-Care Technologies Conference (HI-POCT)_ , pages 188–191. IEEE, 2016.
  * Everson et al. [2018] Luke Everson, Dwaipayan Biswas, Madhuri Panwar, Dimitrios Rodopoulos, Amit Acharyya, Chris H Kim, Chris Van Hoof, Mario Konijnenburg, and Nick Van Helleputte.  Biometricnet: Deep learning based biometric identification using wrist-worn ppg.  In _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_ , pages 1–5. IEEE, 2018.
  * Uçar et al. [2018] Muhammed Kürşad Uçar, Mehmet Recep Bozkurt, Cahit Bilgin, and Kemal Polat.  Automatic sleep staging in obstructive sleep apnea patients using photoplethysmography, heart rate variability signal and machine learning techniques.  _Neural Computing and Applications_ , 29(8):1–16, 2018.
  * Aliamiri and Shen [2018] Alireza Aliamiri and Yichen Shen.  Deep learning based atrial fibrillation detection using wearable photoplethysmography sensor.  In _2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)_, pages 442–445. IEEE, 2018.
  * Gyawali [2021] Prashnna Kumar Gyawali.  _Learning with Limited Labeled Data in Biomedical Domain by Disentanglement and Semi-Supervised Learning_.  Rochester Institute of Technology, 2021.
  * Tishby et al. [2000] Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  _arXiv preprint physics/0004057_ , 2000.
  * Tagliasacchi et al. [2019] Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek.  Self-supervised audio representation learning for mobile devices.  _arXiv preprint arXiv:1905.11796_ , 2019.
  * Kocabas et al. [2019] Muhammed Kocabas, Salih Karagoz, and Emre Akbas.  Self-supervised learning of 3d human pose using multi-view geometry.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1077–1086, 2019.
  * Wang et al. [2019] Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu.  Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4006–4015, 2019.
  * Banville et al. [2021] Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort.  Uncovering the structure of clinical eeg signals with self-supervised learning.  _Journal of Neural Engineering_ , 18(4):046020, 2021.
  * Lan et al. [2022] Xiang Lan, Dianwen Ng, Shenda Hong, and Mengling Feng.  Intra-inter subject self-supervised learning for multivariate cardiac signals.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 4532–4540, 2022.
  * Almanifi et al. [2022] Omair Rashed Abdulwareth Almanifi, Ismail Mohd Khairuddin, Mohd Azraai Mohd Razman, Rabiu Muazu Musa, and Anwar PP Abdul Majeed.  Human activity recognition based on wrist ppg via the ensemble method.  _ICT Express_ , 2022.
  * Saeed et al. [2019] Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien.  Multi-task self-supervised learning for human activity detection.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 3(2):1–30, 2019.
  * Kramer [1991] Mark A Kramer.  Nonlinear principal component analysis using autoassociative neural networks.  _AIChE journal_ , 37(2):233–243, 1991.
  * Reiss et al. [2019] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven.  Deep ppg: Large-scale heart rate estimation with convolutional neural networks.  _Sensors_ , 19(14):3079, 2019.
  * Afzali Arani et al. [2021] Mahsa Sadat Afzali Arani, Diego Elias Costa, and Emad Shihab.  Human activity recognition: A comparative study to assess the contribution level of accelerometer, ecg, and ppg signals.  _Sensors_ , 21(21):6997, 2021.
  * Sakr et al. [2019] Nehal A Sakr, Mervat Abu-Elkheir, Ahmed Atwan, and HH Soliman.  Data driven recognition of interleaved and concurrent human activities with nonlinear characteristics.  _Journal of Intelligent & Fuzzy Systems_, 37(4):5573–5588, 2019.
  * Fan et al. [2008] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin.  Liblinear: A library for large linear classification.  _the Journal of machine Learning research_ , 9:1871–1874, 2008.
  * Dudani [1976] Sahibsingh A Dudani.  The distance-weighted k-nearest-neighbor rule.  _IEEE Transactions on Systems, Man, and Cybernetics_ , (4):325–327, 1976.
  * Gyawali et al. [2019] Prashnna Kumar Gyawali, B Milan Horacek, John L Sapp, and Linwei Wang.  Sequential factorized autoencoder for localizing the origin of ventricular activation from 12-lead electrocardiograms.  _IEEE Transactions on Biomedical Engineering_ , 67(5):1505–1516, 2019.

##  Appendix 0.A Dataset Detailed Information

In the PPG-Dalia dataset, each subject followed a defined data collection
protocol. The duration of these activities was approximately defined as well.
However, since the goal of the data collection was to record data close to the
daily-life setting, subjects were instructed to carry out the activities as
naturally as possible. An overview of the data collection protocol is given in
Table A.

Table A: Dalia Datasets Activity Label Information  Class/Activity |  Duration (Min) | Description
---|---|---
Transient Periods |  – |  Before and after each activity, a transient period was included, in order to arrive at the starting location of the next activity
Sitting still |  10 |  Sitting still while reading. (motion artefact-free baseline)
Ascending/Descending stairs |  5 |  Climbing six floors up and going down again, repeating this twice
Table soccer |  5 |  Playing table soccer, 1 vs. 1 with the supervisor of the data collection
Cycling |  8 |  Performed outdoors, around research campus, following a defined route of about 2 km length with varying road conditions (gravel, paved)
Driving car |  15 |  Started at the parking ground of our research campus and was carried out within the area nearby. Subjects followed a defined route which took about 15 min to complete. The route included driving on different streets in a small city as well as driving on country roads
Lunch break |  30 |  This activity was carried out at the canteen of research campus. The activity included queuing and fetching food, eating, and talking at the table
Walking |  10 |  Walking back from the canteen to the office, with some detour
Working |  20 |  Subjects returned to their desk and worked on a computer

###  0.A.1 Labeling Process

An activity signal containing the label indexes synchronized with the PPG
signal is available. After creating the preprocessed PPG windows based on the
steps in Section 3, PPG windows should be labeled. At this step, the activity
signal is segmented into overlapping windows with the same settings as PPG
windows. Later, the activity windows containing two different label indexes
are removed along with the corresponding PPG windows. The remaining activity
windows have only one activity label index, so this label index is assigned to
the corresponding PPG window.

##  Appendix 0.B Deep Learning Models Architecture

All Deep Learning frameworks are developed and evaluated in python (Version
3.8.8) using Keras API. It should be noted that for the parameters that are
not mentioned in the implementation details, the Keras default settings are
used.

###  0.B.1 CNN-AE Model

The detailed architecture of implemented CNN-AE model for signal
reconstruction pretext task is shown in Table B.1.

Table B.1: The detailed architecture of implemented CNN-AE model  Layer (Type) |  Output Shape |  Param #
---|---|---
Encoder
Input Layer |  (None, 512, 1) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 512, 64) |  2112
Activation Layer |  (None, 512, 64) |  0
Batch Normalization Layer |  (None, 512, 64) |  256
Max Pooling1D Layer |  (None, 256, 64) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 256, 128) |  262272
Activation Layer |  (None, 256, 128) |  0
Batch Normalization Layer |  (None, 256, 128) |  512
Max Pooling1D Layer |  (None, 128, 128) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 128, 1) |  4097
Activation Layer |  (None, 128, 1) |  0
Batch Normalization Layer |  (None, 128, 1) |  4
Max Pooling1D Layer |  (None, 64, 1) |  0
Decoder
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 64, 64) |  2112
Activation Layer |  (None, 64, 64) |  0
Batch Normalization Layer |  (None, 64, 64) |  256
Up Sampling1D Layer |  (None, 128, 64) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 128, 128) |  262272
Activation Layer |  (None, 128, 128) |  0
Batch Normalization Layer |  (None, 128, 128) |  512
Up Sampling1D Layer |  (None, 256, 128) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 256, 1) |  4097
Activation Layer |  (None, 256, 1) |  0
Batch Normalization Layer |  (None, 256, 1) |  4
Up Sampling1D Layer (Output Layer) |  (None, 512, 1) |  0
| Total Params: 538,506
---
Trainable Params: 537,734 & Non-Trainable Params: 772

###  0.B.2 Simple Baseline Model

The detailed architecture of fully supervised model consists of the encoder
part of the CNN-AE model and additional classification layer for HAR
downstream task is shown in Table B.2.

Table B.2: The detailed architecture of implemented Simple Baseline Model Layer (Type) |  Output Shape |  Param #
---|---|---
Input Layer |  (None, 512, 1) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 512, 64) |  2112
Activation Layer |  (None, 512, 64) |  0
Batch Normalization Layer |  (None, 512, 64) |  256
Max Pooling1D Layer |  (None, 256, 64) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 256, 128) |  262272
Activation Layer |  (None, 256, 128) |  0
Batch Normalization Layer |  (None, 256, 128) |  512
Max Pooling1D Layer |  (None, 128, 128) |  0
Conv1D Layer (kernel-size = 32, padding = “same”) |  (None, 128, 1) |  4097
Activation Layer |  (None, 128, 1) |  0
Batch Normalization Layer |  (None, 128, 1) |  4
Max Pooling1D Layer |  (None, 64, 1) |  0
Reshape Layer |  (None, 64) |  0
Output Layer (activation = “softmax”) |  (None, 5) |  325
| Total Params: 269.578
---
Trainable Params: 269,192 & Non-Trainable Params: 386

###  0.B.3 Complex CNN-LSTM Baseline Model

The detailed architecture of the complex CNN-LSTM fully supervised model for
the HAR downstream task is shown in Table B.3.

Table B.3: The detailed architecture of implemented CNN-LSTM Baseline Model Layer (Type) |  Output Shape |  Param #
---|---|---
Input Layer |  (None, 512, 1) |  0
Conv1D Layer (kernel-size = 64, activation = “tanh”, kernel-regularizer = L2(0.01), bias-regularizer = L2(0.01)) |  (None, 449, 32) |  2080
Batch Normalization Layer |  (None, 449, 32) |  128
Max Pooling1D Layer (pool-size = 4) |  (None, 112, 32) |  0
Dropout (rate = 0.5) |  (None, 112, 32) |  0
LSTM Layer (kernel-regularizer = L2(0.01), bias-regularizer = L2(0.01)) |  (None, 32) |  8320
Output Layer (activation = “softmax”) |  (None, 5) |  165
| Total Params: 10,693
---
Trainable Params: 10,629 & Non-Trainable Params: 64

##  Appendix 0.C Effect of SSL-Learned Representation Dimensionality on
Downstream Task Performance

The effect of SSL-learned representation dimensionality on the downstream task
performance using SSL-kNN is shown in Table C. It can be seen that $d=64$,
which has the best mean relative MSE results over test subjects, leads to the
best downstream task performance when limited labeled data is available.

Table C: SSL-kNN AUC performance on Activity Recognition downstream task based on different SSL-learned representation dimensionality, showing the effect of using SSL-learned representation with different dimensionality Samples Per Class | Dimensionality of SSL-Learned Representation $\textbf{h}_{i}$
---|---
$d=2$ |  $d=8$ |  $d=32$ |  $d=64$ |  $d=128$
2 |  $0.50\pm 0.06$ |  $0.54\pm 0.04$ |  $0.57\pm 0.05$ |  $0.60\pm 0.05$ |  $0.57\pm 0.04$
5 |  $0.52\pm 0.05$ |  $0.58\pm 0.04$ |  $0.59\pm 0.06$ |  $0.62\pm 0.05$ |  $0.60\pm 0.05$
10 |  $0.52\pm 0.04$ |  $0.58\pm 0.04$ |  $0.61\pm 0.06$ |  $0.64\pm 0.05$ |  $0.61\pm 0.04$
50 |  $0.55\pm 0.03$ |  $0.58\pm 0.04$ |  $0.64\pm 0.04$ |  $0.63\pm 0.04$ |  $0.62\pm 0.03$
1000 |  $0.57\pm 0.05$ |  $0.62\pm 0.05$ |  $0.67\pm 0.08$ |  $0.69\pm 0.07$ |  $0.67\pm 0.08$


# Can Population-based Engagement
Improve Personalisation?
A Novel Dataset and Experiments

Sahan Bulathwela1 Meghana Verma2 María Pérez-Ortiz1 Emine Yilmaz1 and John
Shawe-Taylor1

1 Centre for Artificial Intelligence, University College London, UK  2 Indian
Institute of Technology Bombay, India<EMAIL_ADDRESS>
###### Abstract

This work explores how population-based engagement prediction can address
cold-start at scale in large learning resource collections. The paper
introduces i) VLE, a novel dataset that consists of content and video based
features extracted from publicly available scientific video lectures coupled
with implicit and explicit signals related to learner engagement, ii) two
standard tasks related to predicting and ranking context-agnostic engagement
in video lectures with preliminary baselines and iii) a set of experiments
that validate the usefulness of the proposed dataset. Our experimental results
indicate that the newly proposed VLE dataset leads to building context-
agnostic engagement prediction models that are significantly performant than
ones based on previous datasets, mainly attributing to the increase of
training examples. VLE dataset’s suitability in building models towards
Computer Science/ Artificial Intelligence education focused on e-learning/
MOOC use-cases is also evidenced. Further experiments in combining the built
model with a personalising algorithm show promising improvements in addressing
the cold-start problem encountered in educational recommenders. This is the
largest and most diverse publicly available dataset to our knowledge that
deals with learner engagement prediction tasks. The dataset, helper tools,
descriptive statistics and example code snippets are available publicly.

###### keywords:

Population-based Engagement, Cold-start, Educational Recommender, Personalised
Education, AI in Education

##  1 Introduction

With the rapid growth of Open Educational Resources (OER) [42] and Massively
Open Online Courses (MOOC) [37, 23], large educational resource repositories
need scalable tools to understand and estimate the engagement potential of
newly added materials [13]. While _contextualised engagement_ can be defined
as learner’s engagement driven by variables related to the context of the
learner at a given time in their learning path (e.g., learning needs/goals,
knowledge state, background on the topic, etc.), _context-agnostic engagement_
aims to capture patterns and features associated with engagement that instead
are applicable to an entire learner population rather than individual contexts
of specific learners [6]. Put simply, context-agnostic engagement is concerned
with the features that generally make a piece of educational material
engaging.

For works such as this, it is important to clarify that Information Retrieval
(IR) in education has very distinct characteristics than conventional web-
searches, as users tend to _gather_ information. Learners discovering
knowledge in an educational IR system usually carry out learning and
familiarising with novel concepts during the search session leading to the
task deviating or expanding into unanticipated sub tasks [15]. This behaviour
is drastically different from typical search engine users who are usually
aware of the exact information they are after. This distinction has led to
many works that are specific to educational IR [21, 32]. This difference also
makes conventional query log datasets sub-optimal when training e-learning IR
models. Prior works have shown this mismatch [11] leading to using education
specific datasets [40] instead of general-purpose query logs.

This work attempts to enrich the educational IR research domain by making a
large educational video dataset available to the community. Video Lecture
Engagement (VLE) is a novel dataset that presents around 12,000 peer-reviewed
scientific video lectures constructed from a popular OER repository,
VideoLectures.NET and contains a variety of lecture types ranging from
scientific talks to expert panels to MOOC-like lectures. The majority of these
lectures belong to Computer Science (CS), Artificial Intelligence (AI) and
Data Science subject areas, making this dataset a great source for
understanding learner engagement with AI/CS related educational videos. The
dataset provides an extensive set of textual and video-specific features
extracted from the lecture transcripts, together with Wikipedia topics covered
in the lecture (via entity linking) and user engagement labels (both explicit
and implicit) for each video.

This dataset is uniquely suited for solving the cold-start problem in
educational recommenders, both i) _user cold-start_ , where new users join the
system and we may not have enough information about their context and ii)
_item cold-start_ , where new educational content is released, for which we
may not have user engagement data yet and thus an engagement predictive model
would be necessary. The effectiveness of using context-agnostic engagement
prediction to address cold-start problem has been empirically demonstrated
before (see Figure 1). To the best of our knowledge, this is the largest
publicly available dataset to tackle such problems in educational
recommenders. The aim of the dataset is not to replace personalised
recommenders, but rather complement them providing meaningful population
baseline/priors to solve the common cold-start problem. While constructing the
VLE dataset is a major contribution of this paper, a series of experimental
results is also included as an additional contribution. These results validate
VLE dataset’s usefulness in i) significantly improving engagement prediction
models, ii) determining how the training data size impacts model improvements,
iv) using the VLN dataset for AI/CS and E-Learning/MOOC educational
recommenders and finally, v) showing the feasibility of fusing context-
agnostic prediction with personalised recommenders to improve overall
prediction performance.

Figure 1: How the difference between Mean Absolute Error (MAE) of population-
based and personalised models change with the number of training events per
learner. Until about 80 events in this plot, population-based predictions are
more accurate. Plot from [6]

##  2 Related Work

The majority of work in Intelligent Tutoring Systems (ITS) and Educational
Recommendation Systems (EduRecSys) revolve around using learner context [26,
8] to predict learner engagement. Although the connection between learner
engagement and learning gains have been explored by many [2, 20, 28], public
datasets in this realm are hard to come by. Many MOOC platforms such as edX
[23] and Khan Academy [29] harvest valuable data created in an _in-the-wild_
setting, yet this data is gated within course owners and consortia[21] (or
heavily anonymised) due to its proprietary nature. However, with the boom of
online education, understanding features of contents leading to context-
agnostic (population-based) engagement, an under researched knowledge area,
has become a critical area to explore. This work marks a significant step in
this direction by publishing a large dataset that the community can use to
push the research frontiers. Other public datasets relating to educational
question generation [11, 36], argument strength [34] and essay scoring [46]
serve different objectives and tasks.

Study of context-agnostic engagement of video lectures so far has been mainly
qualitative, deriving guidelines such as keeping videos short and in parts
[23] and using conversational language [3]. These guidelines are only useful
at the content creation stage and have no use in moderating the mammoth of
materials already circulating in the Internet. Our work, proposes to model
context-agnostic engagement using features associated with the educational
resource itself, which is useful for scalable quality assurance and
recommendation of existing (and newly created) materials.

###  2.1 Related Datasets

A few works on engagement prediction with videos (e.g. modelling watch time)
have been done with YouTube [16, 24], where YouTube specific meta-data
features (e.g. channel reputation, category etc.) are used exclusively.
Although large-scale public datasets relating to engagement prediction are
encountered, they focus on general-purpose videos (largely entertainment)
rather than educational videos [44]. Some of the features used by these works
share similarity with ones proposed in this paper (such as video duration,
language and topic features). However, no textual features (based on video
transcript) relating to understandability and presentation are used in this
prior work, making the methods hard to generalise outside of YouTube. Beyond
videos, educational IR [39, 14] and Wikipedia page quality prediction [19, 43]
has been attempted using features such as text style, readability, structure,
network, recency and review data. Publicly available Wikipedia article quality
dataset [19] with human annotated (explicit) labels is used to tackle the
latter task although implicit labels are not included in this dataset. Similar
datasets are available for automated essay scoring [41]. But, none of these
datasets fill the lack of resources for predicting engagement of educational
videos.

In the context of education, a different line of work looks at learner
engagement using learner-specific multi-modal data (brain waives [45], facial
expressions [25] etc.). While tackling a different task, these datasets are
mainly collected in a controlled lab setting where the in-the-wild factor is
missing [22]. A large number of public datasets and competitions in education
also relate to students interacting with learning problems (e.g. ASSISTments
[30] or multiple choice questions [12]), but these datasets, contrary to the
proposed VLE dataset, do not focus on implicit engagement. More relevant and
similar datasets that address context-agnostic engagement prediction in
education has been emerging with a focus on MOOCs. Studying approximately 800
videos from edX platform, Guo et al. [23] manually processed and provided a
qualitative analysis of engagement, with some features being relatively
subjective and difficult to automate. A similar work [38] takes 22 edX videos,
extracts cross-modal features and manually annotates their quality with no
focus on learner engagement with the videos. Neither dataset is publicly
available. MOOCCube is a recently released dataset that contains a spectrum of
details relating to MOOC interactions [47]. Although large, the video watch
logs in MOOCCube come from 190,000 users where as VLE signals are generated
over 1.1 Million users. As central values (e.g. median) are used for context-
agnostic engagement prediction, larger user base adds stability to the
engagement centres. MOOCCube uses a closed topic taxonomy disconnected from
Wikipedia which prevents the dataset from using all the powerful signals in
Wikipedia (e.g. semantic relatedness, category tree to name a few) to improve
prediction models. No engagement prediction attempts are published thus far to
demonstrate MOOCCube’s promise in the task.

On a more relevant contribution, [6] has demonstrated that context-agnostic
engagement prediction models can be built with implicit watch time based
labels and these models can be used in addressing the cold-start problem (see
Figure 1). We identify this work as the closest contribution to the proposed
dataset. They gather a collection of 4,000 video lectures with engagement
signals generated by 150,000 informal learners. We consider our work, VLE, as
an expansion to this dataset with around 12,000 video lectures where
engagement signals are generated by 7 times as many learners. The new dataset
also restricts itself to strictly English lectures to preserve the
meaningfulness of majority of features that are specific to English language.
VLE also introduces a new set of Wikipedia-based topical features.
Furthermore, this work also accompanies additional experiments that goes
beyond data quantity and validates the usefulness of the dataset.

##  3 VLE Dataset

Figure 2: The video data and the learner interaction logs from
VideoLectures.Net repository are processed separately to create the content-
based, video-specific features and Wikipedia-based Topics. Multiple different
engagement labels are extracted from interaction logs and published in VLE
dataset.

VLE dataset is constructed using the aggregated video lectures consumption
data coming from a popular OER repository,
VideoLectures.Net111www.videolectures.net (VLN). These videos are recorded
when researchers are presenting their work at peer-reviewed conferences.
Lectures are thus reviewed and material is controlled for correctness of
knowledge. The presenters and authors of published work that is recorded by
VLN agree and provide rights to publish their presentation video, slides and
supplementary materials under an open licence in the VLN website which can be
used for educational and research purposes. The majority of research venues
covered by VLN is related to Artificial Intelligence and Computer Science,
making most videos associating to these topics. Although the dataset mainly
consists of scientific talks that are geared towards postgraduate and PhD
level learners, a significant number of tutorials (Table 1) are geared for
university level students. In that aspect, many videos in the dataset draw
similarities to the style of conventional MOOC lectures.

The dataset provides a set of statistics aimed at studying population based
engagement in scientific videos, together with other conventional metrics in
subjective assessment such as star ratings and number of views. We believe the
dataset will serve the community applying AI in Education to further
understand what are the features of educational material that makes it
engaging for learners. The users also agree for the user-generated content in
the VLN website to be available for research purposes. We have anonymised all
user interaction data and aggregated them to ensure privacy of the users.
Figure 2 gives a high level representation of how different data silos within
VLN repository has been used to create the VLE dataset.

###  3.1 Feature Extraction

We process the video meta-data and English transcriptions222provided by
www.translectures.eu. to extract i) content-based textual features, ii)
Wikipedia topic-based features and iii) video-specific features. Majority of
the extracted features (with the exception of a few features in the video-
specific category) are cross modal (e.g. books, websites and audios) and are
easily automatable.

####  3.1.1 Content-based Features

Prior work [6] has proposed a set of effective content-based features for
similar datasets. We use the video meta-data and transcript to extract _Word
Count_ [43], _Title Word Count_ and _Document Entropy_ [1], language style
features [18], _Preposition Rate_ , _Auxiliary Rate_ , _To Be Rate_ ,
_Conjunction Rate_ , _Normalisation Rate_ , _Pronoun Rate_ , readability
related _Easiness (FK Easiness)_[18] and language style related _Stop-word
Presence Rate_ , _Stop-word Coverage Rate_ [1, 31]. To represent _Freshness_
of lectures (recency), we calculate the number of days between January 01,
1960 and the lecture published date which is a proxy for recency of the
lecture [6].

####  3.1.2 Wikipedia-based Features

We use Wikifier333http://wikifier.org [4], a novel entity linking method, on
transcripts to extract topical features. Two main types of novel topical
features that cover topic authority and topic coverage verticals [9] are
proposed through this work.

The _top-5 authoritative topic URLs_ and _top-5 PageRank scores_ features
represent the Topic Authority feature vertical. Wikifier [4] produces a
PageRank score [5] that indicates the marginal authoritativeness of a
Wikipedia concept among all Wikipedia concepts associated with a lecture. We
use this score to rank and identify the most authoritative topics and use the
actual PageRank score as a proxy for quantifying authority. It is noteworthy
that _authority_ of a learning resource entails author, organisation and
content authority [9]. These features represent content authority.

The _top-5 covered topic URLs_ and _top-5 cosine similarity scores_ features
represent _Topic Coverage_ feature vertical. The cosine similarity score
$cos(s_{tr},c)$ between the _Term Frequency-Inverse Document Frequency (TF-
IDF)_ representations of the lecture transcript $s_{tr}$ and the Wikipedia
page of concept $c$ is also an output from the Wikifier. We use these values
to i) rank and identify most covered topics and ii) quantify the topic
coverage [8].

For authority and coverage, the top 5 topic URLs and their scores are included
introducing four additional feature groups providing 20 distinct feature
columns in the final dataset. Figure 3 presents two word clouds that show the
25 most authoritative and covered topics in the VLE dataset. With respect to
the topics in a lecture, the authoritative topics are the ones that have
strong connection (linkage in Wikipedia) with other co-occurring topics in a
lecture whereas the covered topics are the ones with high textual overlap with
Wikipedia concept pages. This figure confirms that these two feature groups
represent different things although there is some overlap between the topics.

Figure 3: WordClouds summarising the distribution of top 25 (i) most
authoritative and (ii) most covered Wikipedia topics in the dataset. Note that
Data Science and Computer Science related topics are most dominant topics.

####  3.1.3 Video-specific Features

We identify a set of easily automatable, prior proposed [6] video specific
features. _Lecture Duration, Is Chunked, Lecture Type_[23], _Silence Period
Rate (SPR)_ and _Speaker Speed_ [6] are calculated based on prior work .
_Lecture Duration_ is reported in seconds. _Is Chunked_ is a binary feature
which indicates if a lecutre has multiple parts. _Lecture type_ value is
derived from the metadata. The possible values for this feature are described
in Table 1.

Table 1 also gives insight into how diverse the VLE dataset is. There are
different types of videos such as research presentations (e.g.
$\texttt{vl,vps,vpr,}\dots$ ), scientific talks (e.g.
$\texttt{vbp,vdm,vkn,vit,}\dots$), dialogues (e.g. $\texttt{vpa,vdb,}\dots$)
and tutorials (e.g. vtt) among the dataset.

Table 1: 14 types of lectures in the VLE dataset and their abbriviation (Abbr.) and frequency (Freq). Abbr. | Description | Freq. | Abbr. | Description | Freq.
---|---|---|---|---|---
vbp | Best Paper | 67 | vdb | Debate | 135
vdm | Demonstration | 315 | viv | Interview | 121
vid | Introduction | 75 | vit | Invited Talk | 609
vkn | Keynote | 274 | vl | Lecture | 7125
vop | Opening | 190 | oth | Other | 58
vpa | Panel | 207 | vps | Poster | 162
vpr | Promotional Video | 69 | vtt | Tutorial | 2142

###  3.2 Labels

Multiple labels based on explicit and implicit feedback are provided with this
dataset which will allow researchers to compare between different labels and
also integrate multiple label types when modelling engagement. Three main
types of quantification of engagement labels are presented.

####  3.2.1 Explicit Ratings

_Mean Star Rating_ based on a rating scale of 1-5 for each lecture is
provided. This value is accompanied by the number of ratings used to calculate
the mean. The proposed dataset has 2,127 ratings (almost 2x than Bulathwela et
al. [6]). Missing ratings are labelled with -1.

####  3.2.2 Popularity

The total number of views, named _View Count_ , for each video lecture as of
February 1, 2021 is extracted from the metadata and provided with the dataset.

####  3.2.3 Watch Time/Engagement

The majority of learner engagement labels in the VLE dataset are based on
watch time. We compute Normalised Engagement Time (NET) to compute the Median
of Normalised Engagement (MNET), as it has been proposed as the gold standard
for engagement with educational materials in previous work [23]. We further
compute Average of Normalised Engagement (ANET). To have the MNET and ANET
labels in the range $[0,1]$, we set the upper bound to 1 and derive Saturated
MNET (_SMNET_) and Saturated ANET (_SANET_) that are included in the dataset.

The standard deviation of NET for each lecture (_Std of Engagement_) is
reported, together with the _Number of User Sessions_ used for calculating
MNET and ANET. These measure allows understanding stability of the centres
published. The set of individual NET values are also published to allow future
researchers to exploit the true distribution of values.

###  3.3 Preserving Anonymity and Ethics

We only publish lectures with more than 5 views to preserve k-anonymity and
avoid revealing learner identities [17]. A regime of additional techniques are
used to preserve lecturer anonymity in order to avoid having unanticipated
effects on lecturer’s reputation by associating implicit learner engagement
values to their content.

Rarely occurring _Lecture Type_ values were grouped together to create the
_other_ category in Table 1. Similarly, subject categories Life Sciences,
Physics, Technology, Mathematics, Computer Science, Data Science and Computers
subjects to stem category and the other subjects to misc category. Rounding is
used with _Freshness_ and _Lecture Duration_ to the nearest 10 days and 10
seconds respectively. Gaussian white noise (10%) is added to _Title Word
Count_ feature and rounded to the nearest integer.

VLE dataset comes from a video lecture collection that mainly belongs to the
Computer Science community, where there is a gender imbalance, both in
audience and presenters. To enhance the neutrality of our findings and
contributions, we have avoided using feature classes that could potentially
reflect gender characteristics. For example, we have avoided using visual
features (facial features, presenter emotions) and sound related features
(pitch) that may actively or passively embed gender characteristics. Instead
we have focused on features that mainly reflect informational content of the
lectures. Furthermore, where video specific features are incorporated, we have
used very generic features such as "speaker speed" that are unlikely to be
correlated with characteristics such as gender or age.

###  3.4 Final Dataset

The final dataset includes lectures that are published between September 1,
1999 and December 31, 2020. The engagement labels are created from events of
over 1.1 Million users logged between December 01, 2016 and February 01, 2020.
The final dataset contains 11,548 lectures across 21 subjects (eg. Computer
Science, Philosophy, etc. with a majority from AI and Computer Science) that
are grouped into STEM and Miscellaneous categories. The collection of videos
span various video lengths with the duration distribution having two modes at
approx. 2000s (33 mins) and 4000s (1hr) time points which align with typical
lengths of research talks and presentations. The mean word count of the videos
is 5347.9. The video lecture collection uses on average 93.9 learners per
video when calculating engagement centres. The dataset, helper tools, example
code snippets and various descriptive statistics related to the VLE dataset
are available publicly444https://github.com/sahanbull/VLE-Dataset.

###  3.5 Supported Tasks

This section introduces the reader to the tasks that the dataset could be used
for. The main application areas of these tasks are i) quality assurance in
educational video repositories and understanding and ii) predicting context-
agnostic engagement in an web-based learning setting. We establish two main
tasks, which we mainly focus on in this paper, that can be objectively
addressed using the VLE dataset using a supervised learning approach. These
are:

  * •

Task 1: Predicting context-agnostic (population-based) engagement of video
lectures: The dataset provides a set of relevant features and labels to
construct machine learning models to predict context-agnostic engagement in
video lectures. The task can be treated as a regression problem.

  * •

Task 2: Ranking of video lectures based on engagement: Building predictive
models that can rank videos based on their context-agnostic engagement could
be useful in the setting of an educational recommendation system, including
tackling the cold-start problem associated to new video lectures. The task can
be treated as a ranking problem to predict the global/relative ranking of
video lectures.

##### Other Tasks

Several auxiliary tasks can also be addressed with this dataset. This dataset
is suitable for, not limiting to, several tasks such as i) understanding
influential features for engagement prediction ii) understanding the strengths
and weaknesses of different implicit/explicit labels, that have been
investigated in prior work with similar datasets [6, 33]. The video
representations, with the use of unsupervised approaches can be used to
understand meaningful hidden patterns contrasting between clusters of videos
(e.g. talks vs. lectures vs. tutorials). With the use of Wikipedia based
topics, we also see opportunities in deducing the structure of knowledge based
on how topics co-occur within videos. Such investigations can be done on this
dataset in isolation or can be fused with similar datasets where this task has
been attempted before [47].

###  3.6 Evaluating Performance

We identify _Root Mean Squared Error (RMSE)_ as a suitable metric for
evaluating Task 1. Measuring RMSE against the original labels published with
the datasets will allow different works to be compared fairly. With reference
to Task 2, we identify _Spearman’s Rank Order Correlation Coefficient (SROCC)_
as a suitable metric. SROCC is suitable for comparing between ranking models
that create global rankings (e.g. point-wise rankers).

We use 5-fold cross validation to evaluate model performance with tasks 1 and
2. The folds are released together with the dataset, to allow to facilitate
fair comparison and reproducability. The five folds can be identified using
the fold column in the dataset. 5-fold cross validation allows reporting the
_confidence intervals (1.96 $\times$ Standard Error)_ of the performance
estimate, which we include in Table 2.

##  4 Baselines and Experiments

Through our experiments we seek answers to multiple research questions, which
we detail below. Note that these research questions overlap only partially
with the proposed supported tasks outlined in section 3.5, as the use purposes
of the dataset go much beyond what we could explore in this paper. The main
research questions of interest are:

  * •

RQ1:  Does the newly constructed VLE dataset lead to training more performant
prediction models?

  * •

RQ2:  How does the larger quantity of training data affect predictive
performance?

  * •

RQ3:  Is the model useful for modelling engagement with Computer Science
materials?

  * •

RQ4:  Is this dataset useful for modelling engagement in E-Learning lectures
and MOOC videos?

  * •

RQ5:  Does context-agnostic engagement prediction help in the cold-start
scenario?

Prior work by Bulathwela et al. [6] demonstrated _Random Forest (RF)_ model
obtains best performance among linear and non-linear models in similar
datasets. Therefore, we use the RF model to benchmark the new VLE dataset for
Tasks 1 and 2 described earlier. In addition, a handful of _Multi Layer
Perceptron (MLP)_ architectures were also experimented with due to their
success in engagement prediction with YouTube videos [16].

The code for extracting the proposed features is also published with this work
in the dataset repository.

###  4.1 Labels and Features for Baseline Models

_SMNET_ label is used as the target variable for both tasks. Preliminary
investigations indicated that SMNET label follows a Log-Normal distribution,
motivating us to use a log transformation on the SMNET values before training
the models. Empirical results further confirmed that this step improves the
final performance of the models. We undo this transformation for computing
$RMSE$ while this transformation doesn’t affect $SROCC$.

All the features outlined as the content-based and video-based sections in
section 3.1 are included in the baseline models. The models are trained with
three different feature sets in an incremental fashion:

  1. 1.

_Content-based_ : Features extracted from lecture metadata and the transcript-
based textual features.

  2. 2.

_\+ Wiki-based_ : Content-based + 2 Wikipedia-based features (Top 1 Most
Authoritative Topic URL and Most Covered Topic URL).

  3. 3.

_\+ Video-based_ : Content-based + Wikipedia-based + Video-specific features.

However, due to the large amount of topics in the Wikipedia-based feature
groups, we restrict to the top 1 authoritative and covered topic features
where they are encoded as binary categorical variables. Our initial attempts
to encode these features in a reduced dimension space (using Singular Value
Decomposition) led to deteriorated results contrary to our expectations.
Practitioners are encouraged to try further encoding of the topic variables,
as it will likely have a positive impact on the performance.

###  4.2 Experiments

Addressing RQ1, both the RF and MLP models are experimented with the proposed
features sets with the smaller 4k dataset [6] and the newly proposed VLE
dataset. 5-fold cross validation is used in this experiment. This setup allows
identifying i) how performance gains are achieved through adding each new
group of features and ii) how performance gains are achieved through adding
new observations. Follow on experiments addressing RQ2 and 3 are run using
folds 1-4 as training data and fold 5 of the dataset as testing data. We
experiment by using varying proportions of training data to train the model.
When selecting training data, random sampling is used to keep the diversity of
the lectures similar to the full dataset. All the trained models using
different quantities of training data are then evaluated on the same held out
test set.

To validate RQ4, we first partition the entire dataset to two parts, i)
tutorial videos (vtt in Table 1) and ii) all other videos, as test and train
data respectively. However, tutorials presented in a research conference may
significantly vary from e-learning videos geared for course learning. To
address this mismatch, we further identify 1,035 videos (among the tutorials)
that exclusively belong to the Open Course Ware Consortium
(OCWC)555http://videolectures.net/ocwc. OCWC contains university lectures that
have been intended for course teaching via e-learning. These lectures are
recorded using a variety of MOOC production techniques such as classroom
lecture, talking head and power point presentation [23]. We define these
videos as ocw lectures. We use the training data (all except tutorials) to
train the engagement model and evaluate prediction performance on i)
OpenCourseWare, ocw lectures, ii) all tutorials but OpenCourseWare, !ocw and
iii) all tutorials vtt, (Entire test data). The follow on experiments (RQ2-4)
are only done with the best performing model from Table 2 (RF model with
_Content + Wiki + Video_ feature group) to reduce computational cost.

A different experiment was run to answer RQ5. We utilise TrueLearn Novel [8]
(hereby referred to as _TrueLearn_), a recently proposed educational
recommender model that learns individualised models to predict engagement with
video lectures. A key limitation with personalised models such as TrueLearn is
that there is no information about the user in the beginning, leading to a
user cold-start problem which effectively means having ill-informed engagement
predictions in the beginning of the learner session. To address this, we
utilise the proposed context-agnostic engagement prediction model where a
hybrid recommendation system (hereby referred to as _TrueLearn ++_) is built
by combining it with TrueLearn model. For simplicity’s sake, TrueLearn++ uses
"switching" [10], where the context agnostic model is used to make a
prediction for the _first_ event of the user (where the personalistion model
has no information to work with) and then switched to TrueLearn model which
can exploit the user watch history. PEEK dataset [7], which includes more than
20,000 user sessions, is used for the experiment where the context agnostic
model is trained using lectures that are not present in the PEEK test data.
Then the predictive performance on the PEEK test data using TrueLearn (The
baseline) and TrueLearn++ (where the first event is predicted using the
context-agnostic model) is measured using Accuracy, Precision, Recall and
F1-Score. To evaluate if the improvement of metrics is statistically
significant, a learner-wise, one-tailed paired t-test is used.

##  5 Results and Discussion

Table 2: RMSE and SROCC with confidence intervals for the engagement prediction (Task 1) and lecture ranking (Task 2) using the Random Forests model with both 4k [6] and 12k (Our VLE) datasets. Better performance per task is highlighted in bold. |  $RMSE$ with Task 1 |  $SROCC$ with Task 2
---|---|---
Feature set | 4k |  12k _(Ours)_ | 4k |  12k _(Ours)_
Content-based |  .1801$\pm$.006 | .1170$\pm$.006 |  .6190$\pm$.011 | .7504$\pm$.013
\+ Wiki-based |  .1798$\pm$.007 | .1178$\pm$.006 |  .6251$\pm$.014 | .7505$\pm$.013
\+ Video-specific |  .1728$\pm$.007 | .1098$\pm$.007 |  .6758$\pm$.020 | .7832$\pm$.009
Figure 4: Predictive performance for (i) engagement prediction and (ii) lecture ranking tasks with varying proportions of randomly sampled training data. The test set performance for full test dataset (Blue) and subsets of test dataset that consists of CS lectures only (Orange) and Non-CS lectures only (Green) are also reported Table 3: Performance for OpenCourseWare (ocw), Non-OpenCourseWare (!ocw) tutorial and All tutorial (vtt) videos for engagement prediction and lecture ranking tasks. Better performance per task is highlighted in bold. | ocw | !ocw | vtt |  From Table 2
---|---|---|---|---
$RMSE$ with Task 1 | .0539 | .0404 | .0406 | .1098
$SROCC$ with Task 2 | .9485 | .9209 | .9223 | .7832
Table 4: Average test set performance for Accuracy (Acc.), Precision (Prec.), Recall (Rec.) and F1-Score (F1). The more performant value is highlighted in bold. The metrics where the proposed model that outperform the baseline counterpart in the PEEK dataset ($p<0.01$ in a one-tailed paired t-test) are marked with $\cdot^{(*)}$. Model | Acc. | Prec. | Rec. | F1
---|---|---|---|---
Truelearn | 62.69 | 57.54 | 81.88 | 64.98
Truelearn++ | 63.51$\cdot^{(*)}$ | 57.91$\cdot^{(*)}$ | 79.13 | 64.39
Table 5: Test set performance for Accuracy (Acc.), Precision (Prec.), Recall (Rec.) and F1-Score (F1) for _first_ event of each learner. The more performant value is highlighted in bold. The metrics where the proposed model that outperform the baseline counterpart in the PEEK dataset ($p<0.01$ in a one-tailed paired t-test) are marked with $\cdot^{(*)}$. Model | Acc. | Prec. | Rec. | F1
---|---|---|---|---
TrueLearn | 44.21 | 44.21 | 100.00 | 61.32
TrueLearn++ | 56.09$\cdot^{(*)}$ | 50.32$\cdot^{(*)}$ |  53.58 | 51.90

The performance metrics observed with the RF model on Task 1 and 2 (RQ1) are
outlined in Table 2. Although we expected competitive results with MLP models,
we failed to observe promising results. We believe this is due to the
restrictive architecture space we used in our experiments. We encourage future
researchers to experiment with more complex and wider range of neural
architectures that may lead to better results.

Figure 4 illustrates how the training data size impacts the i) RMSE and ii)
SROCC (RQ2). It also demonstrate the performance difference between predicting
engagement of Computer Science videos (CS) vs. Non-CS videos (RQ3). Finally,
Table 4 presents predictive performance of the model on e-learning type
lectures and tutorials (RQ4).

The overall performance results relating to the effect of _combining_ the
context agnostic model with personalisation models to battle cold-start
problem (RQ5) is reported in Table 4. A magnified view of the performance of
predicting the outcome of the first event of each user (where the context-
agnostic predictor is supposed to help the personaliser) is reported in Table
5.

###  5.1 Performance Gains and Causes (RQ1-2)

Results in Table 2 clearly shows that the 300% larger VLE lead to significant
performance gains in both engagement prediction and lecture ranking tasks. In
the case where the full feature set is used with the RF model, RMSE on Task 1
drops from .17 to .1 (41%) while SROCC on Task 2 jumps from .68 to .78 (15%).
The labels (both explicit and implicit) themselves are more accurate as they
are calculated using a larger user population. This means that many of the
lectures that already existed in the smaller dataset [6] are likely to get
improved engagement labels as the new labels are calculated using more user
sessions that interacted with the videos during a wider time period (including
very recent sessions until February 2020).

Within the VLE dataset itself, using additional feature groups tend to lead to
better performing models. Results for VLE dataset in Table 2 demonstrates this
trend where a significant jump in performance is evidenced when incorporating
modality specific features (Video-specific features). However, the results
also show that the cross-modal content-based features alone lead to
substantial performance. This is a good indication that easy-to-compute,
cross-modal features alone are sufficient to build a system that can predict
context-agnostic engagement of videos. In a practical viewpoint, the proposed
cross-modal features are computationally light (unlike complex deep models,
e.g. vision models). Wikification, used in generating Wiki-based features,
also operates at web-scale666http://wikifier.org. Although the results show
minute gains by adding the Wiki-based features, we believe that this is due to
the simplicity of the Wiki features used in constructing the baselines leaving
much room for sophistication (e.g. exploiting the semantic relatedness of
topics). The topics, coming from a humanly-intuitive taxonomy, leaves room for
building interpretable features.

Figure 4 confirms that the increase of training data is leading to performance
gains in both tasks. Figure 4(i) suggests that the Root Mean Square Error
(RMSE) can be further improved with more training examples. However, Figure
4(ii) tells rather a different story where the performance gain seem to
saturate around 60% mark. This is an indication that improving ranks of the
test data gets significantly harder at around 5,500 training examples
($\approx$ 60% of the 9,239 training set) .

###  5.2 Relevance to AI/CS Education (RQ3)

The follow up experimental results in Figure 4 demonstrates that this dataset
allows achieving higher performance on CS-only lectures. A likely reason for
this may be the higher diversity of lecture in the non-CS category as it
consists of significantly different subjects. Nevertheless, Figure 4 shows
that a test set RMSE of $\approx.1$ and SROCC of $\approx.8$ is achievable
with CS lectures. Figure 3 further shows that majority of the lectures in the
dataset contains concepts relating to Artificial Intelligence and Data Science
(e.g. Machine Learning, Ontology, Semantic Web …). This indicates that
majority of the CS lectures in the dataset contain topics relating to AI
making this dataset a highly suitable dataset for training engagement
prediction models for AI and CS education.

###  5.3 Relevance to E-Learning and MOOCs (RQ4)

Table 4 shows strong evidence that the models trained with VLE dataset
generalise really well for engagement modelling in e-learning type videos
created for course teaching amid the dataset containing many different video
types (as per Table 1). The models trained are much better at engagement
prediction and ranking of tutorial-like videos than general scientific talks.
Having tested with lectures that have been recorded using different MOOC video
production techniques, the high performance obtained on ocw lectures confirms
that VLE dataset can be highly effective in building context-agnostic
engagement models for e-learning and MOOC systems.

###  5.4 Relevance to Addressing Cold-Start (RQ5)

Table 4 shows that by simply incorporating the context-agnostic engagement
prediction in TrueLearn Novel algorithm (together becoming TrueLearn++) can
lead to significant improvements in accuracy and precision. The same table
also shows that the drop of overall F1 score can be attributed to the steep
drop of Recall Score. Table 5 sheds more light into where this steep drop of
recall occurs. This is, the baseline TrueLearn model always predicts positive
engagement for the first event of the user. As seen in table 5, the recall of
baseline TrueLearn model being 1.0 while the accuracy and precision being the
same depicts this fact. TrueLearn predicts positive in event 1 of each user
because the model has no information to base the prediction on [8]. However,
table 5 shows that the scenario is different in TrueLearn++ as the model has
additional information during the first event. Both accuracy and precision of
predictions in first event of the learner population significantly improves.
The recall will fall as the proposed context-agnostic model only captures a
population based prior which may deviate from the individuality of the
learners. However, it can be argued that making a prediction with additional
information is better than predicting with no prior information. In the bigger
picture, being able to make slightly more informed and varied predictions for
the first event of learners based on lecture content features enable
significantly improving prediction accuracy and precision of TrueLearn++ in
Table 4. It is also noteworthy that our experiment, for the sake of
simplicity, uses a rule that could be significantly improved further, e.g.
using weights of the probabilities of both population-based and personalised
models at the beginning of a user session (using weighting or stacking [10]),
where the weight of population-based engagement decreases as we gather more
information about the user. as we gather more information about the user.

##  6 Opportunities and Limitations

The VLE dataset brings plenty of opportunities to the research community that
is interested in building context-agnostic engagement using content related
features. It is significantly larger (15x than [23] and 3x than [6]) and more
education focused than the contenders [44] focused on engagement with videos.
The larger quantity of examples may even enable more complex model families
(e.g. deep learning) to be used in this task although our limited early
experiments did not reap fruit. Another noteworthy opportunity is that this
dataset can be extended periodically to create even larger and more accurate
evaluations of the dataset.

The Wiki-based features open up limitless possibilities as many sophisticated
feature sets can be built and experimented. Due to the connectivity to
Wikipedia, both its content and link structures can be exploited to invent
meaningful, yet interpretable features. A further step can enable other data
structures such as knowledge bases (e.g Wikidata), category tree etc. to be
used for feature creation.

Support for other tasks beyond the main tasks will allow this dataset to be
useful in creating decision support systems that can help future content
creators and help them create engaging educational videos [27] while
confirming prior findings [23, 3, 6] regarding drivers of learner engagement.
Furthermore, the dataset also enables comparing the findings from studies
outside of education [44]. The Wikipedia features also facilitates grounds to
explore topic related tasks such as learning the structure of knowledge [47].

There also exists limitations in this dataset. VLE dataset is largely
comprised of Computer Science and Data Science materials (Figure 3) that are
delivered all in English. While this is an opportunity for AI and Computer
Science education, results in Figure 4 also shows that this fact leads to
comparatively _less fruitful_ non-CS results. The dataset and its features are
also not suitable for non-English video collections. Amid its size, the
dataset still lacks variety of materials in topical and lingual sense.

At first sight, the majority of the lectures in our dataset come from male
presenters, potentially creating a significant gender imbalance in the
dataset. As pointed out in section 3.3, we have taken some measures to
restrict the feature set to what we believe to be more neutral features.
However, since we do not have access to gender information in the data
collected, it is impossible to test and guarantee that there is zero
correlation between the proposed features in the VLE dataset and negative
gender biases. Care should be taken when enhancing these features and there is
room to do more rigorous tests to understand if any gender biases are present
within the dataset.

_Learner Engagement_ is a loaded concept with many facets. In relation to
consuming videos, many behavioural actions such as pausing, rewinding and
skipping can contribute to latent engagement with a video lecture [28]. Due to
the technical limitations of the platform and privacy concerns, only watch
time, view and mean ratings are included in this dataset. Although watch time
has been used as a representative proxy for learner engagement with videos
[23, 44, 16], we acknowledge that more informative measures may lead to more
complete engagement signals.

##  7 Conclusions

Identifying the need of resources to push the frontiers of context-agnostic
engagement prediction, which is a crucial part of addressing scalable quality
assurance and the cold-start problem in educational recommenders, we release
the VLE dataset. This dataset consists of around 12,000 videos with three
groups of features, namely, i) content-based, ii) Wikipedia-based and iii)
Video-specific features, accompanied as well by iv) multiple implicit and
explicit engagement labels. We establish 2 main tasks, and identify multiple
other tasks that can be tackled with this dataset. Addressing the 2 main tasks
proposed, we benchmark the new dataset to show significant prediction gains
over a similar yet, smaller prior dataset. In follow on experiments, we
investigate how the magnitude of training examples relate to performance gains
while also demonstrating the suitability of this dataset to build models for
AI/CS related video collections. We further validate that the dataset can be
used to model engagement with e-learning type video lectures to show its
relevance to educational recommendation systems in the context of MOOCs. With
the use of a simple experiment, it is also demonstrated that such a model can
be incorporated with a personalised (contextual) engagement prediction model
to significantly improve the predictions.

##### Future Directions

We see several lines of future work addressing current limitations of the
dataset. Adding diverse examples from different subject areas is our top
priority. Experimenting further with more sophisticated Wiki-based features
(e.g. by exploiting the Wikipedia semantic graph [35]) is a promising way
forward. The possibility of including more learner engagement related signals
(e.g.: pauses, replays, skips, etc.) will be explored in the subsequent
version of the dataset, without compromising learner privacy.As more
understanding of engagement with other modalities (such as PDFs and e-Books)
is gained, it is possible to add more learning resources from diverse
modalities to widen the horizons of the dataset and improve understanding of
engagement with different modalities of educational material.

This work only devices a simple mechanism to combine the context-agnostic
model with a personalisation model for the sake of proving that the proposal
has genuine use-cases. The experiment barely scratches the surface on how a
context-agnostic engagement prediction model can be incorporated in an
educational recommendation system. There are a variety of alternatively and
more sophisticated approaches that can potentially have bring larger
performance gains, which we will explore in future work.

##  8 Acknowledgments

This research was partially conducted as part of the X5GON project funded from
the EU’s Horizon 2020 research programme grant No 761758. This work is also
supported by the European Commission funded project "Humane AI: Toward AI
Systems That Augment and Empower Humans by Understanding Us, our Society and
the World Around Us" (grant 820437) and the EPSRC Fellowship titled "Task
Based Information Retrieval" (grant EP/P024289/1).

## References

  * [1] M. Bendersky, W. B. Croft, and Y. Diao.  Quality-biased ranking of web documents.  In Proc. of ACM Int. Conf. on Web Search and Data Mining, 2011.
  * [2] F. Bonafini, C. Chae, E. Park, and K. Jablokow.  How much does student engagement with videos and forums in a mooc affect their achievement?  Online Learning Journal, 21(4), 2017.
  * [3] C. J. Brame.  Effective educational videos: Principles and guidelines for maximizing student learning from video content.  CBE—Life Sciences Education, 15(4), 2016.
  * [4] J. Brank, G. Leban, and M. Grobelnik.  Annotating documents with relevant wikipedia concepts.  In Proc. of Slovenian KDD Conf. on Data Mining and Data Warehouses (SiKDD), 2017.
  * [5] S. Brin and L. Page.  The anatomy of a large-scale hypertextual web search engine.  In Proc. of Int. Conf. on World Wide Web, 1998.
  * [6] S. Bulathwela, M. Perez-Ortiz, A. Lipani, E. Yilmaz, and J. Shawe-Taylor.  Predicting engagement in video lectures.  In Proc. of Int. Conf. on Educational Data Mining, EDM ’20, 2020\.
  * [7] S. Bulathwela, M. Perez-Ortiz, E. Novak, E. Yilmaz, and J. Shawe-Taylor.  Peek: A large dataset of learner engagement with educational videos, 2021\.
  * [8] S. Bulathwela, M. Perez-Ortiz, E. Yilmaz, and J. Shawe-Taylor.  Truelearn: A family of bayesian algorithms to match lifelong learners to open educational resources.  In AAAI Conference on Artificial Intelligence, AAAI ’20, 2020\.
  * [9] S. Bulathwela, E. Yilmaz, and J. Shawe-Taylor.  Towards Automatic, Scalable Quality Assurance in Open Education.  In Workshop on AI and the United Nations SDGs at Int. Joint Conf. on Artificial Intelligence, 2019.
  * [10] R. Burke.  Hybrid recommender systems: Survey and experiments.  User modeling and user-adapted interaction, 12(4):331–370, 2002\.
  * [11] G. Chen, J. Yang, C. Hauff, and G. Houben.  Learningq: A large-scale dataset for educational question generation.  In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25-28, 2018, pages 481–490, 2018.
  * [12] Y. Choi, Y. Lee, D. Shin, J. Cho, S. Park, S. Lee, J. Baek, C. Bae, B. Kim, and J. Heo.  Ednet: A large-scale hierarchical dataset in education.  In International Conference on Artificial Intelligence in Education, pages 69–73. Springer, 2020.
  * [13] K. Clements and J. Pawlowski.  User-oriented quality for oer: understanding teachers’ views on re-use, quality, and trust.  Journal of Computer Assisted Learning, 28(1), 2012.
  * [14] K. Collins-Thompson, P. N. Bennett, R. W. White, S. de la Chica, and D. Sontag.  Personalizing web search results by reading level.  In Proc. of Conf. on Information and Knowledge Management, 2011\.
  * [15] R. Cortinovis, A. Mikroyannidis, J. Domingue, P. Mulholland, and R. Farrow.  Supporting the discoverability of open educational resources.  Education and Information Technologies, 24(5):3129–3161, 2019.
  * [16] P. Covington, J. Adams, and E. Sargin.  Deep neural networks for youtube recommendations.  In Proc. of ACM Conf. on Recommender Systems, 2016.
  * [17] N. Craswell, D. Campos, B. Mitra, E. Yilmaz, and B. Billerbeck.  Orcas: 20 million clicked query-document pairs for analyzing search.  In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM ’20, page 2983–2989, New York, NY, USA, 2020. Association for Computing Machinery.
  * [18] D. H. Dalip, M. A. Gonçalves, M. Cristo, and P. Calado.  A general multiview framework for assessing the quality of collaboratively created content on web 2.0.  Journal of the Association for Information Science and Technology, 2017.
  * [19] D. H. Dalip, M. A. Gonçalves, M. Cristo, and P. Calado.  Automatic assessment of document quality in web collaborative digital libraries.  Journal of Data and Information Quality, 2(3), Dec. 2011.
  * [20] D. Davis, G. Chen, C. Hauff, and G. Houben.  Activating learning at scale: A review of innovations in online learning strategies.  Comput. Educ., 125:327–344, 2018.
  * [21] D. Davis, D. Seaton, C. Hauff, and G. Houben.  Toward large-scale learning design: categorizing course designs in service of supporting learning outcomes.  In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, London, UK, June 26-28, 2018, pages 4:1–4:10, 2018.
  * [22] M. A. Dewan, M. Murshed, and F. Lin.  Engagement detection in online learning: a review.  Smart Learning Environments, 6(1):1, 2019.
  * [23] P. J. Guo, J. Kim, and R. Rubin.  How video production affects student engagement: An empirical study of mooc videos.  In Proc. of the First ACM Conf. on Learning @ Scale, 2014.
  * [24] M. Horta Ribeiro and R. West.  Youniverse: Large-scale channel and video metadata from english-speaking youtube.  Proceedings of the International AAAI Conference on Web and Social Media, 15(1), 2021.
  * [25] A. Kaur, A. Mustafa, L. Mehta, and A. Dhall.  Prediction and localization of student engagement in the wild.  In 2018 Digital Image Computing: Techniques and Applications (DICTA), pages 1–8. IEEE, 2018.
  * [26] S. Kim, W. Kim, Y. Jang, S. Choi, H. Jung, and H. Kim.  Student knowledge prediction for teacher-student interaction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15560–15568, 2021.
  * [27] M. Kurdi, N. Albadi, and S. Mishra.  “think before you upload”: an in-depth analysis of unavailable videos on youtube.  Social Network Analysis and Mining, 11(1):1–21, 2021.
  * [28] A. S. Lan, C. G. Brinton, T.-Y. Yang, and M. Chiang.  Behavior-based latent variable model for learner engagement.  In Proc. of Int. Conf. on Educational Data Mining, 2017.
  * [29] Z. MacHardy and Z. A. Pardos.  Evaluating the relevance of educational videos using bkt and big data.  In Proc. of Int. Conf. on Educational Data Mining, 2015.
  * [30] M. Mendicino, L. Razzaq, and N. T. Heffernan.  A comparison of traditional homework to computer-supported homework.  Journal of Research on Technology in Education, 41(3):331–359, 2009\.
  * [31] A. Ntoulas, M. Najork, M. Manasse, and D. Fetterly.  Detecting spam web pages through content analysis.  In Proc. of Int. Conf. on World Wide Web, 2006.
  * [32] G. Penha and C. Hauff.  Curriculum learning strategies for IR.  In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part I, 2020.
  * [33] M. Perez-Ortiz, A. Mikhailiuk, E. Zerman, V. Hulusic, G. Valenzise, and R. K. Mantiuk.  From pairwise comparisons and rating to a unified quality scale.  IEEE Transactions on Image Processing, 29:1139–1151, 2019.
  * [34] I. Persing and V. Ng.  Modeling argument strength in student essays.  In Association for Computational Linguistics, 2015.
  * [35] M. Ponza, P. Ferragina, and S. Chakrabarti.  On computing entity relatedness in wikipedia, with applications.  Knowledge-Based Systems, 188:105051, 2020.
  * [36] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.  Squad: 100, 000+ questions for machine comprehension of text.  In EMNLP, 2016.
  * [37] A. Ramesh, D. Goldwasser, B. Huang, H. Daume III, and L. Getoor.  Learning latent engagement patterns of students in online courses.  In Proc. of AAAI Conference on Artificial Intelligence, 2014.
  * [38] J. Shi, C. Otto, A. Hoppe, P. Holtz, and R. Ewerth.  Investigating correlations of automatically extracted multimodal features and lecture video quality.  In Proceedings of the 1st International Workshop on Search as Learning with Multimedia Information, SALMM ’19, page 11–19, New York, NY, USA, 2019. Association for Computing Machinery.
  * [39] R. Syed and K. Collins-Thompson.  Optimizing search results for human learning goals.  Inf. Retr. J., 20(5):506–523, 2017.
  * [40] R. Syed and K. Collins-Thompson.  Retrieval algorithms optimized for human learning.  In Proc. of Int. Conf. on Research and Development in Information Retrieval, 2017.
  * [41] K. Taghipour and H. T. Ng.  A neural approach to automated essay scoring.  In Proc. of Conf. on Empirical Methods in Natural Language Processing, 2016.
  * [42] UNESCO.  Open educational resources (oer).  https://en.unesco.org/themes/building-knowledge-societies/oer, 2021\.  Accessed: 2021-04-01.
  * [43] M. Warncke-Wang, D. Cosley, and J. Riedl.  Tell me more: An actionable quality model for wikipedia.  In Proc. of Int. Symposium on Open Collaboration, 2013.
  * [44] S. Wu, M. Rizoiu, and L. Xie.  Beyond views: Measuring and predicting engagement in online videos.  In Proc. of the Twelfth Int. Conf. on Web and Social Media, 2018\.
  * [45] F. Xu, L. Wu, K. P. Thai, C. Hsu, W. Wang, and R. Tong.  MUTLA: A large-scale dataset for multimodal teaching and learning analytics.  CoRR, abs/1910.06078, 2019.
  * [46] H. Yannakoudakis, T. Briscoe, and B. Medlock.  A new dataset and method for automatically grading esol texts.  In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 180–189, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics.
  * [47] J. Yu, G. Luo, T. Xiao, Q. Zhong, Y. Wang, W. Feng, J. Luo, C. Wang, L. Hou, J. Li, et al.  Mooccube: a large-scale data repository for nlp applications in moocs.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3135–3142, 2020.