§ INTRODUCTION

Resistive Plate Chambers (RPCs) are gaseous particle detectors widely employed in the muon systems of high energy physics experiments. In particular, thanks to their timing resolution of 1-2 ns and spatial resolution in the order of few mm, they can be employed for triggering purposes. Furthermore, due to their comparatively low cost, they are suitable to cover large areas. This is the case for the ALICE [1] (A Large Ion Collider Experiment) detector at the CERN Large Hadron Collider (LHC), where a total of 72 RPCs are employed for muon identification and cover an area of around 140 m$^{2}$. These detectors are operated with a mixture of  89.7% C_2H_2F_4 (also known as R134a), 10% i-C_4H_10 and 0.3% SF_6. Such a mixture  provides a working voltage around 9.8 kV and excellent performances [2].

Out of the three gases in the mixture, C_2H_2F_4 and SF_6 are greenhouse gases, meaning that they trap great amounts of heat in the atmosphere for an extended period of time. New European Union regulations [3] have imposed a progressive phase down in the production and use of such gases, reducing their availability and increasing their price. Even though scientific laboratories are not explicitly included in these regulations, CERN has asked its scientific community to start an effort to find a new, more eco-friendly, gas mixture to be used in RPC detectors.

In order to quantify how harmful a gas is for the environment, the global warming potential (GWP) has been introduced. This is a number that compares the amount of heat trapped in the atmosphere by a ton of certain gas (over a given period of time, usually 100 years) if compared to a ton of CO_2. By definition GWP(CO_2) = 1. For reference GWP(C_2H_2F_4) $\simeq$ 1400 and GWP(SF_6) $\simeq$ 22800.

Since each of the employed gases has a unique role in signal development, it is not an easy task to find a suitable replacement. For this reason, most of the efforts have been focused on the replacement of C_2H_2F_4, the main contributor to the gas mixture GWP. A possible candidate for the replacement has been identified in Tetrafluoropopene [4] (C_3H_2F_4), also known as HFO-1234ze (HFO in the following). It has a very similar chemical structure to R134a, with a much lower GWP of around 7. Furthermore, this gas is being employed as an industrial replacement for R134a, making it readily available and cheap.

HFO has a smaller effective first Townsend coefficient, making it impossible to simply replace the R134a in toto, since the working voltage would move to over 15 kV. Such voltages cannot be tolerated by the existing high voltage infrastructures and detectors. Hence, many preliminary tests with cosmics, have been performed in order to find possible ways to work with this gas. In particular, it has been found that diluting HFO with CO_2 gives the best performance in terms of operating voltage and efficiency [5]. A few promising gas mixtures have been identified and the next step is to study the detector's response in more controlled environments (i.e. beam tests), and, at the same time, their long-term behavior (i.e. aging studies) when operated with these new gas mixtures.

§ EXPERIMENTAL SETUP

A cross-experiments collaboration (ALICE, ATLAS, CMS, CERN EP-DT and SHiP/LHCb) was established to perform the tests described above, joining forces to reach the common goal of finding a new gas mixture for RPC detectors. Each group provided an RPC prototype, all installed in a common experimental setup at the CERN Gamma Irradiation Facility (GIF++) [6].

This facility is equipped with a 12.5 TBq ^137Cs source, which provides a uniform gamma irradiation field that can be used to simulate a high background environment to speed the aging process, allowing one to simulate several years of irradiation in a reduced amount of time. In addition, during dedicated beam periods, a muon beam of 100 GeV/c is produced by a hadron beam (circulating in the CERN SPS) impinging on a target. The peculiarity of the GIF++ is the possibility to combine the muon beam with the Cs source, enabling the study of the detector performance under various background conditions, hence giving insights into the RPC rate capability.

The objective of this contribution is on the detector that was provided by the ALICE collaboration. This is a small (50x50 cm$^{2}$) ALICE-like RPC, with 2 mm thick bakelite electrodes and 2 mm thick (single) gas gap. Signal amplification and discrimination were provided by the FEERIC [7] front-end boards, currently employed in ALICE. The readout was executed through a CAEN TDC (model V1190a) and the trigger was provided by the coincidence of 4 scintillators coupled with photomultipliers  (two placed outside the irradiation facility and two inside) for a total trigger area of 10x10 cm$^{2}$.

A total of three gas mixtures were tested, the CMS/ATLAS standard gas mixture (STD)[Note that ALICE standard gas mixture could not be used for flammability reasons] (95.2% C_2H_2F_4, 4.5% i-C_4H_10 and 0.3% SF_6) and two eco-friendly gas mixtures, ECO2 (60 % CO_2, 35 % HFO, 4 % i-c_4H_10 and 1 % SF_6) and ECO3 (69 % CO_2, 25 % HFO, 5 % i-c_4H_10 and 1 % SF_6) in the following, both with a GWP $\simeq$ 230.

§ RESULTS

This contribution shows the latest results from a beam test carried out in October 2021. In particular, the detector was flushed with the standard gas mixture in order to have a reference measurement, which was then compared to the two ecogases. The Cs source can be shielded by means of a set of lead filters, allowing one to test the detectors at different background rates. For the upcoming LHC physics runs, the expected hit rate on the ALICE RPCs is around 100 Hz/cm$^{2}$, which includes a safety factor of 2 [8].

STD gas mixture

In figure <ref>, the performance in terms of efficiency and absorbed current is shown for the standard gas mixture. Since the RPC was read out on both planes, with perpendicular strips, it was considered to be efficient if, for a given trigger, at least one strip per plane had given a signal above the set threshold. Also, the absorbed current was measured and, in the figure, the current density (in $\mu$A/cm$^{2}$) is reported. The black line represents the measurement taken without irradiation (source OFF in the following), while the red and green ones show respectively the behavior of the RPC with a gamma hit rate of 100 and 300 Hz/cm$^{2}$.

It is possible to observe that for higher rates, the efficiency curves are shifted to higher voltages, due to a high drop in the high voltage caused by the higher rate of impinging photons. At the same time, the maximum value of efficiency is slightly reduced. These observations can be explained by taking into consideration the RPC rate capability.

In figures <ref> and <ref> the results obtained under the same experimental conditions as discussed earlier for the standard gas mixture, are reported for the two ecogases under study.

ECO2 gas mixture

ECO3 gas mixture

The results with both ecogases are promising and a few observations can be made: the first one is that the source OFF efficiency curve is shifted towards higher voltages, concerning the standard gas mixture, due to the presence of HFO. In particular, for the ECO2 mixture the shift is 1.2 kV while for ECO3 is 0.4 kV.

The effects of the RPC rate capability is even more visible concerning the standard gas mixture, yielding a greater shift and higher efficiency drop. However the situation at a rate of 100 Hz/cm$^{2}$ (red curve) is still acceptable.

The last point that could be raised regards the current absorbed by the detector, in particular, it is doubled and this needs to be closely studied to investigate if it could lead to a deterioration effect on the detector performance.

§ CONCLUSION

A search for new eco-friendly gas mixtures for RPCs is ongoing. In this context, the ECOgas@GIF++ collaboration was established and systematic aging and beam test studies are ongoing. During a dedicated beam test period the performance of an ALICE-like prototype were tested with two eco-friendly gas mixtures and compared to a well-known C_2H_2F_4-based gas mixture. The detector has shown satisfactory performance for the expected rate in ALICE during the upcoming LHC physics runs but the working point is moved to higher voltages and the absorbed current is doubled with respect to the reference gas mixture. For this reason, it is mandatory to carry out in-depth aging studies to better understand whether or not these issues have any long-term impact on the detector performances.

[1]
ALICE collaboration, “The ALICE experiment at the CERN LHC,” Journal
of Instrumentation, vol. 3, pp. S08002–S08002, 2008.

[2]
G. Fronzé, “Performance of the ALICE muon trigger system in pp and
Pb-Pb collisions at the LHC,” Journal of Instrumentation, vol. 11,
pp. C06003–C06003, Jun 2016.

[3]
Council of European Union, “Council regulation (EU) no 517/2014,” 2014.

[4]
L. Benussi et al., “A study of HFO-1234ze (1,3,3,3-Tetrafluoropropene)
as an eco-friendly replacement in RPC detectors,” 2015.

[5]
A. Bianchi et al., “Environment-friendly gas mixtures for Resistive
Plate Chambers: an experimental and simulation study,” JINST,
vol. 15, no. 09, p. C09006, 2020.

[6]
P. Dorothea et al., “The radiation field in the Gamma Irradiation
Facility GIF++ at CERN,” Nuclear Instruments and Methods in Physics
Research Section A: Accelerators, Spectrometers, Detectors and Associated
Equipment, vol. 866, pp. 91–103, 2017.

[7]
S. Manen et al., “FEERIC, a very-front-end ASIC for the ALICE muon
trigger resistive plate chambers,” IEEE Nuclear Science Symposium
Conference Record, 2013.

[8]
A. Ferretti, “The upgrade of the RPC-based ALICE Muon Trigger,” JINST, vol. 14, no. 06, p. C06011, 2019.


# A Deep Learning Approach for Masking Fetal Gender in Ultrasound Images

Amit Borundiya
<EMAIL_ADDRESS>Arshak Navruzyan
<EMAIL_ADDRESS>Dennis Igoschev
<EMAIL_ADDRESS>Feras C. Oughali
<EMAIL_ADDRESS>Hemanth Pasupuleti
<EMAIL_ADDRESS>Mike Fuller
<EMAIL_ADDRESS>Vinay Kanigicherla
<EMAIL_ADDRESS>T S Aniruddha Kashyap
<EMAIL_ADDRESS>Rishabh Chaurasia
<EMAIL_ADDRESS>Dr. Sonali Vinod Jain
<EMAIL_ADDRESS>

Fellowship.AI

###### Abstract

Ultrasound (US) imaging is highly effective with regards to both cost and
versatility in real-time diagnosis; however, determination of fetal gender by
US scan in the early stages of pregnancy is also a cause of sex-selective
abortion. This work proposes a deep learning object detection approach to
accurately mask fetal gender in US images in order to increase the
accessibility of the technology. We demonstrate how the YOLOv5L architecture
exhibits superior performance relative to other object detection models on
this task. Our model achieves 45.8% $AP_{[0.5:0.95]}$, 92% F1-score and 0.006
False Positive Per Image rate on our test set. Furthermore, we introduce a
bounding box delay rule based on frame-to-frame structural similarity to
reduce the false negative rate by 85%, further improving masking reliability.

_K_ eywords Deep Learning $\cdot$ Computer Vision $\cdot$ Object Detection
$\cdot$ Medical Imaging $\cdot$ Ultrasound

##  1 Introduction

In the medical domain, there are a multitude of options with regards to
tomography diagnostic technologies such as X-Rays, Magnetic Resonance Imaging
(MRI), Ultrasound (US), and Computed Tomography (CT). Among all the diagnostic
imaging options, Ultrasound imaging is highly effective in terms of cost and
its versatility in medical diagnosis on various parts of the body in real-
time. While US imaging is used worldwide, it has a tremendous impact in
developing countries as its portability and cost-effectiveness make it one of
the only widely accessible imaging technologies.

However, US imaging faces some challenges in many developing countries where
the technology is perceived to be responsible for infant foeticide as it can
be used to determine the gender of the fetus early in a pregnancy. To
demonstrate the significance of sex-selective abortion, a study [1] was
conducted to provide information on sex ratio at birth (SRB) reference levels
and SRB imbalance. The study identified 12 countries with strong statistical
evidence of SRB imbalance. To quantify the effect of SRB imbalance due to sex-
selective abortion, the annual number of missing female births (AMFB) and the
cumulative number of missing female births (CMFB) were calculated over time.
Missing female birth estimates for the 12 countries from 1970 to 2017 is 23.1
million. The majority of CMFB are concentrated in China with 11.9 million and
in India with 10.6 million. The CMFB between 1970 and 2017 in China and India
made up 51.40% and 45.94%, respectively, of the total CMFB.

In order to combat sex-selective abortion, many of these countries have
stringent laws regulating the use of US machines. Though put in place for good
reason, these laws hinder the aforementioned accessibility that makes US
imaging such a powerful and versatile technology.

The main contribution of this study is the development of a novel deep
learning method that can mask the fetal genital region in US imaging video in
real-time and thus prevent the detection of fetal gender in the early days of
pregnancy by anyone except an authorized medical practitioner. Solving this
challenge will enable further adoption of US technology in developing
countries while simultaneously preventing the occurrence of sex-selective
abortions.

The paper is organized as follows. Section 2 presents presents a literature
review of related work. Section 3 provides details about the dataset and its
creation. Section 4 reports the results of experimental work and provides a
performance comparison of various DNN architectures. Section 5 investigates
parameter optimizations to reduce the occurrence of false positive and false
negative detections. Section 6 demonstrates the performance of our model’s
masking as reported by an expert sonographer. Section 7 details potential
further work. Lastly, Section 8 concludes this study.

##  2 Related Work

The clinical value of identification of fetal gender by ultrasound lies in
deciding whether to carry out prenatal invasive testing in pregnancies at the
risk of sex-linked genetic abnormalities. A study [2] was carried out to
assess the accuracy of fetal gender determination at 11–14 weeks of gestation.
Results showed that the accuracy of gender determination increased with
gestation from 70.3% at 11 weeks to 98.7% at 12 weeks and 100% at 13 weeks.
Male fetuses were wrongly assigned as female in 56% of cases at 11 weeks, 3%
at 12 weeks and 0% at 13 weeks. In contrast, only 5% of the female fetuses at
11 weeks were incorrectly assigned as male, and this false-positive rate was
0% at 12 and 13 weeks.

Another similar study [3] was performed on 496 singleton pregnancies in the
first trimester. It was possible to determine fetal gender for 441 out of 496
patients. Of the 55 cases where no identification of gender was possible, 39
were in the 11-week gestational age group, representing 40.6% of this
category. The attending obstetrician was a certified sonographer for first-
trimester screening registered with the Fetal Medicine Foundation of England.
Reported results show the success rate for correctly identifying fetal gender
(where identification was possible) increased with gestational age, from 71.9%
at 11 weeks, 92% at 12 weeks, and 98.3% at 13 weeks.

Lakra et al. [4] proposed a deep learning-based method to identify in real-
time frames of B-scans that contain gender revealing features and block those
frames out from the display. A residual network with 23 convolutional layers
was used. The accuracy of the classifier was found to be 83.56% on average,
with a 69.57% F1-score using 5-fold cross-validation. Ultrasound B-mode fetal
scan data were obtained from 50 women 12 to 20 weeks into their pregnancy.
This time frame was selected as the gender of the fetus becomes identifiable
from ultrasound scans during this period, and abortion in India is allowed
until 20 weeks of pregnancy.

##  3 Dataset

It is understood that there is no publicly available dataset of ultrasound
images pertaining to fetal gender determination. Therefore, images of both
transverse and mid-sagittal planes containing the fetal genital tubercle were
collected from public pregnancy forums to compile the labeled data. In order
to simulate a typical fetal US scan in which both gender identification and
non-gender identification frames are visible, negative samples were taken from
a recently published dataset of maternal-fetal ultrasound images [5]. The
maternal-fetal dataset contains images from the most widely used fetal
anatomical planes (Abdomen, Brain, Femur and Thorax), the mother’s cervix, and
a general category to include any other less common image or plane. Images
were selected based on patient ID for both validation and test sets to ensure
there is no data leakage. Images from the classes "Femur" and "Other" of the
maternal-fetal dataset were omitted as those classes contain some images with
transverse and mid-sagittal features or irrelevant frames such as
electrocardiogram plots. Negative samples in train, validation and test sets
comprise 72% of total images [4]. Furthermore, they have been selected using
stratified sampling of the plane classes and sub-classes so as to maintain the
class distribution of the original maternal-fetal dataset. The training set
contains 70% of both labeled images and negative samples, while validation and
test sets both contain 15% each of the labeled images and negative samples.
Table 1 shows the composition of the dataset.

Table 1: Dataset Composition  Dataset | Total | Labeled | Negative
---|---|---|---
|  | Total | Transverse | Mid-Sagittal |
Training | 2534 | 707 | 355 | 352 | 1827
Validation | 543 | 152 | 77 | 75 | 391
Test | 543 | 152 | 76 | 76 | 391

###  3.1 Labeled Data Acquisition

Images of transverse and mid-sagittal planes containing the fetal genital
tubercle section were manually collected from public pregnancy forums
111www.whattoexpect.com and www.ingender.com. Only high-quality images were
selected. Images that were small in size, blurry, or with artifacts such as a
reflection of ambient light were discarded. A rigorous exercise to ensure no
duplicate images was strictly followed.

###  3.2 Data Preprocessing

The collected images went through a pipeline of preprocessing steps to ensure
that the dataset was of high quality. Images that had background noise near
the borders were cropped appropriately. Images that had excessive text or
inpainted shapes, such as arrows near the genital region, were discarded. In
some cases, images with minor inpainted text or shapes were treated using a
segmentation mask followed by applying OpenCV’s inpaint function. Due to
imperfections in the segmentation masks, a top-hat morphological operation
with a $7\times 7$ rectangular structuring element was used, similar to [6]
and the edges were recovered by dilating the image with a $3\times 3$
rectangular structuring element. The thresholded text segmentation mask was
obtained using OTSU thresholding [7] and Hysteresis thresholding [8].

###  3.3 Data Annotation

In this study, two classes of gender identification planes were considered:
the transverse plane and the mid-saggital plane. An effort has been made to
ensure that the dataset is balanced across considered classes. All bounding
boxes were squares such that no obvious differences between the bounding box
shapes are apparent between the sexes.

Regarding the annotation of the transverse plane:

  * •

For females, the edge of the genitalia should sit in the center of the
bounding box. For males, the edge of the penis should sit at the relative edge
of the bounding box.

  * •

The inner edges of either leg should be present towards the boundary of the
bounding box.

  * •

The genital region up to the perineum should be captured.

Regarding the annotation of the mid-sagittal plane:

  * •

The entirety of the typically brighter tubercle length should be captured up
to the edge of the bounding box.

  * •

The bounding boxes should include part of the fetal body.

Fig. 1 depicts a few sample annotations.

Figure 1: Sample Annotations

##  4 Experiments

Several object detection architectures have been considered in this study [9,
10, 11, 12]. Table 2 provides the details of various configurations of
selected architectures.

Table 2: Configuration of Object Detection Architectures Model | Backbone | # params
---|---|---
Faster R-CNN | ResNet-50-FPN-1x | 41 M
Faster R-CNN | ResNet-101-FPN-1x | 60 M
RetinaNet | ResNet-50-FPN-1x | 36 M
RetinaNet | ResNet-101-FPN-1x | 55 M
EfficientDet-D2 (AdvProp) | EfficientNet-B2 | 8.1 M
EfficientDet-D3 (AdvProp) | EfficientNet-B3 | 12 M
EfficientDet-D4 (AdvProp) | EfficientNet-B4 | 21 M
EfficientDet-D5 (AdvProp) | EfficientNet-B5 | 34 M
YOLOv5L | YOLOv5L CSPDarknet | 46.6 M

###  4.1 Training

Models were trained using transfer learning from weights trained on the COCO
dataset [13] for YOLOv5L and the ImageNet dataset [14] for all other models.
The models’ hyperparameters were optimized by observing the validation set
performance. All non-YOLOv5L models were trained for 20 epochs with pretrained
frozen backbones and another 20 epochs with the entire parameters unfrozen,
using the Adam optimizer [15]. Training for additional epochs did not provide
an improvement. The SGD optimizer was used for the YOLOv5L model, and training
was performed for 125 epochs with pretrained and unfrozen backbone parameters.
All models were trained using a one-cycle learning rate scheduler [16]. The
maximum learning rate was set to 0.01 for YOLOv5L, 0.003 for EfficientDet
variants, and 0.0001 for other models. Layer discriminative learning rates
were used for fine-tuning the pretrained backbones. For the EfficientDet
variants, a range of $[3\times 10^{-5},3\times 10^{-4}]$ was used, while a
range of $[1\times 10^{-6},1\times 10^{-5}]$ was used for the Faster R-CNN and
RetinaNet models.

The following data augmentations were used in training: ShiftScaleRotate,
RGBShift, RandomBrightnessContrast, RandomFlip, RandomSizedCrop, and Blur. An
adversarial training procedure (AdvProp) [17] was used to enhance the
performance of EfficientDet models. Additionally, YOLOv5 uses Mosaic [18] and
Mixup [19].

###  4.2 Results

In this section, we evaluate the performance of the models on the test set.
The input image size after augmentation was fixed to $384$ pixels square in
order to directly compare the models. The detection confidence and $IoU$
threshold parameters were both set to $0.5$. Models are compared on average
precision ($AP$), label Precision ($P$), label Recall ($R$), and label
F1-score ($F1$). $AP_{0.5}$ is the average precision of all classes with
bounding box predictions over ground truth of $IoU>0.5$. $AP_{[0.5:0.95]}$ is
the average of the average precision across all classes at $IoU$ between $0.5$
and $0.95$ with a step size of $0.05$. Table 3 shows the metrics on the test
set.

Table 3: Performance on the test set at $conf_{thr}=0.5$, $IoU_{thr}=0.5$ Model | $AP_{[0.5:0.95]}$ | $AP_{0.5}$ | $P$ | $R$ | $F1$
---|---|---|---|---|---
Faster R-CNN (ResNet 50) | 0.359 | 0.795 | 0.756 | 0.821 | 0.787
Faster R-CNN (ResNet 101) | 0.356 | 0.809 | 0.84 | 0.834 | 0.837
RetinaNet (ResNet 50) | 0.307 | 0.704 | 0.931 | 0.711 | 0.806
RetinaNet (ResNet 101) | 0.366 | 0.809 | 0.906 | 0.834 | 0.869
EfficientDet-D2 (AdvProp) | 0.371 | 0.796 | 0.961 | 0.821 | 0.886
EfficientDet-D3 (AdvProp) | 0.404 | 0.845 | 0.963 | 0.861 | 0.909
EfficientDet-D4 (AdvProp) | 0.387 | 0.808 | 0.933 | 0.834 | 0.881
EfficientDet-D5 (AdvProp) | 0.419 | 0.837 | 0.942 | 0.861 | 0.900
YOLOv5L | 0.448 | 0.853 | 0.962 | 0.855 | 0.905

###  4.3 Discussion

The results of the YOLOv5L and EfficientDet-D3 (AdvProp) models show a very
high performance of over 90% label F1-score, with the YOLOv5L model’s higher
$AP_{[0.5:0.95]}$ demonstrating a better average precision over different IoU
thresholds. Fig. 2 depicts sample detection and masking on images from the
test set using YOLOv5L. In order to further analyze the performance of the
best performing YOLOv5L model, further investigation into False Positives and
False Negatives will be made in the following sections.

Figure 2: Sample detection and masking from the test set (YOLOv5L)

##  5 Minimizing False Negatives & False Positives

For any deep learning model, domain-shift between the training data and the
real world is a necessary practical consideration. In object detection, this
can potentially lead to problems caused by false negative and false positive
detections.

For the proposed application, false negatives (FNs) are potentially extremely
costly. If the model fails to mask the genital region, the gender of the fetus
may be revealed. In addition, false positives (FPs) are potentially very
harmful in practice. FPs correspond to incorrect detection and masking of
areas that do not contain the regions of interest. This behavior is not
desired as it may prevent the practitioner from performing a smooth scan by
incorrectly masking areas in other fetal anatomical planes like the abdomen or
the brain.

In order to enhance the performance of our best-performing model and
particularly minimize FNs and FPs, the detection confidence and IoU thresholds
have been investigated. The highest F1-score of 0.92 with $AP_{[0.5:0.95]}$
0.458 and FPPI 0.006, defined as the rate of False Positive detections per
frame [20], is found at detection confidence threshold 0.318 and IoU threshold
0.6 [12], as shown in Fig. 3.

##  6 Masking Performance: Specialist Sonographer

In order to confirm the practical utility of our proposed system, a registered
expert medical practitioner has provided guidance in the development of the
detection masking functionality and further feedback on the final performance.

One hundred ROI-containing images from the test set were provided to the
medical expert, first with ROIs masked by our system, and then subsequently,
the same original (unmasked) images were provided. In both cases, the medical
expert’s task was to attempt to identify the gender of the fetus. In the case
of the masked images, the gender identification rate was 5%. The masking by
our system was shown to provide a reduction in recall rates of 88% for the
male fetal gender identification and 97% for the female gender identification.
However, it should be noted that all of the correct gender identifications
were made on images with false negatives and therefore no masking was present.
As such, the fetal gender within 100% of the correctly masked images was not
identifiable. This further impresses the requirement of reducing FN detections
to negligible amounts.

It should be noted the data set comprises publicly available images and, as
such, the results reported here are for demonstration purposes only and should
not be considered indicative of performance in deployment. Further development
and evaluation of the system under clinical conditions are therefore required.

Figure 3: F1 detection confidence threshold curve for YOLOv5L

##  7 Further Work

###  7.1 Post-processing to Reduce False Negatives in Real-Time Masking

No substantial gaps of consecutive false negatives were observed in the
proposed system when testing on multiple videos of fetal US scans. However,
frames with false negatives were occasionally observed. In addition to the
optimization measures stated in Section 5, empirical testing has shown that
the following two proposed approaches effectively reduce this problem;
BBoxHold and BBoxHoldSim.

The BBoxHold approach is quite simple: Whenever the model predicts a bounding
box, the recently predicted bounding box is held for a few frames if no
predictions are made. This approach has been found to reduce the potential
flickering effect of occasionally missing predictions on true-positive frames.
However, this approach may fail when the model does not predict anything for a
sufficiently long enough time to reveal gender, depending on the hold time.
BBoxHoldSim approach tackles this problem by comparing the current frame with
the most recent frame containing a bounding box prediction using Structural
Similarity Index Measure (SSIM) [21]. If the similarity is greater than a
threshold, BBoxHold is activated.

The two approaches were tested on four sections of US videos comprising gender
identifying frames, where the BBoxHold and BBoxHoldSim approaches were found
to reduce FN rates by 56% and 85%, respectively, compared to the YOLOv5L model
alone at the optimum F1 score detection confidence threshold of 0.318 and IoU
threshold 0.6. However, further investigation into the combined detection
threshold settings and these methods is required.

###  7.2 US Video Test Dataset

In order to aid further development of this system, a test dataset formed from
consecutive video frames of several US fetal scan videos including gender
determination frames is likely to be required. By doing so, the training can
be tuned to maximally reduce any remaining false negatives and false positives
in video streams, that are apparent when training on single non-consecutive
gender-region frames.

##  8 Conclusion

In this paper, a system for masking fetal gender in ultrasound scans during
the early stages of pregnancy using deep learning-based object detection has
been proposed. An F1-score of 92% for detection and masking has been achieved
on our dataset containing both true-positive and negative samples simulating a
full fetal scan. In addition, further delay and Structural Similarity Index
Measure methods have been investigated in order to reduce the levels of
potential false negative detections that could otherwise reveal the fetal
gender. We hope the integration of this approach into ultrasound systems will
allow a more ubiquitous usage of ultrasound scans in regions currently
affected by sex-selective abortions caused by fetal gender determination.

## References

  * [1] Fengqing Chao, Patrick Gerland, Alex R. Cook, and Leontine Alkema.  Systematic assessment of the sex ratio at birth for all countries and estimation of national imbalances and regional reference levels.  Proceedings of the National Academy of Sciences, 116(19):9303–9311, 2019.
  * [2] Z. Efrat, O. O. Akinfenwa, and K. H. Nicolaides.  First-trimester determination of fetal gender by ultrasound.  Ultrasound in Obstetrics & Gynecology, 13(5):305–307, 1999.
  * [3] C.H. HSIAO, H.C. WANG, C.F. HSIEH, and J.J. HSU.  Fetal gender screening by ultrasound at 11 to 13+6 weeks.  Acta Obstetricia et Gynecologica Scandinavica, 87(1):8–13, 2008\.
  * [4] Praveen Paul Lakra, Ashish Kumar, Narayanaswamy Mohanram, Ganapathy Krishnamurthi, and Arun Kumar Thittai.  Deep-learning based identification of frames containing foetal gender region during early second trimester ultrasound scanning.  In 2019 IEEE International Ultrasonics Symposium (IUS), pages 471–474, 2019.
  * [5] Xavier P. Burgos-Artizzu, David Coronado-Gutiérrez, Brenda Valenzuela-Alcaraz, Elisenda Bonet-Carne, Elisenda Eixarch, Fatima Crispi, and Eduard Gratacós.  Evaluation of deep convolutional neural networks for automatic classification of common maternal fetal ultrasound planes.  Nature Scientific Reports, 10(10200), 2020.
  * [6] N. Efford.  Digital image processing: A practical introduction using java.  2000\.
  * [7] Nobuyuki Otsu.  A threshold selection method from gray-level histograms.  IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979.
  * [8] John Canny.  A computational approach to edge detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6):679–698, 1986.
  * [9] Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1137–1149, 2015.
  * [10] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar.  Focal loss for dense object detection.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society.
  * [11] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  Efficientdet: Scalable and efficient object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10778–10787, 2020.
  * [12] Glenn Jocher, Alex Stoken, Jirka Borovec, NanoCode012, Ayush Chaurasia, TaoXie, Liu Changyu, Abhiram V, Laughing, tkianai, yxNONG, Adam Hogan, lorenzomammana, AlexWang1900, Jan Hajek, Laurentiu Diaconu, Marc, Yonghye Kwon, oleg, wanghaoyang0106, Yann Defretin, Aditya Lohia, ml5ah, Ben Milanko, Benjamin Fineran, Daniel Khromov, Ding Yiwei, Doug, Durgesh, and Francisco Ingham.  ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations, April 2021.
  * [13] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [15] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [16] Leslie N. Smith and Nicholay Topin.  Super-convergence: Very fast training of residual networks using large learning rates.  CoRR, abs/1708.07120, 2017.
  * [17] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoc V. Le.  Adversarial examples improve image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [18] Alexey Bochkovskiy, Chien-Yao Wang, and H. Liao.  Yolov4: Optimal speed and accuracy of object detection.  ArXiv, abs/2004.10934, 2020.
  * [19] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  CoRR, abs/1710.09412, 2017.
  * [20] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: A benchmark.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 304–311, 2009.
  * [21] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.


11institutetext: University of Bayreuth, Bayreuth, Germany
11email<EMAIL_ADDRESS>22institutetext: Institute of Computing, University of Campinas, Campinas, SP,
Brazil
22email<EMAIL_ADDRESS>
# How to Surprisingly Consider Recommendations? A Knowledge-Graph-based
Approach Relying on Complex Network Metrics

Oliver Baumann All authors contributed equally11 0000-0003-4919-9033 Durgesh
NandiniHow to Surprisingly Consider Recommendations? A Knowledge-Graph-based
Approach Relying on Complex Network Metrics 11 0000-0002-9416-8554 Anderson
RossanezHow to Surprisingly Consider Recommendations? A Knowledge-Graph-based
Approach Relying on Complex Network Metrics 22 0000-0001-7103-4281 Mirco
Schoenfeld 11 0000-0002-2843-3137 Julio Cesar dos Reis 22 0000-0002-9545-2098

###### Abstract

Traditional recommendation proposals, including content-based and
collaborative filtering, usually focus on similarity between items or users.
Existing approaches lack ways of introducing unexpectedness into
recommendations, prioritizing globally popular items over exposing users to
unforeseen items. This investigation aims to design and evaluate a novel layer
on top of recommender systems suited to incorporate relational information and
suggest items with a user-defined degree of surprise. We propose a Knowledge
Graph (KG) based recommender system by encoding user interactions on item
catalogs. Our study explores whether network-level metrics on KGs can
influence the degree of surprise in recommendations. We hypothesize that
surprisingness correlates with certain network metrics, treating user profiles
as subgraphs within a larger catalog KG. The achieved solution reranks
recommendations based on their impact on structural graph metrics. Our
research contributes to optimizing recommendations to reflect the metrics. We
experimentally evaluate our approach on two datasets of LastFM listening
histories and synthetic Netflix viewing profiles. We find that reranking items
based on complex network metrics leads to a more unexpected and surprising
composition of recommendation lists.

###### Keywords:

Recommender Systems Knowledge Graphs Complex Network Metrics.

##  1 Introduction

Recommender Systems (RSs) aim to offer a personalized view of large complex
spaces, prioritizing items likely to interest the user by analyzing user
preferences, historical behavior, and item characteristics [4].
Recommendations can expose users to relevant items and expand their
understanding of the catalog, regardless of whether in an e-commerce, media-
streaming, or GLAM setting. The most popular approaches for recommender
systems are collaborative filtering and content-based filtering [19, 18].
User-item recommendations are an important part of the discovery process of
large collections. In content-based filtering, item characteristics are used
to determine the similarity between items rated (viewed, listened, bought,
etc.) by a user, and “unseen” items. Collaborative filtering, on the other
hand, determines users similar to the target user and predicts ratings on
unseen items by the target user.

While existing approaches have been shown to produce meaningful
recommendations, the items they recommend tend to be expected and located in
whatever portion of the catalog is considered “mainstream”. These approaches
do not consider the rich relations between items beyond the realm of
similarity alone. We argue that users may profit from recommendations that
present a surprise element, as they may come in touch with concepts they have
been unaware of.

Knowledge Graph RSs combine the capabilities of RSs and Knowledge Graphs (KGs)
by incorporating and analyzing the structured representation of information in
KGs. These systems leverage the interconnected nature of entities and their
attributes within the KG to enhance the accuracy and relevance of
recommendations. Using KGs, RSs can go beyond simple user-item interactions
and incorporate a broader understanding of the relationships among items,
users, and other entities. This allows for more sophisticated recommendation
approaches that consider not only the user’s preferences. In this sense,
contextual information encoded in KGs influences recommendation items. For
example, in a movie recommendation scenario, a KG-based recommender system
could consider not only the user’s past viewing history and ratings. It can
consider, for instance, the genre of the movie, the actors and directors
involved, and the relationships between movies based on shared themes or
motifs.

In this study, we propose a layer on top of recommender systems, extending
their functionality by a configurable degree of surprise. Our approach
considers relational information among items encoded in KGs and suggests items
with a user-defined degree of surprise relying on results generated by a
recommender system. The main research question guiding our investigation is
whether network metrics computed on the KG influence the degree of surprise
within the recommendations. We propose taking advantage of KG’s graph
structure, employing complex network measurements [17] as a resource for
encoding entity relevance in a KG. Centrality measurements denote different
meanings of relevance for graph nodes, bringing novelty aspects for analyses
over KGs. Our assumption highlights that the “surprisingness” of
recommendations is reflected in the network-level metrics of the KG.

Figure 1 provides a high-level overview of our approach. We construct KGs from
two distinct catalogs: users’ listening events on the platform
LastFM111https://www.last.fm/, and TV shows and movies on
Netflix222https://www.netflix.com. User profiles for LastFM are available
through the LFM-1b dataset[20]; for Netflix, we generate synthetic profiles.
Recommendations for these profiles are then generated through state-of-the-art
recommender systems (RS). Our work supports any RS, as we focus on re-ranking
recommendations to surface surprising results. Consequently, a specific RS
optimal for a particular use case can be selected. For each user profile, we
determine the induced subgraph on the catalog-KG that includes all items the
user interacted with and further entities that enrich the model. Then, for
each user and each item in their recommendation list, we assess the impact of
including that item and its KG-informed neighborhood on the user’s subgraph
through pre-determined graph metrics. The original recommendation lists are
then re-ranked according to their relative impact.

Our contributions are summarized as follows:

  * •

Insert a configurable level of surprise to any recommender system by adding a
layer of meta-analysis on obtained recommendations;

  * •

Identify a network metric that correlates with different dimensions of
surprise;

  * •

Provide a comparative study regarding several network-level metrics for
reranking recommendation results;

The remainder of this article is organized as follows: Section 2 discusses
related work. Section 3 presents our proposal. Section 4 reports our
experimental evaluation and its results. Section 5 discusses our findings.
Section 6 wraps up our investigation and points out directions for future
studies.

##  2 Related Work

Joseph & Jiang [10] proposed a graph traversal algorithm novel weighting
scheme for cold-start content-based recommendation by utilizing named
entities. Their work computes the shortest distance between named entities
over large KGs. Wang et al. [24] introduced the KG Attention Network, which
enhances the effectiveness of collaborative filtering in RSs by effectively
modeling the high-order connectivity between users, items, and entities within
a KG. Their research investigated how different levels of connectivity, first-
order, second-order, third-order, etc. impact the model’s effectiveness. They
discussed the findings of using attention mechanisms and KG embeddings.

Hui et al. [8] presented a RS called ReBKC that uses auxiliary information
such as historical user behavior and KGs to provide personalized suggestions.
Their investigation integrates KG embeddings and user-item interaction data to
address issues like sparse data and cold starts. ReBKC suggests using KGs as
heterogeneous networks to incorporate additional information to unify
embeddings of user behavior and knowledge features. Their proposed algorithm
employs collaborative filtering, enhanced by the rich semantic associations in
KGs, to mine user preferences more deeply. The system learns from historical
user interactions and multiple types of relationships within the KG.

Zhang et al. [27] addressed the limitations of collaborative filtering in RSs
by leveraging heterogeneous information in a knowledge base to improve the
quality of RSs. Their proposed framework – Collaborative Knowledge Base
Embedding (CKE) – comprises three components to extract semantic
representations from the structural, textual, and visual content of items.
These components employ techniques such as heterogeneous network embedding,
stacked denoising auto-encoders, and stacked convolutional auto-encoders to
extract textual representations and visual representations. It then jointly
learns the latent representations in collaborative filtering as well as items’
semantic representations from the knowledge base. Kaminskas and Bridge [11]
looked into the aspects of diversity, serendipity, novelty, and coverage and
explained that introducing surprise in RS can burst the “user filter bubble”
by finding interesting items that the user might not have otherwise
discovered.

Kotkov et al. [13] examined the concept of serendipity in the context of RS.
Their work discussed different approaches to measure and enhance serendipity
in RS, including the use of algorithms that utilize uncommon similarity
measures or adapt based on user feedback. Their investigation looked at the
balance between accuracy and novelty in recommendations and explored both
offline and online evaluation strategies for assessing the effectiveness of RS
in delivering serendipitous results. On the other hand, De Gemminis et al. [3]
proposed to produce serendipitous suggestions by utilizing the knowledge
infusion process. Their investigation addressed the overspecialization issue
in RSs, proposing to enhance serendipity by suggesting surprising items. Their
approach enrichs a graph-based recommendation algorithm with background
knowledge to uncover hidden correlations among items.

Baumann and Schoenfeld [1] used a KG-based RS to evaluate diversity and
novelty of recommendations, both on a content- and a network-level. Using
subgraphs constructed from user profiles, they generated recommendations by
favoring unpopular items in the catalog that exhibit a high distance from a
user’s profile in terms of content-based features. Apart from unexpectedness
and diversity on a content level, they found this approach to result in a more
fair degree distribution on the individual profile subgraphs.

To the best of our knowledge, our present study is the first to apply complex
network measurements to rerank the order of RS results. Our approach looks at
the graph structure within the KG changes to compute the metrics for obtaining
surprising recommendations.

##  3 KG-Informed Recommendation (Re-)Ranking

We propose a recommendation process as a two-step approach consisting of
retrieval and ranking steps. In the retrieval step, recommendation candidates
are determined by an existing RS. These candidates are ordered in the ranking
step, and the top $N$ elements are returned to the user. Figure 1 presents an
overview of the recommendation process.

Figure 1: Overview of the proposed knowledge-graph informed recommendations.
KGs are constructed for the item catalog and all user profiles. The latter
serve as input to an arbitrary state-of-the-art RS, whose results are re-
ranked according to the impact the items would have were they included in the
original user profile.

This investigation treats the RS algorithm as a closed system over which we
can not exert any influence. Our solution emphasizes and contributes to the
ranking stage. We determine an item’s rank based on its impact on network
metrics correlating with surprise. Section 4.2 presents a list of metrics
investigated.

To evaluate such metrics, we construct KGs from datasets suited for the
recommendation task (cf. Section 4.1). Two types of KGs are constructed. The
first type is a KG representing the catalog, i.e., containing the entire
knowledge about the catalog. This includes the whole set of recommendable
items and all the metadata describing them. The second type stands for the
user-profile KGs. They are subgraphs of the catalog KG, representing items
users have already interacted with. Such KGs are constructed considering TBox
statements representing the domain of their datasets. Therefore, including
recommendable entities, as well as additional entities, along with
heterogeneous relations.

The recommendable entities are evaluated by including them in the user-profile
KGs. According to those existing in the catalog KG, a recommendable entity is
included along with its relationships and further entities. From the updated
user-profile KG, we compute complex network metrics (cf. Figure 2). The
process is conducted for all the recommendable items and all available
metrics. At the final stage, our solution provides a re-ranked recommendation
list sorted according to each metric.

Figure 2: KG-Informed Recommendation. The sub-KG represents a user profile. A
node representing a recommendation is obtained from the catalog KG and
included in the sub-KG along with applicable edges. We then compute network
metrics from the sub-KG.

Where network metrics do not result in scalar values, but in distributions, we
calculate the Herfindahl-Hirshman-Index (HHI) [6] to obtain a single value
representing the concentration of the network (cf. [22]). Let $s$ be the
relative centrality score over all vertices, and $N$ the number of vertices,
then the index and its normalized form are given by

$\displaystyle HHI$ $\displaystyle=\sum_{i=1}^{N}s^{2}$ $\displaystyle
HHI^{*}$ $\displaystyle=\frac{HHI-1/N}{1-1/N}$

Values of $HHI^{*}$ range in $[0,1]$, with $0.0$ corresponding to a balanced
network with no monopolies and a value of $1.0$ indicating a strongly
centralized network.

Formally, let $KG$ be a domain KG, consisting of concepts $C$ and relations
$R$. A user profile $U=\\{u_{1},u_{2},\dots,u_{n}\\}$ is a subset $U\subset C$
of concepts a user has interacted with. Each user profile constitutes an
induced subgraph $SG\subset KG$ containing the history items and further
concepts and relations.

For a recommender system RS, let $I=\\{i_{1},i_{2},\dots,i_{n}\\}$ be the list
of recommended concepts, ordered by a score assigned through RS. Then,
$N_{KG}[i_{n}]$ denotes the closed neighborhood of the vertex $i_{n}$ in $KG$
that corresponds to this recommendation. Let $SG^{\prime}$ denote the induced
subgraph produced by including $N_{KG}[i_{n}]$ in $SG$, i.e., by adding a
recommendation to the user subgraph.

Lastly, let $m$ denote any graph-metric, $m_{baseline}=m(SG)$ the value of
this metric on the original user subgraph, and $m_{update}=m(SG^{\prime})$ its
value after incorporating the recommendation in the subgraph. Then, our method
for re-ranking recommendations is as follows:

  1. 1.

Given a dataset, construct a KG $KG$;

  2. 2.

Given a user profile $U$, determine the subgraph $SG$ of $KG$ that contains
all items in the user’s profile and all relations and intermediate entities;

  3. 3.

Given a set of items in $U$, determine recommendations using any RS;

  4. 4.

For each recommended item $i_{n}$, obtain the metric $m$ on the subgraph
$SG^{\prime}$ including $i_{n}$;

  5. 5.

Re-rank all items according to their impact on the given computed metric;

Algorithm 1 provides a formal formulation of our solution.

Algorithm 1 KG-informed recommendation

0: $KG$ {Catalog KG}

0: $SG$ {User profile subgraph}

0: $RS$ {Blackbox recommender system}

0: $m$ {A graph-metric}

1: $v_{user\\_nodes}\leftarrow SG.get\\_all\\_nodes()$

2: $I\leftarrow RS(v_{user\\_nodes})$

3: $I^{\prime}\leftarrow\emptyset$

4: for all $i\in I$ do

5: $SG^{\prime}\leftarrow SG.copy()$ {Evaluate always on the same instance}

6: $SG^{\prime}.add\\_node(i)$

7: $edge\\_list\leftarrow\emptyset$

8: for all $neighbor\in KG.get\\_neighbors(i)$ do

9: if $neighbor\in v_{user\\_nodes}$ then

10: $edges\leftarrow KG.get\\_edges(i,neighbor)$ {May be more than one}

11: $edge\\_list.insert(edges)$

12: end if

13: end for

14: $SG^{\prime}.add\\_edges(edge\\_list)$

15: $metric\\_value\leftarrow metric(i,SG^{\prime})$

16: $I^{\prime}.insert(metric\\_value,i)$

17: end for

18: $kginformed\\_recommendation\\_list\leftarrow
sort(I^{\prime},metric\\_value)$

19: return $kginformed\\_recommendation\\_list$

##  4 Experimental Evaluation

We evaluate our proposed approach on two distinct music- and movie-domain
datasets. Proceeding according to our defined method (cf. Section 3), we
obtain re-ranked recommendation lists for a set of users. As we focus our
attention on “surprisingness” of recommendations, rather than measuring
precision/accuracy, we turn to “beyond accuracy”-metrics commonly used in the
evaluation of surprise and serendipity in RS, such as novelty and diversity
(cf. [2, 5]).

To back up the insights obtained in this sense, we further measure the
agreement of the re-ranked recommendation lists with those generated through
the SOTA RS, which we treat as ground truth for “expectable” recommendations.
Measuring normalized discounted cumulative gain (nDCG) on the two lists allows
us to identify whether the re-ranked variant deviates from the expectable
recommendations.

Our study addresses the following specific research questions:

RQ1

Which network-level metrics correlate with key surprise elements such as
novelty, unexpectedness, and novelty in recommendations?

RQ2

Can these metrics be used to introduce more surprise into state-of-the-art
recommender systems?

###  4.1 Datasets

We report on data collection and curation for the two domains investigated.

####  4.1.1 LastFM

##### LFM-1b

We base our analysis of recommendations for the music domain on the LFM-1b
dataset [20], which we enrich with two further datasets: acoustic features for
a selection of tracks (the CultMRS dataset) curated by Zangerle et al. [26],
and musical genres annotating a subset of tracks within LFM-1b, kindly
provided by Schedl et al. [21]. The acoustic features contained in the dataset
were retrieved via the Spotify
API333https://developer.spotify.com/documentation/web-api/reference/get-
several-audio-features and serve as content-based features describing the
nature of a track. Examples for these features are a track’s tempo, or
danceability. After merging the three datasets, we are left with 379 million
listening events (cf. Table 1).

Table 1: Statistics of LFM-1b after merging with other datasets. # listening events | 379.754.730
---|---
# users | 120.053
# artists | 26.129
# tracks | 282.011
# genres | 2.137

##### KG construction.

From the merged LFM-1b dataset, we construct a KG consisting of artists,
tracks, and genres. To model the relations among these entities, we use
classes and properties provided by three different ontologies:
FOAF444http://xmlns.com/foaf/0.1/, Dublin
Core555http://purl.org/dc/elements/1.1/ and Music
Ontology666http://purl.org/ontology/mo/ [16]; we define an auxiliary URI to
identify entities from LastFM777http://last.fm/lfm-resource. For instance, a
description of the track “Never Gonna Give You Up” by Rick Astley in Turtle
would be:

    lfmr:disco a mo:Genre ;
        dc:title "disco" .
    lfmr:15160 a mo:MusicArtist ;
        foaf:name "Rick Astley" .
    lfmr:t_4471632 a mo:Track ;
        dc:title "Never Gonna Give You Up" ;
        mo:genre lfmr:disco ;
        foaf:maker lfmr:15160 .

##### Recommendations.

We sub-sample the listening events to 1000 users with at least 100 unique
tracks in their profile. The mean number of tracks listened to is 1076 ($\pm
1194$), with a median of 656 tracks. We use the Python library Surprise [7],
which relies on explicit user-item-ratings to determine the base
recommendations. As our data contains implicit ratings as the number of times
a track was listened to by a user, we follow the approach outlined in Kowald
et al. [14] and scale these play-counts into the range $[1,1000]$ using min-
max-normalization; a user’s most-listened track will thus receive an explicit
rating of 1000.

We evaluate six recommendation models provided by Surprise: BaselineOnly,
which predicts a baseline rating estimate from global averages and user/item
deviations (c.f. Koren [12]); KNNBasic, a user-based collaborative filtering
approach using kNN; KNNBaseline, KNNWithMeans, and KNNWithZScore, extensions
of the base kNN model taking into account baselines, mean ratings, and z-score
normalized ratings, respectively; and NMF, a non-negative matrix factorization
model. We use the default parameters provided by the library and employ cosine
similarity as the distance measure for the kNN-based approaches.

Using 5-fold cross-validation, we evaluate each algorithm’s mean absolute
error (MAE), and pick NMF as our final model; Table 2 presents MAE for all
models.

We train NMF on the full data and retrieve rating predictions on the anti-
testset, i.e., on all items present in the training data that the user has not
rated. The recommendation lists obtained this way are truncated to the top 100
items, sorted by descending predicted rating.

Table 2: Evaluation of prediction algorithms. Model | MAE
---|---
NMF | 54.82
BaselineOnly | 62.38
KNNWithZScore | 65.74
KNNBaseline | 67.01
KNNWithMeans | 67.47
KNNBasic | 71.1

####  4.1.2 Netflix

##### Netflix titles dataset

Our evaluation includes the domain of movies and TV shows. For this matter, we
considered the “Netflix titles” dataset, available on
Kaggle888https://www.kaggle.com/datasets/shivamb/netflix-shows. Such dataset
provides a set of 8808 titles available on Netflix, considering movies and TV
shows’ titles, cast, directors, countries, release dates, ratings, and a brief
description. All data is provided in a comma-separated value (CSV) file.

##### KG construction.

The catalog KG was created considering TBox statements representing
relationships and types as provided in the CSV file. Such TBox contains the
type of each entry, which can be either a movie or a TV show. An actor acts on
entries, and a director directs entries. Entries have an English title, a
brief description, a country of origin, a rating, and a duration. All classes
are of the rdf:Class type, and properties of the rdf:Property type.

The dataset provides no user data, therefore, we randomly generated 88 user-
profiles, ranging from a minumum of 5 to 55 entries, representing watched
movies and TV shows. From such, user-profile KGs were generated according to
the same TBox as the catalog KG.

##### Recommendations.

The recommendations were generated with the help of a state-of-the-art RS
[24]. We used the same parameters for the configuration of graph
convolutaional layers, decay factors, and learning rates as the authors of the
paper used for their evaluations.

The data set was cleaned up to obtain meaningful yet compact KGs. To do this,
rdf:label-entities and nodes with a degree of $1$ were removed. In addition,
the rdf:Class and rdf:Property nodes were removed to prevent the knowledge
graph from becoming too centralized. Certain entries in the KGs were labeled
as recommendable items, i.e., only movies and TV shows.

From user-profile KGs, the interactions on recommended items were registered
and divided into training and test data using a 90/10 split, i.e., 90% of the
interactions of a usage profile were listed in the training set.

###  4.2 Experimental Procedure

For both datasets, we employed the following evaluation procedure:

  1. 1.

Obtain base recommendations through SOTA model.

  2. 2.

Re-rank base recommendations according to graph metric (as outlined in Section
3); ranking proceeds in ascending and descending order of item relevance.

  3. 3.

(LFM-1b only) For each metric and each sort order, measure Unexpectedness and
Intra List Diversity using item features.

  4. 4.

For each metric and each sort order, compare the re-ranked with the base lists
using nDCG@10 via TREC_EVAL999https://trec.nist.gov/trec_eval/ (As of Apr.
2024)..

The network metrics applied in this evaluation are the number of nodes, number
of edges, density, PageRank, average degree, {in,out}-degree, betweenness, and
closeness centrality. Such metrics adhere to standard metrics in the field of
social network analysis [25].

To evaluate Unexpectedness and Intra List Diversity, we represent each track
as an 8D vector of acoustic features. The features we use are danceability,
energy, speechiness, acousticness, instrumentalness, liveness, valence and
tempo. In the original dataset, these features range in $[0,1]$, except for
tempo, which we scale into this range using min-max normalization following
prior research [26, 14].

Intra List Diversity (ILD) measures the pairwise distance of all items in a
recommendation list $I$ concerning a distance $d$ (c.f. [2]):

$ILD(I)=\frac{1}{|I|\cdot(|I|-1)}\sum_{i\in I}\sum_{j\in I}d(i,j)$

We measure Unexpectedness on a user-profile level to determine how different a
recommendation is from the user’s previous history. Essentially, this is the
mean distance of each new item to each item the user has interacted with.
Thus, for a user-profile $H$, a recommendation list $I$ and a distance $d$,
Unexpectedness can be expressed as (cf. [2]):

$Unexpectedness(R)=\frac{1}{|I|\cdot|H|}\sum_{i\in I}\sum_{h\in H}d(i,h)$

For both ILD and Unexpectedness, we employed cosine distance between feature
vectors as $d$. In these measures and in nDCG, we limit the recommendation
lists to the top $10$ items, in line with previous findings on users’
searching behaviour [9, 23].

To further assess the rank-based dynamics underlying this re-ordering, we
measure nDCG@10 for each re-ranking. The base recommendations serve as a
ground truth of expectable recommendations for our purposes. Their ranking
thus serves as the relevance judgement of items. High nDCG indicates that the
same items are ranked highly in the base and re-ordered recommendations,
whereas low nDCG indicates more perturbation in the second list. Our
assumption here is that low nDCG indicates that highly expectable items are
ranked lower after re-ordering.

###  4.3 Experimental Results

We present the results achieved from the experimental procedure for both
datasets.

####  4.3.1 LastFM.

We first review the findings from measuring surprise on the re-ranked list of
recommendations before evaluating list perturbation. Section 5 discusses
obtained results and how they can be further interpreted from a network
perspective.

(a) Unexpectedness

(b) Diversity

Figure 3: Measuring surprise on feature-level for recommendations re-ranked by
metric. For Unexpectedness, the highlighted bars denote the comparison to the
SOTA recommendations. For Diversity, the highlighted bars provide the intra-
list values of SOTA and original user profiles (base and profile, resp.).

We evaluate Unexpectedness and Intra List Diversity on all re-ranked
recommendation lists and the two possible ranking orders. We include the
measurements obtained on the original SOTA recommendations as a baseline; in
addition, the users’ mean profile diversity serves as a reference point for
diversity. Figure 3 plots the mean measures against all metrics, split by
ranking order. For Unexpectedness, sorting ascendingly by betweenness results
in the largest deviations from the user’s history, as shown in Figure 3(a).

“Betweenness” in this case corresponds to the Herfindahl-Hirshman-Index (HHI)
of the distribution. Sorting in ascending order thus places low index values
at the top of the list, indicating a fairer distribution and therefore an
overall less centralized network. The opposite holds true for descending
order, where favouring high betweenness-indexes and thus more centralized
networks results more expectable recommendations. We observe that increasing
the number of nodes and edges in the users’ subgraphs has the highest effect
on Unexpectedness.

Turning to Diversity, we first observed that users’ listening behavior seems
largely uniform, as indicated by the profile bars in Figure 3(b). As the
measure of ILD on the user profile is expressed as the mean pairwise distance
between items in the list, a low distance on average indicates the presence of
tracks with similar acoustic features.

We found that preferring a low betweenness index and thus decentralized
networks results in diverse tracks being recommended for ascending sort order,
whereas for the descending case, increasing the number of nodes, edges, and
out-degree produces the most diverse lists out of our approaches but does not
outperform the baseline recommendations. An interesting observation is that
the base recommendations already contain very diverse items. This is in line
with previous findings of the underlying algorithm, NMF, being able to
recommend items from the long-tail of user-item-interactions [15].

To assess the extent to which re-ranked lists correspond with the original,
expectable ranking, Figure 4 plots nDCG@10 for all metrics. We found that out-
degree and betweenness, particularly, result in a high perturbation of ranks.
We highlight that items considered relevant by an expectable RS are not ranked
highly after optimizing for one of the network metrics.

Figure 4: nDCG@10 for LastFM. Panels show nDCG scores obtained on
recommendation lists ranked by metric in ascending and descending order.

####  4.3.2 Netflix.

Unlike LastFM, Netflix titles dataset provides no additional features (such as
acoustic features) to enable performing observations based on Unexpectedness
and Diversity. The analysis on this dataset is entirely based in the nDCG
score, considering the initial 10 elements of the recommendation list. Figure
5 summarizes such results by metrics, both in ascending and descending orders
of relevance.

Figure 5: nDCG@10 for Netflix. Panels show nDCG scores obtained on
recommendation lists ranked by metric in ascending and descending order.

Similarly to LastFM, we observe that betweenness centrality is the metric that
most introduces surprise into the list of recommendations obtained from the
RS. To further illustrate this finding, Table 3 presents the first three
recommendations offered by the RS, compared with those re-ranked through
betweenness centrality in particular user-profile KGs. Table 3 illustrates a
sample for each dataset.

Table 3: Comparing the top three recommended items obtained from state-of-the-
art recommender, against those re-ranked using betweenness centrality applied
on a user-profile KG.

Dataset | SOTA recommender | Re-ranked (betweenness) |
---|---|---|---
Netflix | Bakugan: Armored Alliance | Creeped Out |
The C Word | Black Mirror |
Weird Wonders of the World | Arthur Christmas |
LFM-1b | Iron Maiden, The Talisman | Shakira, Spotlight |
Iron Maiden, When the Wild Wind Blows | Here We Go Magic, Make Up Your Mind |
Shakira, Spotlight | Here We Go Magic, Alone But Moving |

##  5 Discussion

We observe that recommendations sorted by betweenness in ascending order of
the associated HHI exhibit high Unexpectedness and Diversity. Ranking in this
way favors nodes that result in a lower HHI, thus, revealing a more
decentralized user subgraph. In such a KG, many paths among concepts exist,
and there is low monopolization. The opposite holds for a highly centralized
KG, in which a small number of concepts appear along many paths and carry high
importance.

Figure 6 illustrates this effect, presenting an example user subgraph and two
extensions arising from incorporating more diverse (cf. Figure 6(b)) or more
similar (cf. Figure 6(c)) recommendations. Interactions and recommendations
are shown as solid colored circles; related concepts are light colors with an
outline. Diverse items will likely be loosely connected to existing concepts
the user is familiar with and bring along further related nodes, thus
expanding the user’s exposure. Contrast this with the second example, where
similar items are introduced that only exhibit relations to concepts familiar
to the user. These examples illustrate the effect on the number of edges,
nodes, and degree-related measures. In the diverse case, adding two
recommendations results in four nodes and seven edges added to the graph
versus two nodes and two edges for the case of similar items.

(a) Profile subgraph…

(b) …with diverse…

(c) …and similar items

Figure 6: Illustration of the effect of recommendations on betweenness
centrality in profile subgraphs. Diverse items being recommended open
alternative paths in the resulting profile subgraph lowering the betweenness
of all nodes (6(b)), whereas similar items tend to increase the betweenness of
a few nodes (6(c))

Considering the results from evaluating nDCG, we observed that ranking by
betweenness, node-/edge counts, or degree-based metrics yields lists with low-
rank correlation compared to expectable recommendations.

Our study demonstrated that network-level metrics correlate with key surprise
elements such as diversity and unexpectedness (RQ1). We found betweenness
resulting in the most diverse and unexpected recommendations that rank
expectable items lower than a state-of-the-art baseline. We showed that adding
a KG-informed re-ranking model on top of an existing recommender system can
thus introduce a level of surprise into user-item-recommendations (RQ2).

Results highlighted that calculating betweenness may not be computationally
feasible in constrained environments, especially on large profile subgraphs.
Besides truncating user profiles to the most recent interactions as a solution
in this case, our findings suggest that node-/edge counts or degree-based
features are viable alternatives to betweenness.

We identify the Netflix dataset’s lack of rich content-based features,
prohibiting a similar investigation of surprise-related measures as performed
for the enriched LFM-1b dataset. Furthermore, a user study should evaluate the
degree of surprise, as listening and viewing behaviors are governed by highly
subjective user dynamics. We plan to address it in future studies.

The catalog KGs employed in our study only contain intra-domain concepts
(artists, music genres, directors, actors, etc.). However, KGs are well suited
for linking cross-domain concepts, e.g., tracks that appear in a movie’s
score, or actors who are musicians. Not only does this result in a richer
representation of domains, it also enables cross-domain recommendations. We
defer an analysis of surprising recommendations in such settings to future
work.

##  6 Conclusion

We still encounter open research challenges in how systems may deal with and
benefit from surprise recommendations. This investigation designed a solution
incorporating network-level metrics to introduce personalized yet unexpected
recommendations to users. We evaluated the LastFM music and Netflix movies
datasets to determine the extent of intra-list diversity, unexpectedness, and
comparison to nDCG, respectively, affect the degree of surprise in
recommendations. We found that network-level metrics indeed influence the
degree of surprise in recommendations. Our results demonstrated that
betweenness centrality showed a stronger influence when reranking
recommendations for a surprise. Future work involves additional analysis of
surprising recommendations and how textual features from items can be combined
with our designed approach.

## Supplemental Material Statement

Source code and data for the experiments and evaluations conducted in this
work are available at https://anonymous.4open.science/r/kg-recommender-273D/.
The LFM-1b dataset is available at http://www.cp.jku.at/datasets/LFM-1b/, the
CultMRS dataset at https://zenodo.org/records/3477842, and the Netflix titles
dataset at https://www.kaggle.com/datasets/shivamb/netflix-shows.

####  6.0.1 Acknowledgements

This article is the outcome of research conducted within the Africa Multiple
Cluster of Excellence at the University of Bayreuth, funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s
Excellence Strategy – EXC 2052/1 – 390713894.

## References

  * [1] Baumann, O., Schoenfeld, M.: Supporting Serendipitous Recommendations with Knowledge Graphs. In: Tamine, L., Amigó, E., Mothe, J. (eds.) 2nd Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2022). No. 3178 in CEUR Workshop Proceedings, Aachen (2022), http://ceur-ws.org/Vol-3178
  * [2] Castells, P., Hurley, N.J., Vargas, S.: Novelty and Diversity in Recommender Systems. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 881–918. Springer US, Boston, MA (2015). https://doi.org/10.1007/978-1-4899-7637-6_26
  * [3] De Gemmis, M., Lops, P., Semeraro, G., Musto, C.: An investigation on the serendipity problem in recommender systems. Information Processing & Management 51(5), 695–717 (2015)
  * [4] Felfernig, A., Burke, R.: Constraint-based recommender systems: technologies and research issues. In: Proceedings of the 10th international conference on Electronic commerce. pp. 1–10 (2008)
  * [5] Ge, M., Delgado-Battenfeld, C., Jannach, D.: Beyond accuracy: Evaluating recommender systems by coverage and serendipity. In: Proceedings of the fourth ACM conference on Recommender systems. pp. 257–260. RecSys ’10, Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1864708.1864761
  * [6] Hirschman, A.O.: The paternity of an index. The American Economic Review 54(5), 761–762 (1964)
  * [7] Hug, N.: NicolasHug/Surprise (2022), https://github.com/NicolasHug/Surprise
  * [8] Hui, B., Zhang, L., Zhou, X., Wen, X., Nian, Y.: Personalized recommendation system based on knowledge embedding and historical behavior. Applied Intelligence pp. 1–13 (2022)
  * [9] Jansen, B.J., Spink, A., Saracevic, T.: Real life, real users, and real needs: A study and analysis of user queries on the web. Information Processing & Management 36(2), 207–227 (2000). https://doi.org/10.1016/S0306-4573(99)00056-4
  * [10] Joseph, K., Jiang, H.: Content based news recommendation via shortest entity distance over knowledge graphs. In: Companion Proceedings of The 2019 World Wide Web Conference. pp. 690–699 (2019)
  * [11] Kaminskas, M., Bridge, D.: Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 7(1), 1–42 (2016)
  * [12] Koren, Y.: Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Transactions on Knowledge Discovery from Data 4(1), 1–24 (2010). https://doi.org/10.1145/1644873.1644874
  * [13] Kotkov, D., Wang, S., Veijalainen, J.: A survey of serendipity in recommender systems. Knowledge-Based Systems 111, 180–192 (2016)
  * [14] Kowald, D., Muellner, P., Zangerle, E., Bauer, C., Schedl, M., Lex, E.: Support the underground: Characteristics of beyond-mainstream music listeners. EPJ Data Science 10(1) (2021). https://doi.org/10.1140/epjds/s13688-021-00268-9
  * [15] Kowald, D., Schedl, M., Lex, E.: The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study. Advances in Information Retrieval 12036, 35–42 (2020). https://doi.org/10.1007/978-3-030-45442-5_5
  * [16] Raimond, Y., Abdallah, S.A., Sandler, M.B., Giasson, F.: The music ontology. In: Dixon, S., Bainbridge, D., Typke, R. (eds.) Proceedings of the 8th International Conference on Music Information Retrieval, ISMIR 2007, Vienna, Austria, September 23-27, 2007. pp. 417–422. Austrian Computer Society (2007), http://ismir2007.ismir.net/proceedings/ISMIR2007_p417_raimond.pdf
  * [17] Rossanez, A., da Silva Torres, R., dos Reis, J.C.: Characterizing complex network properties of knowledge graphs. In: Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KEOD. pp. 119–128. INSTICC, SciTePress (2023). https://doi.org/10.5220/0012257300003598
  * [18] Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th international conference on World Wide Web. pp. 285–295 (2001)
  * [19] Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: The adaptive web: methods and strategies of web personalization, pp. 291–324. Springer (2007)
  * [20] Schedl, M.: The LFM-1b Dataset for Music Retrieval and Recommendation. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. ACM (2016). https://doi.org/10.1145/2911996.2912004
  * [21] Schedl, M., Mayr, M., Knees, P.: Music Tower Blocks: Multi-Faceted Exploration Interface for Web-Scale Music Access. In: Proceedings of the 2020 International Conference on Multimedia Retrieval, pp. 388–392. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3372278.3391928
  * [22] Schoenfeld, M., Pfeffer, J.: Shortest path-based centrality metrics in attributed graphs with node-individual context constraints. Social Networks (2021). https://doi.org/10.1016/j.socnet.2021.10.004
  * [23] Silverstein, C., Marais, H., Henzinger, M., Moricz, M.: Analysis of a very large web search engine query log. ACM SIGIR Forum 33(1), 6–12 (1999). https://doi.org/10.1145/331403.331405
  * [24] Wang, X., He, X., Cao, Y., Liu, M., Chua, T.S.: Kgat: Knowledge graph attention network for recommendation. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 950–958 (2019)
  * [25] Wasserman, S., Faust, K., Urbana-Champaign), S.U.o.I.W.: Social Network Analysis: Methods and Applications. Cambridge University Press (1994)
  * [26] Zangerle, E., Pichl, M., Schedl, M.: User Models for Culture-Aware Music Recommendation: Fusing Acoustic and Cultural Cues. Transactions of the International Society for Music Information Retrieval 3(1), 1–16 (2020). https://doi.org/10.5334/tismir.37
  * [27] Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.Y.: Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM (2016). https://doi.org/10.1145/2939672.2939673