# Diachronic Analysis of German Parliamentary Proceedings: Ideological Shifts
through the Lens of Political Biases

Tobias Walter,1 Celina Kirschner,1 Steffen Eger,2 Goran Glavaš,1 Anne
Lauscher,1 Simone Paolo Ponzetto1 1Data and Web Science Group, University of
Mannheim, Mannheim, Germany
2Natural Language Learning Group, Technische Universität Darmstadt, Darmstadt,
Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{goran, anne<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We analyze bias in historical corpora as encoded in diachronic distributional
semantic models by focusing on two specific forms of bias, namely a political
(i.e., anti-communism) and racist (i.e., antisemitism) one. For this, we use a
new corpus of German parliamentary proceedings, DeuPARL, spanning the period
1867–2020. We complement this analysis of historical biases in diachronic word
embeddings with a novel measure of bias on the basis of term co-occurrences
and graph-based label propagation. The results of our bias measurements align
with commonly perceived historical trends of antisemitic and anti-communist
biases in German politics in different time periods, thus indicating the
viability of analyzing historical bias trends using semantic spaces induced
from historical corpora.

##  I Introduction

Recent years have seen much work on the topic of bias in data-driven
Artificial Intelligence [28] and Machine Learning [24]. In the case of Natural
Language Processing, researchers have investigated the bias encoded within
semantic spaces induced by both non-contextualized [5, 13] and contextualized
embeddings [20], and a variety of methods has been developed to debias such
embedding spaces in such a way as to make them fairer [4, 15, 22, _inter
alia_].

However, despite much interest on studying and mitigating bias in semantic
spaces, few works – [13, 33] being notable exceptions – have taken a
historical perspective and looked at ways to study bias in diachronic corpora
through the lens of distributional embedding models. In this paper, we create
a new historical corpus of parliamentary proceedings spanning three different
centuries and set to quantify different kinds of bias in the underlying
historical periods. Our DeuPARL corpus covers the protocols of both the German
Reichstag and the Bundestag from 1867 until 2020, thus spanning a large
timeline that covers many crucial modern and contemporary events (e.g., two
world wars): it consists of the digitized scans of the older German Reichstag
and the digitally-born newer German Bundestag parliamentary proceedings. In
this corpus, we focus on political and racist forms of bias, i.e., anti-
communism and antisemitism, respectively. We achieve this by building upon
previous work on quantifying the biases in textual corpora, that is: (1) we
devise a number of term-based bias specifications that provide us with an
operational definition of antisemitic and anti-communist bias; (2) we quantify
the degree of such biases using measures applied to word embeddings induced
from different time slices of our corpus; (3) we complement this study of bias
in dense semantic vectors with an analysis of bias in more ‘classic’ (i.e.,
sparse) distributional word representations using a label propagation
algorithm applied to word co-occurrence graphs. As a result, we are able to
provide a diachronic analysis of the bias trends in our historical corpus for
different kinds of biases and word representations on the basis of our
specifications.

The contributions of this work are the following ones:

  * •

We present a new historical corpus of parliamentary proceedings built from the
protocols of both German Reichstag and Bundestag between 1867 and 2020.

  * •

We induce a variety of distributional semantic spaces, both dense and sparse,
from different slices of our historical corpus and observe bias trends across
time for different kinds of bias (i.e., antisemitism and anti-communism).

  * •

We introduce a new measure of bias on the basis of term co-occurrences and
graph-based label propagation, which makes it possible to quantify bias in
sparse distributional spaces and thus in an arguably more interpretable way.

Our results are consistent with common historical interpretation of
development of bias in German history, i.e., we identify an increase in
antisemitism towards the NS period, with a peak in the Weimar Republic. This
indicates the viability of studying the evolution of bias from a historical
perspective on the basis of word embeddings, on associated tests, from
historical corpora. We make all code and data (including DeuPARL) available at
https://github.com/umanlp/crosstemporal_bias.

##  II DeuPARL: German Parliamentary Corpus

Our new historical corpus consisting of German parliamentary proceedings from
1867 until 2020, which we refer to as DeuPARL,111We make DeuPARL available at:
https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2889 is built from two
main sources.

TABLE I: Excerpt of a speech in a parliamentary session, from 1914. Left: OCR-scanned document. Right: Corrected version. Words containing OCR errors are underlined. Gröber, Abgeordneter: Meine Herren, der Gedanke | Gröber, Abgeordneter: Meine Herren, der Gedanke
---|---
des Herrn Abgeordneten Gothetn, der dem Beschluß deS | des Herrn Abgeordneten Gothein, der dem Beschluß des
hohen Hauses in der letzten Sitzung zu Grunde lag, ist | hohen Hauses in der letzten Sitzung zu Grunde lag, ist
meines Erachtens zweifellos richtig. Es ist nicht er- | meines Erachtens zweifellos richtig. Es ist nicht er-

Reichstagsprotokolle. For parliamentary speeches before 1945, we use the
German _Reichstagsprotokolle_ which are accessible at
https://www.reichstagsprotokolle.de/. The _Bayerische Staatsbibliothek_
distributes the digitized data upon request, arranged in three splits covering
the years 1867–1895 (Norddeutscher Bund / Zollparlamente), 1895–1918
(Kaiserreich), and 1918–1942 (Weimarer Republik / Nationalsozialismus),
respectively. These were OCR-scanned using the Abbyy FineReader
software.222https://pdf.abbyy.com/ To check data quality, we extracted a
sample of several thousand lines which we corrected by hand. In this sample,
about 23% of all lines contained one or more OCR errors. A sample of the data
is shown in Table I: we found the data to be of sufficiently high quality for
our purposes, not urgently necessitating further OCR post-correction steps
[11]. Overall, we extracted 5,446 individual protocols in the
Reichstagsprotokolle, i.e., stenographic documents corresponding to the
parliamentary sessions.

Bundestagsprotokolle. For parliamentary speeches after 1945, we use the
_Bundestagsprotokolle_ from https://www.bundestag.de/protokolle.333Note that,
for the time period of the Division of Germany between 1949 and 1990, the
Bundestagsprotokolle only cover the protocols of the Federal Republic of
Germany (also known as “West Germany”). We extracted 4,260 protocols in 19
legislative periods from 1949 to March 2020.

Preprocessing. We conducted preprocessing steps for DeuPARL (mostly affecting
the Reichstagsprotokolle) including:444We also aimed at historical spelling
normalization, but found it to lead to substantial amount of undesired text
modifications.

  * •

We fixed word segmentations at line breaks
(e.g., “Poli- tik” $\rightarrow$ “Politik”);

  * •

We lowercased and lemmatized all words (the latter using GermaLemma++ [29]) to
increase statistical support for all subsequent models built on top of our
data;

  * •

We resolved cases such as “Luft- und Raumfahrt” into
“Luftfahrt und Raumfahrt” and fixed erroneous word segmentations (e.g., where
numbers and words were merged)

  * •

We took care to keep only text from the individual parliamentary sessions, not
considering various attachments to the protocols such as lists of names. To
this end, we used regular expressions to extract the text within boundaries
such as “Die Sitzung ist eröffnet” (“ _the session is openend_ ”) and “Schluss
der Sitzung _hh_ Uhr _mm_ Minute(n)” (“ _end of session hh:mm_ ”)

Temporal splits. We defined our own temporal splits for a historical analysis
of bias, taking historically accepted time periods in modern German history as
reference. These are: the Kaiserreich I (KR1, 1867–1890), Kaiserreich II (KR2,
1890–1918), Weimarer Republik (WR, 1918–1933), Nationalsozialismus (NS,
1933–1945). In the case of the Bundesrepublik (1949–2020), we divided the data
into slices of contiguous time periods in which one of the two major German
parties (Volksparteien, i.e., liberal-conservative CDU or social-democratic
SPD) was in charge. Table II shows statistics of our data. For the
Bundestagsprotokolle, slices range from between $\sim$30 million to $\sim$66
million tokens. For the Reichstagsprotokolle, the NS era is severely
restricted in size, congruent with the observation that the parliamentary
processes during the NS time were largely
abolished.555https://www.bundestag.de/parlament/geschichte/parlamentarismus/drittes_reich

Party distribution. The number of parties in our historical time periods with
active speakers ranges from 2 (during the NS period, including SPD and NSDAP)
to 10 (during KR2).666In the original data, party affiliation is attached to
the speaker names. Of course, the party distribution may decisively affect our
results, i.e., distribution of biases found. In this context, it is worth
noting that certain parties were banned during the time span of our data. For
example, the right-wing party NSDAP was forbidden after 1945. The communist
party KPD was forbidden in 1956. Our data thus contains communist parties only
during the time period of WR and CDU1.

TABLE II: Labels for time periods, range of time periods, and sizes of the corresponding data slices. Period | Years | Tokens
---|---|---
KR1 | 1867-1890 | 40,585,912
KR2 | 1890-1918 | 77,175,976
WR | 1918-1933 | 35,838,922
NS | 1933-1942 | 230,018
CDU1 | 1949-1969 | 43,337,027
SPD1 | 1969-1982 | 35,208,879
CDU2 | 1982-1998 | 55,451,433
SPD2 | 1998-2005 | 28,614,189
CDU3 | 2005-2020 | 66,192,033

Figure 1: Word clouds showing relative frequency of words in text data for our
nine periods: KR1, KR2, WR, NS (top, from left to right) and CDU1, SPD1, CDU2,
SPD2, CDU3 (bottom).

Corpus analysis. In Figure 1, we show so-called ‘word clouds’ for each slice
in our timeline (1867–2020).777We use https://github.com/amueller/word_cloud
for drawing. The word clouds graphically illustrate the frequency of words in
text data, assigning more frequent words larger font sizes (we removed stop
words and only consider words with a minimum frequency of 50 occurrences). As
expected, vocabulary relevant for parliamentary debates is prominent in our
textual material, e.g., ‘Abgeordneter’ (_representative_), ‘Herr’ (_mr._),
‘Antrag’ (_petition_). Names for political parties (‘SPD’, ‘FDP’, ‘CDU/CSU’)
become more frequent in more recent time periods. Interestingly, the word
‘deutsch’ (_German_) becomes more prominent from KR1 to KR2 to WR and then
peaks in the NS period, where the word ‘Volk’ (ideologically-loaded _people_
as nation) along with ‘Führer’ (_leader_ , used as title for Nazi dictator
Adolf Hitler) and ‘Berlin’ also become very prominent. The trend for ‘deutsch’
then reverses with the onset of the Bundesrepublik, i.e., after 1949, thus
intuitively aligning with general understatement of national and cultural
identity in Germany after World War II.

Figure 2: Histogram summarizing the number of occurrence of the German word
‘Jude’ (_jew_) per 100k tokens across different time slices of our corpus.
Figure 3:  The semantic representations of the word ‘Judentum’ (denoted by the
numbers in chronological order, i.e., 1:KR1, 2:KR2, 3: WR) and their 15
nearest neighbors in the semantic space of each Reichstag period (red: KR1,
orange:KR2, gray:WR).

A limitation of word clouds is that they can highlight only a few frequent
words from the corpus. To also analyze trends for lower frequency and highly
relevant words, e.g., ‘Jude’ (_jew_) – due to its central relevance for one of
the two kinds of bias that we focus on (i.e., antisemitism) – we show a
temporal curve in Figure 2. Like ‘deutsch’, ‘Jude’ peaks in the NS period,
with a frequency of almost 6 per 100k tokens, then sharply declines in the
post war period to regain prominence again after 1982, though not on the same
level as before 1949 (disregarding KR1). Finally, Figure 3 shows semantic
shift [10] of the word ‘Judentum’ (_Judaism_) in our data. The plot is
obtained by semantically embedding words from different time periods in a
shared vector space using the model of [6] and then tracking in the semantic
space how the word changes its fine-granular meaning over time periods using
t-SNE [34] to visualize the embeddings (cf. also HistWords [17]). In this
case, for instance, we see how ‘Judentum’ seemingly takes on (putatively
negative) associations of ‘Bolschewismus’ (_Bolshevism_) and
‘Weltkapitalismus’ (_world capitalism_) in the time of the Weimar Republic.

##  III Capturing Political Biases: Methodology

Figure 4: Overview of our bias evaluation pipeline.

Our bias evaluation pipeline is illustrated in Figure 4. The methodology
consists of three main steps. We first slice DeuPARL according to the
dimension of interest: in this paper, our focus is on politically well-defined
historical periods. We then induce computational text representations, namely
(1) dense distributional word vector spaces (i.e., word embeddings) and (2)
word co-occurrence graphs, for each of the corpus slices. Finally, given
human-designed bias specifications for antisemitism and anti-communism, we
compute bias scores with several measures on the basis of either word
embedding spaces or word coocurrence graphs induced in the previous step. We
next describe our bias specifications, followed by the detailed description of
our word embedding-based and co-occurrence graph-based approaches for
quantifying the bias effects across different time periods.

###  III-A Bias Specifications

In order to quantify the degree of presence of some societal bias (e.g.,
racial, religious, political, or gender bias), one must first specify (i.e.,
formalize) that bias. In this work, we follow the established body of work on
measuring biases in word embedding spaces [5, 9, 22] and adopt the so-called
explicit bias specification $B_{E}=(T_{1},T_{2},A_{1},A_{2})$, consisting of
two target term sets ($T_{1}$ and $T_{2}$), between which a significantly
different association with respect to two sets of attribute terms ($A_{1}$ and
$A_{2}$) is expected to exist. In this work, we investigate two dimensions of
political biases, which arguably played the most important role in (most
recent) German history: (1) antisemitic bias, i.e., the bias between the
dominant Christian group ($T_{1}$) and the targeted Jewish group
($T_{2}$),888We decided to define Judaism along a religious dimension, rather
than along a racial dimension, as often perceived e.g. during the NS times.
Thus, we contrasted Jews to Christians rather than Germans or Aryans. We
speculate that this could lead to an underestimation of antisemitic bias. and
(2) anti-communist bias, as the bias between the mainstream political
conservatism ($T_{1}$) and the targeted communism ($T_{2}$). We expect the
diachronic analysis of the degree of presence (or lack thereof) of these two
bias types to point to ideological shifts in German political history. In what
follows, we describe concrete bias specifications for each of the two
dimensions of analysis. Table III provides an overview on all bias
specifications employed in our study.

Antisemitism. We quantify the presence of antisemitism as the difference in
association between ‘Christian’ terms ($T_{1}$) and ‘Jewish ’ terms ($T_{2}$)
with respect to some attribute term sets $A_{1}$ (e.g., terms of positive
sentiment like love) and $A_{2}$ (e.g., negative sentiment terms like hate).
We obtain the terms for $T_{1}$ (i.e., Christian terms) by translating the
Christian bias terms introduced by [13] from English to German (e.g., church
$\rightarrow$ Kirche). We directly translated each term from the original
English set and then judged the suitability of the translation for our use-
case of diachronic bias analyses, resulting in minor modifications with
respect to the original set. For instance, we discard the term ‘Kreuz’ (cross)
due to its participation in the collocation ‘Eisernes Kreuz’, one of the
highest Prussian military accolades. Besides, we added to $T_{1}$ the
following terms: ‘Pfarrer’ (pastor), ‘Ostern’ (_Easter_) and ‘Bibel’
(_Bible_). We devised the target term set for Judaism from scratch, starting
from central terms that clearly denote this religion, e.g., ‘jüdisch’ (Jewish)
and ‘Jude’ (Jew).

We observe the differences between Christian and Jewish target sets through
seven different views, each of which is defined with one pair of attribute
sets ($A_{1}$, $A_{2}$). Views b–g) correspond to five bias lines identified
by [33]:

  * a)

Sentiment. We rely on the widely-used attribute set of pleasant ($A_{1}$) and
unpleasant ($A_{2}$) terms introduced by [5] and translated to German by [21];

  * b)

Religious. Terms capturing a religious view of what is “good” and “bad”, e.g.,
‘Gläubige’ (believers, $A_{1}$);

  * c)

Economic. Terms capturing the “jew as greedy” stereotype;

  * d)

Patriotic. Nationally-loaded terms, e.g., ‘patriotisch’ (_patriotic_ ,
$A_{1}$), ‘fremd’ (foreign, $A_{2}$);

  * e)

Racial. Racist-related terminology capturing the idea of superior and inferior
races;

  * f)

Conspirational. Terms related to “jew world conspiracy’ theories, e.g.,
‘loyal’ ($A_{1}$), betrügerisch’ (deceitful, $A_{2}$);

  * g)

Ethic. Includes ethically-loaded terms describing what is “good” and “bad”,
e.g., ‘tugendhaft’ (virtuous, $A_{1}$).

For each of these views, terms in $A_{2}$ have been recognized by [33] to be
commonly used in constructions expressing negative stereotypes about Jews: for
example, the negatively biased view of greedy and economically influential
Jews symbolized by the Rothschilds (economic); the Jews involved in political
machinations and secret plots symbolized by George Soros (conspiratorial); or
the Nazi myth of Jews as distinct and inferior race (racial).

Anti-communism. In order to capture anti-communist bias, we devise two target
term sets representing mainstream political conservatism ($T_{1}$) and the
targeted communism ($T_{2}$). To this end, we resort to political and
historical literature [32]. For both bias specifications, we account for
linguistic changes across the analyzed range of $153$ years by adapting the
bias specifications after $1949$ (starting with the German Bundestag). Here,
we focus on three views:

  * a)

Sentiment. We employ the same set of terms capturing general sentiment as
above;

  * b)

Politics. Terms relating to a political view of “good” and “bad”, e.g.,
‘wirksam’ (effective, $A_{1}$)

  * c)

Propaganda. Terms capturing anti-communist propaganda.

TABLE III: Bias specifications for representing antisemitism and anti-communism employed in our study. Superscript - indicates terms which we remove and superscript + indicates terms which we add for tests on historical slices starting from $1949$ (German Bundestag). Bias Type/View |  Specification terms
---|---
Antisemitism |  $T_{1}$ (Christentum): Taufe, Katholizismus, Christentum, evangelisch, Evangelium, Jesus, Christ, christlich, katholisch, Kirche, Pfarrer, Ostern, Bibel
$T_{2}$ (Judentum): Rabbiner, Synagoge, koscher, Talmud, orthodox, Judentum,
Jude, jüdisch, Mose, mosaisch, Israel, Abraham, zionistisch, israelitisch,
Israelis
View: Sentiment |  $A_{1}$ (+): streicheln, Freiheit, Gesundheit, Liebe, Frieden, Freude, Freund, Himmel, loyal, Vergnügen, Diamant, sanft, ehrlich, glücklich, Regenbogen, Geschenk, Ehre, Wunder, Sonnenaufgang, Familie, Lachen, Paradies, Ferien
$A_{2}$ (-): Missbrauch, Absturz, Schmutz, Mord, Krankheit, Tod, Trauer,
vergiften, stinken, Angriff, Katastrophe, Haß, verschmutzen, Tragödie,
Scheidung, Gefängnis, Armut, häßlich, Krebs, töten, faul, erbrechen, Qual
2-2 View: Patriotism |  $A_{1}$ (+): patriotisch, vaterlandsliebe, volksbewußtsein, volksgeist, germanische, deutschnational, nationalbewußstsein, vaterländisch, reichstreu, nationalgesinnt, nationalstolz, königstreu, staatstreu, nationalgefühl, Volkstum+, Patriotismus+, Patriot+
$A_{2}$ (-): nichtdeutsch, fremdländisch, fremd, undeutsch, vaterlandslos,
reichsfeind, landesverräter reichsfeindlich, unpatriotisch, antideutsch,
deutschfeindlich, umstürzler, staatsfeindlich, dissident, heimatlos,
separatistisch+, staatenlos+
2-2 View: Economy |  $A_{1}$ (+): geben, Großzügigkeit, großzügig, selbstlos, genügsam, Großmut, uneigennützig, sparsam, Proletariat, Armut, Industriearbeiter
$A_{2}$ (-): nehmen, Gier, gierig, egoistisch, habgierig, Habsucht,
eigennützig, verschwenderisch, Bourgeoisie, Wohlstand, Bankier, Wucher+
2-2 View: Conspiracy |  $A_{1}$ (+): loyal, Kamerad, Ehrlichkeit, ersichtlich, aufrichtig, vertrauenswürdig, wahr, ehrlich, unschuldig, freundschaftlich, hell, zugänglich, machtlos, ohnmacht, untertan
$A_{2}$ (-): illoyal, Spitzel, Verrat, geheim, hinterlistig, betrügerisch,
unwahr, zweifelhaft, Verbrecher, bedrohlich, dunkel, geheimnis, einflussreich,
weltmacht, herrschaft, verschwoerung
2-2 View: Religion |  $A_{1}$ (+): Gläubige, geistlich, Engel, heilig, fromm, geheiligt, göttlich, ehrwürdig, treu, gläubig, religiös
$A_{2}$ (-): Atheist, weltlich, Teufel, irdisch, atheistisch, heidnisch,
gottlos, verflucht, untreu, ungläubig, irreligiös, Gotteslästerung
2-2 View: Race |  $A_{1}$ (+): normal, überlegenheit, gleichheit, angenehm, freundlich, ehrenwert, sympathie, akzeptiert, besser, national, rein, überlegen, sauber, ehrenhaft
$A_{2}$ (-): seltsam, unterlegenheit, ungleichheit, unangenehm, boshaft,
schändlich, hass, abgelehnt, schlechter, fremdländisch, unrein, unterlegen,
schmutzig, verseucht, schädlich, niederträchtig
2-2 View: Ethics |  $A_{1}$ (+): bescheiden, sittlich, anständig, tugendhaft, charakterfest, würdig, treu, moralisch, ehrlich, gesittet, gewissenhaft, vorbildlich
$A_{2}$ (-): unbescheiden, unsittlich, unanständig, lüstern, korrupt,
unwürdig, untreu, unmoralisch, unehrlich, verdorben, gewissenlos, barbarisch
Anti-communism |  $T_{1}$ (Conservatism): Konservatismus, Tradition, Geschichte, Christentum-, Adel-, Monarchie-, Mittelalter-, Stände-, Werte, Moral, König-, Kaiser-, Hierarchie, Identität, Kontinuität, Sicherheit, Grundbesitz-, Autorität, Legitimität, Ordnung, Religion-, Kirche-, Erhaltung, Treue-, Tugend-, Bräuche, Sitten, Bewahrung, Gottesgnadentum-, Ständeordnung-, Restauration-, Bürger+, Bürgertum+, Regierung+, Wertordnung+, Bürgerlichkeit+, Stabilität+, Wohlstand+
$T_{2}$ (Communism): Sozialismus-, Kommunismus, Proletariat, Arbeiter-,
Klassengesellschaft, Klasse-, Revolution, Aufklärung-, Gemeinschaft-,
Gerechtigkeit-, Armut-, Kapital, Gleichheit-, Chancen-, Freiheit-,
Arbeiterklasse-, Solidarität-, Partei-, Verstaatlichung, Gewerkschaft-, Marx,
Engels, Vergesellschaftung, Gemeineigentum, Widerstand,
Kollektivierung,Arbeiterbewegung-, Aufstand-, Lenin+, Planwirtschaft+,
Klassenkampf+, Proletariat+, Revolution+, Produktionsmittel+, Diktatur+,
Bolschewiki+ ,Oktoberrevolution+,Räte+, Sowjetunion+
View: Sentiment |  $A_{1}$ (+): streicheln, Freiheit, Gesundheit, Liebe, Frieden, Freude, Freund, Himmel, loyal, Vergnügen, Diamant, sanft, ehrlich, glücklich, Regenbogen, Geschenk, Ehre, Wunder, Sonnenaufgang, Familie, Lachen, Paradies, Ferien
$A_{2}$ (-): Missbrauch, Absturz, Schmutz, Mord, Krankheit, Tod, Trauer,
vergiften, stinken, Angriff, Katastrophe, Haß, verschmutzen, Tragödie,
Scheidung, Gefängnis, Armut, häßlich, Krebs, töten, faul, erbrechen, Qual
2-2 View: Politics |  $A_{1}$ (+): sozial, progressiv, gemeinschaftlich, gemeinsam, zivilisiert, bewährt, wirksam, etabliert, demokratisch, hoch, möglich, fortschrittlich, gemäßigt, machbar, realistisch, früh, kontinuierlich, legitim, verlässlich, aufrichtig, intellektuell, sicher, Sicherheit, Fortschritt, pragmatisch, Vertrauen, Wandel+, sachlich+, Gewinn+, fähig+
$A_{2}$ (-): unsozial, radikal, extrem, gefährlich, gefährdend, niedrig,
nieder, unmöglich, undemokratisch, unrealistisch, spät, unlegitim, Gefahr,
unehrlich, unaufrichtig, unintellektuell, unsicher, schwer, schwierig,
Misstrauen, Stillstand+, Skandal+, skandalös+, Zukunft+, unsachlich+,
Verlust+, unfähig+
2-2 View: Propaganda |  $A_{1}$ (+): Kamerad-, Kameraden-, Kameradschaft-, kameradschaftlich-, Vaterland-, Patriot-, Ehre, ehrlich, Einsatz, Untertan-, rein-, wir, Heimat-, deutsch, Deutschland-, Truppe-, Nationalstolz-, patriotisch, Volk, Befreiung-, Front-, Wahrheit, wahr, aufrichtig+, gemeinschaftlich+, Wertegemeinschaft+, Mitte+, Frieden+, Partnerschaft+, Integration+, Wandel+
$A_{2}$ (-): Sabotage-, Saboteur-, Betrüger, Betrug, Gauner-, Schwindel-,
Schwindler-, Parasit-, Volksfeind-, Reichsfeind-, undeutsch, unpatriotisch,
reichsfeindlich, Volksverräter-, Spion-, Bolschewist-, fremd, unrein-,
Kommunist, Spitzel-, anders, Lüge, Lügner, Dissident, Feind, Diktatur,
Verschwörung-, verschwörerisch-, unehrlich+, feindlich+, Schmarotzer+, Elite+,
Kriminelle+, kriminell+

###  III-B Measuring Bias in Semantic Spaces.

Most existing measures for capturing bias in text corpora (e.g., [4, 5, 13,
15, 9, 21, 22], _inter alia_), given an explicit bias specification, operate
on word embeddings, namely dense vector representations of meaning in context
[25, 30, 3]. In this work, we adopt two of the arguably most established bias
measures based on word embeddings: Word Embedding Association Test (WEAT) [5]
and Embedding Coherence Test (ECT) [9]. Training reliable word embeddings,
however, requires substantial amounts of text. Given that some of the temporal
slices in our DeuPARL corpus are of fairly limited size (see Table II),
questions remain on whether the word embeddings induced from those slices can
be sufficiently reliable. Because of this, we couple the embedding-based bias
measures with a novel bias measure based on term co-occurrences and graph-
based label propagation, which we dub Co-occurrence Bias Test (CBT).

Quantifying Bias in Word Embeddings. Word embeddings are dense numeric vector
representations of words that aim to capture word meaning insofar that words
with similar meaning get assigned vectors that are close to each other in the
vector space. While there exist several different well-known algorithms for
inducing word embeddings from a given corpus (see, e.g., [25, 30, 3]), they
all rely on a distributional hypothesis [18] and, one way or the other,
exploit the information about local word co-occurrences to derive dense
semantic vectors of words, i.e., word embeddings.

_Word Embedding Association Test (WEAT)._ Introduced by [5], the WEAT test is
an adaptation of the Implicit Association Test (IAT) [26, 27]. Whereas IAT
measures biases based on response times of human subjects to provided stimuli,
WEAT quantifies the biases using semantic similarities between word embeddings
of the same stimuli (i.e., terms in our bias specifications, see Table III).
Given some word embedding space and an explicit bias specification
$B_{E}$=$(T_{1},T_{2},A_{1},A_{2})$, the WEAT test statistic is computed as
the differential of the associativity between $T_{1}$ and $T_{2}$ w.r.t.
$A_{1}$ and $A_{2}$, stemming from the mean similarity of their terms with
terms in the attribute sets $A_{1}$ and $A_{2}$, respectively:

$s(B_{E})=\sum_{t_{1}\in T_{1}}{s(t_{1},A_{1},A_{2})}-\sum_{t_{2}\in
T_{2}}{s(t_{2},A_{1},A_{2})}\,$

The association score $s$ for an individual target term $t\in T_{i}$ is
obtained as:

$s(t,A_{1},A_{2})=\frac{1}{|A_{1}|}\sum_{a_{1}\in
A_{1}}{\cos(\mathbf{t},\mathbf{a_{1}})}-\frac{1}{|A_{2}|}\sum_{a_{2}\in
A_{2}}{\cos(\mathbf{t},\mathbf{a_{2}})}$

where $\mathbf{t}$, $\mathbf{a_{1}}$, $\mathbf{a_{2}}$ are word embeddings of
terms $t$, $a_{1}$, and $a_{2}$, respectively and
$\cos{(\mathbf{x},\mathbf{y})}\in[-1,1]$ is the cosine of the angle enclosed
by the vectors $\mathbf{x}$ and $\mathbf{y}$. We note that $s(B_{E})$ is
large, for example, when terms in $T_{1}$ are positively associated with
attributes in $A_{1}$ and negatively with attributes in $A_{2}$ _and_ vice
versa for $T_{2}$. In this case, there is a bias for $T_{1}$ to be associated
with $A_{1}$ and $T_{2}$ to be associated with $A_{2}$.

The significance of the above statistic is computed by comparing $s(B_{E})$
with the scores $s(B^{*}_{E})$ obtained for permutations of $B_{E}$,
$B^{*}_{E}=(T^{*}_{1},T^{*}_{2},A_{1},A_{2})$, where $T^{*}_{1}$ and
$T^{*}_{2}$ are equally sized partitions of $T_{1}\cup T_{2}$. The $p$-value
of the test is the probability of $s(B^{*}_{E})>s(B_{E})$. Finally, the bias
effect size, i.e., the “amount” of bias, is computed as the normalized measure
of separation between association distributions:

$\frac{\mu\left(\\{s(t_{1},A_{1},A_{2})\\}_{t_{1}\in
T_{1}}\right)-\mu\left(\\{s(t_{2},A_{1},A_{2})\\}_{t_{2}\in
T_{2}}\right)}{\sigma\left(\\{s(t,A_{1},A_{2})\\}_{t\in T_{1}\cup
T_{2}}\right)}$

with $\mu$ as the mean value and $\sigma$ as the standard deviation.

_Embedding Coherence Test (ECT)._ The ECT test, originally proposed by [9] and
later adjusted by [22] quantifies the amount of explicit bias by comparing
vectors of target sets $T_{1}$ and $T_{2}$, obtained by averaging the
embeddings of their constituent terms, with the vectors from a single
attribute set $A=A_{1}\cup A_{2}$. We first compute the embeddings for target
sets $T_{1}$ and $T_{2}$ by averaging the vectors of their terms:

$\mathbf{\mu}_{1}=\frac{1}{|T_{1}|}\sum_{t_{1}\in
T_{1}}{\mathbf{t}_{1}};\hskip 5.0pt\hskip
5.0pt\mathbf{\mu}_{2}=\frac{1}{|T_{2}|}\sum_{t_{2}\in T_{2}}{\mathbf{t}_{2}}.$

Next, for both $\mathbf{\mu}_{1}$ and $\mathbf{\mu}_{2}$ it computes the
(cosine) similarities with vectors of all attribute terms $a\in A$:

$\mathbf{s}_{1}=\left[\cos(\mathbf{\mu}_{1},\mathbf{a}_{i})\right]^{|A|}_{i=1};\hskip
5.0pt\mathbf{s}_{2}=\left[\cos(\mathbf{\mu}_{2},\mathbf{a}_{i})\right]^{|A|}_{i=1}.$

The two resulting vectors of similarity scores, $\mathbf{s}_{1}$ (for $T_{1}$)
and $\mathbf{s}_{2}$ (for $T_{2}$) are used to obtain the final ECT score: the
Spearman’s correlation between the rank orders of $\mathbf{s}_{1}$ and
$\mathbf{s}_{2}$. Unlike for WEAT where a larger bias effect implies larger
bias, the higher the ECT correlation, the lower the bias.

Measuring Bias with Co-occurrence Graphs. Both ECT and WEAT crucially depend
on the quality of the semantic information encoded in the underlying word
embedding space, which typically require large text corpora. We now propose a
novel bias measure based on word co-occurrence graphs and graph-based label
propagation, specifically designed to be applicable to small corpora as well.

We first compute scores between terms in the corpus indicating the level of
their lexical association, based on their co-occurrence frequency. Concretely,
we adopt Positive Pointwise Mutual Information (PPMI) as the measure of
lexical association between terms. For a pair of words $(w_{1},w_{2})$,
pointwise mutual information (PMI) is computed as the probability of joint
occurrence $P(w_{1},w_{2})$ of $w_{1}$ and $w_{2}$ (in a coocurrence window of
fixed size), normalized with the product of the probabilities of individual
terms, $P(w_{1})$ and $P(w_{2})$:

$\displaystyle\text{PMI}=\log\frac{P(w_{1},w_{2})}{P(w_{1})\cdot P(w_{2})}$

A PMI score of $0$ (i.e., probability ratio of $1$) means two words appear
together exactly as frequently as one would expect from their individual
frequencies. Scores lower than zero imply a reliable lack of association
between terms. This is why PPMI replaces negative PMI scores with $0$, i.e.,
$\text{PPMI}=\max\\{0,\text{PMI}\\}$.

_Co-occurrence Bias Test (CBT)._ The PPMI scores computed between all pairs of
terms effectively define an undirected weighted graph which can then be used
for semi-supervised label propagation of scores – from a small subset of nodes
for which such scores are known to all other (unlabeled) nodes in the graph.
We assign the labels to the nodes corresponding to the terms of the two
attribute lists: the terms from the positive attribute list $A_{1}$ are
assigned the score $1$, whereas the terms from the negative attribute lists
are labeled with $0$. We next employ the semi-supervised graph-based label
propagation algorithm named Harmonic Function Label Propagation (HFLP) [37,
38, 14] to induce the scores for all other nodes (i.e., terms) in the PPMI
graph. Let $\mathbf{W}$ be the weighted adjacency matrix of the PPMI graph
with $w_{ij}$ as the PPMI score between the $i$-th and $j$-th vocabulary term,
and let $\mathbf{D}$ be the corresponding diagonal degree matrix with entries
$\mathbf{D}_{ii}=\sum_{j}w_{ij}$ and $\mathbf{D}_{ij}=0$ for $i\neq j$.
Assuming we index all labeled nodes (i.e., nodes corresponding to terms from
$A_{1}$ and $A_{2}$) in $W$ before all unlabeled nodes, the Laplacian of the
graph, i.e., $\mathbf{\Delta}=\mathbf{W}-\mathbf{D}$, can then be partitioned
as:

$\mathbf{\Delta}=\begin{pmatrix}\mathbf{\Delta_{ll}}&\mathbf{\Delta_{lu}}\\\
\mathbf{\Delta_{ul}}&\mathbf{\Delta_{uu}}\end{pmatrix}$

Let $\mathbf{f}_{l}$ be the binary vector of labels of labeled nodes (i.e.,
terms from $A_{1}$ and $A_{2}$). HFLP then offers a closed form solution for
the scores of unlabeled nodes:

$\mathbf{f}_{u}=-\mathbf{\Delta}_{uu}^{-1}\mathbf{\Delta}_{ul}\mathbf{f}_{l}.$

The scores $f\in\mathbf{f}_{u}$ assigned to unlabeled nodes will then be in
between $0$ and $1$. We are now interested in the scores assigned to the terms
from the two target lists $T_{1}$ and $T_{2}$, respectively. Concretely, we
are interested in whether the mean of the scores for terms in $T_{1}$
statistically significantly differs from the mean of the scores assigned to
terms in $T_{2}$: if so, then we are observing a significant bias effect with
respect to the attribute sets $A_{1}$ and $A_{2}$. To this end, we apply the
Student’s (unpaired, two-tailed) $t$-test and interpret the value of the
$t$-statistic directly as the size bias effect:

$t(T_{1},T_{2})=\frac{\mathbf{\mu}_{T_{1}}-\mathbf{\mu}_{T_{2}}}{s\cdot\sqrt{\frac{1}{|T_{1}|}+\frac{1}{|T_{2}|}}}$

with $\mathbf{\mu}(T)$ as the mean score that HFLP assigned to the terms in
$T$ and $s$ as the estimator of the pooled standard deviation of the two sets
of target scores:

$s=\sqrt{\frac{(|T_{1}|-1)\cdot s^{2}_{T_{1}}+(|T_{2}|-1)\cdot
s^{2}_{T_{2}}}{|T_{1}|+|T_{2}|-2}}$

where $s^{2}_{T_{1}}$ and $s^{2}_{T_{2}}$ are the variances of the scores
induced with HFLP for terms in $T_{1}$ and $T_{2}$, respectively.

##  IV Experiments

We present our experiments on measuring bias in our DeuPARL corpus of German
parliamentary proceedings.

###  IV-A Experimental Setup

For the embedding-based approach, we train Word2Vec CBOW models [25] on each
historical slice independently. However, here, we exclude the NS slice due to
its small size. We employ the _gensim_ framework for inducing the embeddings
with an embedding size of $200$ and the window size as well as the minimum
count for unigrams set to $5$. For computing the PPMI matrices, we set the
window size to $5$ and the minimum term count to $10$ with the exception of
the NS slice for which we set the threshold to $2$.

###  IV-B Results

The results for our embedding-based and the graph-based bias measurement
approaches are depicted in Figures 5 and Figures 6, respectively. In the
following, we first focus on antisemitic bias and then discuss the results for
the anti-communist bias.

(a) Aggregated WEAT scores (antisemitism).

(b) WEAT scores (antisemitism).

(c) Aggregated ECT scores (antisemitism).

(d) Aggregated WEAT scores (anti-communism).

(e) WEAT scores (anti-communism).

(f) Aggregated ECT scores (anti-communism).

Figure 5: Results of the embedding-based approach. WEAT: higher scores means
lower bias; ECT: higher scores, lower bias.

Antisemitic Bias. Figure 5(a) shows the mean and $95\%$ confidence interval
aggregated across the WEAT effect sizes for all bias views. By our
specification, $T_{1}$ refers to the Christian group, $T_{2}$ to the Jewish
group, $A_{1}$ to positively connotated words and $A_{2}$ to negatively
connotated words. Thus, a large value of WEAT can be interpreted as
antisemitic bias. As it can be seen, antisemitism measured by WEAT
continuously rises starting from KR1 (Kaiserreich 1, $1867-1890$) over KR2
(Kaiserreich 2, $1890-1918$) until it reaches its peak in WR ($1918-1942$).
Afterwards, except for a slight increase in CDU2 ($1982-1998$), the measured
bias continuously drops and falls under the original level from KR1. Although
there are slight differences, overall trends measured by ECT are similar
(lower correlation indicates higher bias): the global minimum of the ECT mean
can be found in the historical slice from the Weimar Republik ($1918-1942$),
and the estimated bias from CDU2 ($1982-1998$) is smaller than the estimates
obtained between KR1 and WR. The different bias views draw an interesting
picture: for instance, while patriotic bias seems to drop, the bias measured
towards attributes reflecting sentiment appears to stay consistent, even in
the period of the Bundesrepublik. A reason for this is the fact that the
discourse in the German parliament shifted towards discussing Germany’s
responsibility for the Holocaust and the need for remembrance, which is _per
se_ not antisemitically biased, but carries a negative sentiment. Generally,
the results obtained using the graph-based approach (CBT), shown in Figure
6(b), are less significant but confirm the observations obtained using the
embedding-based approach.

Anti-communist Bias. The mean of the aggregated WEAT scores for anti-communist
bias obtained with different views is starting from a local optimum in KR1
($1867-1890$), and then decreasing until they peak in CDU1 ($1949-1969$, see
Figure 5(d)). Interestingly, compared to this finding, the ECT scores depicted
in Figure 5(f)) show a different picture, while the overall trend remains
similar: generally, the scores seem to vary more heavily and, measured with
ECT (all scores are significant at $\alpha<0.05$), the bias seems to steadily
rise until SPD1, decrease a bit and then increase again to CDU3 ($2005-2020$).
Intuitively, though the results are not entirely conclusive, all of these
peaks as well as the shared trend of the increased bias over the years make
sense: (i) We speculate that the peak in KR1 reflects the prominence of anti-
socialist laws from that period; (ii) After the end of the monarchy in
Germany, the communists formally founded the communist party Kommunistische
Partei Deutschlands (KPD) and the SPD continued to establish itself as a
moderate-leftist party, entering into the Weimarer Coalition. Until the $1932$
election (6th legislative period in WR), the SPD remained the strongest
political force in the Reichstag. Generally, the SPD gained a reputation for
pursuing compromises between the different ideological factions while the KPD
strongly advocated for abolishing the parliamentary system and even
constituted an anti-parliamentary alliance with the NSDAP in the late WR years
[16]. In a qualitative analysis, we found evidence indicating the moderate
left’s desire to distance themselves from extremist communism. For instance,
they denounce the bad reflection of the communist tactics on honest desire to
improve conditions for the working class; (ii) after World War II (CDU1), the
establishment of the Iron Curtain led to a clash of Western and Eastern
ideologies [8]. In our study, we only include protocols from the
Bundesrepublik, which reflect just the Western perspective. (iii) The reason
for the peak in CDU3 may be attributed to the major parties’ general stance
against populist and anti-democratic forces, which have been gaining momentum
in recent years [19]. The different views indicate that, initially (KR1), the
bias with respect to the sentiment attribute sets is higher, while later on,
and especially in the last periods, it gets surpassed by the bias with respect
to politics and propaganda terms.

Consistency across Measures. As already noticed by Lauscher et al. [22],
different bias measurement approaches capture different aspects of bias in the
distributional space and therefore sometimes deviate in terms of the amount of
bias they reveal. In line with their findings, we note some inconsistencies:
for instance, the amount of bias measured with CBT is generally lower than the
one estimated with WEAT. This is not surprising, since WEAT has been shown to
overestimate bias [12]. We therefore acknowledge the importance of running
several measures in parallel in order to capture the most pervasive trends.

(a) CBT scores (antisemitism).

(b) CBT scores (anti-communism).

Figure 6: Results for the graph-based approach.

##  V Related Work

Measuring Bias in Semantic Spaces. Bolukbasi et al. [4] were the first to
demonstrate that word embeddings induced from human-created corpora encode
stereotypical human biases. More specifically, they showed that many
distributional vector spaces allow for algebraically building biased
analogies, such as the famous example _“man is to computer programmer as woman
is to homemaker”_ , an example of a sexist analogy. Following up on this,
Caliskan et al. [5] presented WEAT, inspired from the Implicit Association
Test [26] from psychology, which measures biases in terms of associative
difference in similarity between term sets. The WEAT bias specifications were
later translated to several languages, e.g., by Chaloner and Maldonado [7] and
Lauscher et al. [21, 23]. We use the German sentiment attribute sets from the
latter. A similar group of authors assembled a larger framework of bias
evaluation measures [22] and proposed the notion of implicit and explicit bias
specifications. This framework includes, for instance, the Embedding Coherence
Test [9] and the Implicit Bias Test [15]. We adopt their notion of bias.
Follow up research also proposed evaluation methods and resources for
contextualized embedding methods [36, 20, 1, _inter alia_].

Natural Language Processing for the Analysis of Historical Corpora. The
studies closest to ours were presented by Garg et al. [13] and Tripodi et al.
[33]. Similar to us, Garg et al. quantify bias in historical corpora employing
word vector spaces as proxies. However, they analyze gender and ethnic bias in
U.S.-centric corpora, while we specifically focus on antisemitic and anti-
communist bias in German parliamentary data, which allows us to capture
ideological trends. Tripodi et al. focus, similar to us, on antisemitic bias,
but they analyze French books and periodicals issues. We additionally propose
a graph-based bias evaluation, which allows us to deal with smaller historical
slices. Using word embeddings for studying change in historical corpora was
introduced by Hamilton et al. [17] and Eger and Mehler [10]: however, while
their contribution is on determining statistical laws of semantic change in
diachronic word embeddings, we focus here instead on whether changes in bias
measurements from dense and sparse embeddings align against known historical
changes. Despite their enormous popularity, there exist alternatives to word
embeddings as methodology to analyze historical corpora, including topic
modeling [35, 2] and event extraction [31].

##  VI Conclusion

In this work, we have investigated how to employ the notion of bias in
distributional word vector spaces for understanding ideological shifts in a
large historical corpus of German parliamentary debates. The results obtained
with the various available measures sometimes exhibit different amounts of
bias. However, many shared trends can be found, which can, in turn, be
attributed to events related to ideological shifts within the German history,
e.g., the establishment of the iron curtain or peaks in antisemitism leading
up to the NS time. Along with our study, we release DeuPARL hoping to fuel
further research on computational understanding of German parliamentary
proceedings.

Using the notion of bias, we are able to capture commonly accepted historical
trends. While the embedding-based method has the advantage that a large range
of bias evaluation measures for estimating the extend of stereotyping present
in text representation models is available, it has the drawback in that the
sizes of the slices need to be sufficiently large to induce reliable
embeddings. In these cases, we proposed to ‘fall-back’ to more traditional
meaning representations, i.e., co-occurrence graphs, and leverage label
propagation algorithms to infer bias from sparse co-occurrence counts.

Our analysis shows that main ideological shifts can be read from historical
corpora through the quantification of bias of semantic spaces induced from
them. Our method has the potential to enable a bias-centric exploration of
textual collections from experts and thus calls for the future integration of
our quantitative analysis with qualitative ones in the manner of _distant
reading_ from historians. We also plan to tackle other languages than German,
for a contrastive evaluation of multilingual cross-temporal bias, in future
work.

## Acknowledgments

We thank the reviewers for insightful comments. We thank Niklas Friedrich for
the independent reimplementation of the approaches and reproduction of the
empirical results.

## References

  * [1] C. Basta, M. R. Costa-jussà, and N. Casas, “Evaluating the underlying gender bias in contextualized word embeddings,” in _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , 2019, pp. 33–39.
  * [2] D. M. Blei, “Topic modeling and digital humanities,” _Journal of Digital Humanities_ , vol. 2, no. 1, 2012.
  * [3] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017.
  * [4] T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” in _Advances in Neural Information Processing Systems 29_ , D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016, pp. 4349–4357.
  * [5] A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” _Science_ , vol. 356, no. 6334, pp. 183–186, 2017.
  * [6] V. D. Carlo, F. Bianchi, and M. Palmonari, “Training temporal word embeddings with a compass,” in _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019_ , 2019, pp. 6326–6334.
  * [7] K. Chaloner and A. Maldonado, “Measuring gender bias in word embeddings across domains and discovering new gender bias word categories,” in _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , 2019, pp. 25–32.
  * [8] S. Creuzberger and D. Hoffmann, “Antikommunismus und politische kultur in der bundesrepublik deutschland,” _Geistige Gefahr und Immunisierung der Gesellschaft, München_ , 2014.
  * [9] S. Dev and J. Phillips, “Attenuating bias in word vectors,” in _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , 2019, pp. 879–887.
  * [10] S. Eger and A. Mehler, “On the linearity of semantic change: Investigating meaning variation via dynamic graph models,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 2016, pp. 52–58.
  * [11] S. Eger, T. vor der Brück, and A. Mehler, “A comparison of four character-level string-to-string translation models for (OCR) spelling error correction,” _The Prague bulletin of mathematical linguistics_ , vol. 105, no. 1, pp. 77 – 99, 2017.
  * [12] K. Ethayarajh, D. Duvenaud, and G. Hirst, “Understanding undesirable word embedding associations,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 1696–1705.
  * [13] N. Garg, L. Schiebinger, D. Jurafsky, and J. Zou, “Word embeddings quantify 100 years of gender and ethnic stereotypes,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 115, no. 16, pp. E3635–E3644, 2018.
  * [14] G. Glavaš, F. Nanni, and S. P. Ponzetto, “Unsupervised cross-lingual scaling of political texts,” in _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 2017, pp. 688–693.
  * [15] H. Gonen and Y. Goldberg, “Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers)_ , J. Burstein, C. Doran, and T. Solorio, Eds., 2019, pp. 609–614.
  * [16] W. Halder, _Innenpolitik im Kaiserreich: 1871-1914_. Wissenschaft Buchgesellschaft, 2003.
  * [17] W. L. Hamilton, J. Leskovec, and D. Jurafsky, “Diachronic word embeddings reveal statistical laws of semantic change,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 1489–1501.
  * [18] Z. S. Harris, “Distributional structure,” _Word_ , vol. 10, no. 2-3, pp. 146–162, 1954.
  * [19] U. Jun, _Die Parteien nach der Bundestagswahl 2017: Aktuelle Entwicklungen des Parteienwettbewerbs in Deutschland_. Springer-Verlag, 2020.
  * [20] K. Kurita, N. Vyas, A. Pareek, A. W. Black, and Y. Tsvetkov, “Measuring bias in contextualized word representations,” in _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , 2019, pp. 166–172.
  * [21] A. Lauscher and G. Glavaš, “Are we consistently biased? multidimensional analysis of biases in distributional word vectors,” in _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_ , 2019, pp. 85–91.
  * [22] A. Lauscher, G. Glavaš, S. P. Ponzetto, and I. Vulić, “A general framework for implicit and explicit debiasing of distributional word vector spaces,” in _Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020_ , 2020, pp. 8131–8138.
  * [23] A. Lauscher, R. Takieddin, S. P. Ponzetto, and G. Glavaš, “AraWEAT: Multidimensional analysis of biases in Arabic word embeddings,” in _Proceedings of the Fifth Arabic Natural Language Processing Workshop_ , 2020, pp. 192–199.
  * [24] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” 2019. [Online]. Available: http://arxiv.org/abs/1908.09635
  * [25] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2_ , 2013, pp. 3111–3119.
  * [26] B. A. Nosek, M. R. Banaji, and A. G. Greenwald, “Harvesting implicit group attitudes and beliefs from a demonstration web site.” _Group Dynamics: Theory, Research, and Practice_ , vol. 6, no. 1, p. 101, 2002.
  * [27] B. A. Nosek, A. G. Greenwald, and M. R. Banaji, “Understanding and using the implicit association test: Ii. method variables and construct validity,” _Personality and Social Psychology Bulletin_ , vol. 31, no. 2, pp. 166–180, 2005.
  * [28] E. Ntoutsi, P. Fafalios, U. Gadiraju, V. Iosifidis, W. Nejdl, M. Vidal, S. Ruggieri, F. Turini, S. Papadopoulos, E. Krasanakis, I. Kompatsiaris, K. Kinder-Kurlanda, C. Wagner, F. Karimi, M. Fernández, H. Alani, B. Berendt, T. Kruegel, C. Heinze, K. Broelemann, G. Kasneci, T. Tiropanis, and S. Staab, “Bias in data-driven AI systems - an introductory survey,” _CoRR_ , vol. abs/2001.09762, 2020. [Online]. Available: https://arxiv.org/abs/2001.09762
  * [29] K. Ortmann, A. Roussel, and S. Dipper, “Evaluating off-the-shelf nlp tools for german.” in _KONVENS_ , 2019.
  * [30] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 1532–1543.
  * [31] M. Rovera, F. Nanni, and S. P. Ponzetto, “Event-based access to historical italian war memoirs,” _Journal on Computing and Cultural Heritage_ , vol. 14, no. 1, 2021.
  * [32] A. Schildt, _Konservatismus in Deutschland: von den Anfängen im 18. Jahrhundert bis zur Gegenwart_. Beck, 1998\.
  * [33] R. Tripodi, M. Warglien, S. Levis Sullam, and D. Paci, “Tracing antisemitic language through diachronic embedding projections: France 1789-1914,” in _Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change_ , 2019, pp. 115–125.
  * [34] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, pp. 2579–2605, 2008.
  * [35] T.-I. Yang, A. Torget, and R. Mihalcea, “Topic modeling on historical newspapers,” in _Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities_ , 2011, pp. 96–104.
  * [36] J. Zhao, T. Wang, M. Yatskar, R. Cotterell, V. Ordonez, and K.-W. Chang, “Gender bias in contextualized word embeddings,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 629–634.
  * [37] X. Zhu, Z. Ghahramani, and J. D. Lafferty, “Semi-supervised learning using gaussian fields and harmonic functions,” in _Proceedings of the 20th International conference on Machine learning (ICML-03)_ , 2003, pp. 912–919.
  * [38] X. Zhu and A. B. Goldberg, “Introduction to semi-supervised learning,” _Synthesis lectures on artificial intelligence and machine learning_ , vol. 3, no. 1, pp. 1–130, 2009.


# URIR: Recommendation algorithm of user RNN encoder and item encoder based on
knowledge graph

Na Zhao  Key Laboratory in Software Engineering of Yunnan Province, School of
Software, Yunnan University, Kunming 650504, People’s Republic of China
Electric Power Research Institute of Yunnan Power Grid Co., Ltd, Kunming
650200, People’s Republic of China  Zhen Long  Key Laboratory in Software
Engineering of Yunnan Province, School of Software, Yunnan University, Kunming
650504, People’s Republic of China  Zhi-Dan Zhao Zhi-Dan Zhao:
<EMAIL_ADDRESS>Department of Computer Science, School of Engineering,
Shantou University, Shantou 515063, China  Key Laboratory of Intelligent
Manufacturing Technology (Ministry of Education), Shantou University, Shantou
515063, China  Jian Wang Jian Wang<EMAIL_ADDRESS>College of
Information Engineering and Automation, Kunming University of Science and
Technology, Kunming 650217, People’s Republic of China

###### Abstract

Due to a large amount of information, it is difficult for users to find what
they are interested in among the many choices. In order to improve users’
experience, recommendation systems have been widely used in music
recommendations, movie recommendations, online shopping, and other scenarios.
Recently, Knowledge Graph (KG) has been proven to be an effective tool to
improve the performance of recommendation systems. However, a huge challenge
in applying knowledge graphs for recommendation is how to use knowledge graphs
to obtain better user codes and item codes. In response to this problem, this
research proposes a user Recurrent Neural Network (RNN) encoder and item
encoder recommendation algorithm based on Knowledge Graph (URIR). This study
encodes items by capturing high-level neighbor information to generate items’
representation vectors and applies an RNN and items’ representation vectors to
encode users to generate users’ representation vectors, and then perform inner
product operation on users’ representation vectors and items’ representation
vectors to get probabilities of users interaction with items. Numerical
experiments on three real-world datasets demonstrate that URIR is superior
performance to state-of-the-art algorithms in indicators such as AUC,
Precision, Recall, and MRR. This implies that URIR can effectively use
knowledge graph to obtain better user codes and item codes, thereby obtaining
better recommendation results.

Keywords: KG; RNN; encode

##  1 Introduction

With the rapid development of Internet, the amount of data has increased
exponentially. Due to information overload, it is difficult for users to
single out the ones they are interested in from a large number of choices [1,
2]. In order to make it easier for users to obtain the items and information
they need, recommendation systems have been applied to scenarios such as music
recommendation [3], movie recommendation [4, 5], and online shopping [6, 7].
The recommendation algorithm is the core of recommendation system. One widely
used recommendation algorithm is collaborative filtering (CF), which assigns
representation vectors based on users’ IDs and items’ IDs [1]. Then it models
their interaction through specific operations (such as inner product [8] or
neural network [9]). In addition, methods based on CF are often plagued by the
sparsity of users interaction with items and cold start problem. In order to
solve these limitations, researchers usually turn to feature-rich scenarios,
where the attributes of users and items are applied to compensate for the
sparsity and improve performance of recommendation systems [10, 11, 12, 13].

Some studies [14, 15, 16, 17, 18] go further than simply using attribute: they
point out that attributes are not isolated but connected, which forms
knowledge graph (KG). Generally, KG is a heterogeneous graph, where nodes
correspond to entities (item or attribute of item), and edges correspond to
relationships. Compared with KG-free methods, combining KG into recommendation
has three benefits [15]: (1) The rich semantic relevance between items in KG
helps to explore their potential connections and improve the accuracy of
results; (2) The various types of relationships in a KG help to reasonably
expand users’ interests and increase the diversity of recommended items; (3)
KG connects the items that users likes and the items recommended to users,
which brings interpretability to recommendation systems.

The recommendation algorithm based on knowledge graph is mainly considered
from the following three aspects: a) Method based on graph neural network. It
captures the high-level structure in graph and refines the embedding of users
and items. For example, RippleNet [15] spreads users’ potential preferences in
KG and explores their hierarchical interests. Wang et al. [18] use a KG graph
convolutional network (GCN [19]), which is incorporated to generate high-level
term connectivity features. KGAT [20] explicitly models the high-level
connections by an end-to-end manner. b) Method based on embedding, which
combines the entities and relationships of KG into a continuous vector space,
and helps recommendation system by enhancing semantic representation. For
example, DKN [21] combines the semantic and knowledge-level representations of
news, and incorporates KG representation into news recommendations. In
addition, CKE [17] combines CF module with structure, text and visual
knowledge into a unified recommendation framework. c) Method based on path. It
uses paths and related user-item pairs to explore the connection mode between
items in KG. For example, MCRec [3] learns the explicit representation of
meta-paths in recommendation. In addition, it also considers meta-paths and
the interactions between users and items. Compared with the method based on
embedding, the method based on path develops KG structure more naturally and
intuitively. RKGE [22] and KPRN [23] that needn’t to design meta-paths
automatically collect the paths between users and items, and use Recurrent
Neural Network (RNN) to encode the paths.

However, the above algorithms have some problems such as the following: 1)
users’ representation vectors are obtained by simply coding in the methods
based graph neural network and embedding, which may not be enough to encode
users; 2) method based on embedding is more suitable for in-graph application,
such as link prediction or KG completion, rather than recommendation
algorithm; 3) method based on path relies heavily on meta-paths and ignores
the topological structure of the connections between users and items, and only
aggregates the impact of each path on user preferences in the final stage. In
order to solve the above-mentioned problems, this study propose a user RNN
encoder and item encoder recommendation algorithm (URIR) based on knowledge
graph. The algorithm aggregates and merges a fixed number of high-level
neighborhood information with deviations when calculating the representation
of a given entity in KG, and tries to encode users by an improved RNN and
users’ embeddings.

We apply URIR to three data sets: job (resume), ml (movie), and yelp
(business). The experimental results show that URIR can obtain better
recommendation results. Our contributions in this article are summarized as
follows:
a) We propose URIR, which is an end-to-end framework for exploring user
preferences. By expanding the neighborhood of each entity in KG, URIR can
capture the user’s high-level personal interests.
b) We conduct experiments on three recommendation scenarios, and the
experimental results demonstrate the effectiveness of the URIR algorithm.
c) We use RNN to encode users to obtain better users’ representation vectors.
As far as we know, it’s the first time that RNN has been used to encode users
in KG’s algorithms.

The remainder of this paper is organized as follows: In section 2, we specify
the Item Encoding Layer 2.1, the User Encoding Layer 2.2, the Prediction Layer
2.3, the Model Optimization 2.4, and the Experimental Setup 2.5. In section 3,
we conduct extensive experiments to evaluate the proposed method for URIR.
Finally, in section 4, we discuss and conclude this work.

##  2 Material and methods

Figure 1: Model Framework

We first introduce the commonly used mathematical symbols in the text and
their meanings. The detailed information is shown in Table 1. Additionally,
the model framework is shown in Figure 1, which is mainly composed of 3 parts:
1) Item encoding layer 2.1, which responsible for encoding item $v$ and
generating item $v$’s representation vector v̂. 2) User encoding layer 2.2,
which illustrate the coding process of user $u$ and generate user $u$’s
representation vector û. 3) Prediction layer 2.3, it generate the probability
$\tilde{r}_{u,v}$ of the interaction between user $u$ and item $v$.

Table 1: Model symbol introduction Notation |  Description
---|---
$\mathcal{E}_{v}^{l}$ |  v’s l-level neighbsor
${S}_{v}^{l}$ |  v’s l-level triples
$(h,r,t)$ |  KG’s triples
$\pi_{(h,r,t)}$ |  $(h,r,t)$ weight
$\tilde{\pi}_{(h,r,t)}$ |  $(h,r,t)$ normalized weight
$\mathbf{W}_{*}$ |  learnable parameter
$\mathbf{b}_{*}$ |  learnable parameter
$\mathbf{z}_{*}$ |  hidden layer output
$\mathbf{v}_{l}$ |  $v$’s l-level neighbor vector
$\hat{\mathbf{v}}$ |  $v$’s representation vector
$L$ |  the order of the high-level neighbor
$\mathbf{v}$ |  $v$’s embedding
$H_{u}$ |  the set of the items that user $u$ has interacted with
$r_{u,v}$ |  $r_{u,v}=1$ represents that user $u$ has interacted with item $v$, $r_{u,v}=0$ represents that user $u$ has not interacted with item $v$
$\hat{\mathbf{u}}$ |  $u$’s representation vector
$n$ |  the number of elements of $H_{u}$
$\mathbf{v}_{i}^{u}$ |  the representation of the item in $H_{u}$
$\mathbf{h}_{i}^{u}$ |  $\mathbf{v}_{i}^{u}$’s RNN output

###  2.1 Item Encoding Layer

The purpose of item encoding layer is to capture the high-level information of
item $v$ in knowledge graph and generate a better item’s representation
vector. Here, item $v$ captures high-level neighbor information through v’s
l-level neighbors ($\mathcal{E}_{v}^{l}$) and v’s l-level triples
($S_{v}^{l}$). As $l$ increases, the elements in $S_{v}^{l}$ increase
exponentially. In order to reduce the calculation time, we collect $k$
neighbors of each node. This method is a commonly used simple but effective
strategy to reduce computational cost [15, 18].

$\mathcal{E}_{v}^{l}=\\{t|t\in\mathcal{G}\quad
and\quad(h,r,t)\in{S_{v}^{l}}\\}\quad l=1,2,3,...$ (1)
$S_{v}^{l}=\\{(h,r,t)|h\in\mathcal{E}_{v}^{l-1}\quad
and\quad(h,r,t)\in\mathcal{G}\quad and\quad t\notin\mathcal{E}_{v}^{j}\\}\quad
l=1,2,3,...\quad j<l$ (2) $\mathcal{E}_{v}^{0}=\\{v\\}$ (3)

We use triples in $S_{v}^{l}$ to capture high-level neighbor information. Due
to the different head entities and relationships in KG, each tail entity of
$S_{v}^{l}$ has different meanings and latent vector representations. For
example, Forrest Gump and Cast Away have more in common in terms of their
directors or casts, but are less similar if measured by genres or writers. As
a result, we propose a knowledge-aware attentive embedding method to generate
different attentive weights of the tail entity to reveal the different
meanings, when it gets different head entities and relations.

$\mathbf{z}_{0}=\mathbf{h\oplus r\oplus t}$ (4)
$\mathbf{z}_{1}={\rm{Relu}}(\mathbf{W}_{1}{\mathbf{z}_{0}}+\mathbf{b}_{1})$
(5)
$\mathbf{z}_{2}={\rm{Relu}}(\mathbf{W}_{2}{\mathbf{z}_{1}}+\mathbf{b}_{2})$
(6) $\pi_{(h,r,t)}={\rm{Relu}}(\mathbf{W}_{3}{\mathbf{z}_{2}}+\mathbf{b}_{3})$
(7)
$\tilde{\pi}_{(h,r,t)}=\frac{\exp(\pi_{(h,r,t)})}{\sum_{(h^{\prime},r^{\prime},t^{\prime})\in
S_{v}^{l}}\exp(\pi_{(h^{\prime},r^{\prime},t^{\prime}))}}$ (8)

where $\tilde{\pi}_{(h,r,t)}$ is the normalized weight that generated from
$(h,r,t)$, ${\pi}_{(h,r,t)}$ is the weight that generated from $(h,r,t)$,
$\mathbf{h}$ is the embedding of $h$, $\mathbf{r}$ is the embedding of $r$,
$\mathbf{t}$ is the embedding of $t$, $\mathbf{W}_{*}$ is the learnable
parameter, $\mathbf{b}_{*}$ is the learnable parameter, $\oplus$ is the
concatenation of vector.

Combine weights with corresponding tail entities to obtain high-level neighbor
information, as shown in Eq. 9:

$\mathbf{v}_{l}=\sum_{(h^{\prime},r^{\prime},t^{\prime})\in
S_{v}^{l}}\tilde{\pi}_{(h^{\prime},r^{\prime},t^{\prime})}\mathbf{t}^{\prime}$
(9)

where $\mathbf{v}_{l}$ is the $l$-level neighbor information of item $v$.
Because the item itself has vital information, the high-level neighbor
information and the information of the item itself are combined to obtain the
item’ representation vector, as shown in Eq. 10:

$\hat{\mathbf{v}}=\frac{\sum_{i=1}^{L}\mathbf{v}_{l}+\mathbf{v}}{L+1}~{}$ (10)

where $\hat{\mathbf{v}}$ is the item $v$’s representation vector, $\mathbf{v}$
is the embedding of item $v$.

###  2.2 User Encoding Layer

The purpose of user encoding layer is to better encode users and generate
better users’ representation vectors. The user encoding layer consists of four
parts: 1) item encoding layer: Input the items that the user has interacted
with, and then generate the corresponding item’s representation vector; 2)
user embedding layer: Generate users’ embedding vectors; 4) RNN layer: Input
user’s embedding vector and the representation vectors of the items in $H_{u}$
to obtain the user $u$’s representation vector. $H_{u}$ represent the set of
the items that user $u$ has interacted with, as demonstrated in Eq. 11.

$H_{u}=\\{v|r_{uv}=1\\}$ (11)

Item encoding layer were introduced in 2.1, and user embedding layer is
actually to obtain users’ embedding. In order to improve the performance of
the URIR, we use an improved RNN model. The improved RNN model is introduced
as follows. URIR takes the representation vector of the item in $H_{u}$ as a
word embedding vector and inputs them into the RNN. URIR considers the
information carried by the user itself, that is, the user $u$’s embedding
vector to encode user $u$. This is in line with daily habits. For example, two
users have watched the same movie, but there is no guarantee that the movie
they watch next time is the same. The RNN layer is represented by the
following formulas for Eq. 12 to 14:

$\hat{\mathbf{u}}=\rm{RNN}([\mathbf{v}_{1}^{\emph{u}},\mathbf{v}_{2}^{\emph{u}},...,\mathbf{v}_{\emph{n}}^{\emph{u}}],\mathbf{u})$
(12)
$\mathbf{h}_{i}^{u}=\rm{ReLu}(\mathbf{W}\mathbf{v}_{i}^{\emph{u}}+\mathbf{H}\mathbf{h}_{i-1}^{\emph{u}}+\mathbf{U}\mathbf{u})$
(13) $\hat{\mathbf{u}}=\mathbf{h}_{n}^{\emph{u}}$ (14)

Where $\hat{\mathbf{u}}$ is the user $u$’s representation vector,
$[\mathbf{v}_{1}^{\emph{u}},\mathbf{v}_{2}^{\emph{u}},...,\mathbf{v}_{n}^{\emph{u}}]$
are the representation vectors of the items in $H_{u}$, u is the user $u$’s
embedding, $\mathbf{h}_{0}^{u}=\mathbf{0}$. URIR takes $\mathbf{h}_{n}^{u}$ as
the user $u$’s representation vector, because it already contains the
information previously entered.

###  2.3 Prediction Layer

Prediction layer obtains the probability of interaction between $u$ and $v$
through inner product of $\hat{\mathbf{u}}$ and $\hat{\mathbf{v}}$, and uses
sigmoid function to control the probability to between 0 and 1, as shown in
the following formula Eq. 15:

$\tilde{r}_{u,v}=\rm{sigmoid}(\hat{\mathbf{u}}\hat{\mathbf{v}}^{\rm{T}})$ (15)

Where $\tilde{r}_{u,v}$ is the predicted probability of interaction between
user $u$ and item $v$.

###  2.4 Model Optimization

Objective function: According to the literature [9], we treat top-K
recommendation task as a binary classification problem, where 1 means that
user has interacted with item, and 0 means that user has not interacted with
item. In objective function, we use cross entropy as objective function, the
formula is as follows Eq. 16:

$\mathcal{J}=\sum_{(u,v)\in\mathcal{D}_{train}}\rm{BCELOSS}(\tilde{\emph{r}}_{\emph{u,v}},\emph{r}_{\emph{u,v}})+\lambda||\mathcal{F}||_{2}^{2}$
(16)

Where $\lambda$ is L2 regularization coefficient, $\mathcal{F}$ are the
parameters of the model, and $\mathcal{D}_{train}$ is training set. We collect
the items that have not been interacted with for each user as negative
samples, and negative samples are collected according to the ratio of the
number of positive samples to the number of negative samples of 4:1 in
training set.

###  2.5 Experimental Setup

To verify the performance of algorithm URIR, We use three data sets: (1) Job
data set describes some occupations that users have carried on; (2) Ml [22] is
a data set widely used in movie recommendation; (3) Yelp [22] records users
ratings of some businesses. In order to better verify the performance of the
model in solving the cold start problem, we limit the number of user
interactions with items to less than 20 [20]. The attributes of the data sets
are shown in Table 2. Users represents the number of users, items represents
the number of items, relations represents the number of relation types, edges
represents the number of edges, interactions represents the number of
interactions, entities represents the number of entities (nodes).

Table 2: Data set attributes data set | users | items | relations | edges | interactions | entities
---|---|---|---|---|---|---
job | 5967 | 85 | 2 | 360 | 13220 | 194
ml | 943 | 1674 | 3 | 13202 | 12434 | 6800
yelp | 1936 | 11515 | 2 | 42875 | 18146 | 12097

Evaluation indicators: We use widely applied indicators to evaluate models’
performance. For each user, we divide him`\`her history records according to
the ratio of the number of items in the training set to the number of items in
the test set of 7:3, and We collect negative samples according to the ratio of
the number of positive samples to the number of negative samples of 4:1 in
training set.

According to the literature [9, 23, 24], in the test process, for each user,
we randomly select 50 items that the user has not interacted with, and
randomly select 1 item that has interacted with the user in test set. The
recommended list is composed of these 51 items, and the 51 items are sorted to
reduce test’s complexity. In model training, we use AUC [25] as an evaluation
indicator to determine the best parameters. When evaluating model performance,
we use Precision@K [25], Recall@K [25]and MRR@K [25] as evaluation indicators.
We calculate these indicators for each user, and calculate the average value
when K = {1, 2, 4, 5, 6, 8, 10}. Since situation does not recommend too many
items in actual, we choose these values for K. Generally, higher values of
these indicators indicate better performance. AUC characterizes the overall
performance of recommendation algorithm and covers the performance of all
different recommendation list lengths [25]. Therefore, we use AUC as an
evaluation indicator during training.

In order to verify the effectiveness of URIR, we select state-of-the-art
algorithms recently proposed for comparison. These algorithms are: NFM [26],
KPRN [23], Wide&Deep [10], RKGE [22], RippleNet [15], KGCN [18]. These
algorithms were chosen because they are all influential algorithms based on
knowledge graph, and some of the algorithm ideas have provided useful
inspiration for us to propose the URIR algorithm.

Parameter settings: The optimal parameters of the above algorithms are set by
experimental verification or the conclusions obtained of the original paper
[26, 23, 10, 22, 15, 18]. For URIR, we use Adam as optimizer, and use mini-
batch gradient descent algorithm. These methods are some commonly used
optimization strategies [15, 20, 18]. In view of the influence of different
hyperparameters the embedding dimension of entity d, the number of neighbors
collected k, the order of the high-level neighbor L and the number of elements
of $H_{u}$ n, we also made a comprehensive comparison of the system. The
learning rate, batch size, and L2 regularization coefficient are determined
according to training time. The initialization parameters of the model are:
d=4, k=4, L=2, n=5. Hyperparameter adjustment sequence is d, L, k, n. The
detailed comparison process and comparison results are introduced in Section
3.3. The default values of these parameters are shown in Table 3.

Table 3: Parameter settings on each data set data set | learning rate | number of iterations | d | k | L | n | $\lambda$ | batch size
---|---|---|---|---|---|---|---|---
job | 0.02 | 11 | 8 | 4 | 1 | 5 | 0.001 | 256
ml | 0.05 | 10 | 64 | 16 | 1 | 5 | 0.0001 | 256
yelp | 0.05 | 6 | 64 | 16 | 1 | 5 | 0.0001 | 256

##  3 Results

We conducted extensive experiments on three data sets to answer the following
three questions: 1) Does the URIR model perform better than other advanced
recommendation algorithms? 2) What is the impact of RNN on the performance of
the URIR algorithm? 3) How does the hyperparameter affect the performance of
the URIR?

###  3.1 Performance of URIR

In order to fully understand the performance of the URIR algorithm, we
compared the performance of the aforementioned eight different algorithms on
three different data sets, and the results are shown in Figure 2. First of
all, Figure 2 (a), Figure 2 (d) and Figure 2 (g) display the variation trend
of the Precision of different algorithms on three different data sets as K
increases, respectively. All the algorithm result curves in Figure 2 (a) show
that as K increases, the overall Precision shows a downward trend. Although
the URIR algorithm curve and other algorithm curves is getting closer, the
performance of URIR is still better than other algorithms on the whole. For
example, when K={1, 2, 4, 5, 6, 8, 10}, compared with the best results of
state-of-the-art algorithms, URIR increases by 26.9%, 36.9%, 5.5%, 1.7% and
0.0%, respectively. Additionally, compared with the worst case of these
algorithms, when K={1, 2, 4, 5, 6, 8, 10}, URIR increased by 147.5%, 190%,
28.9%, 9.6% and 0.9%, respectively. Compared with the average values of other
models when K={1, 2, 4, 5, 6, 8, 10}, URIR increased by 61.0%, 64.9%, 14.6%,
4.3% and 0.5%, respectively. Similarly, both the ml and yelp data sets show
the same characteristics. This phenomenon is consistent with previous studies
[15, 20, 18, 27].

Secondly, Figure 2 (b), Figure 2 (e) and Figure 2 (h) demonstrate the
evolution trend of Recall and K of different algorithms on 3 different data
sets. For example, Figure 2 (b) shows that the Recall of different algorithms
on the job data set shows an overall upward trend as K increases. Although the
Recall of all algorithms shows a trend of increasing with K, the Recall value
of the URIR algorithm is always higher than that of other methods.

Thirdly, Figure 2 (c), Figure 2 (f) and Figure 2 (i) illustrate the changes in
the MRR of different algorithms on three different data sets as K increases.
Figure 2 (c) shows the changing trend of MRR of different algorithms on the
job data set. It is very interesting that we observe that all the curves in
Figure 2 (c) show an upward trend as K increases. At the same time, although
all models have the same trend, the MRR value of the URIR algorithm has always
been higher than other models. For example, when URIR is K={1, 2, 4, 5, 6, 8,
10}, compared with the best model, it is improved by 26.9%, 43.1%, 32.8%,
32.1% and 31.8%, respectively. Compared with the worst model when K={1, 2, 4,
5, 6, 8, 10}, URIR increased by 147.5%, 168.4%, 114.1%, 100.8% and 100.0%,
respectively. Compared with the average value of other models, When K={1, 2,
4, 5, 6, 8, 10}, URIR increased by 61.0%, 74.1%, 56.9%, 53.9% and 53.0%,
respectively.

In general, our experimental results show that compared with other models,
URIR can better find items that users are interested in and put them at the
top of the recommendation list. For example, if the user only requests to
recommend fewer items, compared to other algorithms, URIR better meets the
needs of users. In addition, as the value of K increases, the advantage of the
URIR algorithm gradually becomes smaller. This is due to URIR put the items
that users like at the top of the recommended list. As K increases, other
algorithms also predict the items users like. In addition, when using ml data,
the performance of URIR is not as good as the other two data. The possible
factor is that there are enough interaction records between users and items of
ml data. There is no need to use RNN to capture more information to encode
users, which implies that the URIR algorithm is a useful attempt to solve the
cold start problem of collaborative filtering recommendation algorithm.

Figure 2: URIR’s performance on three data sets

###  3.2 The impact of RNN

In order to generate a better user representation vector, the URIR model
considers the introduction of RNN to encode users. Therefore, whether the use
of RNN has an impact on user coding is an important issue. Here, we answer the
above question by comparing the recommendation results of URIR and the URIR
algorithm without the RNN (URIR-RNN). At this time, user $u$’s representation
vector is $\hat{\mathbf{u}}=\mathbf{u}+\sum_{v\in H_{u}}\hat{\mathbf{v}}$. The
experimental results are shown in Table 4. The experimental results are shown
in Table 4. The research results indicate that after the application of RNN
user encoding, the AUC results of the two datasets job and ml of URIR have
been significantly improved. Especially on the job data set, the AUC value of
the URIR algorithm is about 0.84, and the AUC value of the URIR-RNN is about
0.75, an increase of up to 10%. This result verifies that the use of RNN
proposed in URIR to encode users is very necessary and effective. On Yelp
data, the reason why the AUC values of the two algorithms are not
significantly different may be that there are many interaction records between
users and items in the yelp data and the KG of yelp is so big that URIR-RNN
can get enough auxiliary information from the KG. The result also reflects
that the item encoder of the URIR algorithm effectively obtain auxiliary
information from KG.

Table 4: Comparison of AUC between URIR and URIR-RNN model | job | ml | yelp
---|---|---|---
URIR | 0.8351 | 0.8544 | 0.8039
URIR-RNN | 0.7544(-9.66%) | 0.8059(-5.68%) | 0.8072 (0.41%)

###  3.3 Parameter Sensitivity

In order to better understand the role of different hyperparameters in the
URIR algorithm, here we separately study the impact of these parameter changes
on the performance of the URIR algorithm: the embedding dimension of entity
$d$, the order of the high-level neighbor L, the number of neighbors collected
k and the number of elements of $H_{u}$ $n$.

Table 5: Comparison of AUC for different $d$ of URIR data set | 4 | 8 | 16 | 32 | 64
---|---|---|---|---|---
job | 0.8031 | 0.8100 | 0.7879 | 0.7588 | 0.8039
ml | 0.8160 | 0.8132 | 0.8248 | 0.8257 | 0.8292
yelp | 0.7761 | 0.7751 | 0.7805 | 0.7900 | 0.7967

Embedding dimension of entity $d$: The role of entity embedding dimension $d$
in the knowledge graph is a general problem [15, 20, 18, 27]. Based on this
assumption, we give the performance of $d$ taking different values {4, 8, 16,
32, 64} in the URIR algorithm, and the experimental results are shown in table
5. We noticed that the performance of URIR increase with the $d$ growing, and
reach the best performance when $d$ reaches a certain value. This result
implies that a larger embedding dimension $d$ can significantly help to encode
more useful information. However, with the further increase of $d$, the
performance of some data sets decrease, such as job data sets. As shown in
Table 2, the sparsity of the job graph is significantly smaller than the other
two data of ml and yelp. Future research should focus on the influence of
graph sparsity on the selection of embedding dimensions [18, 27]. This result
indicates that too large embedding dimensions may cause the entity to be
overrepresent and introduce noise, which affects the performance of the
algorithm.

Table 6: Comparison of AUC for different $L$ of URIR data set | 1 | 2 | 3 | 4
---|---|---|---|---
job | 0.8351 | 0.8100 | 0.8049 | 0.7938
ml | 0.8542 | 0.8292 | 0.8377 | 0.8375
yelp | 0.7991 | 0.7967 | 0.7830 | 0.7789

The order of the high-level neighbor L: We studied the impact of the order of
the high-level neighbor L on URIR. The results are shown in Table 6. The
experimental results show that when $L=1$, the AUC value reaches the highest
value in all three data sets. This phenomenon is in line with our intuition.
Because as $L$ gets larger, the less node information the project $v$ can
obtain. However, the correlation between further nodes and $v$ will not be
very high, but some noise is transmitted to item $v$’s representation vector.
Another reason is the role of the RNN encoder. Because URIR-RNN’s $L$ is not
always one.

Table 7: Comparison of AUC for different $k$ of URIR data set | 2 | 4 | 8 | 16
---|---|---|---|---
job | 0.8231 | 0.8351 | 0.8181 | 0.0.8193
ml | 0.8429 | 0.8542 | 0.8523 | 0.8544
yelp | 0.7755 | 0.7991 | 0.8032 | 0.8039

The number of neighbors collected k: The influence of the number of neighbors
on the effect of the algorithm is a hot research issue [15, 18, 27]. In this
section, we study the influence of the number of neighbors on the
recommendation based on the KG by changing the number of neighbors collected
k. The results are shown in the table 7. The experimental results clearly show
that as the number of neighbors k increases, the AUC value also increases.
This is because the small $k$ does not have enough capacity to merge
neighborhood information. However, it is observed that the AUC value in the
job data set becomes smaller. This may be similar to the reason for the
dimension $d$, that is, the job data set is too sparse, and increasing the
value of k cannot obtain enough domain information, but as k increases, it
brings noise .

Table 8: Comparison of AUC for different $n$ of URIR data set | 5 | 10 | 15 | 20
---|---|---|---|---
job | 0.8351 | 0.8341 | 0.8341 | 0.8341
ml | 0.8544 | 0.8488 | 0.8472 | 0.8472
yelp | 0.8039 | 0.8037 | 0.8038 | 0.8038

The number of elements of $H_{u}$ $n$: We studied the impact of the number of
elements of $H_{u}$ $n$. The results are shown in Table 8. When $n$=5, AUC is
the highest. This is not difficult to understand. Because we speculate
people’s preferences from a few things. For example, if a user has watched
four movies that a actor stars, we can infer that the user likes the actor.
However, the user not only watched these movies, but also watched a certain
director’s movie. The actor also star the movie. We may also think that the
user like the director and recommend the director’s movie. However, the user
only likes the actor and does not like the director in actual. In other words,
an appropriate $n$ can help the model understand the user’s preferences. But
an excessively large $n$ makes the model misunderstand the user’s preferences.

##  4 Discussion and conclusion

This paper proposes a recommendation algorithm for user RNN encoder and item
encoder based on knowledge graph, which integrates the structured information
of knowledge graph into the algorithm. The RNN encoder helps to understand
users’ preferences, and the item encoder helps to mine $v$’s high-level
information. These are beneficial to the generation of better users’
representation vectors and items’ representation vectors, make full use of
knowledge graphs, and solve the limitations of the three methods. URIR
effectively alleviates the sparsity problem and improves the accuracy of
recommendation systems. Experiments were carried out on 3 real-world data
sets. we use AUC, Precision, Recall and MRR as the evaluation indicators of
the models. URIR was compared with FM, NFM [26], KPRN [23], Wide&Deep [10],
RKGE [22], RippleNet [15], KGCN [18]. Our experiment results demonstrate that
URIR is generally better than the advanced algorithm on three data sets, can
provide users with more accurate recommendation results, and has strong
flexibility in fusing heterogeneous data. Good performance in AUC, Precision
and Recall shows that it can be applied to classification tasks or tasks where
users rate items. Achieving better performance in MRR indicates that it can be
applied to the task of sorting items.

The URIR algorithm understands users’ preferences through users’ historical
behaviors, which is very dependent on whether the historical behaviors
collected contain all preferences and contains redundant information. Future
work should focus on how to improve the sampling strategy and user encoding
layer to generate better users’ representation vectors. In addition, future
recommendation methods based on knowledge graphs should include specific
network details, including user rating deviations and rating features [13],
network node degrees and H-index [28], and hypernetwork loop structure [29]
etc.

## References

  * [1] Linyuan Lü, Matúš Medo, Chi Ho Yeung, Yi-Cheng Zhang, Zi-Ke Zhang, and Tao Zhou.  Recommender systems.  Physics Reports, 519(1):1–49, 2012.
  * [2] Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong, and Qing He.  A survey on knowledge graph-based recommender systems.  IEEE Transactions on Knowledge and Data Engineering, 2020.
  * [3] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu.  Leveraging meta-path based context for top- n recommendation with a neural co-attention model.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1531–1540, New York, NY, USA, 2018. Association for Computing Machinery.
  * [4] Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, and Hui Xiong.  Setrank: A setwise bayesian approach for collaborative ranking from implicit feedback.  In Proceedings of the AAAI Conference on Artificial Intelligence, number 04, pages 6127–6136, 2020.
  * [5] Fuzhen Zhuang, Zhiqiang Zhang, Mingda Qian, Chuan Shi, Xing Xie, and Qing He.  Representation learning via dual-autoencoder for recommendation.  Neural Networks, 90:83–89, 2017.
  * [6] Jiayu Han, Lei Zheng, Yuanbo Xu, Bangzuo Zhang, Fuzhen Zhuang, S Yu Philip, and Wanli Zuo.  Adaptive deep modeling of users and items using side information for recommendation.  IEEE transactions on neural networks and learning systems, 31(3):737–748, 2019.
  * [7] Yuanbo Xu, Yongjian Yang, Jiayu Han, En Wang, Fuzhen Zhuang, and Hui Xiong.  Exploiting the sentimental bias between ratings and reviews for enhancing recommendation.  In 2018 IEEE International Conference on Data Mining (ICDM), pages 1356–1361. IEEE, 2018.
  * [8] Hongwei Wang, Jia Wang, Miao Zhao, Jiannong Cao, and Minyi Guo.  Joint topic-semantic-aware social recommendation for online voting.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 347–356, 2017.
  * [9] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In Proceedings of the 26th international conference on world wide web, pages 173–182, 2017.
  * [10] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al.  Wide & deep learning for recommender systems.  In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7–10, 2016.
  * [11] Hongwei Wang, Fuzheng Zhang, Min Hou, Xing Xie, Minyi Guo, and Qi Liu.  Shine: Signed heterogeneous information network embedding for sentiment link prediction.  In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 592–600, 2018.
  * [12] Ming-yang Zhou, Rong-qin Xu, Zi-ming Wang, and Hao Liao.  A generic bayesian-based framework for enhancing top-n recommender algorithms.  Information Sciences, 580:460–477, 2021.
  * [13] Hong-Liang Sun, Kai-Ping Liang, Hao Liao, and Duan-Bing Chen.  Evaluating user reputation of online rating systems by rating statistical patterns.  Knowledge-Based Systems, 219:106895, 2021.
  * [14] Jin Huang, Wayne Xin Zhao, Hongjian Dou, Ji-Rong Wen, and Edward Y Chang.  Improving sequential recommendation with knowledge-enhanced memory networks.  In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 505–514, 2018.
  * [15] Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo.  Ripplenet: Propagating user preferences on the knowledge graph for recommender systems.  In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 417–426, 2018.
  * [16] Xiao Yu, Xiang Ren, Yizhou Sun, Quanquan Gu, Bradley Sturt, Urvashi Khandelwal, Brandon Norick, and Jiawei Han.  Personalized entity recommendation: A heterogeneous information network approach.  In Proceedings of the 7th ACM international conference on Web search and data mining, pages 283–292, 2014.
  * [17] Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma.  Collaborative knowledge base embedding for recommender systems.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 353–362, 2016.
  * [18] Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo.  Knowledge graph convolutional networks for recommender systems. corr abs/1904.12575 (2019).  arXiv preprint arXiv:1904.12575, 2019.
  * [19] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [20] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua.  Kgat: Knowledge graph attention network for recommendation.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 950–958, 2019.
  * [21] Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo.  Dkn: Deep knowledge-aware network for news recommendation.  In Proceedings of the 2018 world wide web conference, pages 1835–1844, 2018.
  * [22] Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu.  Recurrent knowledge graph embedding for effective recommendation.  In Proceedings of the 12th ACM Conference on Recommender Systems, pages 297–305, 2018.
  * [23] Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, and Tat-Seng Chua.  Explainable reasoning over knowledge graphs for recommendation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5329–5336, 2019.
  * [24] Ali Mamdouh Elkahky, Yang Song, and Xiaodong He.  A multi-view deep learning approach for cross domain user modeling in recommendation systems.  In Proceedings of the 24th international conference on world wide web, pages 278–288, 2015.
  * [25] Yuxiao Zhu and Linyuan Lü.  Overview of recommendation system evaluation indicators.  Journal of University of Electronic Science and Technology of China, 02:163–175, 2012.
  * [26] Xiangnan He and Tat-Seng Chua.  Neural factorization machines for sparse predictive analytics.  In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 355–364, 2017.
  * [27] Ze Wang, Guangyan Lin, Huobin Tan, Qinghong Chen, and Xiyang Liu.  Ckan: Collaborative knowledge-aware attentive network for recommender systems.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 219–228, New York, NY, USA, 2020. Association for Computing Machinery.
  * [28] Xuzhen Zhu, Wenya Li, Hui Tian, and Shimin Cai.  Hybrid influence of degree and h-index in the link prediction of complex networks.  EPL (Europhysics Letters), 122(6):68003, 2018.
  * [29] Liming Pan, Hui-Juan Shang, Peiyan Li, Haixing Dai, Wei Wang, and Lixin Tian.  Predicting hyperlinks via hypernetwork loop structure.  EPL (Europhysics Letters), 2021.