#  A Game-Theoretic Analysis of Auditing Differentially Private Algorithms
with Epistemically Disparate Herd

Ya-Ting Yang, Tao Zhang, and Quanyan Zhu The Authors are with the Department
of Electrical and Computer Engineering, New York University, Brooklyn, NY,
11201, USA; E-mail: {yy4348, tz636<EMAIL_ADDRESS>
###### Abstract

Privacy-preserving AI algorithms are widely adopted in various domains, but
the lack of transparency might pose accountability issues. While auditing
algorithms can address this issue, machine-based audit approaches are often
costly and time-consuming. Herd audit, on the other hand, offers an
alternative solution by harnessing collective intelligence. Nevertheless, the
presence of epistemic disparity among auditors, resulting in varying levels of
expertise and access to knowledge, may impact audit performance. An effective
herd audit will establish a credible accountability threat for algorithm
developers, incentivizing them to uphold their claims. In this study, our
objective is to develop a systematic framework that examines the impact of
herd audits on algorithm developers using the Stackelberg game approach. The
optimal strategy for auditors emphasizes the importance of easy access to
relevant information, as it increases the auditors’ confidence in the audit
process. Similarly, the optimal choice for developers suggests that herd audit
is viable when auditors face lower costs in acquiring knowledge. By enhancing
transparency and accountability, herd audit contributes to the responsible
development of privacy-preserving algorithms.

##  I Introduction

AI and algorithmic decision-making have become pervasive in both business and
society. However, when algorithms are treated as “black boxes” and their inner
workings remain undisclosed, it becomes difficult to ensure that they perform
as intended and adhere to necessary standards [1]. One specific category of
algorithms that exemplifies this challenge is privacy-preserving algorithms
[2]. For instance, platforms like Facebook Ad Recommendation Systems, Google
SQL, and Safari have integrated differential privacy into their products to
provide privacy protection. Nevertheless, verifying such claims can be arduous
and intricate, for example, see [3, 4, 5].

Herd Audit: Auditing algorithms [6], [7] play a crucial role in tackling this
challenge. However, traditional machine-based audit methods like direct
scraping, sock puppet, and carrier puppet often necessitate the development of
custom computer programs to gather data. Not only can these approaches be
expensive, but they also consume a significant amount of time. A cost-
effective alternative approach to auditing involves leveraging citizen science
and crowd-sourcing principles to establish a democratic audit process that
engages a diverse population of end users [8]. This concept gives rise to
herd-audit (or group-audit) approaches. By empowering end users as auditors,
we can foster a more democratic approach to algorithmic auditing while
minimizing costs and time investments.

Figure 1: A herd of diverse end-users act as auditors to inspect the AI
algorithm used in the developed product.

Epistemic Disparity: One significant challenge in implementing herd-audit
approaches is the presence of epistemic disparity [9, 10]. Not all users
possess the same level of expertise or information required to conduct
comprehensive audits of algorithms. A user-auditor with limited cognitive
resources may inadvertently provide opportunities for algorithm developers to
evade their responsibility. To some extent, incorporating audit into the
algorithm design process itself establishes an accountability mechanism for
developers. This accountability mechanism acts as an incentive for algorithm
developers to uphold their claims and create responsible algorithms.

Game-Theoretic Framework: To design an effective herd-audit mechanism, this
work aims to develop a comprehensive system framework that investigates the
influence of herd-audit on algorithm developers. To accomplish the goal, the
system framework adopts a Stackelberg game approach [11, 12]. In this
approach, the developer assumes the role of the leader and determines the
desired level of performance for differential privacy. The followers,
comprising idiosyncratic end-users or auditors, are selected from a user
population characterized by varying levels of epistemic capabilities. The
proposed framework assumes that algorithms and their associated guarantees are
clearly communicated to the end-users through a privacy protection agreement.
This leader-and-follower structure allows us to analyze the optimal strategies
employed by both the developer and the auditors, providing insights into the
potential noncompliant behaviors of developers.

In order to capture the epistemic disparity experienced by end-users
(auditors), this work employs a rational inattention model [13, 14], which
takes into account the costs associated with accessing information during the
decision-making process. We analyze the epistemic disparity among auditors,
characterized by the epistemic factor, which measures the difficulty of
accessing information. We find that auditors with lower epistemic factors
exhibit higher audit confidence, indicating a better audit performance.
Furthermore, our investigation reveals that a herd audit is a viable approach
when auditors face lower costs in accessing information. In such
circumstances, the algorithm developer is less likely to deviate significantly
from their claims. Our findings highlight the importance of reducing epistemic
injustice as well as lowering information costs to enhance the effectiveness
of herd audits. By doing so, we can foster a more reliable and accountable
environment for the development of algorithms.

Related Works Algorithm auditing refers to the process of evaluating the
algorithms used in systems or applications to ensure they are fair,
transparent, unbiased, and comply with ethical standards [6]. In differential
privacy, several machine-based verification methods have been proposed [3, 4,
5]. While there has been a rich literature on citizen science and its
applications in crowdsensing [15], crowdsourcing [16], and crowd defense [17],
herd audit is a concept in its infancy. It reduces auditing costs and poses a
threat to developers, as public perception [18] can be influenced by the audit
results.

The disparity in the capability of herd behaviors has been studied in
collective intelligence [19, 8, 20, 21]. The literature has examined the
performance [22], reliability [23], and trustworthiness [24] of participants
engaged in outsourced tasks. Processes such as risk and reputation management
[25, 26] have been utilized to understand the differences among participants.
Numerous studies have focused on different cognitive behaviors in humans,
including cognitive-behavioral theory [27, 28, 29] which elucidates how
thoughts, beliefs, and cognitive processes shape behavior, and the theory of
mind [30] that attributes mental states such as beliefs and emotions to
predict individuals’ behavior. In our work, we employ the concept of rational
inattention, as studied in [31], which provides a framework that analyzes how
decision-makers acquire information while considering associated costs,
enabling investigations into cognitive impacts on audit decisions.

A game-theoretic approach is commonly employed to capture the threat posed by
followers in dynamic games, such as ultimatum games [32], Stackelberg games
[33], bargaining games [34], as well as contract [35, 36] and incentive
mechanisms designs [37, 38]. Recently, there has been increased interest in
the investigation of evasion behaviors [39]. This includes exploiting evasion-
aware detection methods [40] and developing evaders for subsequent tests of
collaborative cognition-assisted detector [41].

##  II Herd Auditors with Epistemic Disparity

In the context of herd-auditing an algorithm, the auditor is uncertain about
the true state $\omega\in\Omega=\\{g,b\\}$, where $g$ indicates the null
hypothesis, implying that the algorithm is consistent with the claim, while
$b$ is for the alternative hypothesis, meaning that the algorithm does not
comply. The prior belief of state $\omega$ can be denoted as $\mu(\omega)$,
implying the auditor’s uncertainty in the algorithm’s compliance.

In order to reduce the uncertainty, the auditor can obtain information $s$
about the state according to the information-obtaining strategy $d(s|\omega)$.
More specifically, $s$ can be viewed as the outcome of the algorithm, and
$d(s|\omega)$ indicates how the auditor accesses (obtains) it. The information
$s$ together with the obtaining strategy leads to a posterior belief of the
state
$\mu(\omega|s)=\frac{\mu(\omega)d(s|\omega)}{\sum_{\omega}\mu(\omega)d(s|\omega)}$.

Based on the information $s$ (correspondingly, the posterior belief
$\mu(\omega|s)$), the auditor can select an element from a finite action set
$a\in\mathcal{A}=\\{T,F\\}$, where $T$ means reporting algorithm compliance,
while $F$ indicates reporting non-compliance. The decision rule
$\delta:\mathcal{S}\mapsto\mathcal{A}$ aims to maximize the expected utility
of $u(\omega,a)$, where $u:\Omega\times\mathcal{A}\mapsto\mathbb{R}$ is the
utility of choosing action $a$ when the state is $\omega$.

Figure 2: An illustration of how the auditor performs audit. The auditor
acquires information $s$ about the unknown state $\omega$ using strategy $d$
and then makes a decision $a$ using strategy $\delta$.

However, the acquisition of information can incur costs, which can be viewed
as the discrepancy between the prior belief $\mu(\omega)$ and the posterior
belief $\mu(\omega|s)$ regarding the state $\omega$. In conventional rational
inattention research, a common method to model the cost is through the lens of
Shannon mutual information. Furthermore, due to variations in epistemic
disparities, the cost incurred for accessing information (i.e., reduction in
uncertainty) differs among auditors. To account for this, we introduce the
concept of an epistemic factor for each auditor, denoted as $\lambda$, which
quantifies the differences in the cost experienced by different auditors when
reducing the same amount of uncertainty. The larger value of $\lambda$ implies
harder access to relevant information, as the cost for the same amount of
uncertainty reduction becomes higher. To this end, the auditor’s objective
becomes

$\max_{d,\delta}\mathbb{E}[u(\omega,a)]-\lambda I(\omega;s),$ (1)

where the expected utility is given by

$\mathbb{E}[u(\omega,a)]=\sum_{\omega}\sum_{a}\mu(\omega)u(\omega,a)\sum_{s:\delta(s)=a}d(s|\omega),$
(2)

and the information cost is expressed as

$I(\omega;s)=\sum_{\omega}\sum_{s}d(s|\omega)\mu(\omega)\ln\frac{d(s|\omega)}{\sum_{\omega}d(s|\omega)\mu(\omega)}.$
(3)

###  II-A Bayes hypothesis testing as the auditor’s decision rule

Conventionally, Bayes hypothesis testing deals with the optimization problem

$\max_{\delta}\mathbb{E}[u(\omega,a)]=\sum_{\omega}\sum_{a}\mu(\omega)u(\omega,a)\sum_{\delta(s)=a}d(s|\omega)$
(4)

with given distributions for both hypotheses $d(s|g)$ and $d(s|b)$ during
decision-making, which coincides with the first term in the auditor’s
objective (1). According to detailed derivation in Appendix -A, the optimal
decision rule can be written as

$\delta^{*}(s)=\begin{cases}T,\
&\frac{\mu(b)d(s|b)}{\mu(g)d(s|g)}<\frac{u(g,T)-u(g,F)}{u(b,F)-u(b,T)},\\\ F,\
&\frac{\mu(b)d(s|b)}{\mu(g)d(s|g)}>\frac{u(g,T)-u(g,F)}{u(b,F)-u(b,T)},\\\
\\{T,F\\},&\frac{\mu(b)d(s|b)}{\mu(g)d(s|g)}=\frac{u(g,T)-u(g,F)}{u(b,F)-u(b,T)},\end{cases}$
(5)

which leads us to a threshold decision rule and can be viewed as making a
decision based on the posteriors. We represent the optimal decision rule with
given $d(s|g)$ and $d(s|b)$ as $\delta^{*}_{d}(s)$, and denote the information
set partitioned by $\delta^{*}_{d}(s)$ as

$\begin{cases}S_{d,T}=\\{s:\delta^{*}_{d}(s)=T\\},\\\
S_{d,F}=\\{s:\delta^{*}_{d}(s)=F\\}.\end{cases}$ (6)

###  II-B Auditor’s choice of the information strategy

With the optimal decision rule $\delta^{*}_{d}$, the auditor’s objective:

$\displaystyle\max_{d,\delta}$ $\displaystyle\mathbb{E}[u(\omega,a)]-\lambda
I(\omega;s),\ \text{with}\ \delta=\delta^{*}_{d},$ (7)

which leads to the constrained optimization problem

$\displaystyle\max_{d}$
$\displaystyle\sum_{\omega}\sum_{a}\mu(\omega)u(\omega,a)\sum_{s:\delta^{*}_{d}(s)=a}d(s|\omega)$
(8)
$\displaystyle-\lambda\sum_{\omega}\sum_{s}d(s|\omega)\mu(\omega)\ln\frac{d(s|\omega)}{\sum_{\omega}d(s|\omega)\mu(\omega)},$
s.t. $\displaystyle\sum_{s}d(s|\omega)=1,d(s|\omega)\geq 0,\forall
s\in\mathcal{S},\forall\omega\in\Omega.$

With detailed derivations in Appendix -B we arrive at:

$\displaystyle
d(s|g)=\begin{cases}\frac{v(s)\exp(\frac{u(g,T)}{\lambda})}{y^{\prime}(g)},\
s\in S_{d,T},\\\ \frac{v(s)\exp(\frac{u(g,F)}{\lambda})}{y^{\prime}(g)},\ s\in
S_{d,F},\end{cases}$ (9) $\displaystyle
d(s|b)=\begin{cases}\frac{v(s)\exp(\frac{u(b,T)}{\lambda})}{y^{\prime}(b)},\
s\in S_{d,T},\\\ \frac{v(s)\exp(\frac{u(b,F)}{\lambda})}{y^{\prime}(b)},\ s\in
S_{d,F}.\end{cases}$ (10)

The corresponding posterior belief
$\mu(g|s)=\frac{\mu(g)d(s|g)}{\sum_{\omega}\mu(\omega)d(s|\omega)}=\frac{\mu(g)d(s|g)}{v(s)}$
can then be written as

$\displaystyle\mu(g|s)=\begin{cases}\frac{\mu(g)\exp(u(g,T)/\lambda)}{y^{\prime}(g)},\
s\in S_{d,T},\\\ \frac{\mu(g)\exp(u(g,F)/\lambda)}{y^{\prime}(g)},\ s\in
S_{d,F},\end{cases}$ (11)

Note that the $s\in S_{d,T}$ case can be viewed as the posterior belief
$\mu(g|s)$ given $s$ that results in an action $a=T$ (i.e.,
$\mu(g|s)=\mu(g|T),\ s\in S_{d,T}$), while the $s\in S_{d,F}$ case can be
viewed as the posterior belief $\mu(g|s)$ given $s$ that results in an action
$a=F$ (i.e., $\mu(g|s)=\mu(g|F),\ s\in S_{d,F}$). A similar expression can be
found for $\mu(b|s)$.

$\displaystyle\mu(b|s)=\begin{cases}\frac{\mu(b)\exp(u(b,T)/\lambda)}{y^{\prime}(b)},\
s\in S_{d,T},\\\ \frac{\mu(b)\exp(u(b,F)/\lambda)}{y^{\prime}(b)},\ s\in
S_{d,F},\end{cases}$ (12)

where $y^{\prime}(g)$ and $y^{\prime}(b)$ are corresponding normalization
terms.

######  Remark 1.

For an auditor with epistemic factor $\lambda$, the information-obtaining
strategy represented by the conditional probability $d(s|\omega)$ is chosen if
its resulting posterior belief $\mu(\omega|s)$ maximizes the value of
$\mathbb{E}[u(\omega,a)]-\lambda I(\omega;s)$.

The $\mu(g|s),\forall s\in S_{d,T}$, can also be interpreted as the audit
confidence for making the decision $a=T$ when observing the information $s$.
Since $u(g,T)>u(g,F)$, it is evident that auditors with a smaller epistemic
factor $\lambda$ have higher confidence in the audit process. This implies
that auditors who can easily access relative information are more likely to
perform better in the audit.

##  III Stackelberg Herd Audit Game

To examine the impact of herd audit on the developer’s incentive to behave
irresponsibly, we formulate the interplay between the herd auditor (she) and
the algorithm developer (he) as a Stackelberg herd audit game, depicted in
Fig. 3.

###  III-A Connection to differential privacy

We begin with the definition of $\epsilon$-differential privacy.

######  Definition 1 ($\epsilon$-DP).

A (randomized) mechanism $M:\mathcal{D}\mapsto\mathcal{B}$ is
$\epsilon$-differentially private ($\epsilon$-DP) if for every pair of
neighboring inputs $D_{1},D_{2}\in\mathcal{D}$, and for every (measurable)
output set $B\in\mathcal{B}$, the probabilities of events
$M(D_{1};F,\epsilon)\in B$ and $M(D_{2};F,\epsilon)\in B$ are closer than a
factor of $e^{\epsilon}$:

$Pr(M(D_{1};F,\epsilon)\in B)\leq e^{\epsilon}\cdot Pr(M(D_{2};F,\epsilon)\in
B).$ (13)

In the context of differential privacy, consider a scenario in which there is
a public-known privacy protection agreement that requires $\epsilon^{\prime}$
privacy budget. However, since more privacy budget (which means decreasing the
privacy protection and making the results more distinguishable) often leads to
better algorithm accuracy, the algorithm developer has the incentive to use
some $\epsilon>\epsilon^{\prime}$ when performing the algorithm, which creates
irresponsibility. Hence, we consider the state $\omega=g$ means
$\epsilon=\epsilon^{\prime}$ and the state $\omega=b$ means
$\epsilon>\epsilon^{\prime}$. Since privacy protection is often achieved by
adding noise, it is assumed that for an algorithm $M$ with input dataset $D$,
the privacy budget $\epsilon$ results in an output distribution
$p(M(D)|\epsilon)$ for later usage.

Figure 3: The Stackelberg herd audit game.

###  III-B Problem Setting for the Developer

Consider two types of algorithm developers $g$ and $b$, and they play a mixed
strategy for executing $\epsilon$, which are $q(\epsilon|g)$ and
$q(\epsilon|b)$, respectively (for discrete choices of
$\epsilon\in\mathcal{E}$). Each $\epsilon$ results in an algorithm accuracy
$A(\epsilon)$, where $A:\mathcal{E}\mapsto\mathbb{R}$, under the assumption
that a larger $\epsilon$ leads to better accuracy.

######  Assumption 1.

Given algorithm $M$ and input set $D$, a privacy budget $\epsilon$ leads to a
unique output distribution $p(M(D)|\epsilon)$.

######  Assumption 2.

For a given algorithm, the algorithmic accuracy under the privacy budget
$\epsilon\in\mathcal{E}\subseteq\mathbb{R}$ is governed by
$A:\mathcal{E}\mapsto\mathbb{R}$, and it is increasing in
$\epsilon\in\mathcal{E}$.

In this context, the developer’s strategy $q(\epsilon|\omega)$ given his type
$\omega$ will lead to the distributions for the two hypotheses

$\displaystyle Q_{g}(s)$
$\displaystyle=\sum\nolimits_{\epsilon}p(s|\epsilon)q(\epsilon|g),$ (14)
$\displaystyle Q_{b}(s)$
$\displaystyle=\sum\nolimits_{\epsilon}p(s|\epsilon)q(\epsilon|b),$ (15)

where $p(s|\epsilon)$ is the distribution $p(M(D)|\epsilon)$ in Assumption 1.

####  III-B1 Responsible developer

For a responsible algorithm developer, the mixed strategy $q(\epsilon|g)$
should have mass $0$ for $\epsilon>\epsilon^{\prime}$, which means that he
always provides privacy protection at least complies with the agreement.
Moreover, in order to maximize $A(\epsilon)$, a responsible algorithm
developer tends to put all the mass on $\epsilon=\epsilon^{\prime}$ since
$A(\epsilon)<A(\epsilon^{\prime}),\forall\epsilon<\epsilon^{\prime}$.

######  Proposition 1 (Responsible Developer’s Strategy).

A responsible developer’s mixed strategy reduces to a pure strategy by letting
all the mass on $\epsilon=\epsilon^{\prime}$. Hence,
$Q_{g}(s)=\sum_{\epsilon}p(s|\epsilon)q(\epsilon|g)=p(s|\epsilon^{\prime})$.

####  III-B2 Irresponsible developer

However, it is important to consider various scenarios involving an
irresponsible algorithm developer who prioritizes algorithm performance and
disregards compliance with the agreement. If there is no auditor or no penalty
imposed when the developer fails to pass the audit (i.e., when the auditor
determines that $a=F$), the irresponsible developer can choose an extremely
large value for $\epsilon$. Consequently, it is reasonable to assume that a
penalty will be enforced if the irresponsible developer is detected. In such a
situation, the irresponsible developer may attempt to maximize the probability
of avoiding penalties, which is the probability of the auditor deciding $a=T$.

######  Assumption 3.

The irresponsible algorithm developer’s mixed strategy will not put any mass
on $\epsilon=\epsilon^{\prime}$. That is, $q(\epsilon^{\prime}|b)=0$.

###  III-C Revisiting the Auditor’s Problem

Considering that the penalty term for the irresponsible developer is
influenced by the actions of the auditor, in terms of whether the
irresponsible developer is caught or not, it is necessary to reexamine the
problem from the auditor’s perspective when the developer is also a strategic
player aiming to evade the audit. We reformulate the auditor’s problem by
letting $u(g,T)=u(b,F)=0$ and setting the penalty terms $u(b,T)$ and $u(g,F)$
to negative values. However, within the context of DP, it is important to note
that the distributions for these hypotheses are predefined by the output
distribution $p(s|\epsilon)$ and the developer’s mixed strategy given the
observed information $s$. The audit confidences are analogous to those
provided in (11) and (12).

######  Assumption 4.

Assume that $u(b,T)<0$ and $u(g,F)<0$ are the negative utilities for making
wrong audit decisions.

Given the distributions for the two hypotheses $Q_{g}(s)$ and $Q_{b}(s)$, the
auditor aims to achieve the following:

$\displaystyle\max_{r}$ $\displaystyle
u(g,F)\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|g)\bigg{]}r(b|s)$
(16)
$\displaystyle+u(b,T)\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|b)\bigg{]}r(g|s)$
$\displaystyle-\lambda\
\mathbb{E}_{s}\left[D_{kL}(r(\omega|s)\|\mu(\omega))\right],$
$\displaystyle\text{s.t.}\sum_{\omega}$ $\displaystyle
r(\omega|s)=1,r(w|s)\geq 0,\forall w\in\\{g,b\\},\forall s\in\mathcal{S},$

where the first two terms put negative weights on the audit error, and the
last term quantifies the expected reduction in uncertainty for the state
$\omega$, measured in terms of the Kullback–Leibler (KL) divergence:

$\lambda\sum_{s}\bigg{[}Q_{g}(s)+Q_{b}(s)\bigg{]}\sum_{\omega}r(\omega|s)\log\frac{r(\omega|s)}{\mu(\omega)}.$

The decision of $r(g|s)$, $r(b|s)$ already incorporate the auditor’s
information strategy $d(s|\omega)$ since
$r(\omega|s)=\frac{\mu(\omega)d(s|\omega)}{\sum_{\omega}\mu(\omega)d(s|\omega)}$.

###  III-D Revisit the Irresponsible Developer’s Problem

Until now, the irresponsible developer’s objective has become the following.

$\displaystyle\max_{q(\cdot|b)}$
$\displaystyle\sum_{\epsilon}q(\epsilon|b)A(\epsilon)+\beta\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|b)\bigg{]}r(g|s),$
(17)

with $r(g|s)$ comes from the auditor’s problem. The former term is the
expected algorithm accuracy, and the latter term corresponds to the false
negative rate of the auditor’s decision, which is the rate of the
irresponsible developer successfully passing the audit (and thus, the
irresponsible developer seeks to maximize it). Note that $\beta>0$ indicates
the irresponsible developer’s preference for the two goals.

##  IV Equilibrium Analysis

For illustrative purposes, we work through an example where $|\mathcal{E}|=3$
in Appendix -C. Besides, we assume that the distinguishability—quantified by
distance measures such as the Kullback–Leibler divergence—between the output
distributions $p(\cdot|\epsilon)$ and $p(\cdot|\epsilon^{\prime})$ increases
when the difference between $\epsilon$ and $\epsilon^{\prime}$ expands.

###  IV-A The auditor’s optimal strategy

With the example in Appendix -C and derivations in Appendix -D, the auditor’s
$r(g|s)$ and $r(b|s)$ that optimally solves problem (16) can be written as:

$r(g|s)=\mu(g)\exp\left(u(b,T)Q_{b}(s)/\lambda v(s)\right)/y^{\prime}(s),$
(18) $r(b|s)=\mu(b)\exp\left(u(g,F)Q_{g}(s)/\lambda
v(s)\right)/y^{\prime}(s),$ (19)

and $y^{\prime}(s)=\mu(g)\exp\big{(}\frac{u(b,T)Q_{b}(s)}{\lambda
v(s)}\big{)}+\mu(b)\exp\big{(}\frac{u(g,F)Q_{g}(s)}{\lambda v(s)}\big{)}$
denotes the normalization term. We can observe that $r(g|s)$ and $r(b|s)$
share a similar form as (11) and (12).

######  Proposition 2.

The strategy specified by (18) and (19) is optimal for the auditor with
epistemic factor $\lambda$.

######  Remark 2.

The results coincide with the intuition. We first take a look at $r(g|s)$.
Recall that $\frac{u(b,T)}{\lambda}$ is negative. If the penalty term $u(b,T)$
is the same across all the auditors, the auditor with a larger epistemic
factor $\lambda$ achieves $r(g|s)$ that is closer to $\mu(g)$. Combining with
the auditor’s objective in the maximization problem (16), it means that the
larger-$\lambda$ auditor might have a larger false negative error. Similarly,
for $r(b|s)$, the larger-$\lambda$ auditor might have a larger false positive
error.

###  IV-B The irresponsible developer’s optimal strategy

The irresponsible developer’s problem (17) is organized into

$\displaystyle\sum_{\epsilon}$ $\displaystyle q(\epsilon|b)A(\epsilon)+\beta\
\sum_{s}\left[\sum_{\epsilon}p(s|\epsilon)q(\epsilon|b)\right]r(g|s)$ (20)
$\displaystyle=\sum\limits_{\epsilon_{i}\in\mathcal{E}}q(\epsilon_{i}|b)\bigg{[}A(\epsilon_{i})+\beta\sum_{s}r(g|s)p(s|\epsilon_{i})\bigg{]}.$

The irresponsible developer determines his optimal pure strategy $\epsilon$ to
maximize (20). Specifically, the irresponsible developer assigns
$q(\epsilon|b)=1$ to the $\epsilon$ that achieves the largest
$\big{[}A(\epsilon)+\beta\sum_{s}r(g|s)p(s|\epsilon)\big{]}$. This leads us to
the following propositions and remarks.

######  Proposition 3.

The irresponsible developer’s optimal strategy is choosing the $\epsilon$ that
maximizes $\big{[}A(\epsilon)+\beta\sum_{s}r(g|s)p(s|\epsilon)\big{]}$.

######  Proposition 4.

If algorithm accuracy $A(\epsilon)$ is increasing in $\epsilon$, the
irresponsible developer always chooses the largest $\epsilon$ if
$r(g|s)=r(g|s^{\prime}),\forall s,s^{\prime}\in\mathcal{S}$.

###### Proof.

We sketch the proof in Appendix -E. ∎

######  Remark 3.

The irresponsible developer violates as much as possible when the epistemic
factor for the auditor $\lambda=\infty$.

######  Remark 4.

When the auditor’s epistemic factor $\lambda$ is small, indicating easy access
to relevant information, an irresponsible developer is more likely to violate
the claim.

######  Remark 5.

If the auditor’s epistemic factor $\lambda$ is large, it is likely that an
irresponsible developer with a larger $\beta$ (placing more value on the
success rate of passing audits) will also tend to violate the claim more
severely.

###  IV-C Auditor’s audit confidence and epistemic factor

With respect to Fig. 4, the optimal solution to the auditor’s problem given by
(18) and (19) establishes a relationship between the epistemic factor
$\lambda$ and the auditor’s confidence $r(\cdot|s)$ under fixed utilities
$u(\omega,a)$. Let $\chi(s)=[u(g,F)Q_{g}(s)-u(b,T)Q_{b}(s)]/v(s)$ and
$\phi(s)=[u(g,F)Q_{g}(s)+u(b,T)Q_{b}(s)]/(\lambda v(s))$ . Taking the partial
derivative of $r(g|s)$ with respect to $\lambda$ yields $\frac{\partial
r(g|s)}{\partial\lambda}=\frac{\mu(g)\mu(b)\chi(s)\exp\left(\phi(s)\right)}{\lambda^{2}y^{\prime}(s)^{2}}.$
Here, if the developer is irresponsible, then he never chooses a privacy
budget $\epsilon$ that is equal to the claimed budget $\epsilon_{0}$. Hence,
$\chi(s)\neq 0$. The term $\partial r(g|s)/\partial\lambda$ is (strictly)
positive if $\chi(s)>0$ and (strictly) negative otherwise. When $\chi(s)<0$,
$r(g|s)$ is close to $1$ when $\lambda$ goes close to $0$. Furthermore, the
audit confidences for $g$ and $b$ become close to $0.5$ when $\lambda$
increases, which reveals that higher $\lambda$ leads to a weaker incentive to
acquire more accurate information, thereby inducing lower audit confidences.

Similarly, $\frac{\partial
r(b|s)}{\partial\lambda}=\frac{-\mu(g)\mu(b)\chi(s)\exp\left(\phi(s)\right)}{\lambda^{2}y^{\prime}(s)^{2}}.$
The term $\partial r(b|s)/\partial\lambda$ is positive if $-\chi(s)>0$ and
negative otherwise. When $-\chi(s)<0$, $r(b|s)$ is close to $1$ when $\lambda$
goes close to $0$. Furthermore, the audit confidences for $g$ and $b$ become
closer to $0.5$ when $\lambda$ increases, which coincides with the setting
that higher $\lambda$ leads to a weaker incentive to acquire more accurate
information, thereby inducing lower audit confidences.

(a)

(b)

Figure 4: Trends between the auditor’s confidence (posterior belief) and the
auditor’s epistemic factor ($\mu(g)=\mu(b)=0.5,u(b,T)=u(g,F)=-1$). Left:
$\chi(s)<0$ ($Q_{b}(s)/v(s)=0.25$). Right: $-\chi(s)<0$
($Q_{b}(s)/v(s)=0.85$).

Note that audit confidence is determined by optimizing the objective, which
consists of penalties for audit errors and costs associated with information
acquisition. In this context, it is important to carefully select reasonable
intervals for $u(\omega,a)$ and $\lambda$. In practice, as auditors are end-
users for the algorithm, and given the disparities in end-users across
different algorithms, the range for the epistemic factor needs to be
contingent upon the ease with which corresponding end-users of the algorithm
can access relevant information.

###  IV-D Irresponsible developer’s choice and auditor’s confidence

According to (14) and (15), the irresponsible developer’s budget choice
determines $Q_{b}(\cdot)$ given $p(\cdot)$. Hence, (18) and (19) (shown in
Fig. 5) also establish a relationship between the irresponsible developer’s
choice and the auditor’s confidence.

Figure 5: The trend between the auditor’s confidence and the irresponsible
developer’s choice that leads to $Q_{b}(s)/v(s)$. Here,
$\mu(g)=\mu(b)=0.5,u(b,T)=u(g,F)=-1,\lambda=1$.

By taking partial derivative of $r(g|s)$ with respect to
$\frac{Q_{b}(s)}{v(s)}$, we obtain $\frac{\partial r(g|s)}{\partial
Q_{b}(s)/v(s)}=\frac{\mu(g)\mu(b)(u(g,F)+u(b,T))\exp\left(\phi(s)\right)}{\lambda
y^{\prime}(s)^{2}},$ which is negative since $u(g,F)+u(b,T)<0$. Additionally,
as the value of $\lambda$ increases (when auditors incur higher costs for
information acquisition), the magnitude of $\partial
r(g|s)/\partial(\frac{Q_{b}(s)}{v(s)})$ decreases, implying relatively less
influence on audit confidence. This trend is evident in Figure 5, where a
greater $\lambda$ corresponds to a flatter curve for $r(g|s)$.

##  V Discussion and Conclusions

Herd audit is a collective mechanism that empowers users to hold algorithm
developers accountable, fostering the development of compliant and responsible
digital products for the betterment of society. In this study, we examine herd
audit through a game-theoretic lens, capturing the interactions between an
idiosyncratic user and a privacy-preserving algorithm developer. Our framework
adopts a Stackelberg game approach, enabling us to assess the impact of herd
audit on responsible algorithm design and understand selfish and irresponsible
strategies in worst-case scenarios.

We have specifically explored the presence of auditors with varying cognitive
and reasoning capabilities, capturing epistemic disparities. Within our game-
theoretic framework, we have consolidated the concept of rational inattention.
The optimal strategy for auditors underscores the importance of easy access to
relevant information, which enhances their confidence in the herd-audit
process. Similarly, the optimal decision for algorithm developers has revealed
that herd audit is a viable approach when auditors face lower costs in
accessing knowledge, as denoted by smaller epistemic factors. Based on our
findings, we conclude that herd audit poses a credit threat to developers and
plays a vital role in promoting the responsible development of privacy-
preserving algorithms. In future work, we aim to enrich the game-theoretic
framework by incorporating end-users’ incentives. This extension allows us to
design an incentive mechanism that encourages participation in herd audits.
Additionally, we plan to explore the fusion of distributed audits alongside a
central audit center. Leveraging tools from decentralized hypothesis testing,
game theory, information theory, and differential privacy, this research
direction holds promise for advancing the field further.

## References

  * [1] J. Guszcza, I. Rahwan, W. Bible, M. Cebrian, and V. Katyal, “Why we need to audit algorithms,” 2018. [Online]. Available: https://hdl.handle.net/21.11116/0000-0003-1C9E-D
  * [2] C. Dwork, “Differential privacy: A survey of results,” in _Theory and Applications of Models of Computation: 5th International Conference, TAMC 2008, Xi’an, China, April 25-29, 2008. Proceedings 5_. Springer, 2008, pp. 1–19.
  * [3] Z. Ding, Y. Wang, G. Wang, D. Zhang, and D. Kifer, “Detecting violations of differential privacy,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 475–489.
  * [4] B. Bichsel, T. Gehr, D. Drachsler-Cohen, P. Tsankov, and M. Vechev, “Dp-finder: Finding differential privacy violations by sampling and optimization,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 508–524.
  * [5] Y. Han and S. Martínez, “A numerical verification framework for differential privacy in estimation,” _IEEE Control Systems Letters_ , vol. 6, pp. 1712–1717, 2021.
  * [6] J. Bandy, “Problematic machine behavior: A systematic literature review of algorithm audits,” _Proceedings of the acm on human-computer interaction_ , vol. 5, no. CSCW1, pp. 1–34, 2021.
  * [7] B. Mittelstadt, “Automation, algorithms, and politics— auditing for transparency in content personalization systems,” _International Journal of Communication_ , vol. 10, p. 12, 2016.
  * [8] J. M. Leimeister, “Collective intelligence,” _Business & Information Systems Engineering_, vol. 2, pp. 245–248, 2010.
  * [9] M. Fricker, _Epistemic injustice: Power and the ethics of knowing_. Oxford University Press, 2007.
  * [10] H. Grasswick, “Epistemic injustice in science,” in _The Routledge handbook of epistemic injustice_. Routledge, 2017, pp. 313–323.
  * [11] M. H. Manshaei, Q. Zhu, T. Alpcan, T. Bacşar, and J.-P. Hubaux, “Game theory meets network security and privacy,” _ACM Computing Surveys (CSUR)_ , vol. 45, no. 3, pp. 1–39, 2013.
  * [12] F. Fang, S. Liu, A. Basak, Q. Zhu, C. D. Kiekintveld, and C. A. Kamhoua, “Introduction to game theory,” _Game Theory and Machine Learning for Cyber Security_ , pp. 21–46, 2021.
  * [13] F. Matějka and A. McKay, “Rational inattention to discrete choices: A new foundation for the multinomial logit model,” _American Economic Review_ , vol. 105, no. 1, pp. 272–298, 2015.
  * [14] A. Caplin and M. Dean, “Revealed preference, rational inattention, and costly information acquisition,” _American Economic Review_ , vol. 105, no. 7, pp. 2183–2203, July 2015.
  * [15] F. Restuccia, N. Ghosh, S. Bhattacharjee, S. K. Das, and T. Melodia, “Quality of information in mobile crowdsensing: Survey and research challenges,” _ACM Transactions on Sensor Networks (TOSN)_ , vol. 13, no. 4, pp. 1–43, 2017.
  * [16] Y. Zhao and Q. Zhu, “Evaluation on crowdsourcing research: Current status and future direction,” _Information systems frontiers_ , vol. 16, pp. 417–434, 2014.
  * [17] J. Pawlick and Q. Zhu, “Active crowd defense,” _Game Theory for Cyber Deception: From Theory to Applications_ , pp. 147–167, 2021.
  * [18] H. Frye, “The technology of public shaming,” _Social Philosophy and Policy_ , vol. 38, no. 2, pp. 128–145, 2021.
  * [19] H. Yu, C. Miao, C. Leung, Y. Chen, S. Fauvel, V. R. Lesser, and Q. Yang, “Mitigating herding in hierarchical crowdsourcing networks,” _Scientific reports_ , vol. 6, no. 1, p. 4, 2016.
  * [20] I. Comeig, E. Mesa-Vázquez, P. Sendra-Pons, and A. Urbano, “Rational herding in reward-based crowdfunding: An mturk experiment,” _Sustainability_ , vol. 12, no. 23, p. 9827, 2020.
  * [21] C. Eickhoff, “Cognitive biases in crowdsourcing,” in _Proceedings of the eleventh ACM international conference on web search and data mining_ , 2018, pp. 162–170.
  * [22] R. R. Morris, M. Dontcheva, and E. M. Gerber, “Priming for better performance in microtask crowdsourcing environments,” _IEEE Internet Computing_ , vol. 16, no. 5, pp. 13–19, 2012.
  * [23] D. R. Karger, S. Oh, and D. Shah, “Budget-optimal task allocation for reliable crowdsourcing systems,” _Operations Research_ , vol. 62, no. 1, pp. 1–24, 2014.
  * [24] K. Wang, X. Qi, L. Shu, D.-j. Deng, and J. J. Rodrigues, “Toward trustworthy crowdsourcing in the social internet of things,” _IEEE Wireless Communications_ , vol. 23, no. 5, pp. 30–36, 2016.
  * [25] M. Allahbakhsh, A. Ignjatovic, B. Benatallah, E. Bertino, N. Foo _et al._ , “Reputation management in crowdsourcing systems,” in _8th International conference on collaborative computing: networking, applications and worksharing (CollaborateCom)_. IEEE, 2012, pp. 664–671.
  * [26] Y. Yu, S. Liu, L. Guo, P. L. Yeoh, B. Vucetic, and Y. Li, “Crowdr-fbc: A distributed fog-blockchains for mobile crowdsourcing reputation management,” _IEEE Internet of Things Journal_ , vol. 7, no. 9, pp. 8722–8735, 2020.
  * [27] A. A. González-Prendes and S. M. Resko, “Cognitive-behavioral theory,” 2012.
  * [28] D. Fum, F. Del Missier, A. Stocco _et al._ , “The cognitive modeling of human behavior: Why a model is (sometimes) better than 10,000 words,” _Cognitive Systems Research_ , vol. 8, no. 3, pp. 135–142, 2007.
  * [29] L. Huang and Q. Zhu, _Cognitive Security: A System-Scientific Approach_. Springer Nature, 2023.
  * [30] J. R. Anderson, D. Bothell, M. D. Byrne, S. Douglass, C. Lebiere, and Y. Qin, “An integrated theory of the mind.” _Psychological review_ , vol. 111, no. 4, p. 1036, 2004.
  * [31] C. A. Sims, “Implications of rational inattention,” _Journal of monetary Economics_ , vol. 50, no. 3, pp. 665–690, 2003.
  * [32] S. Rajtmajer, A. Squicciarini, J. M. Such, J. Semonsen, and A. Belmonte, “An ultimatum game model for the evolution of privacy in jointly managed content,” in _Decision and Game Theory for Security: 8th International Conference, GameSec 2017, Vienna, Austria, October 23-25, 2017, Proceedings_. Springer, 2017, pp. 112–130.
  * [33] C. Casorrán, B. Fortz, M. Labbé, and F. Ordóñez, “A study of general and security stackelberg game formulations,” _European journal of operational research_ , vol. 278, no. 3, pp. 855–868, 2019.
  * [34] D. Guerrero, A. A. Carsteanu, and J. B. Clempner, “Solving stackelberg security markov games employing the bargaining nash approach: Convergence analysis,” _Computers & Security_, vol. 74, pp. 240–257, 2018.
  * [35] J. Chen and Q. Zhu, “Optimal contract design under asymmetric information for cloud-enabled internet of controlled things,” in _Decision and Game Theory for Security: 7th International Conference, GameSec 2016, New York, NY, USA, November 2-4, 2016, Proceedings_. Springer, 2016, pp. 329–348.
  * [36] R. Zhang and Q. Zhu, “Flipin: A game-theoretic cyber insurance framework for incentive-compatible cyber risk management of internet of things,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 2026–2041, 2019.
  * [37] Q. Zhu, C. Fung, R. Boutaba, and T. Basar, “Guidex: A game-theoretic incentive-based mechanism for intrusion detection networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 30, no. 11, pp. 2220–2230, 2012.
  * [38] L. Huang and Q. Zhu, “Duplicity games for deception design with an application to insider threat mitigation,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 4843–4856, 2021.
  * [39] K. Horák, Q. Zhu, and B. Bošanskỳ, “Manipulating adversary’s belief: A dynamic game approach to deception by design for proactive network security,” in _Decision and Game Theory for Security: 8th International Conference, GameSec 2017, Vienna, Austria, October 23-25, 2017, Proceedings_. Springer, 2017, pp. 273–294.
  * [40] Y. Hu and Q. Zhu, “Evasion-aware neyman-pearson detectors: A game-theoretic approach,” in _2022 IEEE 61st Conference on Decision and Control (CDC)_ , 2022, pp. 6111–6117.
  * [41] S. N. Narayanan, A. Ganesan, K. Joshi, T. Oates, A. Joshi, and T. Finin, “Early detection of cybersecurity threats using collaborative cognition,” in _2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC)_ , 2018, pp. 354–363.

###  -A Proof of optimal decision rule $\delta^{*}(s)$

For the null hypothesis, $d(s|g)$, and the alternative hypothesis, $d(s|b)$,
the expected utility for problem $HT$ in (LABEL:eq:4) can be reformulated as

$\displaystyle\mathbb{E}[u(\omega,a)]$
$\displaystyle=\mu(g)u(g,F)+\mu(b)u(b,F)$
$\displaystyle+\sum_{\delta(s)=T}\bigg{\\{}\mu(g)\big{[}u(g,T)-u(g,F)\big{]}d(s|g)$
$\displaystyle-\mu(b)\big{[}u(b,F)-u(b,T)\big{]}d(s|b)\bigg{\\}}.$

Therefore, to maximize the expected utility, the auditor must decide
$\delta(s)=T$ if
$\mu(g)\big{[}u(g,T)-u(g,F)\big{]}d(s|g)>\mu(b)\big{[}u(b,F)-u(b,T)\big{]}d(s|b)$.
This completes the proof.

###  -B Proof of optimal information strategy $d$

To analyze the problem, we use the method of Lagrange multipliers and denote

$\displaystyle
J(d,y)=\sum_{\omega}\sum_{a}\mu(\omega)u(\omega,a)\sum_{s:\delta^{*}_{d}(s)=a}d(s|\omega)$
$\displaystyle-\lambda\sum_{\omega}\sum_{s}d(s|\omega)\mu(\omega)\ln\frac{d(s|\omega)}{\sum_{\omega}d(s|\omega)\mu(\omega)}-\sum_{\omega}y(\omega)d(s|\omega),$

with the last term corresponding to the constraint that $d(s|\omega)$ should
be a conditional probability mass function.

Then, for $d(s|g)$ with $s\in S_{d,T}$, according to the first-order and the
second-order condition,

$\displaystyle\frac{\partial J(d,y)}{\partial
d(s|g)}=\mu(g)u(g,T)-\lambda\mu(g)\log(\frac{d(s|g)}{v(s)})-y(g)=0,$
$\displaystyle\frac{\partial^{2}J(d,y)}{\partial
d(s|g)^{2}}=-\lambda\mu(g)\frac{\mu(b)d(s|b)}{d(s|g)v(s)}\leq 0,$

where $v(s)=\sum_{\omega}d(s|\omega)\mu(\omega)$. Letting
$\log(y^{\prime}(g))=\frac{y(g)}{\lambda\mu(g)}$ leads to the following
$d(s|g)$ that maximizes (8).

$\displaystyle\lambda\mu(g)\bigg{[}\frac{u(g,T)}{\lambda}-\log(\frac{d(s|g)}{v(s)})-\log
y^{\prime}(g)\bigg{]}=0,$ $\displaystyle
d(s|g)=\frac{v(s)\exp(\frac{u(g,T)}{\lambda})}{y^{\prime}(g)}.$

Note that $y^{\prime}(g)$ is the normalization term. Similarly, for $d(s|g)$
with $s\in S_{d,F}$ and $d(s|b)$, we can get the information-obtaining
strategy in (9) and (10)

###  -C An illustrative example for equilibrium analysis

We consider a scenario where the cardinality of the set $\mathcal{E}$ is
three; i.e., $|\mathcal{E}|=3$ with
$\mathcal{E}=\\{\epsilon_{l},\epsilon_{m},\epsilon_{h}\\}$, where
$\epsilon_{l}<\epsilon_{m}<\epsilon_{h}$ and it’s assumed that the claimed
differential privacy budget is $\epsilon^{\prime}=\epsilon_{l}$. Then, the two
hypotheses become

$Q_{g}(s)=\sum_{\epsilon}p(s|\epsilon)q(\epsilon|g)=p(s|\epsilon^{\prime})=p(s|\epsilon_{l}),$
$Q_{b}(s)=\sum_{\epsilon}p(s|\epsilon)q(\epsilon|b)=p(s|\epsilon_{m})q(\epsilon_{m}|b)+p(s|\epsilon_{h})q(\epsilon_{h}|b).$

According to derivations in Appendix -D, the strategy specified by (18) and
(19) is optimal for the auditor with epistemic factor $\lambda$.

We then shift our focus to the irresponsible developer’s strategy. The
irresponsible developer endeavors to enhance algorithmic accuracy while
concurrently maximizing the probability of evading detection by the auditor,
thereby increasing the likelihood of being perceived as a responsible
developer. Hence, the irresponsible developer’s decision-making can be
described by the following optimization problem:

$\displaystyle\max_{q(\cdot|b)}$
$\displaystyle\big{[}q(\epsilon_{m}|b)A(\epsilon_{m})+q(\epsilon_{h}|b)A(\epsilon_{h})\big{]}$
(21) $\displaystyle+\beta\
\sum_{s}\bigg{[}p(s|\epsilon_{m})q(\epsilon_{m}|b)+p(s|\epsilon_{h})q(\epsilon_{h}|b)\bigg{]}r(g|s).$

By leveraging $q(\epsilon_{m}|b)=1-q(\epsilon_{h}|b)$, we rewrite the problem
(21) as follows:

$\displaystyle\max_{q(\epsilon_{l}|b)}$ $\displaystyle A(\epsilon_{h})+\beta\
\sum_{s}r(g|s)p(s|\epsilon_{h})+\bigg{\\{}\big{[}A(\epsilon_{m})-A(\epsilon_{h})\big{]}$
(22)
$\displaystyle+\beta\sum_{s}r(g|s)\big{[}p(s|\epsilon_{m})-p(s|\epsilon_{h})\big{]}\bigg{\\}}q(\epsilon_{m}|b).$

Since the first two terms $A(\epsilon_{h})+\beta\
\sum_{s}r(g|s)p(s|\epsilon_{h})$ are independent of $q(\cdot|b)$, (22)
suggests the following strategy for the irresponsible developer: let $\Delta
A=A(\epsilon_{m})-A(\epsilon_{h})$,

$\begin{cases}q(\epsilon_{m}|b)=1,&\Delta
A+\beta\sum_{s}r(g|s)\big{[}p(s|\epsilon_{m})-p(s|\epsilon_{h})\big{]}>0\\\
q(\epsilon_{h}|b)=1,&\textup{otherwise.}\end{cases}$

That is, the irresponsible developer has a pure strategy by choosing either
$q(\epsilon_{m}|b)=1$ or $q(\epsilon_{h}|b)=1$.

###  -D Proof of auditor’s strategy $r$

In (16), the KL divergence term with a negative sign is concave with respect
to the decision variables $r(\cdot)$ given fixed priors $\mu(\cdot)$.
Therefore, the combination of the terms in the objective function forms a
weighted sum of concave functions. This makes the overall objective function
concave. Given the linear constraints, the feasibility set is convex. Hence,
the optimization problem (16) is a concave maximization over a convex set.

The Lagrangian corresponding to (16) is then given by

$\displaystyle J(r,y,z)=$ $\displaystyle\
u(g,F)\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|g)\bigg{]}r(b|s)$
(23)
$\displaystyle+u(b,T)\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|b)\bigg{]}r(g|s)$
$\displaystyle-\lambda\
\sum_{s}\bigg{[}\sum_{\epsilon}p(s|\epsilon)q(\epsilon|g)+\sum_{\epsilon}p(s|\varepsilon)q(\epsilon|b)\bigg{]}$
$\displaystyle\sum_{\omega}r(\omega|s)\log\frac{r(\omega|s)}{\mu(\omega)}-yr(g|s)-zr(b|s),$

where $y\in\mathbb{R}$ and $z\in\mathbb{R}$ are the associated Lagrange
multipliers. Then, the first-order condition concerning $r(g|s)$ implies

$\displaystyle\frac{\partial J(r)}{\partial r(g|s)}$
$\displaystyle=u(b,T)\bigg{[}p(s|\epsilon_{m})q(\epsilon_{m}|b)+p(s|\epsilon_{h})q(\epsilon_{h}|b)\bigg{]}$
$\displaystyle-\lambda\bigg{[}p(s|\epsilon_{l})+p(s|\epsilon_{m})q(\epsilon_{m}|b)+p(s|\epsilon_{h})q(\epsilon_{h}|b)\bigg{]}$
$\displaystyle\bigg{(}\log\frac{r(g|s)}{\mu(g)}+1\bigg{)}-y$
$\displaystyle=0.$

Hence, we obtain

$\displaystyle u(b,T)Q_{b}(s)-\lambda
v(s)\bigg{(}\log\frac{r(g|s)}{\mu(g)}+1\bigg{)}-y=0,$
$\displaystyle\frac{u(b,T)Q_{b}(s)}{\lambda v(s)}-\bigg{(}\frac{y}{\lambda
v(s)}+1\bigg{)}=\log\frac{r(g|s)}{\mu(g)}.$

By letting $\log y^{\prime}(s)=\big{(}\frac{y}{\lambda v(s)}+1\big{)}$,
$r(g|s)$ can then be written as (18). We can get $r(b|s)$ described in (19)
with a similar process.

###  -E Proof of Proposition 4

We sketch the proof for $|\mathcal{S}|=2$ with
$\mathcal{S}=\\{s_{1},s_{2}\\}$. In this example,
$p(s_{1}|\epsilon_{m})+p(s_{2}|\epsilon_{m})=1$ and
$p(s_{1}|\epsilon_{h})+p(s_{2}|\epsilon_{h})=1$, then
$r(g|s_{1})[p(s_{1}|\epsilon_{m})-p(s_{1}|\epsilon_{h})]+r(g|s_{2})[p(s_{2}|\epsilon_{m})-p(s_{2}|\epsilon_{h})]=1-1=0$
if $r(g|s_{1})=r(g|s_{2})$.

Hence,
$A(\epsilon_{m})-A(\epsilon_{h})+\sum_{s}r(g|s)\big{[}p(s|\epsilon_{m})-p(s|\epsilon_{h})\big{]}<0$
in the case where $A(\epsilon_{m})<A(\epsilon_{h})$, which leads to
$q(\epsilon_{m}|b)=0$ and $q(\epsilon_{h}|b)=1$. This completes the proof.


# Toxic Bias: Perspective API Misreads German as More Toxic

Gianluca Nogara<EMAIL_ADDRESS>University of Applied Sciences and
Arts of Southern SwitzerlandSwitzerland ,  Francesco Pierri  Politecnico di
MilanoItaly ,  Stefano Cresci  IIT-CNRItaly ,  Luca Luceri  USC Information
Sciences InstituteUSA ,  Petter Törnberg  University of AmsterdamNetherlands
and  Silvia Giordano  University of Applied Sciences and Arts of Southern
SwitzerlandSwitzerland

###### Abstract.

Proprietary public APIs play a crucial and growing role as research tools
among social scientists. Among such APIs, Google’s machine learning-based
Perspective API is extensively utilized for assessing the toxicity of social
media messages, providing both an important resource for researchers and
automatic content moderation. However, this paper exposes an important bias in
Perspective API concerning German language text. Through an in-depth
examination of several datasets, we uncover intrinsic language biases within
the multilingual model of Perspective API. We find that the toxicity
assessment of German content produces significantly higher toxicity levels
than other languages. This finding is robust across various translations,
topics, and data sources, and has significant consequences for both research
and moderation strategies that rely on Perspective API. For instance, we show
that, on average, four times more tweets and users would be moderated when
using the German language compared to their English translation. Our findings
point to broader risks associated with the widespread use of proprietary APIs
within the computational social sciences.

##  1\. Introduction

In recent years, abusive interactions have become a fixture of the online
experience, with approximately 40% of U.S. adults self-reporting as victims of
online abuse (vogels2021state, ). The prevalence of such language on social
media has become a major concern for both researchers and the public, often
under the label of online ”toxicity”. Currently, the leading approach to
detecting toxicity is the Perspective API, offered by the Jigsaw unit of
Google (lees2022new, ), and designed to ”help mitigate toxicity and ensure
healthy dialogue online” (perspectiveHelpMitigate, ). The API is widely used
both within academic research efforts and for content moderation on news and
social media platforms. At the time of this writing, there are over 1,400
documents on Google Scholar mentioning the term ”Perspective API”, and the API
is used by major platforms such as Reddit, The New York Times, El Pais, and
Faceit (perspectiveCaseStudies, ).

Perspective API’s machine learning model is developed through supervised
learning, utilizing a vast dataset of millions of comments gathered from
diverse online platforms, including forums like Wikipedia and The New York
Times, and encompassing more than 20 languages. Perspective API defines toxic
messages as texts that use ”rude, disrespectful, or unreasonable language that
is likely to make someone leave a discussion”
(Gehman2020RealToxicityPromptsEN, ; Welbl2021ChallengesID, ). The API provides
several different toxicity scores: Severe toxicity, Insult, Profanity,
Identity attack, Threat, and Sexually explicit. In a 2019 SemEval Task,
Perspective API has been shown to perform better than other transformer-based
models on canonical datasets for hate speech detection (pavlopoulos-
etal-2019-convai, ). More recent work found that its scores differ from human
labels (Welbl2021ChallengesID, ). The toxicity score, which ranges between 0
and 1, does not have a meaning in absolute terms, and the traditional approach
to identify toxic or hateful users in online social platforms involves
defining a threshold on the toxicity score, which is generally in the region
of 0.5-0.7 (hua2020towards, ; hua2020characterizing, ), above which content is
considered to be toxic.

While technologies such as Perspective API promise a safer, more respectful
digital environment, their effectiveness is contingent on their accuracy and
impartiality across diverse languages and cultural contexts. This paper
identifies a systemic bias within Perspective API when dealing with German
texts, showing a tendency of the tool to assign higher toxicity scores to
German content compared to other languages. By leveraging two large-scale
datasets of multilingual Twitter conversations and random Wikipedia summaries,
we show that toxicity scores for texts in German are consistently
significantly higher than those for any other language, regardless of the
content. Our findings suggest a strong negative bias against the German
language which might be due to artifacts/biases in the training dataset or the
model itself, which is challenging to further investigate given the black-box
nature of the tool. Our findings not only question the reliability of a tool
employed by several large online platforms – potentially leading to unjust
censorship or misrepresentation in online spaces – but also aggravate the
growing reliance on proprietary, black-boxed APIs like Perspective within the
academic realm. These systems present a fundamental problem as researchers and
users are unable to scrutinize or understand the inner workings of the
algorithms, leading to a substantial risk of systemic biases being embedded
and perpetuated through their employment across news outlets, social media,
and various online platforms.

##  2\. Perspective API and toxicity research

Perspective API has become a foundation for academic research on online abuse
and incivility (rieder2021fabrics, ), as well as a relevant case-study for
legal and social scholars interested at the regulatory nuances of socially-
relevant AI systems (friedl2023dis, ). A diverse and rich research field
relies on the capacity of the API to reliably assess these aspects of online
text. Rather than reviewing the diverse contributions that employ Perspective
API to study online toxicity, we will here examine some examples of research
that draws on the API concerning the analysis of German language messages, to
assess the potential implications of the findings of this paper.

There are many examples of research using Perspective API for cross-national
comparison of toxic communication on social media. Rye et al. (rye2020reading,
), for instance, analyze 14 months of data activity on the far-right platform
Dissenter, using Perspective API to measure the toxicity of messages and
derive the propensity of specific content to elicit toxic comments. They find
that ”deutschland.de” is the second most toxic domain, with most angry
comments about the Muslim diaspora in Europe. Similarly, Hoseini et al.
(hoseini2023globalization, ), analyze Telegram messages related to the QAnon
conspiracy shared between September 2019 and March 2021, finding that German
and Portuguese messages are more toxic than those in English.

Even when studies are not explicitly comparative, they often include cross-
language analysis of toxicity, which may potentially be vulnerable to the type
of model biases discussed in this paper. Smirnov et al. (smirnov2023toxic, ),
for instance, study the effect of toxic comments directed to Wikipedia’s
volunteer editors. Analyzing over 57 million comments on the six most active
language editions of Wikipedia (English, German, French, Spanish, Italian, and
Russian), they find that toxic speech could lead to a substantial loss of
productivity among editors. Differently from the majority of other works,
Ghosh Roy et al. (ghosh2020leveraging, ) do not directly use Perspective API
to identify toxic tweets, but rather use toxicity scores from the API as
features in a transformers-based hate speech classifier for English, German,
and Hindi. Also this usage pattern, however, could leak Perspective API’s
biases into the trained classifier.

Many studies focus specifically on German-language debates, seeking to draw
out the dynamics of toxicity within these social spaces. Lasser et al.
(lasser2023collective, ) study more than 1M tweets posted on Twitter from 2015
to 2018 in German, a period characterized by a ”large influx of refugees into
Germany which sparked heated discussions and political polarization”. They
measure the interaction between incivility (hatefulness and toxicity) and
extremity (of speech and speakers) with other dimensions of discourse. For
this purpose, they use Perspective API to measure the overall toxicity score
as the average of five attributes (toxicity, severe toxicity, profanity,
insult, and identity attack). They report very high values of average toxicity
and highlight which behaviors can reduce hate speech. Another example is
Czymara et al.’s analysis (czymara2023catalyst, ) of the eﬀect of Islamist
terror attacks on ethnic insulting comments on YouTube, showing how terror
attacks boost interest in immigration-related topics in general, and lead to a
disproportional increase in hate speech in particular. Focusing on the
”Identity attack” attribute of Perspective API, they classify insulting
comments as those with a score surpassing 0.9. The study reveals varying
proportions of ethnically insulting comments, ranging from 2.7% in the French
data to 5.4% in the German data and 6.3% in the UK data.

While Perspective API has seen widespread use, few studies have systematically
evaluated the accuracy and reliability of the model either across languages or
specifically in German. Among them, (gargee2022analyzing, ) highlights a
discrepancy in Perspective API’s toxicity assessments of sentences that are
semantically equivalent, but with different arrangements of words. In detail,
they show that switching from active to passive voice, as well as reordering
words in a sentence, could lead to significantly different toxicity scores.
Then, Wich et al. study toxic German-language speech on Telegram
(wich2022introducing, ). They develop their own classifier and compare its
performance with Perspective API, finding their model exhibits a higher F1
score ($54.75\%$ vs. $53.50\%$) and macro F1 score ($73.20\%$ vs. $70.51\%$).
Mihaljevic et al. (mihaljevic2023toxic, ) use antisemitism in the German
language on Telegram to examine how Perspective API scores correlate with
different subforms of antisemitism. They also demonstrate that manipulating
messages with widely known antisemitic codes can substantially reduce toxicity
scores, making it easy to bypass content moderation based on Perspective API.
While such studies suggest the need for more research on potential biases of
the model, the black-box nature of the model limits the capacity of
researchers to understand the limitations and dynamics of the classification.
Research therefore has to draw on observational data to seek to identify
biases in model classification.

##  3\. Data

Our analyses are based on a combination of real-world data from Twitter and
Wikipedia, as described in the following sections.

###  3.1. Dataset 1: Representative sample of international tweets across
languages

Our first Twitter dataset contains a random sample of tweets shared between
2018 and 2021 in 16 countries, collected using the Twitter Academic API on
Sept. 12, 2022. The collection was done by repeatedly selecting a random time
point between 1/1/2018 and 1/1/2022, and fetching the 500 most recent messages
sent in a particular country with a particular language, until 250,000
messages were collected for each region. The country and language were
filtered using the Twitter API place_country parameter, combined with the lang
parameter to specify the language. The resulting dataset represents a roughly
representative sample of Twitter messages uniformly distributed over time. The
procedure thus produces a dataset with a pseudo-random sample of Twitter
messages from 16 countries over the specified time period. The total number of
tweets equals 3,878,751, while those shared in German-speaking countries, such
as Austria (AT), Germany (DE), and Switzerland (CH), are 733,451 (18.9%).
Having multiple countries and languages allows for showing the unique biases
of German classification, and including several German-speaking countries
allows verifying that the results do not follow from the dynamics of online
conversation in a specific country.

###  3.2. Dataset 2: COVID-19 tweets in Italian and German

The second Twitter dataset contains tweets related to COVID-19 vaccines in the
Italian and German languages, shared between November 1, 2020, and June 30,
2021 – a time when vaccination campaigns kicked off in many European
countries. The resulting number of tweets is 1,453,450 for Italian and
1,219,445 for German (45.6%). The data were collected through Twitter’s
Historical Search API in July 2021 and are described in detail in
(giovanni2022vaccineu, ). These larger-scale data enable a deeper comparison
of Perspective API’s labeling dynamics between German and another language.

Figure 1. Distribution of toxicity scores for tweets shared in German-speaking
countries (Austria, Switzerland, and Germany) versus those in other EU
countries. Distributions are statistically different at $\alpha=0.05$
according to a Kruskal-Wallis test. Median toxicity is 0.075 for German-
speaking countries and 0.023 otherwise.

###  3.3. Dataset 3: Random sample of Wikipedia texts

The third dataset consists of a random sample of summaries from Wikipedia
pages in English and German language. Specifically, we collected $12.279$
summaries of a random sample of pages available in both languages leveraging
Wikipedia Python library111https://pypi.org/project/wikipedia/. The median
length of texts in characters is EN = 242, DE = 198. The median number of
words is EN = 39, DE = 28. Including random texts allows verifying that the
biases identified do not apply only to Twitter messages but to any text.

Figure 2. Top 10 most frequent Perspective API scores rounded at 8 decimal
digits for tweets shared in German-speaking countries (Austria, Switzerland
and Germany) and those in other EU countries, from Dataset 1.

##  4\. Analyses and Results

This section presents the results of the analyses conducted on the three
aforementioned datasets. In each case, we ran Perspective API on all available
texts focusing on the attribute ”Toxicity” – the estimated probability for a
comment to be perceived as toxic – as it is the most general and widely
employed in the literature. Specifically, we employed Perspective API in
September 2022 to obtain toxicity scores for Dataset 1, whereas we ran it
between September and December 2023 for labeling Datasets 2 and 3.

Our analyses first investigate the distribution of toxicity scores in a random
sample of multilingual tweets (Dataset 1). Next, we perform a similar analysis
for tweets in Italian and German language related to COVID-19 vaccines
(Dataset 2). For this dataset, we also assess the potential impact of
Perspective API’s language bias when it is used for automatically flagging
toxic messages for removal – a common content moderation strategy
(rieder2021fabrics, ). Lastly, we analyze biases in Perspective API scores
when considering a random set of Wikipedia summaries in their corresponding
English and German language versions (Dataset 3).

Figure 3. Distribution of toxicity scores for COVID-19 vaccine-related tweets
(top) and users (bottom) for German and Italian language. Median values for
tweet toxicity are DE = 0.132, IT = 0.026; for user toxicity DE = 0.212, IT =
0.048.

###  4.1. German tweets are labeled as more toxic than those from other
languages

We first study the distribution of toxicity scores for tweets shared in
German-speaking countries – namely Austria, Germany, and Switzerland –
compared to other countries. As shown in Figure 1, tweets from German-speaking
countries (which are mostly in the German language) are significantly more
toxic (Kruskal-Wallis $P<.001$) than those from other countries, with a median
toxicity of 0.075 versus 0.023. The latter distribution closely resembles a
smooth exponential distribution, which is typical of a range of phenomena on
social media, whereas the former is spiky – suggesting the presence of some
type of error or classification artifact.

We further look into the top 10 most frequent toxicity scores returned by the
API after rounding them at 8 decimal digits. As shown in Figure 2, tweets in
the German language exhibit an unusual pattern, as some very specific scores
are extremely common. While a smooth distribution would imply that each
individual score should account for a negligible proportion of all scores, as
in the case of the distribution for all other languages (in grey in Figure 2),
a few scores are disproportionately frequent for German texts – in line with
abnormal spikes highlighted in Figure 1, accounting for more than 10% of the
total toxicity scores returned by the Perspective API. This offers further
evidence of the presence of some internal issues in the toxicity model leading
to erroneous labeling of German language text.

###  4.2. German vaccine tweets are more prone to removal

To examine the implications of the identified biases in toxicity scores for
realistic real-world scenarios of content moderation, we leverage the dataset
on COVID-19 vaccine-related tweets. COVID-19 information has frequently been
raised as an example in which content moderation is particularly crucial
(ferrara2020misinformation, ; krishnan2021research, ).

####  4.2.1. Toxicity of tweets and users

We first analyze the distribution of toxicity scores for tweets shared in
German and Italian. As shown in the top panel of Figure 3, German tweets are
significantly more toxic than Italian tweets. The bottom panel of Figure 3
highlights that the same result also holds when considering user-level
toxicity, which we computed as the average of all tweets shared by each user.
The Italian and German distributions are statistically different in both cases
according to a Kruskal-Wallis test ($P<.001$).

####  4.2.2. Toxicity of tweets translated in English

To rule out the possibility that the higher toxicity scores assigned to German
texts are due to more toxic content, we translate both German and Italian
tweets into English and we subsequently use Perspective API to compute
toxicity scores for the English texts. We leverage Argos Translate
(argotranslate, ) to obtain English translations of our tweets. Figure 4 shows
the distribution of toxicity scores for all German tweets (solid line) and
their English translation equivalents (dashed line). We observe that the
tweets translated in English are much less toxic, with their distribution
being similar to that of generic tweets in non-German languages, presented in
Figure 1.

Figure 4. Distribution of toxicity scores for German COVID-19 vaccine-related
tweets and for their English translation. Median values are: DE = 0.132, EN =
0.012. Figure 5. Distribution of the differences in toxicity scores for German
and Italian tweets compared to their English translation. Median values are DE
= +0.085, IT = -0.006.

However, it may be that the machine-translation tends to reduce the toxicity
of messages, for instance by refusing to translate certain swearwords or
slang. To examine whether this is the case, we also compute the difference in
toxicity between a random sample of $10\,000$ tweets in German compared to
their respective English translation; we also translate them from German to
Italian and compute their difference in toxicity with the English version.
Figure 5 shows the distribution of these differences. We observe that, on
average, German messages are 8.5% more toxic (i.e., median difference =
+0.085) than the corresponding English messages, as also shown by the skewed
orange-colored distribution. Contrarily, the differences between Italian and
English-translated messages appear to follow a normal distribution centered
around 0 and, in fact, Italian messages only show negligible differences
(median difference = 0.01) when compared to their English translations. These
results unequivocally demonstrate that the larger toxicity scores for German
texts do not depend on the content of such texts, but are rather the
consequence of an inherent language bias in Perspective API. To rule out
potential biases in the translation that might modify the toxicity, we perform
the same experiments by starting from a random sample of $10\,000$ Italian
tweets and translating them to German and English to compute differences. Also
in this case, we obtained qualitatively similar results, but omit the figure
for the sake of brevity.

####  4.2.3. Implications for content moderation

To assess the implications of this bias, we analyze how moderation strategies
aimed at minimizing online toxicity might affect decision-making processes,
especially when a moderator or an automatic system relies on toxicity scores
from the Perspective API to oversee and regulate discussions on a social media
platform (rieder2021fabrics, ).

Specifically, we focus on how different toxicity thresholds used for removing
tweets or users might affect COVID-19 vaccine conversations in German when
compared to their translation in English, using the random sample of $10\,000$
tweets employed in the previous analysis. In the top panel of Figure 6, we
simulate removing all tweets whose toxicity score exceeds a given threshold,
and we show the percentage of extra tweets removed when considering German
scores compared to English scores. For instance, by adopting the threshold
value of $0.7$ recommended by Perspective API (thresholdperspective, ), 10
times more German tweets would be removed, and similarly for the number of
German users (as shown in the bottom panel of the figure). The median increase
in percentage points is 429.94 for tweets and 409.84 for users, meaning that
on average more than four times German tweets and users would be removed
compared to English.

Figure 6. Proportion of extra tweets (top) and users (bottom) removed in
German compared to English for different choices of toxicity score as a
threshold, i.e., tweets/users above a given score would be removed. The median
increase in percentage points is 429.94 for tweets and 409.84 for users,
meaning that on average more than 4 times German tweets and users would be
removed compared to English.

###  4.3. Wikipedia texts in German are more toxic than in English

To verify that the identified biases do not merely apply to Twitter messages,
we also examine the distribution of toxicity scores when classifying a random
sample of summaries from Wikipedia. For this dataset, each summary is
available in English and German language. As shown in Figure 7, also for these
texts, German exhibits significantly larger toxicity scores than those in
English (Kruskal-Wallis $P<.001$). We also observe peaks in the distribution
of German scores which resemble those of previous distributions, such as
Figure 1.

Figure 7. Distribution of toxicity for $12\,000$ random Wikipedia page
summaries in English and German language. Median values are: DE = 0.03, EN =
0.01.

##  5\. Discussion

Perspective API has emerged as the technological foundation for a broad
spectrum of applications, stretching from academic research to practical
implementations in content moderation across various online platforms. In the
realm of research, scholars and data scientists utilize Perspective API to
analyze online discourse, study patterns of hate speech and harassment, and
explore the dynamics of digital communication. Beyond the academic sphere,
Perspective API plays a pivotal role in content moderation for websites,
forums, and large-scale social media platforms. By automatically flagging
potentially harmful or abusive comments, it aids moderators in maintaining a
constructive and respectful environment, enabling large-scale platforms to
manage large volumes of user-generated content that would make manual
moderation impractical. Perspective API is often integrated into comment
systems and discussion forums, enabling preemptive identification and removal
of messages that are classified as toxic. While the widespread application of
the API means that the system provides important support for researchers and
platforms, it also highlights the potential negative impact of any biases or
inaccuracies in its algorithm.

This paper has identified a significant bias within the model when it comes to
processing German-language messages: our analysis reveals that the algorithm
tends to assign disproportionately higher toxicity scores to text written in
German, compared to similar content in other languages. Specifically, our
examination of Twitter posts revealed that the same content reported in German
was more toxic than its English-translated version. This pattern was further
validated by analyzing random texts from Wikipedia summaries in both
languages. The distribution of toxicity returned by the API for German appears
spiky, rather than following the smooth distribution of other languages –
suggesting some errors or artifacts stemming from the underlying machine
learning model. Due to the black box and proprietary nature of the model, we
unfortunately cannot examine the underlying reasons for these results.

This bias raises profound concerns, as it not only skews research findings but
also impacts the efficacy of content moderation on digital platforms. In the
academic context, such a bias can lead to misleading conclusions about the
nature and prevalence of toxicity within German-speaking online communities.
For content moderators and platform administrators, this skewed assessment
could result in unjust censorship or the inadvertent promotion of a hostile
online environment for German-speaking users. For academic and journalistic
research relying on the API, the implications are far-reaching, potentially
affecting the perception and treatment of German digital discourse globally.

The problems identified in this paper highlight the inherent risks as
researchers become increasing dependent on proprietary models for data
analysis and AI. The closed and proprietary APIs offered by major tech
companies provide researchers with easy access to vast amounts of data and
sophisticated analysis methods, which can otherwise be challenging to access.
This ready availability is invaluable for studying complex social phenomena,
trends, and behaviors in the digital age. However, the growing dependency on
closed models also raises critical concerns. The opaque nature of these
proprietary systems means that researchers are unable to scrutinize or
understand the inner workings of the algorithms, and identify potential biases
and limitations. The result is a substantial risk of systemic biases being
embedded and perpetuated through their employment. This lack of transparency
can furthermore inadvertently influence the direction and conclusions of
studies, as researchers must operate within the constraints and methodologies
predefined by these tools. While proprietary APIs have opened new avenues for
exploring social sciences, they also necessitate a cautious and critical
approach to ensure the integrity of research and its reproducibility
(pozzobon2023challenges, ) – rather than taking the technology firms by their
word.

Ethical considerations. We envision positive ethical implications for this
work, particularly concerning future research on online toxicity as well as on
the policing of online platforms. Our findings underscore the potential
consequences of relying on black-box models and APIs, without a thorough
understanding of their language biases. The identification of intrinsic biases
within the multilingual Perspective API highlights the risk of inadvertently
amplifying or suppressing certain voices in online spaces. The observed higher
levels of toxicity assigned to German content compared to other languages
raise concerns about the fairness and equity of ongoing content moderation
processes, which could potentially lead to disproportionate censorship or
scrutiny of specific linguistic communities. This study urges a reevaluation
of the ethical dimensions surrounding the deployment of such tools,
emphasizing the need for transparency, accountability, and ongoing refinement
to ensure that AI-driven content moderation aligns with principles of
impartiality and cultural sensitivity, thereby mitigating the inadvertent
amplification of biases in online discourse.

## References

  * [1] Emily A Vogels.  The state of online harassment.  Pew Research Center, 13:625, 2021.
  * [2] Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman.  A new generation of Perspective API: Efficient multilingual character-level transformers.  In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’22), pages 3197–3207, 2022.
  * [3] Using machine learning to reduce toxicity online.  https://www.perspectiveapi.com/.  Accessed: 2023-10-31.
  * [4] Perspective API.  https://www.perspectiveapi.com/case-studies/.  Accessed: 2023-10-31.
  * [5] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20), pages 3356–3369, 2020.
  * [6] Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang.  Challenges in detoxifying language models.  In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP’21), pages 2447–2469, 2021.
  * [7] John Pavlopoulos, Nithum Thain, Lucas Dixon, and Ion Androutsopoulos.  ConvAI at SemEval-2019 task 6: Offensive language identification and categorization with Perspective and BERT.  In Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval’19), pages 571–576, 2019.
  * [8] Yiqing Hua, Thomas Ristenpart, and Mor Naaman.  Towards measuring adversarial Twitter interactions against candidates in the US midterm elections.  In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM’20), volume 14, pages 272–282, 2020.
  * [9] Yiqing Hua, Mor Naaman, and Thomas Ristenpart.  Characterizing Twitter users who engage in adversarial interactions against political candidates.  In The 2020 CHI Conference on Human Factors in Computing Systems (CHI’20), pages 1–13, 2020.
  * [10] Bernhard Rieder and Yarden Skop.  The fabrics of machine moderation: Studying the technical, normative, and organizational structure of Perspective API.  Big Data & Society, 8(2), 2021.
  * [11] Paul Friedl.  Dis/similarities in the design and development of legal and algorithmic normative systems: The case of Perspective API.  Law, Innovation and Technology, 15(1):25–59, 2023.
  * [12] Erik Rye, Jeremy Blackburn, and Robert Beverly.  Reading in-between the lines: An analysis of Dissenter.  In Proceedings of the ACM Internet Measurement Conference (IMC’20), pages 133–146, 2020.
  * [13] Mohamad Hoseini, Philipe Melo, Fabricio Benevenuto, Anja Feldmann, and Savvas Zannettou.  On the globalization of the QAnon conspiracy theory through Telegram.  In Proceedings of the 15th ACM Web Science Conference (WebSci’23), pages 75–85, 2023.
  * [14] Ivan Smirnov, Camelia Oprea, and Markus Strohmaier.  Toxic comments reduce the activity of volunteer editors on Wikipedia.  arXiv preprint:2304.13568, 2023.
  * [15] Sayar Ghosh Roy, Ujwal Narayan, Tathagata Raha, Zubair Abid, and Vasudeva Varma.  Leveraging multilingual transformers for hate speech detection.  In The 12th meeting of Forum for Information Retrieval Evaluation (FIRE’20), 2020.
  * [16] Jana Lasser, Alina Herderich, Joshia Garland, Segun Taofeek Aroyehun, David Garcia, and Mirta Galesic.  Collective moderation of hate, toxicity, and extremity in online discussions.  arXiv preprint:2303.00357, 2023.
  * [17] Christian S Czymara, Stephan Dochow-Sondershaus, Lucas G Drouhot, Müge Simsek, and Christoph Spörlein.  Catalyst of hate? Ethnic insulting on YouTube in the aftermath of terror attacks in France, Germany and the United Kingdom 2014–2017.  Journal of Ethnic and Migration Studies, 49(2):535–553, 2023.
  * [18] SK Gargee, Pranav Bhargav Gopinath, Shridhar Reddy SR Kancharla, CR Anand, and Anoop S Babu.  Analyzing and addressing the difference in toxicity prediction between different comments with same semantic meaning in Google’s Perspective API.  In The 7th International Conference on ICT for Sustainable Development (ICT4SD’22), pages 455–464, 2022.
  * [19] Maximilian Wich, Adrian Gorniak, Tobias Eder, Daniel Bartmann, Burak Enes Cakici, and Georg Groh.  Introducing an abusive language classification framework for Telegram to investigate the German hater community.  In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM’22), volume 16, pages 1133–1144, 2022.
  * [20] Helena Mihaljević and Elisabeth Steffen.  How toxic is antisemitism? Potentials and limitations of automated toxicity scoring for antisemitic online content.  arXiv preprint:2310.04465, 2023.
  * [21] Marco Di Giovanni, Francesco Pierri, Christopher Torres-Lugo, and Marco Brambilla.  VaccinEU: COVID-19 vaccine conversations on Twitter in French, German and Italian.  Proceedings of the International AAAI Conference on Web and Social Media (ICWSM’22), 2022.
  * [22] Emilio Ferrara, Stefano Cresci, and Luca Luceri.  Misinformation, manipulation, and abuse on social media in the era of COVID-19.  Journal of Computational Social Science, 3:271–277, 2020.
  * [23] Nandita Krishnan, Jiayan Gu, Rebekah Tromble, and Lorien C Abroms.  Research note: Examining how various social media platforms have responded to COVID-19 misinformation.  Harvard Kennedy School Misinformation Review, 2(6):1–25, 2021.
  * [24] Argos Translate.  http://www.argosopentech.com/.  Accessed: 2023-09-21.
  * [25] Perspective API threshold.  https://developers.perspectiveapi.com/s/about-the-api-score?language=en_US.  Accessed: 2023-09-21.
  * [26] Luiza Amador Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker.  On the challenges of using black-box apis for toxicity evaluation in research.  In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.