# Quantifying fairness and discrimination
in predictive models

Arthur Charpentiera∗
a Université du Québec à Montréal (UQAM), Montréal (Québec), Canada
∗Corresponding author<EMAIL_ADDRESS>
###### Abstract

The analysis of discrimination has long interested economists and lawyers. In
recent years, the literature in computer science and machine learning has
become interested in the subject, offering an interesting re-reading of the
topic. These questions are the consequences of numerous criticisms of
algorithms used to translate texts or to identify people in images. With the
arrival of massive data, and the use of increasingly opaque algorithms, it is
not surprising to have discriminatory algorithms, because it has become easy
to have a proxy of a sensitive variable, by enriching the data indefinitely.
According to Kranzberg, (1986), “technology is neither good nor bad, nor is it
neutral ", and therefore, “machine learning won’t give you anything like
gender neutrality ‘for free’ that you didn’t explicitely ask for ", as claimed
by Kearns and Roth, (2019). In this article, we will come back to the general
context, for predictive models in classification. We will present the main
concepts of fairness, called group fairness, based on independence between the
sensitive variable and the prediction, possibly conditioned on this or that
information. We will finish by going further, by presenting the concepts of
individual fairness. Finally, we will see how to correct a potential
discrimination, in order to guarantee that a model is more ethical.

Keywords Classifier; Demographic Parity; Discrimination; Equal Opportunity;
Fairness; Penalized regression; Proxy; Statistical Discrimination

##  1 Introduction

In a classification problem, given a collection of covariates
$\bm{x}\in\mathcal{X}\subset\mathbb{R}^{p}$, we will estimate a model (such as
a logistic regression) to compute a score $m(\bm{x})$ that will be used to
predict a binary outcome $y\in\\{0,1\\}$. In standard econometrics,
$m(\bm{x})$ is interpreted as a probability, $m:\mathcal{X}\to[0,1]$. The
assignment to classes, $\widehat{y}$ will be done according to the
interpretation of $y$ (here the occurrence – or not – of a specific risk) and
$m(\bm{x})$, though a function $m_{\tau}:\mathcal{X}\to\\{0,1\\}$, where
$\tau$ is some chosen probability level.

  * •

In credit risk, $y$ will designate a credit default ($y=1$ in case of
default), and often the credit score (in the FICO sense) will be all the
higher that the risk is low. In other words, $y$ and the “credit score",
evolve in opposite directions: with this interpretation of the score,
$\widehat{y}=m_{\tau}(\bm{x})=\bm{1}_{m(\bm{x})<\tau}$ (for some appropriate
threshold $\tau$).

  * •

In “death insurance”, $y$ denotes a death ($y=1$ in case of death of a
specific individual) and the “score” $m(\bm{x})$ will be the probability of
dying within a specific time window. In other words, here, $m(\bm{x})$ and $y$
evolve in the same direction: with this interpretation of the score,
$\widehat{y}=m_{\tau}(\bm{x})=\bm{1}_{m(\bm{x})>\tau}$.

For a regression problem, we will predict $y\in\mathbb{R}$ using a model
$m(\bm{x})$, and we will consider $\widehat{y}=m(\bm{x})$. The subtle part in
a classification problem is the intermediate passage through this score
$m(\bm{x})$ (which will not have values in $\\{0,1\\}$ \- like $y$ \- but in
$[0,1]$ or in $\mathbb{R}$). To avoid any confusion, the score $m(\bm{x})$
will be supposed to be evolve in the same direction as $y$ : large values of
$m(\bm{x})$ are supposed to be associated with $y=1$. Finally, we suppose that
there is a sensitive (or protected) attribute $s$, that is supposed to be
binary, with $s\in\\{0,1\\}$, or
$\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle},\leavevmode\nobreak\
{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
with bullets and squares) in illustrations.

###  1.1 Sensitive attribute and discrimination

For economists, the study of discrimination is an old problem, as recalled in
Charles and Guryan, (2011), whose theoretical foundations were laid by Becker,
(1957), and earlier considerations can be found in Edgeworth, (1922) for
instance. Other examples include Phelps, (1972), who had tried to understand
the origins of discrimination, and who had argued (in the context of racial
discrimination) “what has been called racism – similar remarks apply to sexism
– can be hypothesized to be the consequence of scientific management in the
impersonal pursuit of maximum profit, not racial hostility or intolerance".
This idea will be the basis of “statistical discrimination”, where the central
question was to link discrimination with a rational behaviour, and therefore a
notion of efficiency. Bohren et al., (2019) reminds us that statistical
discrimination is sometimes qualified as “efficient discrimination”, insofar
as it is the optimal answer to a signal extraction problem. Phelps, (1972) and
Aigner and Cain, (1977) have laid the groundwork for this statistical
discrimination. For example, if the probability of being an “offender" and of
belonging to a group with a visible characteristic is higher, on average, than
for other groups, then a policeman will be more likely to control an
“offender" if he controls a member of this group. There is therefore, at the
group level, a “statistical reason" which will be opposed to the individual
principle of non-discrimination. Also, in the name of the efficiency of the
procedure, Gary Becker defended “racial profiling". One finds this
argumentation in the fight against terrorism. As Becker, (2005) says “if young
Moslem Middle Eastern males were in fact much more likely to commit terrorism
against U.S. than were other groups, putting them through tighter security
clearance would reduce current airport terrorism”, in other words, “racial
profiling” is “effective”, even though “such profiling is ‘unfair’ to the many
young male Moslems who are not terrorists, and to the many minority shoppers
who are honest”. And he proposes a method for “testing" efficiency “some
profiling by governments and the private sector has been due to prejudice
against various groups, not as a way of achieving efficiency. So it is crucial
to be able to distinguish whether a profiling is efficient from whether it is
evidence of discrimination. This distinction can be made in the terrorist
field by keeping records on the fractions of young Moslem males and others who
were searched and found with weapons or other evidence of intent to commit a
terrorist act”. Another typical example of “statistical discrimination”,
justified by economists on the grounds of economic efficiency, is
discrimination in the hiring of young women (who might become pregnant, and
(temporarily) interrupt her work). In the latter case, there is no need for
statistics, since only a woman can become pregnant. Many countries now offer
long parental leave, which can be taken by any parent, regardless of gender,
which questions the economic efficiency of this profiling. For Gary Becker,
this “statistical reason" is, and must be, the only decision criterion used.
That is, more or less, what actuaries have in mind when they mention “a fair
actuarial classification". As we can see, this search for efficiency raises
many moral and ethical questions.

From the legal point of view, Cornu, (2016), presents discrimination as a
“differentiation contrary to the principle of civil equality consisting in
breaking this one to the detriment of certain physical persons because of
their racial or confessional belonging, more generally of criteria on which
the law forbids to found legal distinctions” (while admitting “more rarely, in
a neutral sense, synonymous with distinction (not necessarily hateful)” which
recalls the statistician’s vision). Equality, which we find stated in the
Universal Declaration of Human Rights, was initially perceived in a “vertical"
sense, imposing a constraint on the behavior of the State towards citizens.
The law will impose “horizontal" constraints in private law, whether in the
labor market (during job interviews in particular) or in real estate (for
renting housing). Article 23 of the Charter of Fundamental Rights of the
European Union imposes the general principle that “equality between women and
men must be ensured in all areas, including employment, work and pay". This
right cannot be invoked by “women", but individually, by everyone (for example
in a dispute with an employer). This “right to equal treatment" belongs to a
person, as an individual, and not in his capacity as a member of a sexual
group (for example). The law says that an individual cannot be treated
differently because of his or her membership to such a group, especially a
group to which he or she has not chosen to belong to. This individualistic
view of the law is strongly opposed to the mutual and collective concept of
insurance, as insurers aim for a kind of equality within the group, based on
averages, and not at the individual level (see Charpentier, (2022) for a
specific discussion in the context of insurance).

From a more moral point of view, according to the principle of choice
mentioned by Lippert-Rasmussen, (2007), people should not be subjected to
disadvantageous treatment because of something that does not reflect their own
choices. This may explain why discrimination on the basis of gender, race,
ethnicity, or genetics is widely seen as morally problematic, as argued by
Daniels, (2004), Palmer, (2007), or Avraham et al., (2014). However, the
principle of choice is not violated if individuals have imposed additional
costs as a result of their choice. How about discrimination on the basis of
religious beliefs? As we can see, the term “discrimination" seems to encompass
all sorts of realities. In particular, using the typology of Thomsen, (2017)
and Khaitan, (2017), we can distinguish direct discrimination and indirect
discrimination. In a nutshell, indirect discrimination (proxy discrimination
or statistical discrimination) means that instead of discriminating according
to a sensitive attribute $s$ (which would not be allowed by law), a variable
$x_{j}$, which is highly correlated with $s$, is used.

Big data and machine learning have provided an opportunity to revisit this
topic that has been explored by lawyers, economists, philosophers and
statisticians for the past fifty years, or longer. The aim here is to revisit
these ideas, to shed new light on them, with a focus on risk management, and
explore possible solutions. Lawyers, in particular, have discussed these
predictive models in the context of justice, also called “actuarial justice",
as Thomas, (2007), Harcourt, (2011) or Rothschild-Elyassi et al., (2018).

The idea of bias and algorithmic discrimination is not a new one, as shown for
instance by Pedreshi et al., (2008). However, since then the number of
examples has continued to increase. “AI biases caused 80% of black mortgage
applicants to be rejected” claimed Hale, (2021), or “How the use of AI risks
recreating the inequity of the insurance industry of the previous century"
from Ito, (2021). Pursuing David,’s 2015 analysis, McKinsey, (2017) announced
that artificial intelligence would disrupt the workplace (including the
insurance and banking sectors). These replacements raise questions, and compel
the market and the regulator to be cautious. Bergstrom and West, (2021) note,
with a touch of irony, that there are people writing a bill of rights for
robots, or devising ways to protect humanity from super-intelligent,
Terminator-like machines, but that getting into the details of algorithmic
auditing is often seen as boring, but necessary. To solve the problems that AI
is creating now, we need to understand the data and algorithms we already use
for more mundane purposes.

###  1.2 Examples of discrimination

In many countries, there are legal texts stipulating that all individuals must
have equal opportunities with other individuals, without being hindered by
discriminatory practices based on race, national or ethnic origin, color,
religion, age, sex, sexual orientation, gender identity or expression, marital
status, family status, genetic characteristics, disability, among others. A
challenge for econometricians and statisticians is that those protected or
sensitive attributes are personal, with strong concern with privacy. Kelly,
(2021) reminds that “often the location data is used to determine what stores
people visit. Things like sexual orientation are used to determine what
demographics to target". Each type of data can reveal something about our
interests and preferences, our opinions, our hobbies and our social
interactions. For example, a MIT study111Project
https://immersion.media.mit.edu/. demonstrated how email metadata can be used
to map our lives, showing the changing dynamics of our professional and
personal networks. This data can be used to infer personal information,
including a person’s background, religion or beliefs, political views, sexual
orientation and gender identity, social relationships or health. For example,
it is possible to infer our specific health conditions simply by connecting
the dots between a series of phone calls. For Mayer et al., (2016), the law
currently treats call content and metadata separately and makes it easier for
government agencies to obtain metadata, in part because it assumes that it
should not be possible to infer specific sensitive details about individuals
from metadata alone.Chakraborty et al., (2013) reminds us that current
approaches to privacy protection, typically defined in multi-user contexts,
rely on anonymization to prevent such sensitive behavior from being traced
back to the user - a strategy that does not apply if the user’s identity is
already known. In 2015, as told in Miracle, (2016), Noah Deneau wondered if it
would be possible to identify devout Muslim drivers in New York City by
looking at anonymized data and inactive drivers during the five times of the
day they are supposed to pray. He quickly searched for drivers who were not
very active during the 30-45 minute Muslim prayer period and was able to find
four examples of drivers who might fit this pattern. This brings to mind Gambs
et al., (2010), who conducted an investigation on a dataset containing
mobility data of taxi drivers in the San Francisco Bay Area. By finding places
where the taxi’s GPS sensor was turned off for a long period of time (e.g. two
hours), they were able to infer the interests of the drivers. For 20 of the 90
analyzed users, they were able to locate a plausible home in a small
neighborhood. They even confirmed these results for 10 users by using a
satellite view of the area: It showed the presence of a yellow taxi parked in
front of the driver’s supposed home. Dalenius, (1977) introduced an
interesting concept of privacy. Nothing about an individual should be learned
from a dataset if it cannot be learned without having access to the dataset.
We will return to this idea when we define the fairness criteria, and when we
require that the protected variable $s$ cannot be predicted from the data, and
from the predictions. This idea will be used later on to insure that a model
is non-discriminatory. Recently, Barry and Charpentier, (2022) provided
several examples of possible sensitive attribute in the context of insurance,
where predictive models are used constantly (for pricing, fraud detection,
etc).

In the discrimination literature, we note either $p$ (for protected variable),
$s$ (for sensitive variable), or even $a$ (for protected attribute). In the
causal inference literature, we use $t$ (for treatment). We will use here the
notation $s$ for the sensitive variable, while $m$ will be used for the score.
We will refer to $s=0$ as the (assumed) advantaged population, and $s=1$ as
the disadvantaged population, as in the literature in causal inference, where
$t=0$ is the control group, and $t=1$ the treated group. Scott and Marshall,
(2009) point out in their dictionary, that in common language, discrimination
is about “treating unfairly". But in social sciences and humanities, “most
sociological analyses of discrimination concentrate on patterns of dominance
and oppression, viewed as expressions of a struggle for power and privilege".
In other words, for discrimination to occur, there must be a “favored" and a
“disadvantaged" group, a “dominant" and a “dominated" group, according to a
(often implicit) power criterion, or even a “majority" group and a “minority"
group (this terminology, introduced by Wirth, (1941), can lead to confusion
because these terms are often defined according to a criterion of group size).
Here, we assume that the favored and disadvantaged group are not related by
some domination power, only to risk occurrence, $y$. More specifically, since
here $y=1$ denotes the occurrence of a “bad” event (not a moral sense, but
related to a big economic loss), $\mathbb{P}[Y=1|S=1]>\mathbb{P}[Y=1|S=0]$
means that $0$ is the favored group (the “less risky"), while $1$ is the
disadvantaged one (the “more risky").

###  1.3 Notations on classifiers

As we have already mentioned, the binary classification (when $y\in\\{0,1\\}$)
is a bit particular because of the intermediate construction of a score
$m(\bm{x})$, before constructing the classifier $\widehat{y}$ stricto sensu
(and $\widehat{y}\in\\{0,1\\})$. By analogy with logistic or probit
regressions, as discussed earlier, the score $m$ will be a function
$\mathcal{X}\to[0,1]$, where $\mathcal{X}\subset\mathbb{R}^{p}$, and with, for
example222Some articles define the score as $\bm{x}^{\top}\bm{\beta}$, which
has values in $\mathbb{R}$. The score we define is an increasing function of
this linear combination. Note that here, $\Phi$ denotes the cumulative
distribution function of some $\mathcal{N}(0,1)$ variable.

$m(\bm{x})=\frac{\exp[\bm{x}^{\top}\bm{\beta}]}{1+\exp[\bm{x}^{\top}\bm{\beta}]}\text{
or }m(\bm{x})=\Phi(\bm{x}^{\top}\bm{\beta}),$

respectively for the logistic model and the probit model.

A standard tool to describe the performance of $m$ is to use the ROC curve, as
discussed in Charpentier et al., (2018). Figure 1 schematically resumes the
analysis of a (linear) classifier, with a confusion matrix on the right, which
will be used as a basis to build the ROC curve,

$(\mathbb{P}[m(\bm{X})>t|Y=0],\mathbb{P}[m(\bm{X})>t|Y=1])_{t\in[0,1]},$

or, noting $\widehat{y}=m_{t}(\bm{x})=\bm{1}_{m(\bm{x})>t}$ for threshold $t$,

$(\mathbb{P}[\widehat{Y}=1|Y=0],\mathbb{P}[\widehat{Y}=1|Y=1])=(\text{FPR},\text{TPR}),$

meaning that the ROC curve is the true positive rate (TPR) plotted against the
false positive rate (FPR), as the threshold $t$ varies from $0$ to $1$. In the
example of Figure 1, the false positive rate (FPR) is 2/7 (out of the 7 blue
bullets , 2 points are misclassified (since they are in the red region) and
announced as positive), that is 28.57%; the true positive rate (TPR) is 5/6
(out of 6 red squares , 1 point is misclassified (being in the blue region),
and announced negative), that is 83.33%.

$x_{1}$$x_{2}$$\hat{y}=1$$\hat{y}=0$$\hat{y}=1$$\hat{y}=0$$y=0$$y=1$positivenegativefalse
negativefalse positivepositiveFPTPFNTNPN Figure 1: Construction of the
confusion matrix for a classifier, $\widehat{y}=\bm{1}(x_{1}+x_{2}>t)$, where
the bullets  represent points $y=0$ and the squares  represent points $y=1$.
The zone below, in the south-west part, corresponds to the predictions
$\widehat{y}=0$ and the upper zone, in the north-east part, to the predictions
$\widehat{y}=1$. The square points in the lower area, and the bullet points in
the upper area are bad classifications, corresponding to errors, respectively
false negatives and false positives.

In Figure 2, we can visualize the distribution of a credit score (in
accordance with the approach in econometrics, statistics and machine learning,
a low $m(\bm{x})$ score indicates a good risk, and therefore less chance of
occurrence of that risk).We can visualize on the left the distributions of the
score, conditionally on $y$, with respectively in dotted line the density of
$m(\bm{X})$ when $y=1$ and in solid line, that of $m(\bm{X})$ when $y=0$. Let
us suppose that the threshold allowing to change class is $60\%$, so that
$\widehat{y}=1$ if $m(\bm{x})>60\%$. We can observe on the left that

$\begin{cases}\mathbb{P}[m(\bm{X})>60\%|Y=1]=\mathbb{P}[\widehat{Y}=1|Y=1]\sim
66.3\%\leavevmode\nobreak\ \text{true positive rate}\\\
\mathbb{P}[m(\bm{X})>60\%|Y=0]=\mathbb{P}[\widehat{Y}=1|Y=0]\sim
9.6\%\leavevmode\nobreak\ \text{false positive rate}\end{cases}$

(because $\widehat{y}=1$ corresponds to a “positive”).

Figure 2: Distributions of $S$ score conditional on $y=1$ and $y=0$, left, and
ROC curve, right, for a score $n$, with $y=1$ denoting the occurrence of a
risk. The areas on the left and the point on the right of the ROC curve
correspond to a threshold $\tau$ of 60%. For that specific threshold, the true
positive rate (the gray on the right of $0.6$, associated with $y=1$) of 66.3%
while the false positive rate (the gray dashed area on the right of $0.6$,
associated with $y=0$) of 9.6% .

To illustrate more precisely the construction, let us consider a toy dataset,
like the one in Kearns and Roth, (2019), as in Figure 3. In our case, we
consider bullet individuals  ($s=0$) and square individuals  ($s=1$), which
will be our sensitive variable $s$. Bullet individuals  ($s=0$) are said to be
favored since the distribution of $m(\bm{X})$ is less skewed towards that
right than the distribution of $m(\bm{X})$ of square individuals , therefore
considered disadvantaged. Favored individuals are less facing the risk we
model here than disadvantaged ones. In Figure 3, at the
$m(\bm{x}_{i})\in[0,1]$ level, we observe a bullet  or square , locating on
the $[0,1]$ scale according to the value of $m(\bm{x}_{i})$. The value given
below is $y_{i}\in\\{0,1\\}$.

11111111111111000000000001thresold Figure 3: Scale of the score
$m(\bm{x})\in[0,1]$, with a sensitive attribute
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
and a variable of interest $y\in\\{0,1\\}$, inspired by Kearns and Roth,
(2019). The very first point on the left corresponds to
$(y_{i},s_{i},\bm{x}_{i})$ such that $y_{i}=0$,
$s_{i}=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}$
(or $0$), $\bm{x}_{i}$ are unobserved components in some space $\mathcal{X}$
such that $m(\bm{x}_{i})\approx 10\%$, and since it is lower than $\tau\approx
45\%$, $\widehat{y}_{i}=0$.

The ROC curve is the curve obtained by representing the true positive rates
according to the false positive rates, by changing the threshold. It is
therefore the parametric curve

$C(t)=\\{\mathbb{P}[m(\bm{X})>t|Y=0],\mathbb{P}[m(\bm{X})>t|Y=1]\\},\leavevmode\nobreak\
\text{for }t\in[0,1],$

when the score $m(\bm{X})$ and $Y$ evolve in the same direction (a high score
indicates a high risk). We define the convex envelope $\mathcal{C}$, as in
Hardt et al., (2016). The convex envelope is interesting because it allows us
to describe the set of classifiers that can be constructed from the $s$ score.
On the left of Figure 4, we can see the convex envelope $\mathcal{C}$ of the
ROC curve of the example in Figure 3. $\mathcal{C}$ is here a quadrilateral,
the edges consisting of four segments. For example the segment [AB] is
obtained by using the classifier $s$, but by drawing the threshold at random:
either the threshold associated to the point A, or the threshold associated to
the point B.

For the right-hand side of Figure 4, recall that the accuracy (noted $a$)
associated with a confusion matrix is the proportion of good prediction,
$\mathbb{P}[\widehat{Y}=Y]$, that is

$a=\frac{\text{TP}+\text{TN}}{\text{P}+\text{N}}=\frac{\text{TPR}\cdot\text{TPR}+(1-\text{FPR})\cdot\text{N}}{\text{P}+\text{N}}$

The iso-accuracy curves have the equation

$\text{TPR}=\frac{\text{N}}{\text{P}}\cdot\text{FPR}+\frac{a\cdot[\text{P}+\text{N}]-\text{N}}{\text{P}}$

which is linear in FPR: they are (parallel) lines of slope
$\text{N}/\text{P}$, corresponding to the ratio
$\mathbb{P}[Y=0]/\mathbb{P}[Y=1]$. The curve with the highest accuracy will be
the highest, and it is “tangent" to the ROC curve in B.

Figure 4: On the left, ROC curve $C$ and its convex envelope $\mathcal{C}$. On
the right, optimal precision from the N/P ratio

.

One can also construct $s$-conditional ROC curves, for subgroups characterized
by the value of the sensitive attribute,

$C_{s}(t)=\\{\mathbb{P}[m(\bm{X})>t|Y=0,S=s],\mathbb{P}[m(\bm{X})>t|Y=1,S=s]\\},\leavevmode\nobreak\
\text{for }t\in[0,1],$

for the two classes $s=0$ (or bullet points  in the application in the next
section) and $s=1$ (the square points ), as well as their convex envelope
$\mathcal{C}_{s}$ (as on Figure 5). Note that we can also write
$C(t)=\text{TPR}\circ\text{FPR}^{-1}(t)$, where
$\text{FRP}(t)=\mathbb{P}[m(\bm{X})>t|Y=0]$ and
$\text{TPR}(t)=\mathbb{P}[m(\bm{X})>t|Y=1]$. In other words, the ROC curve is
obtained from the two survival functions of $m(\bm{X})$ FPR and TPR
(respectively conditional on $T=0$ and $T=1)$. For the $s$-conditional ROC
curve, $C_{s}(t)=\text{TPR}_{s}\circ\text{TPR}_{s}^{-1}(t)$, where
$\text{FRP}_{s}(t)=\mathbb{P}[m(\bm{X})>t|Y=0,S=s]$ and
$\text{TPR}_{s}(t)=\mathbb{P}[m(\bm{X})>t|Y=1,S=s]$. An finally, the AUC, the
area under the curve, is then written

$\text{AUC}=\int_{0}^{1}C(t)dt=\int_{0}^{1}\text{TPR}\circ\text{FPR}^{-1}(t)dt.$

Figure 5: ROC curves $C_{s}$ on the left, and the convex envelope on the right
$\mathcal{C}_{s}$, for $s=0$ (bullet individuals ) and $s=1$ (square
individuals ).  Figure 6: Conditional distributions of $m(\bm{X})|Y=1,S=0$ and
$m(\bm{X})|Y=0,S=0$ ($S=0$, or ), on the left (population assumed favored),
and conditional distributions of $m(\bm{X})|Y=1,S=1$ and $m(\bm{X})|Y=0,S=1$
($S=1$, or ), on the right (population assumed defavored).

###  1.4 Agenda

In section 2, we will present the first concept of fairness, called “fairness
by unawareness". The idea will be to remove the sensitive attribute from the
data, and to not use it. Even if that approach is still very popular, it is a
very bad idea: not only it does not make discrimination disappear (with
massive data, one can expect to get easily a proxy of the sensitive
attributed), but it becomes impossible then both to quantify properly
“statistical discrimination", and to correct it. Thus, we will discuss two
classes of fairness principles. The first one, in section 3 is called “group
fairness". In part 3.1, we will recall classical results on independence,
conditional independence, as well as measures of dependence, with the
correlation, as well as the maximal correlation. Then we will present
Demographic Parity in part 3.2 and Equalized Odds in part 3.3, those two
concepts being probably the most popular ones. In part 3.3, we will present
other measures than the one obtained by asking for the equality of “true
positive rates" (for instance the “false positive rates" equality or equality
of AUC). In part 3.4, we will discuss extensions of Demographic Parity by
adding some exogenous variables. In part 3.5 we will discuss the idea of class
balance parity, as well as calibration (or accuracy) parity. The last concept
we will discuss in the concept on non-reconstruction, in part 3.6. In part
3.7, we will give a final overview, as well as a discussion about confidence
intervals and uncertainty in part 3.8, and some practical example in part 3.9.
In section 4, we will introduce “individual fairness" principles. In part 4.1
we will present the Lipschitz property while in part 4.2, we discuss
“counterfactual fairness", and its connections with causal inference (“what
would have been $y$ is $s$ had been 1, instead of 0?"). Finally, in section 5,
we present standard techniques used to correct a possible discrimination. More
precisely, we will focus on three approaches, namely pre-processing in part
5.1, in-processin in part 5.2, and post-processing in part 5.3.

##  2 Fairness by unawareness

The most popular approach to having a fair classifier is to prohibit the use
of a protected variable in a predictive model. This approach is called
“fairness by ignorance” or “fairness through unawareness”.

######  Definition 2.1 (Fairness Through Unawareness, Kusner et al., (2017)).

We will speak of fairness through unawareness if the sensitive attribute $s$
is not explicitly used in the decision function $\widehat{y}$, i.e. neither in
the construction of the score $m$, nor in the choice of the threshold level
$\tau$, allowing to pass from $m$ to $\widehat{y}$.

It is assumed here that the threshold choice does not depend on the sensitive
criterion $s$, as in Figure 7. In this case, for the favored population (curve
blue on the left, $s=0$), more people in this sub-population have a lower
score (and therefore a lower risk) than for the disadvantaged population
(curve red on the right, $s=1$).

Figure 7: Distributions of $m(\bm{X})$ conditional on $y=1,s=1$ and
$m(\bm{X})$ conditional on $y=0,s=1$ on the right (population assumed
disadvantaged), and distributions of $m(\bm{X})$ conditional on $y=1,s=0$ and
$s$ conditional on $y=0,s=1$, on the left (population assumed favored). The
threshold $\tau$ to go from $\widehat{y}=0$ to $\widehat{y}=1$ is set to 60%.

Removing a sensitive variable from the training database might seem like a
step forward, to insure more fairness of models. However, some of the
unprotected predictor variables may in fact be (highly) correlated with the
sensitive variable, allowing discrimination to go on and even eliminate all
highly correlated variables as well. But this comes at a price, because each
deletion of a variable also deletes valuable information for the prediction
task. Gajane and Pechenizkiy, (2017), Žliobaitė, (2017), Verma and Rubin,
(2018), or Friedler et al., (2019) have identified several concepts of
algorithmic fairness. Most definitions of fairness are based on group
fairness, which addresses statistical fairness in the population as a whole.
In addition, individual fairness states that similar individuals should be
treated the same, regardless of their group membership. In this blog, we will
focus primarily on group fairness, which has the following three definitions:
(i) demographic parity, (ii) equality of opportunity, and (iii) equality of
opportunity. We will now examine these in turn, before proposing some
extensions, inspired by Žliobaitė, (2017) (which mentions about twenty
measures) and Gajane and Pechenizkiy, (2017) (which considers seven major
approaches).

##  3 Group-level fairness

###  3.1 Independence and conditional independence

As we will see in this section, all group-level fairness concepts are related
to independence, between the prediction $\widehat{Y}$ and the sensitive
attribute $S$, possibly conditional on other variables, e.g. between
$\widehat{Y}$ and $S$ conditional on $Y$ (“separation condition”), or $Y$ and
$S$ conditional on $\widehat{Y}$ (“sufficiency condition”). Recall that
independence is characterized by a separability condition, for (generally) two
discrete random variables $X$ and $Y$,

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\
\Longleftrightarrow\leavevmode\nobreak\ \exists g,h:\leavevmode\nobreak\
\mathbb{P}[X=x,Y=y]=g(x)\cdot h(y).$

A weaker notion is the second-order version, $X\perp Y$, meaning simply that
the partial correlation is zero,

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\
\Longrightarrow\leavevmode\nobreak\ X\perp Y\leavevmode\nobreak\
\Longleftrightarrow\leavevmode\nobreak\
\mathbb{E}[(X-\mathbb{E}(X))(Y-\mathbb{E}(Y))]=0,$

i.e. $\text{cov}(X,Y)=0$ or $\text{cor}(X,Y)=0$, Pearson’s linear correlation
is null, for continuous random variables. A characterization of independence
can be obtained using the maximal correlation, introduced in Hirschfeld,
(1935), Gebelein, (1941) and Rényi, (1959), defined as the supremum of
$\text{cor}(f(X),g(Y))$, for all functions $f$ and $g$ such that the
correlation exists,

$\text{cor}^{\star}(X,Y)=\underset{f,g}{\max}\left\\{\text{cor}(f(X),g(Y))\right\\}.$

For computational reasons, it is necessary to normalize functions $f$ and $g$
: let
$\mathcal{S}_{x}=\\{f:\mathcal{X}\rightarrow\mathbb{R}:\mathbb{E}[f(X)]=0\text{
and }\mathbb{E}[f(X)^{2}]=1\\}$ and similarly $\mathcal{S}_{y}$, and then

$\text{cor}^{\star}(X,Y)=\underset{f\in\mathcal{S}_{x},g\in\mathcal{S}_{y}}{\max}\left\\{\mathbb{E}[f(X)g(Y)]\right\\}.$

As proved in Rényi, (1959),

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\
\Longleftrightarrow\text{cor}^{\star}(X,Y)=0.$

For the conditional Independence, recall that

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\
Z\leavevmode\nobreak\ \Longleftrightarrow\leavevmode\nobreak\ \exists
g,h:\leavevmode\nobreak\ \mathbb{P}[X=x,Y=y,Z=z]=g(x,z)\cdot h(y,z).$

Note that if $X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\
Z$ and $X\perp\\!\\!\\!\perp Z$ then we have unconditional independence
between our two variables, $X\perp\\!\\!\\!\perp Y$. And if we have both
$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\ Z$ and
$X\perp\\!\\!\\!\perp Y$ then either $X\perp\\!\\!\\!\perp Z$ or
$Y\perp\\!\\!\\!\perp Z$. Note that

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\
Z\leavevmode\nobreak\ \Longleftrightarrow\leavevmode\nobreak\
Y\perp\\!\\!\\!\perp X\leavevmode\nobreak\ |\leavevmode\nobreak\ Z.$

Moreover

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\
Z\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
U=h(Y)\leavevmode\nobreak\ \Longleftrightarrow\leavevmode\nobreak\
X\perp\\!\\!\\!\perp U\leavevmode\nobreak\ |\leavevmode\nobreak\ Z.$

A weaker notion is the second-order version, $X\perp Y\leavevmode\nobreak\
|\leavevmode\nobreak\ Z$, meaning simply that the partial correlation is zero,

$X\perp\\!\\!\\!\perp Y\leavevmode\nobreak\ |\leavevmode\nobreak\
Z\leavevmode\nobreak\
\Longrightarrow\mathbb{E}[(X-\mathbb{E}(X|Z))(Y-\mathbb{E}(Y|Z))]=0,$

meaning that the conditional correlation is null. And as earlier, a
characterization can be obtained using a conditional maximal correlation : let
$\mathcal{S}_{x|z}=\\{f:\mathcal{X}\rightarrow\mathbb{R}:\mathbb{E}[f(X)|Z]=0\text{
and }\mathbb{E}[f(X)^{2}|Z]=1\\}$ and similarly for $\mathcal{S}_{y|z}$,

$\text{cor}^{\star}(X,Y|Z)=\underset{f\in\mathcal{S}_{x|z},g\in\mathcal{S}_{y|z}}{\max}\Big{\\{}\mathbb{E}[f(X)g(Y)|Z]\Big{\\}}.$

###  3.2 Demographic Parity

As pointed out by Caton and Haas, (2020), there are several ways to define
(formally) the fairness of a classifier, or of a model. For example, one can
wish for independence between the score and the group membership,
$m(\bm{X})\perp\\!\\!\\!\perp S$, or between the prediction (as a class) and
the sensitive variable $\widehat{Y}\perp\\!\\!\\!\perp S$.

######  Definition 3.1 (Demographic Parity, Corbett-Davies et al., (2017),
Agarwal, (2021)).

A decision function $\widehat{y}$ satisfies demographic parity if
$\widehat{Y}\perp\\!\\!\\!\perp S$, i.e.

$\mathbb{P}[\widehat{Y}=y|S=0]=\mathbb{P}[\widehat{Y}=y|S=1],\leavevmode\nobreak\
\forall y\in\\{0,1\\}\leavevmode\nobreak\ -\text{ classification},$

or

$\mathbb{P}[\widehat{Y}\leq y|S=0]=\mathbb{P}[\widehat{Y}\leq
y|S=1],\leavevmode\nobreak\ \forall y\in\mathbb{R}\leavevmode\nobreak\ -\text{
regression}.$

For the later, an implication will be
$\mathbb{E}[\widehat{Y}|S=0]=\mathbb{E}[\widehat{Y}|S=1]$.

The last characterization is equivalent to the two others if $y$ and
$\widehat{y}$ are binary variables. In the case where $y$ is continuous, the
second property corresponds to a notion of “strong" demographic fairness while
the last one corresponds to a notion of “weak" demographic fairness (the
second one implying the third one, but not the reverse).

This demographic fairness, also called “statistical parity", simply requires
that the fraction of blue applicants who are granted credit be approximately
the same as the fraction of red applicants who are granted credit. By
symmetry, the rejection proportions must be identical. Using the same
threshold $\tau$ on the score, to grant credit, as in Figure 3, we see that
statistical parity is not achieved:

$\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=\mathbb{P}[m(\bm{X})>\tau|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=\frac{2}{8}=25\%\text{
et
}\mathbb{P}[m(\bm{X})>\tau|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]=\frac{12}{16}=75\%,$

so that

$\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]\neq\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}].$

Statistical parity is certainly a form of fairness, but it is generally weak
and imperfect. And as the left side of Figure 8 shows, it has nothing to do
with the quality of the predictive model.

Figure 8: ROC curve on the example data in Figure 3 with points in
$\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
with two distinct levels for the threshold on the right, respectively the
“fair” (Figure 9) and “optimal” (Figure 11).

Suppose that

$\mathbb{P}[Y=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=\frac{1}{4}=25\%\text{
et
}\mathbb{P}[Y=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]=\frac{3}{4}=75\%,$

and that the law of $Y$ depends only on $S$. In this case, imposing
statistical parity means choosing the wrong model, because the perfect model
would give

$\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=\frac{1}{4}=25\%\text{
et
}\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]=\frac{3}{4}=75\%.$

On Figure 9, we can see the “optimal” threshold, in the sense of maximum
predictive power, minimizing the rate of error committed, visible on Figure
10, on the left.

11111111111111000000000001 optimal thresold Figure 9:
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
$y\in\\{0,1\\}$, via Kearns and Roth, (2019). Figure 10: False-positive and
false-negative rates, left, and evolution of the
$\displaystyle{{{\mathbb{P}}[\widehat{Y}=1|S=0]}/{\mathbb{P}[\widehat{Y}=1|S=1]}}$,
as a function of the threshold level used.

In Figure 11, we can see the “fair" (or at least “fairer") threshold, in the
sense that
$t\mapsto\mathbb{P}[m(\bm{X})>t|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]/\mathbb{P}[m(\bm{X})>t|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]$
is as large as possible (i.e.
$\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]/\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]$
is as large as possible).

11111111111111000000000001 “fair” thresold Figure 11: Choosing a “fair"
fairness threshold, via Kearns and Roth, (2019).

To summarize, minimizing the error rate (and therefore increasing accuracy)
and maximizing fairness are often two irreconcilable goals.

$\begin{cases}\text{optimal threshold\leavevmode\nobreak\
}&:\leavevmode\nobreak\ \mathbb{P}[\widehat{Y}\neq
Y]=\displaystyle{\frac{6+1}{24}=29.17\%}\text{ and
}\displaystyle{\frac{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]}{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]}=\frac{0}{24}=0\%}\\\
\text{``fair'' threshold\leavevmode\nobreak\ }&:\leavevmode\nobreak\
\mathbb{P}[\widehat{Y}\neq Y]=\displaystyle{\frac{4+4}{24}=33.33\%}\text{ and
}\displaystyle{\frac{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]}{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]}=\frac{2\cdot
16}{12\cdot 4}=33.33\%}\\\ \end{cases}$

We will return later to the trade-off that will often exist between the
fairness of the models, and their accuracy (or predictive power), and the
associated efficiency frontier.

Another shortcoming of this approach is that the desired independence between
the sensitive variable $s$ and the prediction $\widehat{y}$ does not take into
account the fact that the outcome $y$ may be correlated with the sensitive
variable $s$. In other words, if the groups induced by $s$ have different
underlying distributions for $y$, ignoring these dependencies may lead to
results that would be considered fair, but not for the groups themselves.
Quite naturally, an extension of the independence property is the “separation"
criterion which requires independence between the prediction $\widehat{y}$ and
the sensitive variable $S$, conditional on the value of the target variable
$y$, i.e. $\widehat{Y}\perp\\!\\!\\!\perp S$ conditional on $Y$.

This approach can amount to choosing a different threshold, with a lower
threshold for blue individuals  than for red individuals , as in Figure 12. In
this case,

$\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=\mathbb{P}[m(\bm{X})>{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\tau}}|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]=50\%=\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]=\mathbb{P}[m(\bm{X})>{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\tau}}|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}].$

11111111111111000000000001 thresold thresold Figure 12:
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
$y\in\\{0,1\\}$, via Kearns and Roth, (2019).

In Figure 13, we can visualize the rate of false positives in each class, as a
function of threshold, on the left, and the rate of true positives on the
right. This last case is called “equality of opportunity” by Hardt et al.,
(2016).

Figure 13: False positive rate, left, and true positive rate, right.

Demographic parity (or statistical parity) suggests that a predictor is
unbiased if the prediction $\widehat{y}$ is independent of the sensitive
attribute $s$,

$\widehat{Y}\perp\\!\\!\\!\perp S$

so that

$\begin{cases}\mathbb{P}(\widehat{Y}=1|S=0)=\mathbb{P}(\widehat{Y}=1|S=1)=\mathbb{P}(\widehat{Y}=1),\\\
\mathbb{P}(m(\bm{X})\leq\mu|S=0)=\mathbb{P}(\widehat{Y}\leq\gamma|S=1)=\mathbb{P}(m(\bm{X})\leq\mu),\leavevmode\nobreak\
\forall\mu\in[0,1],\ \end{cases}$

We will also speak here of “group fairness". Note that in the first case, the
condition is equivalent to having

$\mathbb{E}(\widehat{Y}|S=0)=\mathbb{E}(\widehat{Y}|S=1)=\mathbb{E}(\widehat{Y})$

but not in the second, when $y$ is continuous. The latter will be true, but it
will not be enough. Here, the same proportion of each population is classified
as positive. However, this may result in different false positive and true
positive rates if the true outcome $y$ does indeed vary with the sensitive
attribute $s$.

An alternative to the independence assumption $\widehat{Y}\perp\\!\\!\\!\perp
S$ is to require that $\widehat{Y}$ and $S$ have zero mutual information,

$IM(\widehat{Y},S)=\mathcal{E}(\widehat{Y})+\mathcal{E}(S)-\mathcal{E}(\widehat{Y},S)=0,$

where $\mathcal{E}$ denotes the entropy, that is

$IM(\widehat{Y},S)=\displaystyle{\sum_{\widehat{y},s}\mathbb{P}(\widehat{y},s)\log{\frac{\mathbb{P}(\widehat{y},s)}{\mathbb{P}(\widehat{y}})}},$

For example, if we want to maintain a threshold at 60% for the advantaged
population (curve red on the left, $s=0$), we must slightly lower the
threshold for the disadvantaged population (curve blue on the right, $s=1$),
with here a threshold slightly lower than 50%. In other words, with such a
choice $\mathbb{P}(\widehat{Y}=1|S=1)=\mathbb{P}(\widehat{Y}=1|S=0)$. The
disadvantage of this method is that the true and false positive rates may be
completely different in the two subpopulations.

###  3.3 Equalized odds (and other related concepts)

Equal opportunity and equality of opportunity are not so much a measure of
fairness as a potential definition of fairness. Equal opportunity is achieved
when the predicted target variable of a $\widehat{y}$ model and the label of a
sensitive category $s$ are statistically independent of each other,
conditional on the actual value of the target variable $y$. In a binary
classification task, this can be simplified by requiring that true positive
rates and false positive rates be equal between groups, where the groups are
determined by the protected category. A slightly less demanding fairness
criterion is equal opportunity, in which only the probability of the true
positive is equalized across groups in a protected category. Formally, we have
the following definitions, where we require parity of false or true positives
(Figures 14 and 15 respectively).

######  Definition 3.2 (True positive equality, Equalized Odds, Hardt et al.,
(2016)).

We will speak of equality of opportunity, or parity of true positives, if

$\mathbb{P}[\widehat{Y}=1|S=0,Y=1]=\mathbb{P}[\widehat{Y}=1|S=1,Y=1]$

or equivalently

$\text{TPR}_{0}=\frac{\text{TP}_{0}}{\text{FN}_{0}+\text{TP}_{0}}=\frac{\text{TP}_{1}}{\text{FN}_{1}+\text{TP}_{1}}=\text{TPR}_{1}.$

######  Definition 3.3 (False positive equality, Hardt et al., (2016)).

We will speak of equality of false positives if

$\mathbb{P}[\widehat{Y}=1|S=0,Y=0]=\mathbb{P}[\widehat{Y}=1|S=1,Y=0],$

or equivalently

$\text{FPR}_{0}=\frac{\text{FP}_{0}}{\text{TN}_{0}+\text{FP}_{0}}=\frac{\text{FP}_{1}}{\text{TN}_{1}+\text{FP}_{1}}=\text{FPR}_{1}.$

Figure 14: Distributions of $m(\bm{X})$ conditional on $y=1,s=0$ and $s$
conditional on $y=0,s=0$, left (assumed favored population), and distributions
of $s$ conditional on $y=1,s=1$ and conditional on $y=0,s=1$, on the right
(supposedly disadvantaged population), with
$\mathbb{P}(\widehat{Y}=1|S=1,Y=0)=\mathbb{P}(\widehat{Y}=1|S=0,Y=0)$ (here of
the order of 9.6% false positives).  Figure 15: Distributions of $m(\bm{X})$
conditional on $y=1,s=0$ and $y=0,s=0$, left (assumed favored population), and
distributions of $m(\bm{X})$ conditional on $y=1,s=1$ and $y=0,s=1$, on the
right (supposedly disadvantaged population), with
$\mathbb{P}(\widehat{Y}=1|S=1,Y=1)=\mathbb{P}(\widehat{Y}=1|S=0,Y=1)$ (here of
the order of 66.3% true positives).

######  Definition 3.4 (Equalized of opportunity, Hardt et al., (2016)).

The parity of false positives and true positives is called equality of
opportunity,

$\begin{cases}\mathbb{P}[\widehat{Y}=1|S=0,Y=1]=\mathbb{P}[\widehat{Y}=1|S=1,Y=1]\\\
\mathbb{P}[\widehat{Y}=1|S=0,Y=0]=\mathbb{P}[\widehat{Y}=1|S=1,Y=0]\end{cases}$

or

$\mathbb{P}[\widehat{Y}=1|S=0,Y=y]=\mathbb{P}[\widehat{Y}=1|S=1,Y=y],\leavevmode\nobreak\
\forall y\in\\{0,1\\}$

in other words, $\widehat{Y}\perp\\!\\!\\!\perp S$ conditionally on $Y$.

On can also use any measure based on confusion matrices, such as $\phi$,
introduced by Matthews, (1975),

######  Definition 3.5 ($\phi$-fairness, Chicco and Jurman, (2020)).

We will have $\phi$-fairness if $\phi_{1}=\phi_{0}$, where $\phi_{s}$ denotes
Matthews correlation coefficient for the $s$ group,

$\phi_{s}={\displaystyle={\frac{\text{TP}_{s}\cdot\text{TN}_{s}-\text{FP}_{s}\cdot\text{FN}_{s}}{\sqrt{(\text{TP}_{s}+\text{FP}_{s})(\text{TP}_{s}+\text{FN}_{s})\cdot(\text{TN}_{s}+\text{FP}_{s})(\text{TN}_{s}+\text{FN}_{s})}}}}$

All those measures are based on some choice of thresholds, but it is also
possible to consider a global measures of calibration, such as the area under
the curve,

######  Definition 3.6 (AUC fairness, Borkan et al., (2019)).

We will have AUC fairness if $\text{AUC}_{1}=\text{AUC}_{0}$, where
$\text{AUC}_{s}$ is the AUC for the $s$ group.

We find a similar idea in Beutel et al., (2019). The problem with the AUC is
that we can have identical AUCs, but very different underlying ROC curves. So,
it can be interesting to consider a notion of fairness based on the ROC
curves. As a reminder, we had defined the ROC curve as
$t\mapsto\text{TPR}\circ\text{FPR}^{-1}(t)$.

######  Definition 3.7 (Equality of ROC curves, Vogel et al., (2021)).

Let $\text{FRP}_{s}(t)=\mathbb{P}[m(\bm{X})>t|Y=0,S=s]$ and
$\text{TPR}_{s}(t)=\mathbb{P}[m(\bm{X})>t|Y=1,S=s]$. Set
$\Delta_{TPR}(t)=\text{TPR}_{1}\circ\text{TPR}_{0}^{-1}(t)-t$ et
$\Delta_{FRP}(t)=\text{FPR}_{1}\circ\text{FPR}_{0}^{-1}(t)-t$. We will have an
fairness of ROC curves if
$\|\Delta_{TPR}\|_{\infty}=\|\Delta_{FPR}\|_{\infty}=0$.

An (implicit) assumption made here is that class 1 (or  in our illustration)
in the sensitive attribute $S$ represents a socially sensitive group, i.e., a
minority group that is discriminated against), such that disparate impact is
defined by positive (i.e., desirable) outcomes. Equal opportunity is satisfied
if the prediction $\widehat{Y}$ is conditionally independent of the protected
attribute $S$, given the actual value $Y$,

$\forall y:\leavevmode\nobreak\ \widehat{Y}\perp\\!\\!\\!\perp
S\leavevmode\nobreak\ |\leavevmode\nobreak\ Y=y$

or

$\mathbb{P}(\widehat{Y}=1|S=0,Y=y)=\mathbb{P}(\widehat{Y}=1|S=1,Y=y)=\mathbb{P}(\widehat{Y}=1|Y=y),\leavevmode\nobreak\
\forall y\in\\{0,1\\}$

which is equivalent to having

$\mathbb{E}(\widehat{Y}|S=0,Y=y)=\mathbb{E}(\widehat{Y}|S=1,Y=y)=\mathbb{E}(\widehat{Y}|Y=y),\leavevmode\nobreak\
\forall y\in\\{0,1\\},$

The latter will be true, but it will not be enough. This means that the true
positive rate and the false positive rate will be the same for each
population; each type of error is matched between each group.

In our illustration, equality of opportunity is impossible to achieve. Indeed,
this definition of fairness suggests that the false positive and true positive
rates be the same for both populations. This may be reasonable, but in the
illustrative example, it is impossible because the two ROC curves do not
intersect. Note that if the curves did cross, this could impose threshold
choices that would be unattractive in practice (with acceptance rates
potentially much too low, or too high).

Equality of opportunity, defined by Hardt et al., (2016), has the same
mathematical formulation as equality of opportunity, for a classifier, but it
focuses on a particular label,

$\exists y:\leavevmode\nobreak\ \widehat{Y}\perp\\!\\!\\!\perp
S\leavevmode\nobreak\ |\leavevmode\nobreak\ Y=y$

Typically, we will focus on the $1$ label of the true $y$ value, to define
equal opportunity, so that

$\mathbb{P}(\widehat{Y}=0|Y=1,S=s)=\mathbb{P}(\widehat{Y}=0|Y=1),\leavevmode\nobreak\
\forall Ss\in\\{0,1\\},$

which is the same as comparing the rates of negative rates. In this case, we
want the true positive rate $\mathbb{P}(\widehat{Y}=1|Y=1)$ to be the same for
each population without taking into account the errors when $y=0$. In effect,
this means that the same proportion of each population receives the “good"
result $y=1$.

The deviation from equality of opportunity is measured by the difference in
equality of opportunity:

$EOD=\mathbb{P}(\widehat{Y}=1|Y=1,S=1)-\mathbb{P}(\widehat{Y}=1|Y=1,S=0)$

As we have here a binary variable $y$, the condition on the probability of
$\widehat{y}$ will be here equivalent to the condition on the expectation

$\mathbb{E}(\widehat{Y}|Y=1,S=s)=\mathbb{E}(\widehat{Y}|Y=1),\leavevmode\nobreak\
\forall s\in\\{0,1\\},$

In our illustration, equality of opprtunities is equivalent to finding
equivalent levels of positive true rates on the ROC curves.

Figure 16: ROC curves for the two subpopulations, the assumed disfavoured (or
$s=1$) and the assumed favoured (or $s=0$). On the left, the two points
correspond to a threshold of 60%, identical for the two populations (strategy
of blinding to the protected criterion). On the right, the case of equality of
opportunity, where the threshold for the disadvantaged population is the
threshold induced by
${\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\bullet}$,
chosen so as to have the same rate of true positives as on the favoured
population
${\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\bullet}$.

As visible on the right side of Figure 16, the thresholds are chosen so that
the rate of true positives is the same for both populations. In other words,
we must have the same proportion offered credit in each group (advantaged and
disadvantaged). For example, here we keep the threshold of 60% on the score
for the advantaged population (corresponding to a true positive rate of about
63%), and we must use a threshold of about 70% on the score for the
disadvantaged population (corresponding to a true positive rate of about 63%).

Finally, instead of focusing solely on one quantity (false positive rate for
instance), it is possible to consider some function of a pair of those
quantities.

######  Definition 3.8 (Equal treatment, Berk et al., (2021)).

We have equality of treatment, the rate of false positives and false negatives
are identical in the protected groups,

$\frac{\mathbb{P}[\widehat{Y}=1|S=0,Y=0]}{\mathbb{P}[\widehat{Y}=0|S=0,Y=1]}=\frac{\mathbb{P}[\widehat{Y}=1|S=1,Y=0]}{\mathbb{P}[\widehat{Y}=0|S=1,Y=1]}$

Berk et al., (2021) uses the processing term in connection with causal
inference which we will discuss next. If the classifier produces more false
negatives than false positives for the supposedly privileged group, this means
that more disadvantaged individuals receive a favorable outcome than the
reverse.

A slightly different version had been proposed by Jung et al., (2020),

######  Definition 3.9 (Equalizing Disincentives, Jung et al., (2020)).

The difference between the true positive rate and the false positive rate must
be the same in the protected groups,

${\mathbb{P}[\widehat{Y}=1|S=0,Y=1]}-{\mathbb{P}[\widehat{Y}=1|S=0,Y=0]}={\mathbb{P}[\widehat{Y}=1|S=1,Y=1]}-{\mathbb{P}[\widehat{Y}=1|S=1,Y=0]}$

###  3.4 Conditional Demographic Parity

All the discussion we have just had can be extended by also conditioning on
some explanatory variables

######  Definition 3.10 (Conditional demographic parity, Corbett-Davies et
al., (2017)).

We will have a conditional demographic parity if

$\mathbb{P}[\widehat{Y}=y|\bm{X}_{\ell}=\bm{x},S=0]=\mathbb{P}[\widehat{Y}=y|\bm{X}_{\ell}=\bm{x},S=1],\leavevmode\nobreak\
\forall y\in\\{0,1\\}$

where $\ell$ denotes a “legitimate" subset of unprotected covariates.

###  3.5 Class balance and calibration

Instead of predicting the value of $\widehat{y}$ (conditional on $y$ and $s$),
Kleinberg et al., (2016) had suggested predicting the average value of
$m(\bm{x})$:

######  Definition 3.11 (Class balance, Kleinberg et al., (2016)).

We will have class balance in the weak sense if

$\mathbb{E}[m(\bm{X})|Y=y,S=0]=\mathbb{E}[m(\bm{X})|Y=y,S=1],\leavevmode\nobreak\
\forall y\in\\{0,1\\}$

or in the strong sense if

$\mathbb{P}[m(\bm{X})\leq\mu|Y=y,S=0]=\mathbb{P}[m(\bm{X})\leq\mu|Y=y,S=1],\leavevmode\nobreak\
\forall\mu\in[0,1],\leavevmode\nobreak\ \forall y\in\\{0,1\\}.$

A third commonly used criterion is sometimes called “sufficiency", which
requires independence between the target $Y$ and the sensitive variable $S$,
conditional on a given score $m(\bm{X})$ (or forecast), $\widehat{Y}$,
introduced by Sokolova et al., (2006), and later taken up by Kleinberg et al.,
(2016) and Zafar et al., (2017). In most of the definitions we had seen, we
were interested (only) in $\widehat{y}$, but it is also possible to use the
score $m(\bm{X})$. Therefore, we aim at

$Y\perp\\!\\!\\!\perp m(\bm{X})\leavevmode\nobreak\ |\leavevmode\nobreak\ S$

so that

$\mathbb{P}(Y=1|m(\bm{X})=\mu,S=0)=\mathbb{P}(Y=1|m(\bm{X})=\mu,S=1)=\mathbb{P}(Y=1|m(\bm{X})=\mu),\leavevmode\nobreak\
\forall\mu\in[0,1].$

######  Definition 3.12 (Calibration (or accuracy) parity, Kleinberg et al.,
(2016), Zafar et al., (2017)).

We have calibration parity if

$\mathbb{P}[Y=1|m(\bm{X})=\mu,S=0]=\mathbb{P}[Y=1|s(\bm{X})=\mu,S=1],\leavevmode\nobreak\
\forall\mu\in[0,1].$

We can go further by asking for a little more, by asking not only for parity,
but also for a good calibration

######  Definition 3.13 (Good calibration, Kleinberg et al., (2017)).

We have an fairness of good calibration if

$\mathbb{P}[Y=1|m(\bm{X})=\mu,S=0]=\mathbb{P}[Y=1|m(\bm{X})=\mu,S=1]=\mu,\leavevmode\nobreak\
\forall\mu\in[0,1].$

This “good calibration" property of the model $m$, also called “well-
calibration" in Dawid, (1982), and “autocalibration" in Van Calster et al.,
(2019), Krüger and Ziegel, (2021) and Denuit et al., (2021) in the context of
regression, i.e. $\mathbb{E}[Y|m(\bm{X})=\mu]=\mu$, is a standard property in
econometrics, in generalized linear models, but not in most machine learning
algorithms.

A weaker version would be

######  Definition 3.14 (Predictive parity (1), Chouldechova, (2017)).

We have a predictive parity if

$\mathbb{P}[Y=1|\widehat{Y}=1,S=0]=\mathbb{P}[Y=1|\widehat{Y}=1,S=1].$

Note that if $\widehat{y}$ is not a perfect classifier
($\mathbb{P}[\widehat{Y}\neq Y]>0$), and if the two groups are not balanced
($\mathbb{P}[S=0]\neq\mathbb{P}[S=1]$), then it is impossible to have
predictive parity and equal opportunity at the same time. Note that

$\text{PPV}_{s}=\frac{\text{TPR}\cdot\mathbb{P}[S=s]}{\text{TPR}\cdot\mathbb{P}[S=s]+\text{FPR}\cdot(1-\mathbb{P}[S=s])},\leavevmode\nobreak\
\forall s\in\\{0,1\\},$

such that $\text{PPV}_{0}=\text{PPV}_{1}$ implies that either TPR or FPR is
zero, and since

$\text{NPV}_{s}=\frac{(1-\text{FPR})\cdot(1-\mathbb{P}[S=s])}{(1-\text{TPR})\cdot\mathbb{P}[S=s]+(1-\text{FPR})\cdot(1-\mathbb{P}[S=s])},\leavevmode\nobreak\
\forall s\in\\{0,1\\},$

such that $\text{NPV}_{0}\neq\text{NPV}_{1}$, and predictive parity cannot be
achieved.

Continuing the formalism of Chouldechova, (2017), Barocas et al., (2019)
proposed an extension to predictive parity

######  Definition 3.15 (Predictive parity (2), Barocas et al., (2019)).

$\begin{cases}\mathbb{P}[{Y}=1|S=0,\widehat{Y}=1]=\mathbb{P}[{Y}=1|S=1,\widehat{Y}=1]\leavevmode\nobreak\
&\leavevmode\nobreak\ \text{ positive prediction}\\\
\mathbb{P}[{Y}=1|S=0,\widehat{Y}=0]=\mathbb{P}[{Y}=1|S=1,\widehat{Y}=0]\leavevmode\nobreak\
&\leavevmode\nobreak\ \text{ negative prediction}\end{cases}$

or

$\mathbb{P}[{Y}=1|S=0,\widehat{Y}=\widehat{y}]=\mathbb{P}[{Y}=1|S=1,\widehat{Y}=\widehat{y}],\leavevmode\nobreak\
\forall\widehat{y}\in\\{0,1\\}$

Finally, let us note that Kleinberg et al., (2017) introduced a notion of
balance for positive / negative class.

$\begin{cases}\mathbb{E}(m(\bm{X})|Y=1,S=1)=\mathbb{E}(m(\bm{X})|Y=1,S=0),\leavevmode\nobreak\
\text{balance for positive class}\\\
\mathbb{E}(m(\bm{X})|Y=0,S=1)=\mathbb{E}(m(\bm{X})|Y=0,S=0),\leavevmode\nobreak\
\text{equilibrium for the negative class.}\\\ \end{cases}$

###  3.6 Principle of non-reconstruction

A last approach can be inspired by Kim, (2017), for whom, another way to
define if a classification is fair, or not, is to say that we cannot tell from
the result if the subject was member of a protected group or not. In other
words, if an individual’s score does not allow us to predict that individual’s
attributes better than guessing them without any information, we can say that
the score was assigned fairly.

######  Definition 3.16 (Non-reconstruction of the protected attribute, Kim,
(2017)).

If we cannot tell from the result ($\bm{x}$, $m(\bm{x})$, $y$ and
$\widehat{y}$) whether the subject was a member of a protected group or not,
we will talk about fairness by non-reconstruction of the protected attribute

$\mathbb{P}[S=0|\bm{X},m(\bm{X}),\widehat{Y},Y]=\mathbb{P}[S=1|\bm{X},m(\bm{X}),\widehat{Y},Y].$

###  3.7 Comparison of fairness criteria

Demographic parity would result in

$\mathbb{P}(\widehat{Y}=1|m(\bm{X})=\mu)=\mathbb{P}(\widehat{Y}=1),\leavevmode\nobreak\
\forall\mu,$

or $\text{TP}+\text{FP}$ must be identical on both groups ($s=0$ and $s=1$).
On the confusion matrices of Figure 17, it is the case, because the positive
rate is $50\%$ in both groups. The notion of equality of opportunity means

$\mathbb{P}(\widehat{Y}=1|S=0,Y=y)=\mathbb{P}(\widehat{Y}=1|S=1,Y=y)=\mathbb{P}(\widehat{Y}=1|Y=y),\leavevmode\nobreak\
\forall y\in\\{0,1\\},$

in other words, the false positive and false negative rates must be identical:

$\frac{\text{FP}}{\text{TN}+\text{FP}}\text{ and
}\frac{\text{FN}}{\text{TP}+\text{FN}},$

must be identical on both groups. This is the case on confusion matrices of
Figure 17, because the false positive rate is 50%, whether $s$ is $0$ or $1$
(with 30/60 versus 20/40), and the false negative rate is also 50% in both
groups. For predictive parity

$\mathbb{P}(Y=1|S=0,\widehat{Y}=y)=\mathbb{P}(Y=1|S=1,\widehat{Y}=y)=\mathbb{P}(Y=1|\widehat{Y}=y),\leavevmode\nobreak\
\forall y\leavevmode\nobreak\ $

in other words, the positive and negative predictive values must be identical:

$\frac{\text{TP}}{\text{TP}+\text{FP}}\text{ and
}\frac{\text{TN}}{\text{TN}+\text{FN}}$

must be identical on both groups. But here the positive predictive values are
respectively 60% and 40%, depending on the value of $s$ (respectively 30/50
and 20/50). For the global accuracy,

$\mathbb{P}(\widehat{Y}=Y|S=s)=\mathbb{P}(\widehat{Y}=Y),\leavevmode\nobreak\
\forall s$

or $\text{TP}+\text{TN}$ must be identical on both groups ($s=0$ and $s=1$).
This is the case here because the rate of well classified observations is 50

$\mathbb{P}(\widehat{Y}=Y|m(\bm{X})=\mu)=\mathbb{P}(\widehat{Y}=Y),\leavevmode\nobreak\
\forall\mu\in[0,1],$

or $\text{FP}/\text{FN}$ must be identical for both groups ($s=0$ and $s=1$).
But here, the rates are respectively 3/2 and 2/3, which are not equal.

$\hat{y}=1$$\hat{y}=0$$y=0$$y=1$TN=30%FN=20%FP=30%TP=20%$\hat{y}=1$$\hat{y}=0$$y=0$$y=1$TN=20%FN=30%FP=20%TP=30%$s=0$$s=1$
Figure 17: fairness of a classifier from the confusion matrices, on the two
subpopulations, $s=0$ on the left and $s=1$ on the right.

We will now consider the different possible ways of setting these thresholds
that result in different senses of fairness. We emphasize that we are not
advocating any particular criteria, but simply exploring the ramifications of
different choices. For demographic parity, the threshold could be chosen so
that the same proportion of each group is classified as $\widehat{y}=1$. For
equal opportunity, thresholds are chosen so that the true positive rate is the
same for both populations (Figure 4). Of those who repay the loan, the same
proportion are offered credit in each group. For the two ROC curves, this
means that the thresholds are chosen so that the vertical position on each
curve is the same regardless of the horizontal position (Figure 2c). However,
this means that different proportions of the blue and yellow groups receive
loans (Figure 4b).

The different notions of fairness can be summarized in Table 1.

statistical parity | Dwork et al., (2012) | $\mathbb{P}[\widehat{Y}=1|S=s]=\text{cst},\leavevmode\nobreak\ \forall s$ | independence
---|---|---|---
conditional statistical parity | Corbett-Davies et al., (2017) | $\mathbb{P}[\widehat{Y}=1|S=s,X=x]=\text{cst}_{x},\leavevmode\nobreak\ \forall s,y$ | $\widehat{Y}\perp\\!\\!\\!\perp S$
equalized odds | Hardt et al., (2016) | $\mathbb{P}[\widehat{Y}=1|S=s,Y=y]=\text{cst}_{y},\leavevmode\nobreak\ \forall s,y$ | separation
equalized opportunity | Hardt et al., (2016) | $\mathbb{P}[\widehat{Y}=1|S=s,Y=1]=\text{cst},\leavevmode\nobreak\ \forall s$ |
predictive equality | Corbett-Davies et al., (2017) | $\mathbb{P}[\widehat{Y}=1|S=s,Y=0]=\text{cst},\leavevmode\nobreak\ \forall s$ | $\widehat{Y}\perp\\!\\!\\!\perp S\leavevmode\nobreak\ |\leavevmode\nobreak\ Y$
balance (positive) | Kleinberg et al., (2017) | $\mathbb{E}[m(\bm{X})|S=s,Y=1]=\text{cst},\leavevmode\nobreak\ \forall s$ | $m(\bm{X})\perp\\!\\!\\!\perp S\leavevmode\nobreak\ |\leavevmode\nobreak\ Y$
balance (negative) | Kleinberg et al., (2017) | $\mathbb{E}[m(\bm{X})|S=s,Y=0]=\text{cst},\leavevmode\nobreak\ \forall s$ |
conditional accuracy equality | Berk et al., (2017) | $\mathbb{P}[Y=y|S=s,\widehat{Y}=y]=\text{cst}_{y},\leavevmode\nobreak\ \forall s,y$ | sufficiency
predictive parity | Chouldechova, (2017) | $\mathbb{P}[Y=1|S=s,\widehat{Y}=1]=\text{cst},\leavevmode\nobreak\ \forall s$ |
calibration | Chouldechova, (2017) | $\mathbb{P}[Y=1|m(\bm{X})=\mu,S=s]=\text{cst}_{\mu},\leavevmode\nobreak\ \forall\mu,s$ | $Y\perp\\!\\!\\!\perp S\leavevmode\nobreak\ |\leavevmode\nobreak\ \widehat{Y}$
well-calibration | Chouldechova, (2017) | $\mathbb{P}[Y=1|m(\bm{X})=\mu,S=s]=\mu,\leavevmode\nobreak\ \forall\mu,s$ |
accuracy equality | Berk et al., (2017) | $\mathbb{P}[\widehat{Y}=Y|S=s]=\text{cst},\leavevmode\nobreak\ \forall s$ |
treatment equality | Berk et al., (2017) | $\displaystyle{\frac{\text{FNR}_{s}}{\text{FPR}_{s}}=\text{cst}_{s}},\leavevmode\nobreak\ \forall s$ |
Table 1: Group Fairness Definitions.

###  3.8 Relaxation and confidence intervals

From a statistical perspective, achieving fairness is impossible since it
requires equality between probabilities, or strict independence. Thus, we will
discuss practical use of those concepts.

####  3.8.1 Exogenous threshold and relaxation

We had seen that the demographic fairness is translated by the equality

$\frac{\mathbb{P}[\widehat{Y}=1|S=0]}{\mathbb{P}[\widehat{Y}=1|S=1]}=1=\frac{\mathbb{P}[\widehat{Y}=1|S=1]}{\mathbb{P}[\widehat{Y}=1|S=0]}$

If this approach is intellectually interesting, the statistical reality is
that having a perfect equality between two (predictive) probabilities is often
impossible.

######  Definition 3.17 (Disparate impact, Feldman et al., (2015)).

A decision function $\widehat{Y}$ has a disparate impact, for a given
threshold $d$, if,

$\min\\{\frac{\mathbb{P}[\widehat{Y}=1|S=0]}{\mathbb{P}[\widehat{Y}=1|S=1]},\frac{\mathbb{P}[\widehat{Y}=1|S=1]}{\mathbb{P}[\widehat{Y}=1|S=0]}\\}<d\leavevmode\nobreak\
\text{(usually 80\%)}.$

This so-called “four-fifths rule", coupled with the $d=80\%$ threshold, was
originally defined by the State of California Fair Employment Practice
Commission (FEPC) Technical Advisory Committee on Testing, which issued the
California State Guidelines on Employee Selection Procedures in October 1972,
as recalled in Feldman et al., (2015), Mercat-Bruns, (2016) or Biddle, (2017).
This standard was later adopted in the 1978 Uniform Guidelines on Employee
Selection Procedures, used by the Equal Employment Opportunity Commission
(EEOC), the U.S. Department of Labor, and the U.S. Department of Justice. An
important point here is that this form of discrimination occurred even when
the employer did not intend to discriminate, but by looking at employment
statistics (on gender or racial grounds), it was possible to observe (and
correct) discriminatory bias.

For example, on the data in Figure 9,

$\frac{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}]}{\mathbb{P}[\widehat{Y}=1|S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}]}=\frac{1}{3}\ll
80\%.$

Another approach, suggested to relax the equality
$\mathbb{P}(\widehat{Y}=1|S=0)=\mathbb{P}(\widehat{Y}=1|S=1)$, consists in
introducing a notion of $\varepsilon$-fairness

$|\mathbb{P}(\widehat{Y}=1|S=0)-\mathbb{P}(\widehat{Y}=1|S=1)|<\varepsilon.$

The left deviation is sometimes called “statistical parity difference"
($SPD$). Žliobaite, (2015) suggests normalizing the statistical parity
difference,

$NSPD=\frac{SPD}{D_{\max}}\text{ where
}D_{\max}=\min\\{\frac{\mathbb{P}(\widehat{Y}=1)}{\mathbb{P}(S=1)},\frac{\mathbb{P}(\widehat{Y}=0)}{\mathbb{P}(S=0)}\\}$

so that $NSPD=1$ for maximum discrimination.

####  3.8.2 Endogenous threshold and confidence intervals

Besse et al., (2018) proposes another approach, based on confidence intervals
for fairness criteria. For example, for the disparate impact, we have seen
that we should calculate

$T=\frac{\mathbb{P}[\widehat{Y}=1|S=0]}{\mathbb{P}[\widehat{Y}=1|S=1]}$

whose empirical version is

$\widehat{t}_{n}=\frac{\sum_{i}\widehat{y}_{i}\bm{1}(s_{i}=0)}{\sum_{i}\widehat{y}_{i}\bm{1}(s_{i}=1)}\cdot\frac{\sum_{i}\bm{1}(s_{i}=1)}{\sum_{i}\bm{1}(s_{i}=0)}$

which can be used to construct a confidence interval for $T$ (Besse et al.,
(2018) proposes an asymptotic test, but resampling methods are possible).

###  3.9 Implementation and comparison

On the toy example (with 24 observations) of Figure 3, repeated in table 2, we
obtain the values in table 3. Note that the ‘diff’ column gives the absolute
difference between the two probabilities (expressed as percentages) and the
‘(%)’ column gives the relative difference between the two probabilities,
expressed as a percentage. 3.

$s$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$y$ | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1
$\widehat{y}$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
Table 2: Data from Figure 3, ordered according to their $S$-score (not shown here) with an identical threshold for both groups ($s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$), and with $\widehat{y}=m_{\tau}(\bm{x})$, for some threshold $\tau$. Name | Probabilistic formula |  |  | diff | (%)
---|---|---|---|---|---
statistical parity | $\mathbb{P}[\widehat{Y}=1|S=\circ]$ | 25.0% | 75.0% | 50.0 | +200.0%
equalized opportunity | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=1]$ | 40.0% | 88.9% | 48.9 | +122.2%
predictive equality | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=0]$ | 0.0% | 57.1% | 57.1 | -
conditional accuracy | $\mathbb{P}[Y=0|S=\circ,\widehat{Y}=0]$ | 50.0% | 75.0% | 25.0 | +50.0%
predictive parity | $\mathbb{P}[Y=1|S=\circ,\widehat{Y}=1]$ | 100.0% | 66.7% | -33.3 | -33.3%
accuracy equality | $\mathbb{P}[\widehat{Y}=Y|S=\circ]$ | 62.5% | 68.8% | 6.2 | +10.0%
treatment equality | $\text{FN}_{\circ}/\text{FP}_{\circ}$ | - | 25.0% | - | -
Table 3: Data from Figure 3 and Table 2, with the different concepts of
fairness, the values of the measures for the two groups,
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
the absolute difference, and the relative difference (as a percentage).

On the continuous example of Figure 6, we get the values in the table 4, with
the same threshold of 60% for both groups, as in Figure 18, or with two
different thresholds, with $55\%$ and $65\%$ respectively for the bullet  (or
$s=0$) and square  (or $s=1$) groups, as in Figure 19. In Figures 18 and 19,
at the top are the conditional densities of
$m(\bm{X})|Y=1,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}$
(solid line) and
$m(\bm{X})|Y=0,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}$
(dashed line), on the left (assumed disadvantaged population), and the
conditional densities of
$m(\bm{X})|Y=1,S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}$
and
$m(\bm{X})|Y=0,S=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}$,
on the right (population assumed to be favored) At the bottom are the survival
functions, $t\mapsto\mathbb{P}(m(\bm{X})>t|Y=y,S=\circ)$, with, in Figure 18,

$\widehat{y}=\begin{cases}\bm{1}_{[60\%,100\%]}(s)\text{ if
}s=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}\text{
or }0\\\ \bm{1}_{[60\%,100\%]}(s)\text{ if
}s=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\text{
or }1\end{cases}$

and on Figure 19,

$\widehat{y}=\begin{cases}\bm{1}_{[55\%,100\%]}(s)\text{ if
}s=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}\text{
or }0\\\ \bm{1}_{[65\%,100\%]}(s)\text{ if
}s=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\text{
or }1\\\ \end{cases}$

The probabilities indicated on the survival functions are the probabilities of
“true" positives or negatives, according to, for example A VERIFIER !!!!!!!

$\mathbb{P}(\widehat{Y}=0|Y=0,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}})=90.4\%\text{
et
}\mathbb{P}(\widehat{Y}=1|Y=1,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}})=66.3\%$

while the rates of negative or positive “falses" are respectively

$\mathbb{P}(\widehat{Y}=1|Y=0,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}})=9.6\%\text{
et
}\mathbb{P}(\widehat{Y}=0|Y=1,S=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}})=33.7\%.$

Figure 18: Continuous example, with the same single ($60\%$) in both groups, $s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$ or $p\in\\{0,1\\}$, with the density of $m(\bm{X})$ on top, and its survival function on the bottom, and $\widehat{y}=\bm{1}_{[60\%,100\%]}(m(\bm{x}))$.  Name | Probabilistic formula |  |  | diff | (%)
---|---|---|---|---|---
statistical parity | $\mathbb{P}[\widehat{Y}=1|S=\circ]$ | 38% | 56.8% | 18.8 | 49.5 %
equalized opportunity | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=1]$ | 66.3% | 76.8% | 10.5 | 15.9%
predictive equality | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=0]$ | 9.6% | 36.7% | 27.1 | 281.2%
conditional accuracy | $\mathbb{P}[Y=0|S=\circ,\widehat{Y}=0]$ | 72.8% | 73.2% | 0.4 | 0.5%
predictive parity | $\mathbb{P}[Y=1|S=\circ,\widehat{Y}=1]$ | 87.3% | 67.7% | -19.6 | -22.5%
accuracy equality | $\mathbb{P}[\widehat{Y}=Y|S=\circ]$ | 62% | 56.8% | -5.3 | -8.5%
treatment equality | $\text{FN}_{\circ}/\text{FP}_{\circ}$ | 350.1 | 63.2 | -286.9 | -82%
Table 4: Different concepts of fairness based on Figure 18, with the values of
the measures for both groups,
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$
or $s\in\\{0.1\\}$, the absolute difference and the relative difference (in
percent), with the same cutoff (60%) in both groups to obtain $\widehat{y}$
from the $m$ score.

Figure 19: Continuous example, with different thresholds ($55\%$ and $65\%$) in the two groups, $s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$ or $s\in\\{0,1\\}$, with the density of $m(\bm{X})$ at the top, and its survival function at the bottom, and $\widehat{y}=\bm{1}_{[55\%,100\%]}(m(\bm{x}))$ if $s=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}$ and $\widehat{y}=\bm{1}_{[65\%,100\%]}(m(\bm{x}))$ if $s=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}$. Name | Probabilistic formula |  / 0 |  / 1 | diff | (%)
---|---|---|---|---|---
statistical parity | $\mathbb{P}[\widehat{Y}=1|S=\circ]$ | 25.0% | 75.0% | 50.0 | +200.0%
equalized opportunity | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=1]$ | 40.0% | 88.9% | 48.9 | +122.2%
predictive equality | $\mathbb{P}[\widehat{Y}=1|S=\circ,Y=0]$ | 0.0% | 57.1% | 57.1 | -
conditional accuracy | $\mathbb{P}[Y=0|S=\circ,\widehat{Y}=0]$ | 50.0% | 75.0% | 25.0 | +50.0%
predictive parity | $\mathbb{P}[Y=1|S=\circ,\widehat{Y}=1]$ | 100.0% | 66.7% | -33.3 | -33.3%
accuracy equality | $\mathbb{P}[\widehat{Y}=Y|S=\circ]$ | 62.5% | 68.8% | 6.2 | +10.0%
treatment equality | $\text{FN}_{\circ}/\text{FP}_{\circ}$ | - | 25.0% | - | -
Table 5: Different concepts of fairness based on Figure 19, with the values of
the measures for the two groups,
$s\in\\{\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}},\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}\\}$,
the absolute difference, and the relative difference (in percent), with a
different threshold (55% if
$s=\text{{\color[rgb]{0.2890625,0.56640625,0.88671875}\definecolor[named]{pgfstrokecolor}{rgb}{0.2890625,0.56640625,0.88671875}\scriptsize\faCircle}}$
and 65% if
$s=\text{{\color[rgb]{0.81640625,0.0078125,0.10546875}\definecolor[named]{pgfstrokecolor}{rgb}{0.81640625,0.0078125,0.10546875}\scriptsize\faStop}}$)
in the two groups to obtain $\widehat{y}$ from the score $m(\bm{X})$.

##  4 Individual-level fairness

In the previous section, we were interested in a notion of “group" fairness,
with subgroups constituted by the values of $y$, $\widehat{y}$ and $s$, since
the three variables are categorical. Individual fairness is a relatively
different concept. The criteria discuessed in the previous section were all
group-based, whereas individual fairness, as the name suggests, is based on
the individual observations. It was first proposed in Dwork et al., (2012).
The notion of individual fairness emphasizes that similar individuals (on
unprotected attributes) should be treated similarly.

###  4.1 Lipschitz property

The natural idea, found in Duivesteijn and Feelders, (2008), is that two
“close" individuals (in the sense of unprotected characteristics $\bm{x}$)
must have the same forecast. Formally, let us consider two distance, one on
$\\{0,1\\}\times\\{0,1\\}$ noted $d_{y}$, and one on $\mathcal{X}$ noted
$d_{x}$, such that we will have individual fairness on a database of size $n$
if we have the following property (called Lipschitz property)

######  Definition 4.1 (Lipschitz property, Duivesteijn and Feelders, (2008),
Luong et al., (2011)).

A decision function $\widehat{Y}$ satistfies the Lipschitz property if

$d_{y}(\widehat{y}_{i},\widehat{y}_{j})\leq
d_{x}(\bm{x}_{i},\bm{x}_{j}),\leavevmode\nobreak\ \forall i,j=1,\cdots,n.$

Duivesteijn and Feelders, (2008) talked about monotonic classification. It is
difficult to determine which metric to use to measure the similarity of two
individuals (i.e. between $\bm{x}_{i}$ and $\bm{x}_{j}$), as explained by Kim
et al., (2018). The most usual is to use a Mahalanobis type distance, to take
into account the different scales between the variables.

###  4.2 Counterfactual Fairness

####  4.2.1 Causal inference

Consider, as in Rubin, (1974) or Hernán and Robins, (2010), the following
framework: let $t$ denote some binary treatment ($t\in\\{0,1\\}$, with
respectively, the control and the treatment). Let $\bm{x}$ be some covariates,
$y$ the observed outcome, with $y_{T\leftarrow 1}^{\star}$ and $y_{T\leftarrow
0}^{\star}$ the potential outcomes (also denoted $y(1)$ and $y(0)$ in Imbens
and Rubin, (2015) or Imai, (2018), or $y^{1}$ and $y^{0}$ in Morgan and
Winship, (2015) or Cunningham, (2021), even $y_{t=1}$ and $y_{t=0}$ in Pearl
and Mackenzie, (2018)), realized either under treatment condition ($t=1$) or
under control condition ($t=0$). Note that the observed outcome is
$y=y_{T\leftarrow t}^{\star}$, or $y=t\cdot y_{T\leftarrow
1}^{\star}+(1-t)\cdot y_{T\leftarrow 0}^{\star}$. An illustration is reported
in Table 6.

| Treatment | Outcome | Age | Gender | Height | Weight
---|---|---|---|---|---|---
| $t_{i}$ | $y_{i}$ | $y_{i,T\leftarrow 1}^{\star}$ | $y_{i,T\leftarrow o}^{\star}$ | $x_{1,i}$ | $x_{2,i}$ | $x_{3,i}$ | $x_{4,i}$
1 | 1 | 121 | 121 | ? | 37 | F | 160 | 56
2 | 0 | 109 | ? | 109 | 28 | F | 156 | 54
3 | 1 | 162 | 162 | ? | 53 | M | 190 | 87
Table 6: Potential outcome framework of causal inference, with one binary
treatment $t_{i}$, the observed outcome variable $y_{i}$ and the two potential
outcomes $y_{i,T\leftarrow 1}^{\star}$ and $y_{i,T\leftarrow 0}^{\star}$, as
well as some covariates $\bm{x}_{i}$. One of the two potential outcomes is
observed, and the other is missing, indicated by the question mark in the
table.

We will use the term “treatment” (and letter $t$) even if interventions are
not possible, so it is no per se a “treatment”. In this article, we try to
answer a hypothetical question, like most questions asked at the third level
of the “ladder of causality". For instance, in a context of quantifying
discrimination, the “treatment" will denote the sensitive attribute, such as
the race of an individual, e.g., “what would have been the outcome if that
person had been Afro-American?" Since our approach proposes an improvement on
the metrics used in causal inference literature, we will use similar
notations. There will be a significant impact of treatment $t$ on $y$ if
$y^{\star}_{T\leftarrow 0}\neq y^{\star}_{T\leftarrow 1}$. More specifically,
the causal effect for individual $i$ is ${\tau_{i}=y^{\star}_{i,T\leftarrow
1}-y^{\star}_{i,T\leftarrow 0}}$, as discussed in Charpentier et al., (2023).

A model satisfies the “counterfactual fairness” property if “had the protected
attributes (e.g., race) of the individual been different, other things being
equal, the decision would have remained the same”. Also, a classifier will be
counterfactually fair if for all individuals the outcome is equal to the
outcome of its counterfactual individual (i.e., the same individual with one
protected attribute reversed).

$\mathbb{P}[Y^{\star}_{S\leftarrow
0}=1|\bm{X}=\bm{x}]=\mathbb{P}[Y^{\star}_{S\leftarrow 1}=1|\bm{X}=\bm{x}],$

where $Y^{\star}_{S\leftarrow z}$ is the prediction of the classifier if $s$
takes a specific value (corresponding to some sort of “intervention").

######  Definition 4.2 (Counterfactual fairness, Kusner et al., (2017)).

If the prediction in the real world is the same as the prediction in the
counterfactual world where the individual would have belonged to a different
demographic group, we have counterfactual fairness, i.e.

$\mathbb{P}[Y^{\star}_{S\leftarrow
s}=y|\bm{X}=\bm{x}]=\mathbb{P}[Y^{\star}_{S\leftarrow
s^{\prime}}=y|\bm{X}=\bm{x}],\leavevmode\nobreak\ \forall
s,s^{\prime},\bm{x},y.$

fairness through awareness | Dwork et al., (2012) | $D(\widehat{y}_{i},\widehat{y}_{j})\leq d(\bm{x}_{i},\bm{x}_{j}),\leavevmode\nobreak\ \forall i,j$
---|---|---
counterfactual fairness | Kusner et al., (2017) | $\mathbb{P}[Y^{\star}_{S\leftarrow s}=y|\bm{X}=\bm{x}]=\text{cst}_{y},\leavevmode\nobreak\ \forall s$
no proxy discrimination | Kilbertus et al., (2017) | $\mathbb{P}[\widehat{Y}=y|do(S=s)]=\text{cst}_{y},\leavevmode\nobreak\ \forall s$
Table 7: Definitions of individual fairness, with the $do$ operator of Pearl,
(1988).

##  5 Correcting discrimination

Once a discrimination of a model $m$ with respect to some sensitive attribute
$s$ is observed, one might try to correct the discrimination

###  5.1 Pre-processing approaches

A straightforward approach to removing bias from datasets would be to remove
the protected attribute and other data elements that are suspected of
containing related information. Unfortunately, such removal is rarely
sufficient. In the literature, there are four approaches to removing bias by
manipulating the data set. Respectively, these approaches modify the labels
$y$ the observed data $\bm{x}$, the data/label pairs $\bm{x},y\\}$ and the
weighting of these pairs.

####  5.1.1 Manipulation of labels

Kamiran and Calders, 2009 and 2012 proposed to modify some of the training
labels, which they call data manipulation. They compute a classifier on the
original dataset and find examples close to the decision surface. They then
swap the labels so that a positive outcome for the disadvantaged group is more
likely and retrain. This is a heuristic approach that empirically improves
fairness at the expense of accuracy.

####  5.1.2 Manipulation of observed data

Feldman et al., (2015) proposed to manipulate the individual dimensions of the
$\bm{x}$ data in a way that depends on the sensitive attribute $s$. The idea
is to align the cumulative distributions $F_{0}[\bm{x}]$ and $F_{1}[\bm{x}]$
for the feature $\bm{x}$ when the sensitive attribute $s$ is $0$ and $1$,
respectively, to a median cumulative distribution $F_{m}[\bm{x}]$. This method
is similar to the normalization of test scores across different schools, and
is called “disparate impact suppression”. This approach has the disadvantage
of treating each input variable $\bm{x}$ separately, and ignores their
(possible) interactions.

####  5.1.3 Manipulation of labels and data

Calmon et al., (2017) learns a $\psi$-transformation that transforms
$\\{\bm{x},y\\}$ data pairs into new $\\{\bm{x}^{\prime},y^{\prime}\\}$ data
values in a way that explicitly depends on the sensitive attribute $s$. Calmon
et al., (2017) formulates this problem as an optimization problem in which the
change in data utility must be minimized, subject to bounds on the harm and
distortion of the original values. Unlike disparate impact removal, this
method takes into account interactions between all dimensions of the data.
However, the randomized transformation is formulated as a probability table,
which is only suitable for data sets with a small number of discrete input and
output variables.

####  5.1.4 Pairwise weighting of data

Kamiran and Calders, (2012) proposes to reweight the $\\{\bm{x},y\\}$
observations in the training dataset so that cases where the sensitive
attribute $s$ predicts that the disadvantaged group will have a positive
outcome are more heavily weighted. They then train a classifier that uses
these weights in its cost function. They also propose to resample the training
data according to these weights and use a standard classifier.

###  5.2 Reprocessing or in-processing algorithms

In the previous section, we introduced the latent prejudice measure based on
the mutual information between the data $\bm{x}$ and the sensitive attribute
$s$. Similarly, we can measure the dependency between the labels $y$ and the
sensitive attribute $s$

$IP=\sum_{y,s}\mathbb{P}(y,s)\log\frac{\mathbb{P}(y,s)}{\mathbb{P}(y)\mathbb{P}(s)}$

which Kamishima et al., (2011) calls the “indirect prejudice". Intuitively, if
there is no way to predict the labels from the sensitive attribute and vice
versa, then there is no possibility of bias.

One approach to eliminating bias during training is to explicitly remove this
dependency using adversarial learning. Other approaches involve penalizing
mutual information using regularization, fitting the model under the
constraint that it is unbiased. We will briefly discuss each of these
approaches.

####  5.2.1 Adversarial debiasing

Adversarial debiasing (introduced in Beutel et al., (2017) or Zhang et al.,
(2018)) reduces evidence of sensitive attributes in predictions by
simultaneously trying to fool a second classifier that is trying to guess the
sensitive attribute $s$. Beutel et al., (2017) forces both classifiers to use
a shared representation and therefore minimizing the performance of the
adversary classifier means removing all information about the sensitive
attribute from that representation. Beutel et al., (2017) proposes a
representation for classification that was also used to predict the sensitive
attribute. The system was trained adversarially, encouraging good system
performance but punishing correct classification of the sensitive attribute.
In this way, a representation that does not contain information about the
sensitive attribute is learned.

####  5.2.2 Suppression of bias by regularization

Kamishima et al., (2011) proposed adding an additional regularization
condition to the output of the logistic regression classifier that attempts to
minimize the mutual information between the sensitive attribute and the
$\widehat{y}$ prediction.

In econometrics and machine learning, we try to maximize accuracy by solving

${\underset{\bm{\theta}\in\Theta}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}\right\\}},\text{
where
}\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}=\sum_{i=1}^{n}\ell\big{(}m_{\bm{\theta}}(\bm{x}_{i}),{y_{i}}\big{)}$

for some loss function $\ell$ (that might be related to minus the log-
likelihood). Following Hastie et al., (2015) it is possible achieve parsimony
be introducing some penalty in the objective function : given a penalty
$\mathcal{P}$

$\underset{\bm{\theta}\in\Theta}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}+\lambda\mathcal{P}(m_{\bm{\theta}})\right\\},$

for instance $\mathcal{P}(m_{\bm{\theta}})=\text{dim}(\bm{\theta})$, the
(true) dimension of $\bm{\theta}$, when removing null values. Inspired by
Goodfellow et al., (2018) (but also Bechavod and Ligett, (2017) or Cho et al.,
(2020)), to avoid un-fairness, it is natural penalize according to a
discrimination measure. We have seen (see e.g. Table 1) that most group
fairness principles are related to independence, or conditional independence,
and therefore, can be quantified using correlation related measures, as
discussed in section 3.1. For demographic parity, characterized by
independence between $\widehat{y}$ and the sensitive attribute $s$. For
instance, we can solve

$\underset{\bm{\theta}\in\Theta}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}+\lambda\text{cor}(m_{\bm{\theta}}(\bm{y}),s)\right\\},$

i.e.

$\underset{\bm{\theta}\in\Theta}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}+\lambda\text{cor}^{\star}(m_{\bm{\theta}}(\bm{y}),s)\right\\},$

that can be written (in a very general context, where $s$ can be non binary)

$\underset{\bm{\theta}\in\Theta}{\text{max}}\left\\{\underset{g\in\mathcal{S}_{y},h\in\mathcal{S}_{s}}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}+\lambda\text{cor}(g(m_{\bm{\theta}}(\bm{y})),h(s))\right\\}\right\\},$

that can be solved using either neural networks or some function basis for
$\mathcal{S}_{y}$ and $\mathcal{S}_{s}$, as in Grari et al., (2022). For
equalized odds), characterized by independence between $\widehat{y}$ and the
sensitive attribute $s$, conditional on $y$, i.e. when $y\in\\{0,1\\}$

$\underset{\bm{\theta}\in\Theta}{\text{argmin}}\left\\{\mathcal{L}\big{(}m_{\bm{\theta}}(\bm{x}),{y}\big{)}+\lambda_{0}\text{cor}(m_{\bm{\theta}}(\bm{y}),s|y=0)+\lambda_{1}\text{cor}(m_{\bm{\theta}}(\bm{y}),s|y=1)\right\\},$

that could extended to maximal correlation.

###  5.3 Post-processing

Several techniques used to perform post-processing of the output scores of the
classifier to make decisions fairer were introduced in Corbett-Davies et al.,
(2017), Dwork et al., (2018), Menon and Williamson, (2018), Lohia et al.,
(2019) and Awasthi et al., (2020). For instance, Hardt et al., (2016)
considered a technique for flipping some decisions of a classifier to enhance
equalized odds, or equalized opportunity, while Menon and Williamson, (2018)
suggested selecting district thresholds $\tau$ for each group (as discussed
earlier), in a manner that maximizes accuracy and minimizes demographic
parity.

## Acknowledgments

Arthur Charpentier acknowledges the financial support of the AXA Research Fund
through the joint research initiative use and value of unusual data in
actuarial science, as well as NSERC grant 2019-07077.

## References

  * Agarwal, (2021) Agarwal, S. (2021).  Trade-offs between fairness and interpretability in machine learning.  In IJCAI 2021 Workshop on AI for Social Good.
  * Aigner and Cain, (1977) Aigner, D. J. and Cain, G. G. (1977).  Statistical theories of discrimination in labor markets.  Industrial and Labor Relations Review, 30(2):175–187.
  * Avraham et al., (2014) Avraham, R., Logue, K. D., and Schwarcz, D. (2014).  Towards a universal framework for insurance anti-discrimination laws.  Connecticut Insurance Law Journal, 21:1.
  * Awasthi et al., (2020) Awasthi, P., Kleindessner, M., and Morgenstern, J. (2020).  Equalized odds postprocessing under imperfect group information.  In International Conference on Artificial Intelligence and Statistics, pages 1770–1780. PMLR.
  * Barocas et al., (2019) Barocas, S., Hardt, M., and Narayanan, A. (2019).  Fairness and Machine Learning.  fairmlbook.org.
  * Barry and Charpentier, (2022) Barry, L. and Charpentier, A. (2022).  The Fairness of Machine Learning in Insurance: New Rags for an Old Man? .  ArXiv.
  * Bechavod and Ligett, (2017) Bechavod, Y. and Ligett, K. (2017).  Penalizing unfairness in binary classification.  arXiv preprint arXiv:1707.00044.
  * Becker, (1957) Becker, G. S. (1957).  The economics of discrimination.  University of Chicago press.
  * Becker, (2005) Becker, G. S. (2005).  Is ethnic and other profiling discrimination?  The Becker-Posner Blog, 01-23-2005.
  * Bergstrom and West, (2021) Bergstrom, C. T. and West, J. D. (2021).  Calling bullshit: the art of skepticism in a data-driven world.  Random House Trade Paperbacks.
  * Berk et al., (2017) Berk, R., Heidari, H., Jabbari, S., Joseph, M., Kearns, M., Morgenstern, J., Neel, S., and Roth, A. (2017).  A convex framework for fair regression.  arXiv, 1706.02409.
  * Berk et al., (2021) Berk, R., Heidari, H., Jabbari, S., Kearns, M., and Roth, A. (2021).  Fairness in criminal justice risk assessments: The state of the art.  Sociological Methods & Research, 50(1):3–44.
  * Besse et al., (2018) Besse, P., del Barrio, E., Gordaliza, P., and Loubes, J.-M. (2018).  Confidence intervals for testing disparate impact in fair learning.  arXiv, 1807.06362.
  * Beutel et al., (2019) Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Chi, E. H., et al. (2019).  Fairness in recommendation ranking through pairwise comparisons.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2212–2220.
  * Beutel et al., (2017) Beutel, A., Chen, J., Zhao, Z., and Chi, E. H. (2017).  Data decisions and theoretical implications when adversarially learning fair representations.  arXiv, 1707.00075.
  * Biddle, (2017) Biddle, D. (2017).  Adverse impact and test validation: A practitioner’s guide to valid and defensible employment testing.  Routledge.
  * Bohren et al., (2019) Bohren, J. A., Haggag, K., Imas, A., and Pope, D. G. (2019).  Inaccurate statistical discrimination: An identification problem.  Technical report, National Bureau of Economic Research.
  * Borkan et al., (2019) Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. (2019).  Nuanced metrics for measuring unintended bias with real data for text classification.  In Companion proceedings of the 2019 world wide web conference, pages 491–500.
  * Calmon et al., (2017) Calmon, F. P., Wei, D., Ramamurthy, K. N., and Varshney, K. R. (2017).  Optimized data pre-processing for discrimination prevention.  arXiv, 1704.03354.
  * Caton and Haas, (2020) Caton, S. and Haas, C. (2020).  Fairness in machine learning: A survey.  arXiv, 2010.04053.
  * Chakraborty et al., (2013) Chakraborty, S., Raghavan, K. R., Johnson, M. P., and Srivastava, M. B. (2013).  A framework for context-aware privacy of sensor data on mobile systems.  In Proceedings of the 14th Workshop on Mobile Computing Systems and Applications, HotMobile ’13. Association for Computing Machinery.
  * Charles and Guryan, (2011) Charles, K. K. and Guryan, J. (2011).  Studying discrimination: Fundamental challenges and recent progress.  Annual Review of Economics, 3(1):479–511.
  * Charpentier, (2022) Charpentier, A. (2022).  Insurance: biases, discrimination and fairness.  Institut Louis Bachelier.
  * Charpentier et al., (2023) Charpentier, A., Flachaire, E., and Gallic, E. (2023).  Causal inference with optimal transport.  In Thach, N. N., Kreinovich, V., Ha, D. T., and Trung, N. D., editors, Optimal Transport Statistics for Economics and Related Topics. Springer Verlag.
  * Charpentier et al., (2018) Charpentier, A., Flachaire, E., and Ly, A. (2018).  Econometrics and machine learning.  Economie et Statistique, 505(1):147–169.
  * Chicco and Jurman, (2020) Chicco, D. and Jurman, G. (2020).  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  BMC genomics, 21(1):1–13.
  * Cho et al., (2020) Cho, J., Hwang, G., and Suh, C. (2020).  A fair classifier using mutual information.  In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2521–2526. IEEE.
  * Chouldechova, (2017) Chouldechova, A. (2017).  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5(2):153–163.
  * Corbett-Davies et al., (2017) Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., and Huq, A. (2017).  Algorithmic decision making and the cost of fairness.  arXiv, 1701.08230.
  * Cornu, (2016) Cornu, G. (2016).  Vocabulaire juridique, volume 6.  Presses Universitaires de France.
  * Cunningham, (2021) Cunningham, S. (2021).  Causal inference.  Yale University Press.
  * Dalenius, (1977) Dalenius, T. (1977).  Towards a methodology for statistical disclosure control.  statistik Tidskrift, 15(429-444):2–1.
  * Daniels, (2004) Daniels, N. (2004).  The functions of insurance and the fairness of genetic underwriting.  Genetics and life insurance: Medical underwriting and social policy, pages 119–145.
  * David, (2015) David, H. (2015).  Why are there still so many jobs? the history and future of workplace automation.  Journal of economic perspectives, 29(3):3–30.
  * Dawid, (1982) Dawid, A. P. (1982).  The well-calibrated bayesian.  Journal of the American Statistical Association, 77(379):605–610.
  * Denuit et al., (2021) Denuit, M., Charpentier, A., and Trufin, J. (2021).  Autocalibration and tweedie-dominance for insurance pricing with machine learning.  Insurance: Mathematics & Economics.
  * Duivesteijn and Feelders, (2008) Duivesteijn, W. and Feelders, A. (2008).  Nearest neighbour classification with monotonicity constraints.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 301–316. Springer.
  * Dwork et al., (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012).  Fairness through awareness.  In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226.
  * Dwork et al., (2018) Dwork, C., Immorlica, N., Kalai, A. T., and Leiserson, M. (2018).  Decoupled classifiers for group-fair and efficient machine learning.  In Conference on fairness, accountability and transparency, pages 119–133. PMLR.
  * Edgeworth, (1922) Edgeworth, F. Y. (1922).  Equal pay to men and women for equal work.  The Economic Journal, 32(128):431–457.
  * Feldman et al., (2015) Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015).  Certifying and removing disparate impact.  In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259–268.
  * Friedler et al., (2019) Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., and Roth, D. (2019).  A comparative study of fairness-enhancing interventions in machine learning.  In Proceedings of the conference on fairness, accountability, and transparency, pages 329–338.
  * Gajane and Pechenizkiy, (2017) Gajane, P. and Pechenizkiy, M. (2017).  On formalizing fairness in prediction with machine learning.  arXiv preprint arXiv:1710.03184.
  * Gambs et al., (2010) Gambs, S., Killijian, M.-O., and del Prado Cortez, M. N. n. (2010).  Show me how you move and i will tell you who you are.  In Proceedings of the 3rd ACM International Workshop on Security and Privacy in GIS and LBS.
  * Gebelein, (1941) Gebelein, H. (1941).  Das statistische problem der korrelation als variations- und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung.  ZAMM - Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Mathematik und Mechanik, 21(6):364–379.
  * Goodfellow et al., (2018) Goodfellow, I., McDaniel, P., and Papernot, N. (2018).  Making machine learning robust against adversarial inputs.  Communications of the ACM, 61(7):56––66.
  * Grari et al., (2022) Grari, V., Charpentier, A., Lamprier, S., and Detyniecki, M. (2022).  A fair pricing model via adversarial learning.  ArXiv, 2202.12008.
  * Hale, (2021) Hale, K. (2021).  A.i. bias caused 80% of black mortgage applicants to be denied.  Forbes, 09/2021.
  * Harcourt, (2011) Harcourt, B. E. (2011).  Surveiller et punir à l’âge actuariel.  Déviance et Société, 35:163.
  * Hardt et al., (2016) Hardt, M., Price, E., and Srebro, N. (2016).  Equality of opportunity in supervised learning.  Advances in neural information processing systems, 29:3315–3323.
  * Hastie et al., (2015) Hastie, T., Tibshirani, R., and Wainwright, M. (2015).  Statistical learning with sparsity.  Monographs on statistics and applied probability, 143:143.
  * Hernán and Robins, (2010) Hernán, M. A. and Robins, J. M. (2010).  Causal inference.
  * Hirschfeld, (1935) Hirschfeld, H. O. (1935).  A connection between correlation and contingency.  Mathematical Proceedings of the Cambridge Philosophical Society, 31(4):520–524.
  * Imai, (2018) Imai, K. (2018).  Quantitative social science: an introduction.  Princeton University Press.
  * Imbens and Rubin, (2015) Imbens, G. W. and Rubin, D. B. (2015).  Causal inference in statistics, social, and biomedical sciences.  Cambridge University Press.
  * Ito, (2021) Ito, J. (2021).  Supposedly ‘fair’ algorithms can perpetuate discrimination.  Wired, 02.05.2019.
  * Jung et al., (2020) Jung, C., Kannan, S., Lee, C., Pai, M. M., Roth, A., and Vohra, R. (2020).  Fair prediction with endogenous behavior.  arXiv, 2002.07147.
  * Kamiran and Calders, (2009) Kamiran, F. and Calders, T. (2009).  Classifying without discriminating.  In 2009 2nd international conference on computer, control and communication, pages 1–6. IEEE.
  * Kamiran and Calders, (2012) Kamiran, F. and Calders, T. (2012).  Data preprocessing techniques for classification without discrimination.  Knowledge and Information Systems, 33(1):1–33.
  * Kamishima et al., (2011) Kamishima, T., Akaho, S., and Sakuma, J. (2011).  Fairness-aware learning through regularization approach.  In 2011 IEEE 11th International Conference on Data Mining Workshops, pages 643–650.
  * Kearns and Roth, (2019) Kearns, M. and Roth, A. (2019).  The ethical algorithm: The science of socially aware algorithm design.  Oxford University Press.
  * Kelly, (2021) Kelly, H. (2021).  A priest’s phone location data outed his private life. it could happen to anyone.  The Wshinghton Post, 22-07-2021.
  * Khaitan, (2017) Khaitan, T. (2017).  Indirect discrimination.  In Lippert-Rasmussen, K., editor, Handbook of the Ethics of Discrimination, pages 30–41. Routledge.
  * Kilbertus et al., (2017) Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., and Schölkopf, B. (2017).  Avoiding discrimination through causal reasoning.  arXiv, 1706.02744.
  * Kim et al., (2018) Kim, M. P., Reingold, O., and Rothblum, G. N. (2018).  Fairness through computationally-bounded awareness.  arXiv preprint arXiv:1803.03239.
  * Kim, (2017) Kim, P. T. (2017).  Auditing algorithms for discrimination.  University of Pennsylvania Law Review, 166:189.
  * Kleinberg et al., (2017) Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., and Mullainathan, S. (2017).  Human Decisions and Machine Predictions.  The Quarterly Journal of Economics, 133(1):237–293.
  * Kleinberg et al., (2016) Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016).  Inherent trade-offs in the fair determination of risk scores.  arXiv, 1609.05807.
  * Kranzberg, (1986) Kranzberg, M. (1986).  Technology and history:" kranzberg’s laws".  Technology and culture, 27(3):544–560.
  * Krüger and Ziegel, (2021) Krüger, F. and Ziegel, J. F. (2021).  Generic conditions for forecast dominance.  Journal of Business & Economic Statistics, 39(4):972–983.
  * Kusner et al., (2017) Kusner, M. J., Loftus, J., Russell, C., and Silva, R. (2017).  Counterfactual fairness.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30, pages 4066–4076. NIPS.
  * Lippert-Rasmussen, (2007) Lippert-Rasmussen, K. (2007).  Nothing personal: On statistical discrimination.  Journal of Political Philosophy, 15(4):385–403.
  * Lohia et al., (2019) Lohia, P. K., Ramamurthy, K. N., Bhide, M., Saha, D., Varshney, K. R., and Puri, R. (2019).  Bias mitigation post-processing for individual and group fairness.  In Icassp 2019-2019 ieee international conference on acoustics, speech and signal processing (icassp), pages 2847–2851. IEEE.
  * Luong et al., (2011) Luong, B. T., Ruggieri, S., and Turini, F. (2011).  $k$-nn as an implementation of situation testing for discrimination discovery and prevention.  In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 502–510.
  * Matthews, (1975) Matthews, B. W. (1975).  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451.
  * Mayer et al., (2016) Mayer, J., Mutchler, P., and Mitchell, J. C. (2016).  Evaluating the privacy properties of telephone metadata.  Proceedings of the National Academy of Sciences, 113(20):5536–5541.
  * McKinsey, (2017) McKinsey (2017).  Technology, jobs and the future of work.  McKinsey Global Institute.
  * Menon and Williamson, (2018) Menon, A. K. and Williamson, R. C. (2018).  The cost of fairness in binary classification.  In Conference on Fairness, Accountability and Transparency, pages 107–118. PMLR.
  * Mercat-Bruns, (2016) Mercat-Bruns, M. (2016).  Discrimination at Work.  University of California Press.
  * Miracle, (2016) Miracle, J. M. (2016).  De-Anonymization Attack Anatomy and Analysis of Ohio Nursing Workforce Data Anonymization.  PhD thesis, Wright State University.
  * Morgan and Winship, (2015) Morgan, S. L. and Winship, C. (2015).  Counterfactuals and causal inference.  Cambridge University Press.
  * Palmer, (2007) Palmer, D. E. (2007).  Insurance, risk assessment and fairness: An ethical analysis.  In Insurance ethics for a more ethical world. Emerald Group Publishing Limited.
  * Pearl, (1988) Pearl, J. (1988).  Probabilistic reasoning in intelligent systems: networks of plausible inference.  Morgan kaufmann.
  * Pearl and Mackenzie, (2018) Pearl, J. and Mackenzie, D. (2018).  The book of why: the new science of cause and effect.  Basic books.
  * Pedreshi et al., (2008) Pedreshi, D., Ruggieri, S., and Turini, F. (2008).  Discrimination-aware data mining.  In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 560–568.
  * Phelps, (1972) Phelps, E. S. (1972).  The statistical theory of racism and sexism.  The american economic review, 62(4):659–661.
  * Rényi, (1959) Rényi, A. (1959).  On measures of dependence.  Acta mathematica hungarica, 10(3-4):441–451.
  * Rothschild-Elyassi et al., (2018) Rothschild-Elyassi, G., Koehler, J., and Simon, J. (2018).  Actuarial Justice, chapter 14, pages 194–206.  John Wiley & Sons, Ltd.
  * Rubin, (1974) Rubin, D. B. (1974).  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of educational Psychology, 66(5):688.
  * Scott and Marshall, (2009) Scott, J. and Marshall, G. (2009).  A dictionary of sociology.  Oxford University Press, USA.
  * Sokolova et al., (2006) Sokolova, M., Japkowicz, N., and Szpakowicz, S. (2006).  Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation.  In Australasian joint conference on artificial intelligence, pages 1015–1021. Springer.
  * Thomas, (2007) Thomas, R. G. (2007).  Some novel perspectives on risk classification.  The Geneva Papers on Risk and Insurance-Issues and Practice, 32(1):105–132.
  * Thomsen, (2017) Thomsen, F. K. (2017).  Direct discrimination.  In Lippert-Rasmussen, K., editor, Handbook of the Ethics of Discrimination, pages 19–29. Routledge.
  * Van Calster et al., (2019) Van Calster, B., McLernon, D. J., Van Smeden, M., Wynants, L., and Steyerberg, E. W. (2019).  Calibration: the achilles heel of predictive analytics.  BMC medicine, 17(1):1–7.
  * Verma and Rubin, (2018) Verma, S. and Rubin, J. (2018).  Fairness definitions explained.  In 2018 ieee/acm international workshop on software fairness (fairware), pages 1–7. IEEE.
  * Vogel et al., (2021) Vogel, R., Bellet, A., Clémen, S., et al. (2021).  Learning fair scoring functions: Bipartite ranking under roc-based fairness constraints.  In International Conference on Artificial Intelligence and Statistics, pages 784–792. PMLR.
  * Žliobaite, (2015) Žliobaite, I. (2015).  On the relation between accuracy and fairness in binary classification.  arXiv, 1505.05723.
  * Wirth, (1941) Wirth, L. (1941).  Morale and minority groups.  American journal of sociology, 47(3):415–433.
  * Zafar et al., (2017) Zafar, M. B., Valera, I., Rodriguez, M. G., and Gummadi, K. P. (2017).  Fairness constraints: Mechanisms for fair classification.  arXiv, 1507.05259.
  * Zhang et al., (2018) Zhang, B. H., Lemoine, B., and Mitchell, M. (2018).  Mitigating unwanted biases with adversarial learning.  arXiv, 1801.07593.
  * Žliobaitė, (2017) Žliobaitė, I. (2017).  Measuring discrimination in algorithmic decision making.  Data Mining and Knowledge Discovery, 31(4):1060–1089.