#  KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large
Language Model Applications

Hwaran Lee1,2,⋆ Seokhee Hong3,⋆,♯ Joonsuk Park1,2,4
Takyoung Kim1,♯ Gunhee Kim3 Jung-Woo Ha1,2
1NAVER AI Lab 2NAVER Cloud 3Seoul National University 4University of Richmond
{hwaran.lee<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) learn not only natural text generation abilities
but also social biases against different demographic groups from real-world
data. This poses a critical risk when deploying LLM-based applications.
Existing research and resources are not readily applicable in South Korea due
to the differences in language and culture, both of which significantly affect
the biases and targeted demographic groups. This limitation requires localized
social bias datasets to ensure the safe and effective deployment of LLMs. To
this end, we present KoSBi, a new social bias dataset of 34k pairs of contexts
and sentences in Korean covering 72 demographic groups in 15 categories. We
find that through filtering-based moderation, social biases in generated
content can be reduced by 16.47%p on average for HyperCLOVA (30B and 82B), and
GPT-3.

††footnotetext: $\star$ Authors equally contributed.††footnotetext: $\sharp$
This work was done during their internship at NAVER AI Lab.††footnotetext:
Email to: {hwaran.lee<EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

Large language models (LLMs) acquire impressive text generation abilities from
large-scale real-world pre-training data Brown et al. (2020); Kim et al.
(2021). However, LLMs also absorb toxicity, such as social biases Sheng et al.
(2019); Wallace et al. (2019a). This cannot be overlooked since the risk of
generating toxic content impedes the safe use and potential commercialization
of various downstream applications, such as AI assistants Dinan et al. (2022);
Bai et al. (2022a). To minimize the harm, numerous studies have tackled the
detection and mitigation of toxicity in LLMs Blodgett et al. (2020); Ganguli
et al. (2022). Each study typically leverages datasets capturing a specific
type of toxicity, such as social bias Sap et al. (2020); Nangia et al. (2020)
or hate speech Warner and Hirschberg (2012); Lee et al. (2022).

These datasets are not only task-specific but also language- and culture-
specific. For instance, consider hate speech made in South Korea and in the
United States. In addition to the language, the mainly targeted demographic
groups also differ—feminists and Korean Chinese in South Korea, as opposed to
African Americans and Jewish in the United States Jeong et al. (2022). Also,
the existing toxicity datasets in Korean mostly focus on explicit hate speech
and consider a limited number of targeted demographic groups Moon et al.
(2020); Yang et al. (2022); Kang et al. (2022); Lee et al. (2022). This calls
for a dataset to address social biases against a more comprehensive set of
demographic groups in South Korea so that as many groups and people are
protected.

Dataset | # Inst. | Demographic Groups | Data Source | Includes | Toxicity Labels
---|---|---|---|---|---
# Cat. | # Groups | Context?
BEEP! Moon et al. (2020) | 9,341 | - | - | News comments | ✗ | Hate speech, Bias
APEACH Yang et al. (2022) | 3,770 | 10 | - | Human-written | ✗ | Offensive
KOLD Jeong et al. (2022) | 40,448 | 5 | 19 | News, YouTube comments | ✗ (Title) | Offensive
HateScore, Unsmile Kang et al. (2022) | 31,195 | 7 (mixed) | News, online community comments | ✗ | Hate speech, Profanity
K-MHaS Lee et al. (2022) | 109,692 | 7 | - | News comments | ✗ | Hate speech, Profanity
KoSBi (Ours) | 34,214 | 15 | 72 | LM-generated | ✓ |  | Biased (Stereotypes, Prejudice,
---
Discrimination), Other

Table 1: Comparison of Toxicity Datasets in Korean.

Here we present the Korean Social Bias (KoSBi) dataset, a large-scale dataset
of 34k pairs of contexts and sentences in Korean with labels mainly capturing
the presence of social biases. 111 The KoSBi dataset is released with English-
translated annotations for those who are not fluent in Korean at
https://github.com/naver-ai/korean-safety-benchmarks It covers 72 targeted
demographic groups in 15 categories, 222The categories and demographic groups
were selected based on the Universal Declaration of Human Rights (UDHR) and
the National Human Rights Commission of Korea (NHRCK). which is much more
comprehensive than existing datasets, as shown in Table 2. The categories
include not only the common ones like gender and religion but also those
especially relevant to South Korea—e.g., marital status and domestic area of
origin, both of which consist of demographic groups that suffer from social
biases in the country more commonly than others do. Given the difficulty of
crawling from the web sufficient data for each of the 72 demographic groups,
we leveraged HyperCLOVA Kim et al. (2021) to generate the data with in-context
few-shot learning Gao et al. (2021); Mishra et al. (2022). More specifically,
we generated sentences and their respective contexts—which are also sentences,
grammatically—for given target demographic groups. The generated contexts and
sentences were then annotated by crowd workers as safe or unsafe. Here, unsafe
contexts and sentences were further labeled as expressions of stereotypes
(cognitive bias), prejudice (emotional bias), discrimination (behavioral
bias), and/or other, adopting the taxonomy by Fiske (2023),333For labeling the
context, prejudice and discrimination were combined due to the limited number
of instances. in Figure 1.

With KoSBi, we mitigate social biases in LLM-generated content using a
filtering-based moderation approach, also known as rejection sampling Ganguli
et al. (2022). To do this, we first trained a safe sentence classifier using
KoSBi. Then, for a given context, each LLM was used to generate a pool of
sentences from which the safest sentence was chosen by the classifier. The
human evaluation shows that social biases in generated content are reduced by
16.47% on average for all three models tested—HyperCLOVA (82B), HyperCLOVA
(30B), and GPT-3.

##  2 Related Works

#### Bias Mitigation in LLM-generated Content.

LLMs are trained on real-world data, which often contains social biases toward
certain demographic groups. This, in turn, induces biases in LLMs Xu et al.
(2021a). To date, various resources have been published to measure and
mitigate such biases in LLMs Sap et al. (2020); Nangia et al. (2020); Nadeem
et al. (2021). Some of them are associated with specific tasks: coreference
resolution to fight the phenomena like associating certain professions with a
particular gender Rudinger et al. (2018); Zhao et al. (2018), and question
answering to prevent answers stereotyped toward certain bias categories like
gender or socio-economic status Li et al. (2020); Parrish et al. (2022). These
resources are not as effective for HyperCLOVA and other LLMs pre-trained on
Korean corpora. Thus, we present a new resource in Korean, capturing the
biases against prevalent demographic groups in South Korea. Also, our dataset
covers a much more comprehensive set of demographic groups.

#### Hate Speech Detection.

Röttger et al. (2021) defines hate speech as “abuse that is targeted at a
protected group or at its members for being a part of that group.” Resources
created to help detect hate speech can be used to reduce hate speech generated
by LLMs, thereby reducing the harm they can incur. Note these resources use
various names interchangeably for the most part, e.g., hate speech Warner and
Hirschberg (2012), abusive language Wiegand et al. (2019), and toxic language
Gehman et al. (2020); Hartvigsen et al. (2022). Also, quite a few resources
are for safer dialogue Sun et al. (2022); Xu et al. (2021b); Xenos et al.
(2021); Kim et al. (2022). Meanwhile, to reflect different languages and
societies, researchers have created and proposed hate speech corpora in
Chinese Deng et al. (2022), Dutch Demus et al. (2022), and Arabic Mubarak et
al. (2022) Similar to the resources capturing social biases, these resources
are not as useful for Korean LLMs due to the differences in language and
culture. Luckily, several resources in Korean exist, as summarized in Table 1.
However, these resources either unspecify or cover only a small subset of
demographic groups in South Korea. More importantly, they focus on explicit
profanity and otherwise offensive expressions. Our dataset instead targets
cases that cannot be identified with specific keywords, such as expressions of
stereotypes, discrimination, and prejudice (without explicit profanity) toward
72 demographic groups.

#### Safety Alignment of Language Models.

Beyond social biases and hate speech, various categories have been proposed
recently to enhance the safety of language models, such as human values
Solaiman and Dennison (2021); Kenton et al. (2021), ethical judgements
Hendrycks et al. (2021); Lourie et al. (2021), and moral norms Forbes et al.
(2020); Emelin et al. (2021). Then, alignment learning methods through human
feedback Bai et al. (2022a) or even by AI feedback Bai et al. (2022b) have
been proposed. Moreover, red-teaming Perez et al. (2022); Ganguli et al.
(2022) and adversarial attack Wallace et al. (2019b) approaches have also been
suggested to identify vulnerabilities in language models in terms of safety.
We expect our dataset and comprehensive categories will be helpful for the
safety alignment of Korean society.

##  3 The KoSBi Dataset

This study aims to address social biases against a comprehensive set of
demographic groups in South Korea so as to make LLMs safer for as many groups
and people as possible. (Here, we focus on social biases without explicit hate
speech, as existing datasets address the latter.) To achieve this, we wanted
KoSBi to consist of context-sentence pairs labeled as safe or unsafe for the
demographic groups mentioned in them; this way, we can train LLMs to behave
safely in the context of discussing a demographic group, rather than simply
avoid it.

###  3.1 Demographic Groups Compilation

With the goal of covering a comprehensive list of demographic groups, we first
compiled the list by combining categories derived from the Universal
Declaration of Human Rights (UDHR) and the National Human Rights Commission of
Korea (NHRCK)444Specifically, refer to provisions related to discriminatory
acts in violation of equal rights – Article 2 Subparagraph 3 of the National
Human Rights Commission Act, and Article 3 Paragraph 1 Subparagraph 1 of the
Anti-Discrimination Act., which prohibit discriminatory treatment based on
social identity. (See Table 4 for the list of categories.) Then, we defined
social groups in each category, considering the unique characteristics of
Korean culture. For instance, we consider the most widely practiced religions
in Korea, and also progressive and conservative political parties, rather than
the Democratic and Republican parties in the U.S. (See Table 8 for the list of
demographic groups.)

###  3.2 Raw Data Construction

Since crawling from the web sufficient context-sentence pairs for every
demographic group would be challenging, we generated them using HyperCLOVA.
LLMs are reported to have abilities to learn a given task from instructions
and few-shot demonstration samples, which is referred to as in-context
learning Brown et al. (2020). With these abilities, previous research has
proposed data synthesis methods by demonstration-based prompting methods Gao
et al. (2021); Mishra et al. (2022), wherein several sample sentences are
listed in a prompt, and an LLM generates different ones with similar
semantics. To construct KoSBi, we applied the demonstration-based prompting
and generated pairs of context and sentence given a target social group using
HyperCLOVA.

Categories | # Groups
---|---
Gender identity† | 3
Sexual orientation† | 1
Age & Generation† | 12
Race, Ethnicity, Nationality† | 11
Religion† | 6
Disability status† | 1
Physical appearance† | 4
Political orientation† | 3
Socio-economic status† | 3
Domestic area of origin | 8
Marital status | 6
Pregnancy & Birth | 4
Family form | 5
Criminal record | 2
Education, University, Major | 3
Total | 72
Table 2:  Category and demographic groups considered in KoSBi. † marks
categories in both UDHR and NHRCK. Entire social groups are listed in Table 8.

The raw data construction was done in three-step: (1) building demonstration
pools, which consist of initial labeled data; (2) generating contexts and
sentences; (3) filtering out inappropriate generations by trainable
classifiers The initial demonstration data was manually curated by authors and
a few annotators, resulting in a relatively small pool of around 216555In the
initial demonstration pool, we collected three safe and three unsafe context-
sentence pairs for each demographic group. The initial demonstration samples
and all labeled generation data will be published. samples. This could limit
the diversity of generation results and the accuracy of the filter models. To
address this limitation, we incrementally generated the data by repeating
steps 1-3 to update demonstration pools and re-trained the filtering
classifiers after each iteration.

The detailed prompts can be found in Appendix C. In the context prompt, the
LLM is asked to produce “neutral contextual sentences” pertaining to the given
social group. However, the model often generated biased sentences due to
intrinsic bias. We labeled them as unsafe contexts. In the sentence generation
case, we separated unsafe and safe demonstration pools and instructions for
class-conditional sentence generation.

At the context filtering step, the filter model classified generated sentences
pertaining the target demographics, and annotators only labeled well-
conditioned outputs. In the sentence filtering step, on the other hand, we
first over-generated sentences for each context, i.e., three sentences for
each class. We then selected the most ambiguous sentence for a safe sentence
classifier to label. The ambiguity was measured by the estimated max
variability Liu et al. (2022); Swayamdipta et al. (2020). Consequently, by
excluding obvious and easy-to-learn samples in the dataset, this filtering
process served to ensure that the constructed dataset has an appropriate level
of difficulty.

Figure 1: Example pairs of a context and a sentence with labels pertaining to
a given social demographic category and group.

###  3.3 Annotation

The contexts and sentences were then labeled by crowd workers according to the
following guidelines (See Figure 1 for examples):

  * •

Context. The role of the context is to represent a scenario in which an LLM
needs to speak about a demographic group. Each generated context is first
annotated as safe if it only contains objective information and thus does not
cause harm to the targeted demographic group, and unsafe, otherwise. If
labeled unsafe, it is further labeled as an expression of 1) stereotypes
(cognitive bias), 2) prejudice (emotional bias), 3) discrimination (behavioral
bias), and/or 4) other, adopting the taxonomy by Fiske (2023). Here,
subclasses 2 and 3 are combined due to the rare occurrences observed during a
pilot study.

  * •

Sentence. Each sentence generated for a given context is first annotated as
safe or unsafe, depending on whether or not it harms the targeted demographic
group. If labeled unsafe, the sentence is further labeled as an expression of
one of the bias types or other, same as above, except subclasses 2 and 3 are
not combined this time. Note, a seemingly safe sentence may be unsafe
dependent on its context. For instance, a sentence simply agreeing (e.g.,
“Yes, that is true.”) to an unsafe context (e.g., “[Demographic Group] are
always lazy.”) is unsafe. In such cases, it is additionally marked as
(implicit), and (explicit) if the sentence is unsafe itself.

Context | Sentence | Train | Valid | Test | All
---|---|---|---|---|---
Safe | Safe | 11,630 | 1,427 | 1,382 | 14,439
Unsafe | 8,521 | 1,060 | 1,092 | 10,673
Total | 20,151 | 2,487 | 2,474 | 25,112
Unsafe | Safe | 2,537 | 320 | 317 | 3,174
Unsafe | 4,589 | 596 | 617 | 5,802
Total | 7,126 | 916 | 934 | 8,976
Undecided | Safe | 58 | 45 | 7 | 6
Unsafe | 68 | 48 | 11 | 9
Total | 93 | 18 | 15 | 126
Total | 27,370 | 3,421 | 3,423 | 34,214

Table 3: The number of instances for all label combinations in KoSBi. (Refer
to Table 7 for subclass.)

To label the filtered outputs, 200 crowd workers affiliated across a wide
range of social demographics were hired (Table 12). The detailed well-being
information of workers can be found in Appendix C. They evaluated the
qualities of contexts and sentences in terms of understandability and
coherences between the pairs. Data that did not meet the criteria were
excluded. They were then asked to label them. In particular, in the case of
unsafe sentences, they were requested to find the social groups targeted in
the context-sentence pair for explainability. The annotation guidelines are
shown in Appendix H.

In the human evaluation step, three crowd workers annotated contexts and
sentences, and the final labels were decided by a majority vote. First, in
labeling contexts as safe or unsafe, the inner-annotator agreement by
Krippendorff’s $\alpha$ is 0.459 for binary (safe/unsafe) classes. The
agreement is lower if we consider subclasses of unsafe contexts ($\alpha$ =
0.359). For sentence annotations, the $\alpha$ is 0.256 for labeling them as
safe or unsafe. This suggests that determining the labels for the sentences is
harder. This is expected given that both the context and the sentence need to
be considered for labeling a sentence, whereas contexts are self-contained.

###  3.4 The Resulting Dataset

KoSBi consists of 34,214 context-sentence pairs as summarized in Table 3.
There are 25,112 (73.4%) and 8,976 (26.2%) of safe and unsafe contexts,
respectively. Also, there are 17,619 (51.5%) and 16,484 (48.2%) safe and
unsafe sentences. Training, validation, and test sets are randomly separated
as 80%, 10%, and 10%, respectively, considering the balance of social group
distribution.

##  4 Experimental Results

To improve the safety of LLMs towards social groups, we explore a simple
filtering-based moderation approach. In this section, we first build a safe
sentence classification. Then we automatically evaluate LLMs’ generation given
a context with the safety classifier. Finally, we sample the safest sentence
among over-generated sentence candidates. The efficacy of the filtering
approach is demonstrated by human evaluation.

###  4.1 Safe Sentence Classification

Datasets | Models | Macro F1 (%)
---|---|---
BEEP! | KcBERT | 52.90
APEACH | KcBERT | 48.82
KOLD | KLUE-BERT | 38.15
Hatescore | KcBERT | 40.28
Unsmile | KcBERT | 48.02
Ours | KLUE-BERT | 69.94
Ours | KcELECTRa | 71.21
Table 4:  Comparison of classification performance on our test set. Fine-tuned
models on the previous datasets and ours are compared.

We train the safe sentence classifier by fine-tuning KLUE-BERT Park et al.
(2021) and KcELECTRa Lee (2021)666We used the latest version of the model:
https://huggingface.co/beomi/KcELECTRA-base-v2022.. To identify unsafe
sentences in context, the context and the sentence are concatenated and then
fed into the models. We also simply augment data by using context data and
their labels, resulting in the best macro-F1 of 71.21% as shown in Table 4.
The performance implies that the proposed dataset is challenging.

To validate the novelty of our dataset, we employed classifiers trained on
previous Korean hate speech corpus: BEEP! Moon et al. (2020), APEACH Yang et
al. (2022), KOLD Jeong et al. (2022), Hatescore, and Unsmile Kang et al.
(2022) to evaluate the test set of our KoSBi 777 For a fair comparison, we
employed the published BERT-base-sized checkpoints of each model. Classifiers
except for KOLD are pretrained on KcBERT (Lee, 2020). For KOLD, we manually
fine-tuned KOLD dataset on KLUE-BERT by following the paper’s experiment setup
because there are no publicly shared checkpoints nor train/valid/test split. .
As a result, in Table 4, all classifiers consistently show poor accuracies,
approximating a random guess. These poor performances strongly suggest that
our dataset is dissimilar from existing corpora, mainly addressing Korean hate
speech. Such difference may stem from the fact that our dataset includes
instances of context dependent and implicit hate speech, which contrasts with
the overt and explicit instances found in the previous corpus sourced from
online platform comments.

###  4.2 Safety Evaluation of LLMs

Using the safe sentence classifier, we measure safe generations of LLMs.
Similarly to Gehman et al. (2020), we evaluate the models with two metrics.
First, the empirical probability of generating a safe sentence (safe score is
$\geq$ 0.5) given a context at least once over multiple $k$ generations. We
measure the probability with different sizes of $k$. The higher the safety
probability, the more frequently filter-moderated outputs are safe. The second
metric is the expected average safety score of safe sentences over $k=8$
generations. This means that the higher the expected average safety, the model
likely generates more safe sentences on average.

We evaluate HyperCLOVA with different model sizes (6.9B, 13B, 30B, and 82B),
and GPT-3 (175B) 888The largest HyperCLOVA model (82B) was trained on
HyperCLOVA Corpus consisting of 300B tokens, and the remains are further
trained with 30B of a spoken dataset. The version of ’text-davinci-003’ is
used as the GPT-3 model. Note also that HyperCLOVA models are not trained by
instruct-tuning or reinforcement learning from human feedback, likewise ’text-
davinci-003’.. We sample a subset of the test set to contain 30 contexts per
each demographical category, i.e., a total of 450 contexts. The LLMs generate
sentences given the contexts in a zero-shot generation setup. The prompt used
for this experiment is listed in Appendix C.

Model | Safety Probability | Exp. Avg. Safety
---|---|---
$k=$ 1 | 2 | 4 | 8
GPT-3 (175B) | .809 | .902 | .956 | .969 | .625 $\pm$ .083
HyperClova (6.9B) | .673 | .796 | .796 | .876 | .589 $\pm$ .102
HyperClova (13B) | .713 | .789 | .789 | .862 | .581 $\pm$ .096
HyperClova (30B) | .711 | .844 | .844 | .900 | .588 $\pm$ .105
HyperClova (82B) | .647 | .813 | .813 | .887 | .575 $\pm$ .100

Table 5:  Safety evaluations of LLM’s continuations after given contexts.
Left: The empirical probability of generating safe sentence at lease once over
$k$ generations. Right: Expected average safety score of safe sentences with
standard deviations over 8 generations.

Table 5 presents the evaluation results. First, the empirical probability of
generating safe sentences increases as generation increases for all LLMs. In
other words, when the HyperCLOVA-82B generates 8 sentences per context, 88.7%
of continuations are safe w.r.t the classifier model. Notably, the more over-
generations, the more improved safety. Next, for the expected average of
safety score, we could not find distinguished differences among different
sizes of HyperCLOVA. Overall, GPT-3 shows more improved safety probability and
score than HyperCLOVA by the automatic evaluations.

Furthermore, we divide the results into those generated from a safe context
and an unsafe context in order to measure how the safety of the context
affects the model’s continuation. As can be seen by comparing both results
presented in Table 9, models generate more unsafe sentences when an unsafe
context was given, while all models generate 99% of safe continuations when
conditioned on a safe context in $k=8$ settings.

###  4.3 Filter-based Moderation

| Quality Assessments |
---|---|---
|  | Grammatical
---
Error-Free (%)
| Understandability
---
(%)
| Pertaning to Target
---
Social Group (%)
| Context (%)
---
Coherency
| Overall (%)
---
GPT-3 (175B) | 89.8 | 80.2 | 90.0 | 71.6 | 32.0
GPT-3 (175B) + filtering | 89.3 | 80.9 | 87.3 | 69.1 | 31.6
HyperCLOVA (80B) | 99.1 | 97.1 | 93.6 | 89.6 | 49.3
HyperCLOVA (80B) + filtering | 99.6 | 96.2 | 93.3 | 88.9 | 54.0
HyperCLOVA (30B) | 99.3 | 98.2 | 95.8 | 93.8 | 61.6
HyperCLOVA (30B) + filtering | 100 | 97.3 | 94.7 | 91.6 | 56.9

Table 6: Human evaluation on the subset of test set. Comparisons between
unfiltered responses and filtered responses among 8 generations from GPT-3
(175B; ‘text-davinci-003’), HyperClova (82B and 30B). Overall score denotes
the percentage of instances that are marked as passed all quality assessment
questions by all evaluators.  Figure 2:  Human evaluation on the subset of the
test set. We compared two HyperCLOVA models (82B and 30B) and the GPT-3 (175B;
text-davinci-003) models, for both with and without filtering.

We demonstrate the efficacy of filter-based moderation of unsafe sentence
generation. The filtering approach samples the safest sentence among 8
generations. We conduct a human-evaluation experiment to assess the quality
and safety of generation results. The evaluation results of the three models —
GPT-3, HyperCLOVA 30B, and 82B are compared in Figure 2 and Table 6.

With the filtering process, we find that the ratio of unsafe generations
decreases for all models by 16.47%p on average. We observe that the filter-
based moderation remarkably improves the safety of all LLMs by reducing unsafe
generation as 16%, 15%, and 18.5% and by increasing safe sentences as 15.6%,
15.3%, and 18.7% for GPT-3, 82B-HyperCLOVA, and 30B-HyperCLOVA, respectively.
It is interesting that the ratio of the ambiguous sentences generated by GPT-3
does not decrease despite the filtering.

Table 6 presents qualitative results of sentences generated by each model and
the effects of the filter-based moderation. Inconsistent with the results in
Figure 2, the filter-based moderation does not improve the quality of
generated sentences. This means the filtering is likely to slightly sacrifice
the coherency of generation by playing the role of constraints as a side
effect against enhancing safety. However, overall quality scores of all LLMs
are competitive enough, and HyperCLOVA presents better qualitative performance
than GPT-3, consistent with the results in Figure 2.

Figure 3: Moderation results on each category in the augmented test set. Left:
Safe response ratio from human evaluation results. Right: Safe sentence
classification performance of the best classifier (KcELECTRa). The vertical
lines represent the averages of safe response and accuracy for all categories.
Categories are ordered by descend of the classifier’s accuracy.

###  4.4 Social Bias Mitigation Level by Category

We analyze the moderation results by the 15 demographic categories. Before
getting a result, we augmented the test set with additional annotated data to
increase the number of samples per category and the reliability of the test
results. As a result, our augmented test set consists of 6,801 (context,
sentence) pairs (see Table 10 for detailed statistics for it). For experiments
conducted in this section, we sample a small subset from the augmented test
set to contains at least 48 contexts per category, resulting in 1,746
contexts. All other settings follow of them in Sec 2.

Figure 3 presents the human evaluation results of filter-based moderation by
each demographic category. Each category displays a different ratio of
generated safe sentences. By comparing with and without filter-based
moderation, we can notice that the efficacy of the filtering process also
varies. For example, we find the biggest increase of safe generations ratio in
Disability status category (+64.0%) while the smallest in Marital status
(+0.85%). Within the category, the differences also exist between models; such
as in Disability status category, HyperCLOVA-82B got an increase of 33.3%p but
HyperCLOVA-30B got only 4.1%p (See Figure 6 for the results by the group for
all three models).

Since filter-based moderation utilizes a filter model, it it natural to assume
that there could appear to be a correlation between the performance of the
filter model and the moderation efficacy. To identify any tendencies between
the two, we have also included the accuracy of the filter model in Figure 3.
We, however, couldn’t find a strong correlation between them. We conjecture
the reason is the relatively small differences in accuracy across the
categories or the sampled set used here not being large enough. Further
analysis is expected in future work. Despite this, the filter-based moderation
approach demonstrates the effectiveness for all social demographic categories.
It is crucial to scrutinize and improve the models’ safety for fair
consideration of each demographic category and group.

##  5 Conclusion

To alleviate unsafe social bias of LLMs, we propose a large-scale social bias
dataset related to safety addressing the Korean language and cultures, KoSBi.
Our dataset covers 72 demographic groups in 15 categories, consisting of 34k
of situation context and following sentence pairs. To construct KoSBi, we
employ a human-LLM collaboration framework, where HyperCLOVA generates
contexts and sentences, and human annotators label them as safe or unsafe.
Extensive experiments present our dataset as differentiated from existing
prevalent datasets on social bias and hate speech. Moreover, the results show
the filter model trained with our dataset remarkably improves the ratio of
generating safe sentences for various LLMs such as GPT-3 and HyperCLOVA with
diverse model sizes, which presents the efficacy of our dataset.

## Limitations

The proposed KoSBi addresses social bias based on Korean culture with the
Korean language. This Korean-specific property might restrict the
effectiveness of our dataset in Korea and its similar cultures. However, our
dataset construction and evaluation protocol can contribute to a helpful guide
for other research groups on AI safety to build the datasets for their
cultures and languages.

The performance of the filter models for harmless sentence classification in
this study is not very competitive. We leave it as a future research topic to
make a filter classifier with higher accuracy on our dataset because the goal
of this study is not to make a strong social bias filter itself.

## Ethics Statement

We expect that our KoSBi can considerably contribute to enhancing the safe
usage of LLMs’ applications by reducing risks caused by social bias.
Constructing datasets on harmfulness is likely to cause stress on the
contributors, such as human experts and crowd workers. To minimize their
stress exposure, we use HyperCLOVA to generate contexts and sentences and ask
humans to label them. Furthermore, our study was approved by the public
institutional review board (IRB) affiliated with the Ministry of Health and
Welfare of South Korea (P01-202211-01-016).

## Acknowledgements

The authors would like to thank all committee members of the AI Ethics Forum
for Human at NAVER, including Meeyoung Cha, Byoungpil Kim, Eun-Ju Lee, Yong
Lim, Alice Oh, Sangchul Park, Woochul Park, Joonha Jeon, Jonghyun Kim, Do Hyun
Park, and Eunjung Cho, for their constructive feedback and helpful
discussions. We are also grateful to Ryumin Song, Jaehyeon Kim, and Jisun Kim
at Crowdworks, who cooperated in the data collection process, and the 200
crowdworkers who participated in the process. In addition, the authors thank
the research members of SNU-NAVER Hyperscale AI Center and KAIST-NAVER
Hypercreative AI Center for discussion and thank Haksoo Ko and Yejin Choi for
valuable discussion. This project is financially supported by NAVER Cloud.

## References

  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b.  Constitutional ai: Harmlessness from ai feedback.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Demus et al. (2022) Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022.  Detox: A comprehensive dataset for German offensive language and conversation analysis.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 143–153, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Deng et al. (2022) Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. 2022.  COLD: A benchmark for Chinese offensive language detection.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11580–11599, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Dinan et al. (2022) Emily Dinan, Gavin Abercrombie, A. Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2022.  SafetyKit: First aid for measuring safety in open-domain conversational systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4113–4133, Dublin, Ireland. Association for Computational Linguistics.
  * Emelin et al. (2021) Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021\.  Moral stories: Situated reasoning about norms, intents, actions, and their consequences.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 698–718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Fiske (2023) Susan T. Fiske. 2023.  _Prejudice, discrimination, and stereotyping_.  Noba textbook series: Psychology. DEF Publisher.  http://noba.to/jfkx7nrd.
  * Forbes et al. (2020) Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020\.  Social chemistry 101: Learning to reason about social and moral norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 653–670, Online. Association for Computational Linguistics.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022\.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.  ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3309–3326, Dublin, Ireland. Association for Computational Linguistics.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021.  Aligning AI with shared human values.  In _International Conference on Learning Representations_.
  * Jeong et al. (2022) Younghoon Jeong, Juhyun Oh, Jongwon Lee, Jaimeen Ahn, Jihyung Moon, Sungjoon Park, and Alice Oh. 2022.  KOLD: Korean offensive language dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10818–10833, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kang et al. (2022) TaeYoung Kang, Eunrang Kwon, Junbum Lee, Youngeun Nam, Junmo Song, and JeongKyu Suh. 2022.  Korean online hate speech dataset for multilabel classification: How can social science improve dataset on hate speech?
  * Kenton et al. (2021) Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021.  Alignment of language agents.
  * Kim et al. (2021) Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, and Nako Sung. 2021.  What changes can large-scale language models bring? intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3405–3424, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kim et al. (2022) Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022.  ProsocialDialog: A prosocial backbone for conversational agents.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4005–4029, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lee et al. (2022) Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. 2022.  K-MHaS: A multi-label hate speech detection dataset in Korean online news comment.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3530–3538, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Lee (2020) Junbum Lee. 2020.  Kcbert: Korean comments bert.  In _Proceedings of the 32nd Annual Conference on Human and Cognitive Language Technology_ , pages 437–440.
  * Lee (2021) Junbum Lee. 2021.  Kcelectra: Korean comments electra.  https://github.com/Beomi/KcELECTRA.
  * Li et al. (2020) Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020\.  UNQOVERing stereotyping biases via underspecified questions.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3475–3489, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022.  WANLI: Worker and AI collaboration for natural language inference dataset creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lourie et al. (2021) Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021.  Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(15):13470–13479.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Moon et al. (2020) Jihyung Moon, Won Ik Cho, and Junbum Lee. 2020.  BEEP! Korean corpus of online news comments for toxic speech detection.  In _Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media_ , pages 25–31, Online. Association for Computational Linguistics.
  * Mubarak et al. (2022) Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2022.  Emojis as anchors to detect arabic offensive language and hate speech.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021.  Klue: Korean language understanding evaluation.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022.  BBQ: A hand-built bias benchmark for question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2086–2105, Dublin, Ireland. Association for Computational Linguistics.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022.  Red teaming language models with language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3419–3448, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021.  HateCheck: Functional tests for hate speech detection models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58, Online. Association for Computational Linguistics.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020.  Social bias frames: Reasoning about social and power implications of language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5477–5490, Online. Association for Computational Linguistics.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412, Hong Kong, China. Association for Computational Linguistics.
  * Solaiman and Dennison (2021) Irene Solaiman and Christy Dennison. 2021.  Process for adapting language models to society (palms) with values-targeted datasets.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 5861–5873. Curran Associates, Inc.
  * Sun et al. (2022) Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022.  On the safety of conversational models: Taxonomy, dataset, and benchmark.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3906–3923, Dublin, Ireland. Association for Computational Linguistics.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Wallace et al. (2019a) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wallace et al. (2019b) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019b.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Warner and Hirschberg (2012) William Warner and Julia Hirschberg. 2012.  Detecting hate speech on the world wide web.  In _Proceedings of the Second Workshop on Language in Social Media_ , pages 19–26, Montréal, Canada. Association for Computational Linguistics.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019.  Detection of Abusive Language: the Problem of Biased Datasets.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 602–608, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Xenos et al. (2021) Alexandros Xenos, John Pavlopoulos, Ion Androutsopoulos, Lucas Dixon, Jeffrey Sorensen, and Leo Laugier. 2021.  Toxicity detection can be sensitive to the conversational context.
  * Xu et al. (2021a) Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021a.  Detoxifying language models risks marginalizing minority voices.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2390–2397, Online. Association for Computational Linguistics.
  * Xu et al. (2021b) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021b.  Bot-adversarial dialogue for safe conversational agents.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2950–2968, Online. Association for Computational Linguistics.
  * Yang et al. (2022) Kichang Yang, Wonjun Jang, and Won Ik Cho. 2022.  APEACH: Attacking pejorative expressions with analysis on crowd-generated hate speech evaluation datasets.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7076–7086, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics.

##  Appendix A The KoSBi Dataset

###  A.1 Domain and Categories of Social Demographics

The entire social demographic categories and groups are listed in Table 8.

###  A.2 Example Data

Figure 4: Example pairs of a context and a sentence with labels pertaining to
a given social demographic category and group. Note, "금수저" is a Korean
buzzword, roughly meaning "Silver spoon" or "Privileged background" in
English.

###  A.3 Details of Unsafe Label

| Unsafe sub-labels | # data
---|---|---
Context | Stereotypical | 4,719
Prejudice / Discrimination | 407
Other | 1,590
Undefined | 2,260
Sentence | Stereotypical | 8,197
Prejudice | 1,085
Discrimination | 655
Other | 336
| Undefined | 6,905
Table 7: Distribution of the unsafe sub-labels of context and sentence.
Undefined represents cases where three annotators could not decide the label
through major voting, but 2 or more annotators chose one of the unsafe sub-
labels.

Category | Social Group
---|---
Gender identity† | Male
Female
Others
Sexual orientation† | Homosexual
Age & Generation† | Baby
Childern
Teenagers
Young people
Middle-aged
Old people
Baby bommeres
386 Generation
Generation X
Milennials
Generation Z
Alpha Generation
Race, Ethnicity & Nationality† | South Korean
North Korean
Chinese
Japanese
American (U.S.)
Russian
Asian
African
European
Americans, Oceanians
People of color / White
Religion† | Nonreligious
Protestantism
Buddhism
Catholic
Islam
Others
Disability status† | Disability
Physical appearance† | Face Appearance
Body Type
Sexual Appearance
Others
Political orientation† | Liberal
Conservative
Others
Socio-economic status† | Employment Type
Economic Condition
Occupation
Domestic area of origin | Seoul
Gyeonggi-do
Chungcheong-do
Gyeongsang-do
Jeolla-do
Jeju-do
Gangwon-do
Korean residents from abroad
Marital status | Married
Single
Divorced
Remarriaed
Common-law marriage
Others
Pregnancy & Birth | Pregnant Person
Person Who Gave Birth
Pregnancy Before Marriage
Others
Family form | Foster Family
Broken Family
Multicultural Family
Step Family
Others
Criminal record | Violent Criminal Convictions
Others
Education & University & Major | Academic Background
Universities
Major

Table 8: Category and demographic groups considered in KoSBi. The categories
in both UDHR and NHRCK are marked by †.

##  Appendix B HyperClova Generation

#### Hyperparameters

We use the same hyperparameters for generating contexts and sentences: top-$p$
sampling with $p=0.8$, a temperature of 0.5, repeat penalty of 5, and stop
word of $"{\backslash}n"$. We gradually increased top-$p$ and temperature
every time we met duplication generations.

##  Appendix C Prompts of LLMs

#### Context Generation Prompt

Among 10 demonstration samples, we collect 5 demonstrations from the pool
which has the same demographic category with what we are generating and 3
demonstrations from the pool which has the same demographic group. Detailed
prompt is below:

아래 사회적 집단에 속하는 사람에 대해 특정한 상황이나 전반적인 현상을 중립적으로 작성하세요. $\Rightarrow$ General
Instruction

{category} 분류에 따라 {social group}에 속하는 사람에 대해서 작성 $\Rightarrow$ Category and
demographic group
중립적 상황 문장 : {context}
###
…
###
{category} 분류에 따라 {social group}에 속하는 사람에 대해서 작성
중립적 상황 문장 :

#### Safe Sentence Generation Prompt

Among 10 demonstration samples, we collect 5 demonstrations from the pool
which has the same context label with what we are generating.

아래 사회적 집단에 속하는 사람에 대해 "중립적 상황 문장" 뒤에 이어지는 자연스러운 "다음 문장"을 작성하세요. "다음 문장"은 해당
사회적 집단에 대해 고정관념과 편견이 없는 사람이 표현할 수 있는 안전한 문장입니다. 비하와 차별의 의도가 없고, 공정하며 고정관념적이지
않은 표현, 다양성을 존중하고, 인간의 존엄성과 자유와 기본권을 보장합니다. $\Rightarrow$ General Instruction

{category} 분류에 따라 {social group}인 사람에 대해서 작성 $\Rightarrow$ Category and
demographic group
중립적 상황 문장 : {context}
다음 문장 : {sentence}
###
…
###
{category} 분류에 따라 {social group}에 속하는 사람에 대해서 작성 $\Rightarrow$ Category and
demographic group
중립적 상황 문장 : {context}
다음 문장:

#### Unsafe Sentence Generation Prompt

All process is same with safe sentence generation except the prompt below:

아래 사회적 집단에 속하는 사람에 대해 "중립적 상황 문장" 뒤에 이어지는 자연스러운 "다음 문장"을 작성하세요. "다음 문장"은 해당
사회적 집단에 대해 고정관념과 편견을 가진 사람이 표현할 수 있는 위험한 문장입니다. 비하와 차별의 의도가 있고, 불공정하고 논란을 불러
일으킬 수 있으며, 사회적 집단 내 다양성을 해칠 수 있습니다. $\Rightarrow$ General Instruction

{category} 분류에 따라 {social group}인 사람에 대해서 작성 $\Rightarrow$ Category and
demographic group
중립적 상황 문장 : {context}
다음 문장 : {sentence}
###
…
###
{category} 분류에 따라 {social group}에 속하는 사람에 대해서 작성 $\Rightarrow$ Category and
demographic group
중립적 상황 문장 : {context}
다음 문장:

#### Zero-shot Prompt for Human-evaluation

The following prompt is used for the human-evaluation in Section 2

아래 사회적 집단에 속하는 사람에 대해 ”상황 문장” 뒤에 이어지는 자연스러운 ”다음 문장”을 작성하세요.
##
{category} 분류에 따라 ”{social group}”에 속하는 사람에 대해서 작성
상황 문장: {context}
다음 문장:

##  Appendix D Modeling Details

All the training processes described below are under PyTorch-
Lightning999https://www.pytorchlightning.ai/ and
Huggingface101010https://huggingface.co/ environments. For training, the
search space for hyperparameters is:

  * •

learning rate : [$1e-5$, $2e-5$, $3e-5$, $4e-5$, $5e-5$]

  * •

batch size : [32, 48]

  * •

gradient clipping value : [0.0, 1.0]

  * •

epoch : 15

  * •

early stopping : after 5 epochs without improvement

###  D.1 Context Filter Models

We use KcELECTRa Lee (2021) as a backbone model for our context filter model.
The demographic group and the context concatenated by the separate
token([SEP]) are fed to the model to train the model to predict whether the
demographic group is in the context text. 3,819 and 7,569 data points are used
for training after iterations 1 and 2, respectively (80/10/10 split). The best
configuration is $5e-5$ learning rate, 48 batch size, and 0.0 gradient
clipping value for both iterations 1 and 2, showing 83.51% and 90.75% of
accuracy for each test set, respectively.

###  D.2 Next Sentence Filter Models

We also use KcELECTRa as a backbone model for our next sentence filter model.
Note that the main purpose of the next sentence filtering process is to
leverage the filter model to collect the most ambiguous samples w.r.t the
model. The separate token concatenates the context and the next sentence, and
the model is trained to predict the unsafeness of the text. 4,324 and 11,457
data points are used for training after iterations 1 and 2, respectively
(80/10/10 split). The best hyperparameter setup is ($5e-5$ learning rate, 32
batch size, 0.0 gradient clipping value) and ($2e-5$ learning rate, 48 batch
size, 0.0 gradient clipping value) for iterations 1 and 2, respectively. The
accuracies of the best models are 83.83% (iteration 1) and 69.37% (iteration
2). Due to ambiguous data points being augmented for iteration 2, the later
model shows lower accuracy.

###  D.3 Safe Sentence Classifiers

After collecting all data points, we train a safe sentence classifier. In
addition to the KcELECTRa model, we use KLUE-BERT Park et al. (2021) and
KcBERT Lee (2020) as candidates. As mentioned in Section 4.1, we augment data
by using context data. Among six configurations which consist of three models
and two datasets (with and without augmentation), the best model is KcELECTRa
with augmentation (71.22% accuracy). The hyperparameter setup is $1e-5$
learning rate, 32 batch size, and 0.0 gradient clipping value.

##  Appendix E Safety Evaluations of Continuations

Model | Safety Probability | Exp. Avg. Safety
---|---|---
$k=$ 1 | 2 | 4 | 8
Safe Context |  |  |  |  |
GPT-3 (175B) | .931 | .961 | .984 | .993 | .674 $\pm$ .083
HyperClova (6.9B) | .806 | .931 | .977 | .993 | .626 $\pm$ .103
HyperClova (13B) | .766 | .918 | .974 | .990 | .642 $\pm$ .108
HyperClova (30B) | .809 | .941 | .977 | .990 | .647 $\pm$ .102
HyperClova (82B) | .829 | .918 | .967 | .993 | .660 $\pm$ .106
Unsafe Context |  |  |  |  |
GPT-3 (175B) | .644 | .753 | .870 | .918 | .522 $\pm$ .082
HyperClova (6.9B) | .432 | .616 | .740 | .842 | .469 $\pm$ .093
HyperClova (13B) | .507 | .616 | .767 | .849 | .473 $\pm$ .099
HyperClova (30B) | .363 | .514 | .603 | .712 | .443 $\pm$ .073
HyperClova (82B) | .342 | .493 | .651 | .788 | .441 $\pm$ .093

Table 9:  Safety evaluations of LLM’s continuations after given safe (top) and
unsafe (bottom) contexts, respectively. All metrics are calculated as the same
manner as in Table 5.

Table 9 shows the safety generation results given safe and unsafe contexts,
respectively. As can be seen by comparing both results, models generate more
unsafe sentences when an unsafe context is given, while all models generate
99% of safe continuations when conditioned on a safe context in $k=8$
settings.

##  Appendix F Results and Analyses on Augmented Test Set

Context | Safe | Unsafe | Undecided | Total
---|---|---|---|---
Sentence | Safe | Unsafe | Total | S. | U. | T. | S. | U. | T.
Test set | 1,382 | 1,092 | 2,474 | 317 | 617 | 934 | 7 | 11 | 15 | 3,423
Augmented | 2,681 | 2,268 | 4,949 | 589 | 1,239 | 1,828 | 11 | 13 | 24 | 6,801

Table 10: The number of instances for the test and augmented test sets. Figure
5:  Human evaluation on the subset of the augmented test set. For all three
models, filter-based moderation shows efficacy on reducing unsafe generations.

| Quality Assessments |
---|---|---
|  | Grammatical
---
Error-Free (%)
| Understandability
---
(%)
| Pertaning to Target
---
Social Group (%)
| Context (%)
---
Coherency
| Overall (%)
---
GPT-3 (175B) | 84.4 | 77.4 | 87.3 | 70.8 | 30.2
GPT-3 (175B) + filtering | 86.4 | 79.4 | 86.5 | 71.1 | 30.1
HyperCLOVA (80B) | 98.9 | 97.9 | 93.9 | 90.5 | 56.5
HyperCLOVA (80B) + filtering | 99.3 | 97.5 | 92.5 | 88.9 | 56.0
HyperCLOVA (30B) | 99.0 | 98.3 | 95.4 | 93.0 | 62.6
HyperCLOVA (30B) + filtering | 99.1 | 97.9 | 93.6 | 91.8 | 60.0

Table 11: Human evaluation on the subset of augmented test set. Following the
Table 6, comparisons between unfiltered responses and filtered responses among
8 generations from GPT-3 (175B; ‘text-davinci-003’), HyperClova (82B and 30B)
are shown.

As mentioned in Sec 4.4, we augmented our test set with additional annotated
data to increase the reliability of test results. As a result, the augmented
test set has 6,801 data points (See Table 10). Among them, we randomly sampled
1,746 contexts for the human-evaluation experiments, which is the same
procedure described in Sec 2. As seen in Figure 5, we can still observe that
the filter-based moderation reduces unsafe generations for all three models.
Table 11 presents qualitative results for another subset of the test set.

Figure 6: Moderation results on each category in the augmented test set. Left:
Safe response ratio from human evaluation results. Right: Safe sentence
classification performance of the best classifier (KcELECTRa). The vertical
lines represent the averages of safe response and accuracy for all categories.
Categories are ordered by descend of the classifier’s accuracy.

##  Appendix G Social Bias Mitigation Level by Category

Figure 6 shows all results with and without the filter-based moderation for
GPT-3 (175B), HyperCLOVA (82B), and HyperCLOVA (30B). Although the increment
of safety does not strongly correlate to the performance of the classifier,
the filter-based moderation approach demonstrates the effectiveness for all
social demographic categories. It is crucial to scrutinize and improve the
models’ safety for fair consideration of each demographic category and group.

##  Appendix H Human Annotation

###  H.1 Crowd Worker Compensation

We utilized one of the representative crowdsourcing platforms in South Korea.
Among all applicants to our project, we selected 200 crowd workers. All
workers have received reasonable monetary compensation; 80 KRW per sub-single
question. All workers are expected to finish 2$\sim$3 sub-single questions in
one minute, resulting in the minimum compensation is 9,600 KRW/hour. For
reference, the minimum hourly wage in South Korea is 9260 KRW in 2023. The
annotation guidelines and the interface is depicted in Figure 7 and Figure 8.

###  H.2 Annotation Demographics

The detailed demographics are presented in Table 12. Note that every single
data was annotated by two females and one male or vice versa.

Gender
---
Male | 96 | 48.0%
Female | 103 | 51.5%
Prefer not to mention | 1 | 0.5%
Age
18-24 | 4 | 2.0%
25-34 | 44 | 22.0%
35-44 | 71 | 35.5%
45-54 | 55 | 27.5%
55-64 | 23 | 11.5%
65+ | 2 | 1.0%
Prefer not to mention | 1 | 0.5%
Country of Origin
South Korea | 199 | 99.5%
China | 1 | 0.5%
Domestic Area of Origin
Seoul | 71 | 35.5%
Gyeongsang, Daegu, Busan | 40 | 20.0%
Gyeonggi, Incheon | 42 | 21.0%
Jeolla, Gwangju | 19 | 9.5%
Chungcheong, Daejeon, Sejong | 22 | 11.0%
Gangwon | 2 | 1.0%
Jeju | 3 | 1.5%
Prefer not to mention | 1 | 0.5%
Education
College degree - Associate or Bachelor’s | 147 | 73.5%
Graduate or Professional Degree | 31 | 15.5%
High school, GED, etc. | 21 | 10.5%
Prefer not to mention | 1 | 0.5%
Sexual Orientation
Straight | 187 | 93.5%
LGBTQ+ | 1 | 0.5%
Prefer not to mention | 12 | 6.0%
Disability
No | 194 | 97.0%
Yes | 1 | 0.5%
Prefer not to mention | 5 | 2.5%
Total | 200 |

Table 12: Demographics of the crowd workers.

###  H.3 Annotation Guidelines and Interface

Figure 7:  Question annotation setup. Q1: Quality check (understandability and
grammatically/semantically error-free). Q2: Pertaining to Target Social Group.
Q3: Label of Context (Safe/Unsafe).  Figure 8:  Response annotation setup.
Q1: Quality check (appropriateness to the "Question" and
grammatically/semantically error-free). Q2: Label of Sentence (Safe/Unsafe)
Q2-1: (if the sentence is ‘Unsafe’) Label sub-labels.