# How Knowledge Graph and Attention Help?
A Quantitative Analysis into Bag-level Relation Extraction

Zikun Hu1 , Yixin Cao2 , Lifu Huang3 , Tat-Seng Chua1
1National University of Singapore
2S-Lab, Nanyang Technological University
3Computer Science Department, Virginia Tech
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Knowledge Graph (KG) and attention mechanism have been demonstrated effective
in introducing and selecting useful information for weakly supervised methods.
However, only qualitative analysis and ablation study are provided as
evidence. In this paper, we contribute a dataset and propose a paradigm to
quantitatively evaluate the effect of attention and KG on bag-level relation
extraction (RE). We find that (1) higher attention accuracy may lead to worse
performance as it may harm the model’s ability to extract entity mention
features; (2) the performance of attention is largely influenced by various
noise distribution patterns, which is closely related to real-world datasets;
(3) KG-enhanced attention indeed improves RE performance, while not through
enhanced attention but by incorporating entity prior; and (4) attention
mechanism may exacerbate the issue of insufficient training data. Based on
these findings, we show that a straightforward variant of RE model can achieve
significant improvements (6% AUC on average) on two real-world datasets as
compared with three state-of-the-art baselines. Our codes and datasets are
available at https://github.com/zig-kwin-hu/how-KG-ATT-help.

##  1 Introduction

Relation Extraction (RE) is crucial for Knowledge Graph (KG) construction and
population. Most recent efforts rely on neural networks to learn efficient
features from large-scale annotated data, thus correctly extract the
relationship between entities. To save the manual annotation cost and
alleviate the issue of data scarcity, distant supervision relation extraction
(DSRE) Mintz et al. (2009) is proposed and becomes increasingly popular as it
can automatically generate large-scale labeled data. DSRE is based on a simple
yet effective principle: if there is a relation between two entities in KG,
then all sentences containing mentions of both entities are assumed to express
this relation and will form a sentence bag as its annotations.

Figure 1: Examples of disturbing bags in NYT-FB60K.

Although effective, distant supervision may introduce noise to a sentence bag
when the assumption fails — some sentences are not describing the target
relation Zeng et al. (2015) (a.k.a. noisy annotation). To alleviate the
negative impacts of noise, recent studies Lin et al. (2016); Ji et al. (2017);
Du et al. (2018); Li et al. (2020) leveraged attention to select informative
instances from a bag. Furthermore, researchers introduced KG embeddings to
enhance the attention mechanism Hu et al. (2019); Han et al. (2018a). The
basic idea is to utilize entity embeddings as the query to compute attention
scores, so that the sentences with high attention weights are more likely to
be valid annotations Zhang et al. (2019). Previous studies have shown
performance gain on DSRE with attention module and KG embeddings, however,
it’s still not clear how these mechanisms work, and, are there any limitations
to apply them?

In this paper, we aim to provide a thorough and quantitative analysis about
the impact of both attention mechanism and KG on DSRE. By analyzing several
public benchmarks including NYT-FB60K Han et al. (2018a), we observe lots of
disturbing bags — all of the bag’s sentences are valid or noisy annotations,
which shall lead to the failure of attention. As shown in Figure-1, all of
annotations in the first disturbing bag are valid, while the learned
attentions assign the second annotation with a very low weight, which suggests
an inefficient utilization of annotations and exacerbates the data sparsity
issue. Or, in the second bag, all sentences are noisy, can attention and KG
still improve the performance? If so, how do they work and to what extent can
they tolerate these disturbing bags? Answering these questions is crucial
since this type of noise is common in practice. The unveiling of their working
mechanism shall shed light on future research direction, not limited to DSRE.

To achieve this, we propose a paradigm based on newly curated DSRE benchmark,
BagRel-Wiki73K extracted from FewRel Han et al. (2018b) and Wikidata
111dumps.wikimedia.org/wikidatawiki/entities/20201109/, for quantitative
analysis of attention and KG. With extensive experiments, we conclude the
following innovative and inspiring findings:

(1) The accuracy of attention is inversely proportional to the total noise
ratio and disturbing bag ratio of training data; (2) attention effectively
selects valid annotations by comparing their contexts with the semantics of
relations, thus tends to rely more on the context to make predictions.
However, it somehow lowers the model’s robustness to noisy sentences that do
not express the relation; (3) KG-enhanced attention indeed improves RE
performance, surprisingly not via enhanced attention accuracy, but by
incorporating entity features to reduce the demand of contexts when facing
noise; (4) attention could hurt the performance especially when there is no
sufficient training data.

Based on the above observations, we propose a new straightforward yet
effective model based on pre-trained BERT Devlin et al. (2018) for RE with
Concatenated KG Embedding, namely BRE+CE. Instead of in-bag attention, it
breaks the bag and ensembles the results of all sentences belonging to the
bag. For each sentence, we directly incorporate entity embeddings into BERT,
rather than to enhance attentions, to improve the robustness of extracting
both context and mention features. BRE+CE significantly outperforms existing
state-of-the-arts on two publicly available datasets, NYT-FB60K Han et al.
(2018a) and GIDS-FB8K Jat et al. (2018), by 6% AUC on average. We summarize
our contributions as follows:

  * •

To the best of our knowledge, our proposed framework is the first work to
quantitatively analyze the working mechanism of Knowledge Graph and attention
for bag-level RE.

  * •

We have conducted extensive experiments to inspire and support us with the
above findings.

  * •

We demonstrate that a straightforward method based on the findings can achieve
improvements on public datasets.

##  2 Related Work

To address the issue of insufficient annotations, Mintz et al. (2009) proposed
distant supervision to generate training data automatically, which also
introduces much noise. From then, DSRE becomes a standard solution that relies
on multi-instance learning from a bag of sentences instead of a single
sentence Riedel et al. (2010); Hoffmann et al. (2011). Attention mechanism Lin
et al. (2016) accelerates this trend via strong ability in handling noisy
instances within a bag Liu et al. (2017); Du et al. (2018). Aside from intra-
bag attention, Ye and Ling (2019) also designed inter-bag attention
simultaneously handling bags with the same relation. To deal with only-one-
instance bags, Li et al. (2020) utilized a new selective gate (SeG) framework
to independently assign weights to each sentence. External KG is also
incorporated to enhance the attention module Han et al. (2018a); Hu et al.
(2019). However, due to the lack of sentence-level ground truth, it is
difficult to quantitatively evaluate the performance of the attention module.
Previous researchers tend to provide examples as case study.222Shahbazi et al.
(2020) claim to annotate each positive bag in NYT-FB60K, but haven’t published
their code and dataset. Therefore, we aim to fill in this research gap by
constructing a dataset and providing a framework for thorough analysis.

##  3 Preliminary

Knowledge Graph (KG) is a directed graph $G=\\{E,R,T\\}$, where E denotes the
set of entities, $R$ denotes the set of relation types in $G$, and
$T=\\{(h,r,t)\\}\subseteq E\times R\times E$ denotes the set of triples. KG
embedding models, e.g., RotatE Sun et al. (2019), can preserve the structure
information in the learned vectors $\mathbf{e}_{h}$, $\mathbf{e}_{t}$ and
$\mathbf{e}_{r}$. We adopt TransE Bordes et al. (2013) in experiments.

Bag-level relation extraction (RE) takes a bag of sentences
$B=\\{s_{1},s_{2},\ldots,s_{m}\\}$ as input. Each sentence $s_{i}$ in the bag
contains the same entity pair $(h,t)$, where $h,t\in E$. The goal is to
predict a relation $y\in R$ between $(h,t)$.

Attention-based Bag-level RE uses attention to assign a weight to each
sentence within a bag. Given a bag $B$ from the dataset $D$, an encoder is
first used to encode all sentences from $B$ into vectors
$\\{\mathbf{s^{\prime}}_{1},\mathbf{s^{\prime}}_{2},\ldots,\mathbf{s^{\prime}}_{m}\\}$
separately. Then, an attention module computes an attention weight
$\alpha_{i}$ for each sentence and outputs the weighted sum of
$\\{\mathbf{s^{\prime}}_{i}\\}$ as $\mathbf{\overline{s}}$ to denote $B$:

$\omega_{i}=\mathbf{v}_{y}\cdot\mathbf{s^{\prime}}_{i}$ (1)
$\alpha_{i}=\frac{\exp(\omega_{i})}{\sum\limits_{j=1}^{m}\exp(\omega_{j})}$
(2)
$\mathbf{\overline{s}}=\sum\limits_{i=1}^{m}\alpha_{i}\mathbf{s^{\prime}}_{i}$
(3)

where $\mathbf{v}_{y}$ is the label embedding of relation $y$ in the
classification layer, we denote this attention module as ATT in the rest of
paper.

KG-enhanced attention aims to improve $\mathbf{v}_{y}$ with entities
$\mathbf{e}_{h}$ and $\mathbf{e}_{t}$ Han et al. (2018a):

$\mathbf{r}_{ht}=\mathbf{e}_{h}-\mathbf{e}_{t}$ (4)
$\omega_{i}=\mathbf{r}_{ht}\cdot\tanh(\mathbf{W}_{s}\mathbf{s^{\prime}}_{i}+\mathbf{b}_{s})$
(5)

where $\mathbf{r}_{ht}$ is regarded as latent relation embedding. We mark this
way of computing $\omega_{i}$ as KA. $\mathbf{W}_{s}$ and $\mathbf{b}_{s}$ are
learnable parameters.

Given a bag representation $\mathbf{\overline{s}}$, the classification layer
further predicts a confidence of each relation:

$\overline{\mathbf{o}}=\mathbf{W}_{b}\overline{\mathbf{s}}+\mathbf{b}_{b}$ (6)
$P(y|B)=\text{Softmax}(\overline{\mathbf{o}})$ (7)

where $\overline{\mathbf{o}}$ is a logit vector. $\mathbf{W}_{b}$ and
$\mathbf{b}_{b}$ are learnable parameters. During training, the loss is
computed by:

$L=-\sum\limits_{i=0}^{n}\text{log}(P(y_{i}|B_{i}))$ (8)

where $n$ is the number of training bags in $D$. Since the classification
layer is linear, we can rewrite the bag’s logit vector $\overline{\mathbf{o}}$
using a weighted sum of each sentence’s logit vector $\mathbf{o}$:

$\mathbf{o}_{i}=\mathbf{W}_{b}\mathbf{s^{\prime}}_{i}+\mathbf{b}_{b}$ (9)
$\overline{\mathbf{o}}=\sum\limits_{i=1}^{m}\alpha_{i}\mathbf{o}_{i}$ (10)

From equation 10, we can see that the model’s output on the whole bag depends
on three aspects: (1) the model’s output on valid sentences within the bag;
(2) the model’s output on noisy sentences within the bag; (3) the attention
weight assigned to valid sentences and noisy ones.

##  4 Benchmark

To quantitatively evaluate the effect of attention and KG on Bag-level RE, we
first define two metrics to measure the noise pattern (Section 4.1). Then, we
construct a KG and a Bag-level RE dataset (Section 4.2). Finally, we introduce
a general evaluation framework to assess attention, KG and the entire RE model
(Section 4.3).

###  4.1 Metrics Describing Noise Pattern

To analyze how attention module functions on different noise patterns, we
first design 2 metrics to describe the noise pattern: Noise Ratio (NR) and
Disturbing Bag Ratio (DR).

#### Noise Ratio (NR)

represents the proportion of noisy sentences in the dataset. Given a bag
$B_{i}$ and its relation label $y_{i}$, a sentence $s_{ij}\in B_{i}$ is noisy
if its context does not express $y_{i}$. Suppose $\text{Isn}(s_{ij},y_{i})$ is
an indicator function to tell whether $s_{ij}$ is noise. Then NR is defined
as:

$\text{NR}=\frac{\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{|B_{i}|}\text{Isn}(s_{ij},y_{i})}{\sum\limits_{i=1}^{n}|B_{i}|}$
(11)

where $|B_{i}|$ is the size of $B_{i}$, $n$ is the total number of bags.

#### Disturbing Bag Ratio (DR)

means the proportion of disturbing bags in the dataset. A bag is disturbing if
all sentences in it are valid or all sentences are noisy. Formally, we use
function $\text{Isd}(B_{i})$ to indicate whether a bag is disturbing or not:

$\text{Isd}(B_{i})=\prod_{j=1}^{|B_{i}|}\text{Isn}(s_{ij},y_{i})+\prod_{j=1}^{|B_{i}|}(1-\text{Isn}(s_{ij},y_{i}))$
(12)

Then we define DR as follows:

$\text{DR}=\frac{\sum\limits_{i=1}^{n}\text{Isd}(B_{i})}{n}$ (13)

Figure 2: Left: Process of synthesizing the valid sentence with correct
context and the noisy sentence with wrong context. Right: Visualization of
different train sets of different noise patterns, the four sets from left to
right are named as
$\text{train}_{\frac{2}{3},0}$,$\text{train}_{\frac{1}{2},0}$,$\text{train}_{\frac{1}{2},\frac{1}{2}}$
and $\text{train}_{\frac{1}{2},1}$.

###  4.2 Dataset Construction

Based on FewRel and Wikidata, we construct a Bag-level RE dataset containing
multiple training sets with different noise patterns, a test set and a
development set. For each sentence in the bags, there is a ground truth
attention label indicating whether it is a valid sentence or noise. We also
construct a KG containing all entities in the RE dataset by retrieving one-hop
triples from Wikidata.

#### Synthesize Sentence

FewRel is a sentence-level RE dataset, including 80 relations. For each
relation, there are 700 valid sentences. Each sentence has a unique entity
pair. Every sentence along with its entities and relation label form a tuple
$(s,h,t,y)$. We thus synthesize valid and noisy sentences for the same entity
pair for data augmentation.

The first step is to divide sentences of each relation into 3 sets:
$\text{train}_{\text{FewRel}}$, $\text{test}_{\text{FewRel}}$ and
$\text{dev}_{\text{FewRel}}$, where each set has 500, 100 and 100 sentences.
Then, for each tuple $(s,h,t,y)$ in the set, we aim to augment it to a bag
$B$, where all of its sentences contain $(h,t)$. Besides, the sentences in $B$
are either the original $s$, or a synthesized valid sentence, or a synthesized
noisy sentence. We synthesize sentences in the form of $(s^{\prime},h,t,y,z)$,
where $z$ denotes the attention label (1 for valid, 0 for noisy). In specific,
to synthesize a sentence, we randomly replace the source pair of entity
mentions with other target entity pairs while keeping the context unchanged.
Thus, if the contexts express the same relation type with the entity pair, we
can automatically assign an attention label.

We illustrate the synthesizing process in Figure 2.
$(s_{2},h_{2},t_{2},\textit{crosses})$ is a sentence from
$\text{train}_{\text{FewRel}}$. To generate a valid sentence, we randomly
select another sentence $(s_{1},h_{1},t_{1},\textit{crosses})$ which is
labeled with the same relation as $s_{2}$ from $\text{train}_{\text{FewRel}}$.
Then we replace its entity mentions $h_{1}$ and $t_{1}$ as $h_{2}$ and
$t_{2}$. The output is $(s_{4},h_{2},t_{2},\textit{crosses},1)$. Since its
context correctly describe crosses, we regard $s_{4}$ as valid. For the noisy
sentence, we randomly select a sentence $(s_{3},h_{3},t_{3},\textit{isA})$
under another relation. With similar process for $s_{4}$, we get a synthesize
sentence $(s_{5},h_{2},t_{2},\textit{crosses},0)$. Because the context of
$s_{5}$ does not express target relation, we label it as a noise.

#### Training Sets with Different Noise Patterns

As defined in Section 4.1, we use NR and DR to measure the noise pattern of
Bag-level RE dataset. By controlling the number of synthesized noisy sentences
in each bag and the total ratio of noise among all sentences, we can construct
several training sets with different patterns. In the following sections, we
denote a training set of which the NR is $x$ and DR is $y$ as
$\text{train}_{x,y}$. Higher $x$ and $y$ indicate noisy sentences and
disturbing bags account for larger proportion.

For example, in Figure 2, assuming there are 4 sentences in
$\text{train}_{\text{FewRel}}$, for each sentence, we synthesize two noisy
sentences that form the bag together with the original sentence. Thus each bag
contains 3 sentences: 1 valid and 2 noisy, and its NR is 2/3 and DR is 0. For
the other 3 sets, the number of synthesized noisy sentences equals the sum of
original valid sentences and synthesized valid sentences. Thus they all have a
NR of 1/2. Since we define bags containing no valid sentences or no noisy
sentences as disturbing bags, the third set and fourth set have 2 and 4
disturbing bags, with a DR of 1/2 and 1, respectively.

#### Test Set and Development Set

We also construct a test and a development set. Similar as the second set in
Figure 2, each bag in the test/dev sets contains two sentences, the NR of both
sets is 1/2 while the DR is 0. I.e., in every bag of test/dev sets, there is
one valid sentence and one noisy sentence. Instead of multiple test sets of
different noise patterns, we only have one test set so that the evaluation of
different models is consistent. To avoid information leak, when construct
$\text{train}_{x,y}$, test and development sets, the context of synthesized
sentences only come from $\text{train}_{\text{FewRel}}$,
$\text{test}_{\text{FewRel}}$ and $\text{development}_{\text{FewRel}}$,
respectively.

The final BagRel contains 9 train sets, 1 test and 1 development set, as
listed in Table 1. The NR of the training sets has three options: 1/3, 1/2 or
2/3, and similarly, DR can be 0, 1/2 or 1. The NR of both test and development
sets are 1/2, while their DR are 0. All data sets contain 80 relations. For
training sets whose NR are 1/3, 1/2 and 2/3, every bag in these sets contains
3, 2 and 3 sentences, respectively.

Dataset | # Noisy Sentence | # Sentence | # Bag
---|---|---|---
$\text{train}_{\frac{1}{3},(0,\frac{1}{2},1)}$ | 40K | 120K | 40K
$\text{train}_{\frac{1}{2},(0,\frac{1}{2},1)}$ | 40K | 80K | 40K
$\text{train}_{\frac{2}{3},(0,\frac{1}{2},1)}$ | 80K | 120K | 40K
$\text{dev}_{\frac{1}{2},0}$ | 8K | 16K | 8K
$\text{test}_{\frac{1}{2},0}$ | 8K | 16K | 8K
Table 1: Statistics of 11 sets of BagRel-Wiki73K, where
$\text{train}_{c,(x,y,z)}$ denotes three sets of $\text{train}_{c,x}$,
$\text{train}_{c,y}$, and $\text{train}_{c,z}$.

#### KG Construction

To evaluate the impact of KG on attention mechanism, we also construct a KG
based on Wikidata. Denoting the set of entities appearing in FewRel as $E$, we
link each entity in $E$ to Wikidata by its Freebase ID, and then extract all
triples $T={(h,r,t)}$ in Wikidata where $h,t\in E$. To evaluate the effect of
structural information from KG, we also construct a random KG whose triple set
is $\hat{T}$. Specifically, for each triple $(h,r,t)$ in $T$, we corrupt it
into $(h,\hat{r},t)$ by replacing $r$ with a random relation $\hat{r}\neq r$.
Thus the prior knowledge within the KG is destroyed. KG-73K and KG73K-random
have the same scale: 72,954 entities, 552 relations and 407,821 triples.

Finally, we obtain BagRel-Wiki73K, including the Bag-level RE sets and KG-73K.

###  4.3 Evaluation Framework

We first define several measurements to evaluate the effect of the attention
mechanism and KG: Attention Accuracy (AAcc), Area Under precision-recall Curve
(AUC), AUC on Valid sentences (AUCV) and AUC on Noisy sentences (AUCN).

#### AAcc

measures the attention module’s ability to assign higher weights to valid
sentences than noisy sentences. Given a non-disturbing bag (a bag containing
both valid and noisy sentences) $B_{i}=\\{(s_{j},h_{i},t_{i},y_{i},z_{j})\\}$
and the predicted probability distribution $\mathbf{p_{i}}$, the AAcc of this
bag is calculated by the following formula:

$\text{AAcc}_{i}=\frac{\sum\limits_{j=1}^{m}\sum\limits_{k=1}^{m}\textbf{I}(z_{j})\textbf{I}(1-z_{k})\textbf{I}(p_{ij}>p_{ik})}{\sum\limits_{j=1}^{m}\textbf{I}(z_{j})\sum\limits_{j=1}^{m}\textbf{I}(1-z_{j})}$
(14)

where $m=|B_{i}|$ is the size of $B_{i}$, I($\cdot$) is an indicator function
which returns 1 or 0 if the input is True or False. By
$\sum\limits_{j=1}^{m}\textbf{I}(z_{j})\sum\limits_{j=1}^{m}\textbf{I}(1-z_{j})$,
we count how many valid-noisy sentence pairs contained in $B_{i}$. With
$\sum\limits_{j=1}^{m}\sum\limits_{k=1}^{m}\textbf{I}(z_{j})\textbf{I}(1-z_{k})\textbf{I}(p_{ij}>p_{ik})$,
we count how many pairs show higher weight on the valid sentence. Then the
AAcc of the whole data set is computed as
$\text{AAcc}=(\sum\limits_{i=1}^{n}\text{AAcc}_{i})/n$ where n is the number
of bags in the data set.

AAcc is designed specifically for non-disturbing bags. On disturbing bags,
with all sentences noisy or valid, it is meaningless to evaluate attention
module’s performance. So in test/dev sets of our BagRel-Wiki73k, all bags are
non-disturbing bags. Then without distraction, the evaluation results can
better present how the attention module works.

#### AUC

is a standard metric to evaluate DSRE model’s performance on bag-level test
set. As mentioned in section 3, attention-based model’s performance on non-
disturbing bags relies on three aspects: (1)AAcc, (2) model’s performance on
valid sentences and (3) model’s performance on noisy sentences. So we use AUCV
and AUCN to measure the second and the third aspects, respectively. The
difference between AUC and AUCV is that AUC is computed on the original test
set $D=\\{B_{i}\\}$, while AUCV is AUC computed on the Valid-only test set
$D^{v}=\\{B_{i}^{v}\\}$. Compared with $B_{i}$, $B_{i}^{v}$ has the same label
but removes all noisy sentences within it. Thus there is no noisy context
feature in $D^{v}$, then models can utilize both entity mentions and contexts
to achieve a high AUCV. On the opposite, AUCN is AUC computed on the Noise-
only test set $D^{n}=\\{B_{i}^{n}\\}$, where $B_{i}^{n}$ removes all valid
sentences in $B_{i}$. Since all context features in $D^{n}$ are noisy, to
achieve a high AUCN, models have to ignore context and rely more on mention
features to make predictions.

AUC, AUCV and AUCN range from $0$ to $1$, and a higher value of the 3 metrics
indicates that a model makes better prediction on the whole bag, valid
sentences and noisy sentences, respectively.

##  5 Method

To evaluate the effects of attention and KG, we design two straightforward
Bag-level RE models without the attention module, BRE and BRE+CE. By comparing
their performance with BRE+ATT (BRE with attention module) and BRE+KA (BRE
with KG-enhanced attention module), we can have a better understanding of the
roles of ATT and Knowledge-enhanced ATT.

BRE uses BERT Devlin et al. (2018) as the encoder. Specifically, we follow the
way described in Peng et al. (2020); Soares et al. (2019): entity mentions in
sentences are highlighted with special markers before and after mentions. Then
the concatenation of head and tail entity representations are used as the
representation $s^{\prime}$. Since BRE does not have attention mechanism, it
breaks the bags and compute loss on each sentence:

$L=-\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{|B_{i}|}\text{log}(P(y_{i}|s_{ij}))$
(15)
$P(y_{i}|s_{ij})=\text{softmax}(\mathbf{W}_{b}s^{\prime}_{ij}+\mathbf{b}_{b})$
(16)

BRE can be viewed as a special case of BRE+ATT. Its attention module assigns
all sentences in all bags with the same attention weight 1. During inference,
given a bag, BRE uses the mean of each sentence’s prediction as the whole
bag’s prediction:

$P(y_{i}|B_{i})=(\sum\limits_{j=1}^{|B_{i}|}P(y_{i}|s_{ij}))/|B_{i}|$ (17)

BRE+CE concatenates an additional feature vector $\mathbf{r}_{ht}$ with BERT
output, where $\mathbf{r}_{ht}$ is defined based on entity embeddings of $h$
and $t$. The concatenated vector is used as the representation of the sentence
and fed into the classification layer.

##  6 Experiment

We apply our proposed framework on BagRel-Wiki73K and two real-world datasets
to explore the following questions:

  * •

How noise pattern affects the attention module?

  * •

Whether attention mechanism promotes RE model’s performance?

  * •

How KG affects the attention mechanism?

  * •

Whether attention aggravates data sparsity?

###  6.1 Experimental Setup

For fair comparison, all of baselines share the same encoding structure as
BRE. The attention-based models include BRE+ATT,BRE+KA and BRE+SeG, where SeG
Li et al. (2020) is an advanced attention mechanism which achieves the state-
of-the-art performance on NYT-FB60K. Briefly, SeG uses sigmoid instead of
softmax to compute attention weights of each instance in a bag. The models
without attention are BRE and BRE+CE. To check the effect of noise pattern, we
train model on different train sets. As a reminder, $\text{train}_{x,y}$ is a
train set whose NR and DR is $x$ and $y$, respectively.

###  6.2 Noise Pattern v.s. Attention Accuracy

We train BRE+ATT on 9 different training sets with different noise patterns.
As shown in Figure 3, we can see that: (1) higher noise ratio (NR) makes the
model harder to highlight valid sentences, leading to a lower attention
accuracy (AAcc); (2) higher disturbing bag ratio (DR) results in lower AAcc,
indicating that disturbing bags challenge the attention module. Based on these
results, we claim that the noise pattern within the training set largely
affects the attention module’s effectiveness.

Figure 3: Attention accuracy (AAcc) on the test set of BagRel-Wiki73K. The
results are collected with BRE+ATT trained on train sets of various noise
patterns. The x axis denote train sets of different Disturbing bag Ratio (DR).
The different colors indicate various Noise Ratio (NR).

###  6.3 Attention v.s. RE Performance

Model | AUC | AAcc | AUCV | AUCN
---|---|---|---|---
BRE-$\text{train}_{\frac{1}{2},0}$ | .910 | NA | .932 | .850
BRE+ATT-$\text{train}_{\frac{1}{2},0}$ | .878 | .881 | .941 | .434
BRE+ATT-$\text{train}_{\frac{1}{2},\frac{1}{2}}$ | .897 | .751 | .932 | .711
BRE+ATT-$\text{train}_{\frac{1}{2},1}$ | .896 | .713 | .925 | .759
Table 2: Test results of models trained on different train set. In the Model
column, X-Y means model X trained on train set Y. Among 3 train sets,
$\text{train}_{\frac{1}{2},1}$ has the most disturbing bags, while
$\text{train}_{\frac{1}{2},0}$ has no such bag.

To quantitatively analyze the effect of attention mechanism, we compare the
performance of BRE and BRE+ATT in Table 2, keeping other variables of the
model unchanged. Particularly, a higher AUCV indicates the stronger ability of
the model itself — in an ideal setting without any noise, and a higher AUCN
indicates higher robustness of model to noise. Surprisingly, when using the
same training set $\text{train}_{\frac{1}{2},0}$, the AUC of the attention-
enhanced model is lower than the AUC of the model without attention ($0.878$
v.s. $0.910$). In addition, BRE+ATT has lowest AUC using
$\text{train}_{\frac{1}{2},0}$, which has no disturbing bags. The highest AAcc
($0.881$) also suggests that the attention module does effectively select
valid sentences. Why the most effective attention module leads to the worst
performance? The reason is that BRE+ATT-$\text{train}_{\frac{1}{2},0}$ has a
much lower AUCN, which indicates that it is less robust to noisy sentences.

Is it true that an effective attention module shall hurt model’s robustness to
noise? This is actually against our intuition. To answer it, we draw Figure 4
by assigning fixed attention weights to sentences during training.
Specifically, each bag in $\text{train}_{\frac{1}{2},0}$ has a valid sentence
and a noisy sentence, and we assign fixed attention weight $\alpha$ to the
valid and $1-\alpha$ to the noisy one, instead of computing $\alpha$ with
attention module. Then we test the resulting model’s AUCN and AUCV
performance. We can see that when the valid sentences receive higher attention
weights, the AUCV curve rises slightly, indicating the model’s performance
indeed gets enhanced. Meanwhile, the AUCN curve goes down sharply. This
demonstrates the effective attention weakens the model’s robustness to noise.
The reason is that the model with a high-performance attention module prefers
to utilize context information instead of entity mention features. Thus, it
usually fails if most contexts are noisy.

Figure 4: AUCV and AUCN results of BRE+ATT-$\text{train}_{\frac{1}{2},0}$
trained with fixed attention weights.

Thus we can explain the results in Table 2. $\text{train}_{\frac{1}{2},0}$ has
the highest AAcc, indicating that it assigns very low weights to noisy
sentences. Thus the gain from AUCV can not make up the loss from AUCN,
resulting a worse AUC.

In conclusion, attention module can effectively select valid sentences during
training and test. But it has an underlying drawback that it might hurt the
model’s ability to predict based on entity mention features, which are
important in RE tasks Li et al. (2020) Peng et al. (2020), leading to worse
overall performance.

###  6.4 KG v.s. Attention

Model |  AUC |  AAcc |  AUCV |  AUCN
---|---|---|---|---
BRE+ATT-$\text{train}_{\frac{1}{2},0}$ |  .878 |  .881 |  .941 |  .434
BRE+$\text{KA}_{\text{rand}}$-$\text{train}_{\frac{1}{2},0}$ |  .915 |  .762 |  .936 |  .659
BRE+KA-$\text{train}_{\frac{1}{2},0}$ |  .932 |  .857 |  .936 |  .560
BRE+KA-$\text{train}_{\frac{1}{2},\frac{1}{2}}$ |  .924 |  .720 |  .928 |  .723
BRE+KA-$\text{train}_{\frac{1}{2},1}$ |  .913 |  .617 |  .916 |  .761
BRE+CE-$\text{train}_{\frac{1}{2},0}$ |  .915 |  NA |  .935 |  .856
BRE+CE-$\text{train}_{\frac{1}{2},\frac{1}{2}}$ |  .919 |  NA |  .939 |  .849
BRE+CE-$\text{train}_{\frac{1}{2},1}$ |  .918 |  NA |  .941 |  .845
Table 3: Results of models trained on different train set. In the Model
column, X-Y means model X trained on train set Y.
BRE+$\text{KA}_{\text{rand}}$ uses entity embeddings learned on KG-73K-random
for the attention module.

To measure KG’s effect on the combined with attention mechanism, we compare
the results of KA with ATT, while keeping other parts of the model unchanged.
As shown in Table 3. When trained on $\text{train}_{\frac{1}{2},0}$, the KG-
enhanced model (KA-$\text{train}_{\frac{1}{2},0}$) has lower AAcc than the
model without KG (ATT-$\text{train}_{\frac{1}{2},0}$) ($0.857$ v.s. $0.881$),
while the AUC is higher ($0.932$ v.s. $0.878$). This is because the KA version
has a higher AUCN ($0.560$) and comparable AUCV and AAcc. Thus, the KG-
enhanced model achieves better performance on noisy bags, leading to a better
RE performance.

In addition, comparing Table 2 and Table 3, KA shows lower AAcc and higher
AUCN than ATT on all three train sets. This also demonstrates that KG does not
promote model’s performance by improving attention module’s accuracy, but by
enhancing the encoder and classification layer’s robustness to noisy
sentences. This makes sense because the information from KG focuses on
entities instead of contexts. By incorporating KG, the model relies more on
entity mention features instead of noisy contexts feature, thus becomes better
at classifying noisy sentences.

Moreover, comparing BRE+$\text{KA}_{\text{rand}}$’s performance with BRE+KA on
$\text{train}_{\frac{1}{2},0}$, we can observe that after incorporating entity
embeddings learned from a random KG, BRE+$\text{KA}_{\text{rand}}$ has a much
lower attention accuracy. This indicates that misleading knowledge would hurt
attention mechanism.

###  6.5 Attention v.s. Data Sparsity

Attention module assigns low weights to part of training sentences. When
training data is insufficient, not making full use of all training examples
could aggravate the data sparsity issue. Thus we compare performance of models
trained on subsets of $\text{train}_{\frac{1}{2},\frac{1}{2}}$. From Figure 5,
we can see that along with the decreasing size of training data, the
performance gap between BRE+ATT and BRE+CE becomes larger. This is because the
latter one fully utilizes every example by assigning the same weight 1 to all
sentences. We also check each model’s attention weights. BRE+SeG assigns all
sentences with weights $>0.9$, so its performance drop is similar to the model
without attention. Thus, we claim that traditional attention mechanism could
exacerbate the model’s ability to insufficient data. This motivates us a
better attention mechanism for few-shot settings. We leave it in the future.

Figure 5: AUC test results of models trained on 4 subsets of BagRel-Wiki73K’s
$\text{train}_{\frac{1}{2},\frac{1}{2}}$ set. The 4 subsets contain 2%, 10%,
20% and 100% bags of $\text{train}_{\frac{1}{2},\frac{1}{2}}$ set.

###  6.6 Stability of Attention v.s. Noise Pattern

From results in Table 2 and Table 3, we can see that the performance of BRE+CE
is stable when the ratio of disturbing bags changes. In comparison, BRE+ATT
and BRE+KA show varying results across different train sets. On
$\text{train}_{\frac{1}{2},1}$ which has the most disturbing bags, BRE+CE
outperforms BRE+ATT and BRE+KA, demonstrating that BRE+CE could be a
competitive method for Bag-level DSRE.

###  6.7 Results on Real-world Datasets

Model | NYT-FB60K | GIDS-FB8K
---|---|---
JointE | .408 | .912
RELE | .497 | .905
SeG | .451 | .913
BRE+ATT | .457 | .917
BRE+KA | .480 | .917
BRE | .625 | .910
BRE+CE | .630 | .917
Table 4: AUC on NYT-FB60K and GIDS-FB8K.

Figure 6: Precision/recall curves on NYT-FB60K

Based on previous observations, we find that BRE and BRE+CE could avoid latent
drawbacks of attention mechanism and have a stable performance on datasets
with different noise patterns, thus they are competitive methods compared with
prior baselines. To examine whether they work on the real-world Bag-level DSRE
datasets, we compare our method to 3 previous baselines on NYT-FB60K Han et
al. (2018a) and GIDS-FB8K Jat et al. (2018). We select JointE Han et al.
(2018a), RELE Hu et al. (2019) and SeG Li et al. (2020) as baselines, because
they achieve state-of-the-art performance on bag-level RE. To collect AUC
results, we carefully re-run published codes of them using suggested
hyperparameters from the original papers. We also draw precision-recall curves
following prior works. As shown in Table 4 and Figure 6, our method BRE+CE
largely outperforms existing methods on NYT-FB60K and has comparable
performance on GIDS-FB8K. Such result demonstrates that we avoid attention
mechanism’s latent drawback of hurting model’s robustness. Furthermore, the
model’s improvement on NYT-FB60K is promising (around 13% AUC). This is due to
two reasons: (1) NYT-FB60K is a noisy dataset containing prevalent disturbing
bags, which is similar to our synthesized datasets. (2)NYT-FB60K is highly
imbalanced and most relation types only have limited training data, while all
relation types in our balanced datasets have the same number of training
examples; thus BRE+CE and BRE achieve much higher improvement on NYT-FB60K
compared with synthesized datasets. In conclusion, the high performance not
only validates our claim that attention module may not perform well on noisy
and insufficient training data, but also verifies that our thorough analysis
on attention and KG have practical significance.

###  6.8 Effect of KG

Model |  BagRel |  NYT |  GIDS
---|---|---|---
BRE+ATT |  .878 |  .457 |  .917
BRE+KA |  .932 |  .480 |  .917
BRE |  .910 |  .625 |  .910
BRE+CE |  .915 |  .630 |  .917
Table 5: AUC test results of models on BagRel-Wiki73K, NYT-FB60K and GIDS-
FB8K. In the BagRel column, all models are trained on
$\text{train}_{\frac{1}{2},0}$.

From results in Table 5, we provide a straight comparison between models with
KG (BRE+KA, BRE+CE) and models without KG (BRE+ATT, BRE). Apparently, both
methods of utilizing KG (combined with attention and concatenated as
additional features) outperforms methods not using KG. This demonstrates the
prior knowledge from KG is beneficial for relation extraction task. Except our
naive BRE+CE, we expect that a carefully designed mechanism incorporating KG
can lead to higher improvement. We leave it in the future.

##  7 Conclusion

In conclusion, we construct a set of datasets and propose a framework to
quantitatively evaluate how attention module and KG work in the bag-level RE.
Based on the findings, we demonstrate the effectiveness of a straightforward
solution on this task. Experiment results well support our claims that the
accuracy of attention mechanism depends on the noise pattern of the training
set. In addition, although effectively selecting valid sentences, attention
mechanism could harm model’s robustness to noisy sentences and aggravate the
data sparsity issue. As for KG’s effects on attention, we observe that it
promotes model’s performance by enhancing its robustness with external entity
information, instead of improving attention accuracy.

In the future, we are interested in developing a more general evaluation
framework for other tasks, such as question answering, and improving the
attention mechanism to be robust to noise and insufficient data, and an
effective approach to incorporate the KG knowledge to guide the model
training.

## Acknowledgement

This research/project is supported by NExT Research Centre. This research was
also conducted in collaboration with SenseTime. This work is partially
supported by A*STAR through the Industry Alignment Fund - Industry
Collaboration Projects Grant, by NTU (NTU–ACE2020-01) and Ministry of
Education (RG96/20), and by the National Research Foundation, Prime Minister’s
Office, Singapore under its Energy Programme (EP Award No. NRF2017EWT-
EP003-023) administrated by the Energy Market Authority of Singapore.

## References

  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Neural Information Processing Systems (NIPS)_ , pages 1–9.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du et al. (2018) Jinhua Du, Jingguang Han, Andy Way, and Dadong Wan. 2018.  Multi-level structured self-attentions for distantly supervised relation extraction.  _arXiv preprint arXiv:1809.00699_.
  * Han et al. (2018a) Xu Han, Zhiyuan Liu, and Maosong Sun. 2018a.  Neural knowledge acquisition via mutual attention between knowledge graph and text.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Han et al. (2018b) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018b.  Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.  _arXiv preprint arXiv:1810.10147_.
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011\.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 541–550.
  * Hu et al. (2019) Linmei Hu, Luhao Zhang, Chuan Shi, Liqiang Nie, Weili Guan, and Cheng Yang. 2019\.  Improving distantly-supervised relation extraction with joint label embedding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3812–3820.
  * Jat et al. (2018) Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar. 2018.  Improving distantly supervised relation extraction using word and entity based attention.  _arXiv preprint arXiv:1804.06987_.
  * Ji et al. (2017) Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017.  Distant supervision for relation extraction with sentence-level attention and entity descriptions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Li et al. (2020) Yang Li, Guodong Long, Tao Shen, Tianyi Zhou, Lina Yao, Huan Huo, and Jing Jiang. 2020.  Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8269–8276.
  * Lin et al. (2016) Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.  Neural relation extraction with selective attention over instances.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2124–2133.
  * Liu et al. (2017) Tianyu Liu, Kexiang Wang, Baobao Chang, and Zhifang Sui. 2017.  A soft-label method for noise-tolerant distantly supervised relation extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1790–1795.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011.
  * Peng et al. (2020) Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020.  Learning from context or names? an empirical study on neural relation extraction.  _arXiv preprint arXiv:2010.01923_.
  * Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.  Modeling relations and their mentions without labeled text.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 148–163. Springer.
  * Shahbazi et al. (2020) Hamed Shahbazi, Xiaoli Z Fern, Reza Ghaeini, and Prasad Tadepalli. 2020.  Relation extraction with explanation.  _arXiv preprint arXiv:2005.14271_.
  * Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  _arXiv preprint arXiv:1906.03158_.
  * Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019.  Rotate: Knowledge graph embedding by relational rotation in complex space.  _arXiv preprint arXiv:1902.10197_.
  * Ye and Ling (2019) Zhi-Xiu Ye and Zhen-Hua Ling. 2019.  Distant supervision relation extraction with intra-bag and inter-bag attentions.  _arXiv preprint arXiv:1904.00143_.
  * Zeng et al. (2015) Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015.  Distant supervision for relation extraction via piecewise convolutional neural networks.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 1753–1762.
  * Zhang et al. (2019) Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, and Huajun Chen. 2019.  Long-tail relation extraction via knowledge graph embeddings and graph convolution networks.  _arXiv preprint arXiv:1903.01306_.


# Privacy-Preserving Collaborative Genomic Research: A Real-Life Deployment
and Vision

Zahra Rahmani<EMAIL_ADDRESS>,  Nahal Shahini<EMAIL_ADDRESS>Case Western
Reserve UniversityClevelandOhioUSA ,  Nadav Gat<EMAIL_ADDRESS>,
Zebin Yun<EMAIL_ADDRESS>Tel Aviv UniversityTel AvivIsrael ,  Yuzhou
Jiang<EMAIL_ADDRESS>Case Western Reserve UniversityClevelandOhioUSA ,  Ofir
Farchy<EMAIL_ADDRESS>Lynx.MDPalo AltoCaliforniaUSA ,  Yaniv Harel
<EMAIL_ADDRESS>Tel Aviv UniversityTel AvivIsrael ,  Vipin Chaudhary
<EMAIL_ADDRESS>Case Western Reserve UniversityClevelandOhioUSA ,  Erman Ayday
<EMAIL_ADDRESS>Case Western Reserve UniversityClevelandOhioUSA and  Mahmood
Sharif<EMAIL_ADDRESS>Tel Aviv UniversityTel AvivIsrael

(2024)

###### Abstract.

The data revolution holds a significant promise for the health sector. Vast
amounts of data collected and measured from individuals will be transformed
into knowledge, AI models, predictive systems, and digital best practices. One
area of health that stands to benefit greatly from this advancement is the
genomic domain. The advancement of AI, machine learning, and data science has
opened new opportunities for genomic research, promising breakthroughs in
personalized medicine. However, the increasing awareness of privacy and cyber
security necessitates robust solutions to protect sensitive data in
collaborative research. This paper presents a practical deployment of a
privacy-preserving framework for genomic research, developed in collaboration
with Lynx.MD, a platform designed for secure health data collaboration. The
framework addresses critical cyber security and privacy challenges, enabling
the privacy-preserving sharing and analysis of genomic data while mitigating
risks associated with data breaches. By integrating advanced privacy-
preserving algorithms, the solution ensures the protection of individual
privacy without compromising data utility. A unique feature of the system is
its ability to balance the trade-offs between data sharing and privacy,
providing stakeholders with tools to quantify privacy risks and make informed
decisions. The implementation of the framework within Lynx.MD involves
encoding genomic data into binary formats and applying noise through
controlled perturbation techniques. This approach preserves essential
statistical properties of the data, facilitating effective research and
analysis. Additionally, the system incorporates real-time data monitoring and
advanced visualization tools, enhancing user experience and decision-making
capabilities. The paper highlights the need for tailored privacy attacks and
defenses specific to genomic data, given its unique characteristics compared
to other data types. By addressing these challenges, the proposed solution
aims to foster global collaboration in genomic research, ultimately
contributing to significant advancements in personalized medicine and public
health.

[Data sharing; Genomic privacy; Usable privacy]

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
October 14th, 2024; Salt Lake City, Utah; ††ccs: Security and privacy Privacy-
preserving protocols††ccs: Usability in security and privacy

##  1\. Introduction

The world has an incremental progress in collecting genomic data over the
years. The new generation of capabilities including AI, machine learning, and
data science, provide a new opportunity to investigate and run research that
may identify new variants, connect symptoms to root causes, and develop new
(personalized) medicines. Every healthcare institution, a clinic, a health
research center, and a medical organization, holds unique information that can
highly contribute to the global research of specific cases and problems. While
this is clear, at the same time, humanity increases the awareness and the
perceived importance of privacy and cyber security. Making the maximum effort
to protect the patients’ information and personal data of individuals that
participate in clinical trials is of high importance. An important progress in
defining the liability of these procedures’ organizers helps to protect the
private data against unintentionally leakage or malicious attacks. The outcome
of this progress between other reasons creates a situation that most of this
valuable data stays at the organizations that gathered the information rather
than data sharing and collaborative research across institutions.

In this work, we provide our vision for privacy-preserving collaborative
genomic research and showcase our first attempt for practical, a real-life
deployment. In order to make this work practical and connected to real use-
cases we work with an industry collaborator, Lynx.MD, and use the existing
genomic data storage platform of this partner. Lynx.MD is an American Israeli
start-up that has developed a platform for health data collaboration between
healthcare institutions, pharma companies, and research foundations. The
practical use-cases and the real data are essential for the ability to deeply
study the research goals. The proposed framework mainly addresses the cyber
security and privacy issues that can better protect the data and reduce the
damage in case of any data breach. It also allows researchers (users of
Lynx.MD, who store their datasets in the platform) share their datasets among
each other (e.g., for collaborative research) in a privacy-preserving way and
share their AI models as black-box.

We also propose including a component in the system that demonstrates the
trade-off between information sharing and the associated privacy costs.
Currently, there are no interpretable tools available to measure privacy loss,
resulting in decisions about data sharing being made without full awareness.
Individuals responsible for these decisions are not always privacy and
security experts, making it essential to develop methodologies that clearly
present privacy penalties and provide the necessary context for informed
decision-making.

Overall, this work addresses one of the most innovative domains that requires
solving key challenges in order to make a dramatic change on the global
ability to leverage distributed health data. Doing so will contribute to
creating more collective data repositories that may enable the use of AI to
find solutions, make diagnoses, and develop medicines for the most prioritized
world diseases. To achieve our vision, we have already developed and
integrated privacy-preserving algorithms with the Lynx.MD platform, created
interpretable privacy risk tools, and deployed these mechanisms in high-
performance computing environments. These steps ensure robust data protection
and facilitate collaborative research. Moving forward, we will focus on
evaluating and validating these tools with real-world data, fostering
collaboration and training, and continuously enhancing cybersecurity measures.
This roadmap aims to ensure robust data protection, facilitate collaborative
research, and enable the practical use of AI and machine learning in genomic
research while addressing privacy and security concerns. Note that although
this work focuses on privacy, cybersecurity aspects are an integral part of
protecting assets and data confidentiality. AI models are part of the
protected landscape, along with the technologies used to better defend
processes and data flow (Gandal et al., 2023; Harel et al., 2017; Dayan et
al., 2023). Future work will explore this aspect in greater depth.

##  2\. Background - Lynx.MD Platform

Lynx.MD, founded by experts in healthcare, AI, and cyber security, unlocks the
power of comprehensive healthcare data. The platform, powered by AI, large
language models (LLMs), and computer vision, provides unparalleled access to
real-world medical records, driving advancements in medical technology and the
development of advanced diagnostics and therapies.

Lynx.MD surpasses traditional medical data platforms by incorporating a wide
array of data beyond electronic medical records (EMR) and electronic health
records (EHR). Its cloud-based technology centralizes and streamlines both
structured and unstructured real-world patient data, including clinical notes,
imaging, and video. This fosters data exploration and accelerates the
development of data-driven healthcare applications while safeguarding patient
privacy.

Key components of the platform include:

  * •

Real-World Data Access: The platform provides expansive, multi-sourced data
that is continuously updated and tokenized, ensuring researchers and medical
professionals have the most current and relevant information available.

  * •

Structured and Unstructured Data: The platform offers access to a diverse
range of data types, including images, videos, and physicians’ notes. This
comprehensive dataset reveals vital insights that are often missed by
platforms that only handle structured data.

  * •

Real-Time Live Data: The platform ensures that data is always up-to-date,
supporting continuous monitoring and allowing for customized patient data
tracking. This real-time capability is crucial for timely and effective
medical interventions.

  * •

Unmatched Data Volume and Diversity: The extensive repository of data supports
thorough statistical analysis and generates precise insights. The diversity of
data types and sources enhances the robustness of research outcomes and the
development of innovative healthcare solutions.

##  3\. Related Works

We discuss related work from four aspects: privacy risks for genomic data,
privacy-preserving solutions for genomic research, machine learning (ML) and
AI privacy, and law and policy.

###  3.1. Privacy Risks for Genomic Data

Genomic privacy has recently been explored by many researchers (Erlich and
Narayanan, 2014; Naveed et al., 2015; Ayday et al., 2015; Telenti et al.,
2014; Ayday and Humbert, 2017). Several works have shown that anonymization
does not effectively protect the privacy of genomic data (Gitschier, 2009;
Gymrek et al., 2013; Hayden, 2013; Malin and Sweeney, 2004; Sweeney et al.,
2013; Lin et al., 2004; Kale et al., 2017). Previous research have
demonstrated that the identity of a participant of a genomic study can be
revealed by using a second sample, that is, part of the DNA information from
the individual and the results of the clinical study (Homer et al., 2008; Wang
et al., 2009; Im et al., 2012; Clayton, 2010; Zhou et al., 2011; Shringarpure
and Bustamante, 2015; Raisaro et al., 2016; von Thenen et al., 2018).
Furthermore, several studies have examined phenotype prediction from genomic
data, as a means of tracing identity (Humbert et al., 2015; Lippert et al.,
2017; Kayser and de Knijff, 2011; Zubakov et al., 2010; Ou et al., 2012; Allen
et al., 2010; Manning et al., 2012; Walsh et al., 2011; Claes et al., 2014;
Liu et al., 2012). This line of research highlights the vulnerabilities in
genomic data sharing, particularly when datasets are linked with other
publicly available data sources. The predictive power of genomic data can be
exploited to infer sensitive information about individuals, such as their
susceptibility to certain diseases or traits like eye and hair color, which
can then be used for discriminatory purposes.

Recent studies have also highlighted the potential for privacy breaches
through familial search techniques, where an adversary can identify
individuals by matching their genetic markers with those of their relatives
(Gymrek et al., 2013). This method leverages the genetic similarity among
family members to breach privacy, making it a significant concern for genomic
data sharing practices. The growing availability of public genetic databases
intensifies this risk, as these repositories provide a rich source of genetic
information that can be leveraged to cross-reference and identify individuals
from anonymized datasets.

###  3.2. Privacy-Preserving Solutions for Genomic Research

Most proposed solutions to utilize or share genomic data are either based on
obfuscation or the use of cyptographic techniques. Some researchers have
proposed using differential privacy (DP) concept to mitigate membership
inference attacks when releasing summary statistics, such as minor allele
frequencies and chi-square values (Fienberg et al., 2011; Yu et al., 2014;
Johnson and Shmatikov, 2013). DP provides a mathematical framework that
ensures the addition or removal of a single database item does not
significantly affect the outcome of any analysis, thereby preserving the
privacy of individuals in the dataset.

A significant subset of cryptographic solutions focuses on private pattern-
matching and comparison of genomic (Troncoso-Pastoriza et al., 2007; De
Cristofaro et al., 2013; Jha et al., 2008; Blanton et al., 2012; Naveed et
al., 2014). For privacy-preserving clinical genomics, Baldi _et al._ proposed
private set intersection-based techniques (Baldi et al., 2011). These
techniques allow researchers to identify common genetic variants across
datasets without revealing individual data points. Similarly, partially
homomorphic encryption has been proposed for the privacy-preserving use of
genomic data in clinical settings, enabling computations to be performed on
encrypted data without revealing the underlying information (Ayday et al.,
2013b, c, d, a). Kantarcioglu _et al._ proposed homomorphic encryption-based
techniques for privacy-preserving genomic research, which allow secure
computations on genetic data without revealing the data itself (Kantarcioglu
et al., 2008). Wang _et al._ proposed private edit distance protocols to find
similar patients across multiple hospitals, enhancing collaborative research
without compromising patient privacy (Wang et al., 2015). These cryptographic
methods offer robust privacy guarantees and are particularly suited for
applications where sensitive genomic data must be processed securely.

Despite these advancements, existing cryptographic solutions often face
challenges related to interoperability and practicality. Different genomic
data analysis problems require different cryptosystems, which can lead to
significant security and efficiency issues when integrating multiple systems.
For instance, the use of various encryption schemes for different purposes can
create complexities in maintaining security and ensuring seamless data
processing. Additionally, some cryptographic techniques can be computationally
intensive, posing challenges for their practical implementation in large-scale
genomic studies.

Future research must focus on developing unified frameworks that provide
comprehensive privacy-preserving solutions while maintaining high data utility
and computational efficiency. This includes designing cryptographic systems
that are versatile and efficient enough to handle various genomic data
analysis tasks. Moreover, it is essential to address the scalability of these
solutions to ensure they can manage the increasing volume of genomic data
generated by modern sequencing technologies. By advancing these areas, we can
ensure that genomic research can progress without compromising the privacy and
security of individual data.

###  3.3. ML/AI Privacy

A wide range of privacy attacks have been proposed against ML models in recent
years, demonstrating various ways in which ML may leak sensitive data during
or after training (Carlini et al., 2019; Fredrikson et al., 2015; Shokri et
al., 2017; Zhu et al., 2019). For example, membership inference attacks enable
adversaries with access to a trained model to determine whether a record of
data was used in training (Shokri et al., 2017). As another example, model
inversion allows adversaries to reconstruct training samples from trained
models (Fredrikson et al., 2015). Other attacks include, but are not limited
to unintended memorization (Carlini et al., 2019) and input and label
reconstruction in federated learning (Zhu et al., 2019).

ML privacy attacks can serve as a means to empirically quantify leakage of
private data under various settings. Still, to our knowledge, these attacks
have primarily been explored and used by ML privacy academics. In our vision
(Sec. 4), the attack outcomes would be made available to the stakeholders on
the Lynx.MD platform to aid them in decision making when they need to decide
whether to release a model trained on the platform. We also note that ML
privacy attacks have been mainly studied in the vision and text domains, with
little effort exploring their effectiveness on genomic and health data.
Accordingly, it remains unclear whether established attacks can reliably
assess privacy leakage in ML in these domains, suggesting that new attacks
tailored for genomic and health data may need to be developed.

Researchers have also proposed various methods to enhance ML privacy (Abadi et
al., 2016; Bonawitz et al., 2017; Nasr et al., 2018; Papernot et al., 2018).
While some of these methods do not provably guarantee privacy but have been
demonstrated empirically effective (Nasr et al., 2018), other methods also
carry provable guarantees (Abadi et al., 2016; Papernot et al., 2018).
Notably, the differentially private stochastic gradient descent algorithm
(Abadi et al., 2016) enables training ML models via stochastic gradient
descent while satisfying differential privacy guarantees, thus, in a sense,
providing plausible deniability about whether certain records were used in
training. In our vision, these defenses would be provided as a service to
users of the Lynx.MD platform, thus helping protect data privacy when training
models (Sec. 4).

###  3.4. Law and Policy

Issues about the privacy and healthcare data are mostly covered by Health
Insurance Portability and Accountability Act (HIPAA) and General Data
Protection Regulation (GDPR) requirements (CFR, 2020b). The HIPAA Privacy Rule
governs the disclosure of electronic and hard-copy medical information and
allows patients to control their medical records to a degree. The Privacy Rule
establishes that, with some exceptions, covered entities must obtain patients’
permission before disclosing their medical data to others (CFR, 2020a).
However, the HIPAA Privacy Rule only covers protected health information
(PHI), which is “individually identifiable health information” that is
electronically or otherwise transmitted or maintained (U.S. Department of
Health & Human Services, 2012). Thus, the rule does not cover de-identified
information.

HIPAA and the department of Health and Human Services provide detailed
guidance as to how to de-identify records for purposes of the regulatory
exemption. They can be de-identified by experts using reliable techniques such
as suppression, generalization, and perturbation (CFR, 2017). In the
alternative, the regulations list 18 identifiers that should be removed to
achieve anonymization (National Committee on Vital and Health Statistics,
2007). Nevertheless, experts have determined that even with redaction of all
of these identifiers, there is a small chance that skilled attackers could re-
identify records. They could do so by matching de-identified data to publicly
available information, such as voter registration records or news stories
about individuals with illnesses or injuries (Hoffman and Podgurski, 2012;
Cooke Bailey et al., 2018). Furthermore, multi-modal data that includes a wide
variety of correlations, can more easily be re-identified than other
anonymized data. It is also noteworthy that the HIPAA Privacy Rule contains
many other exceptions that allow disclosure of PHI without patient
authorization. Examples are disclosures for treatment, payment, and health
care operations, as well as for law enforcement or public health purposes.

The General Data Protection Regulation (GDPR), on the other hand, provides
more extensive protections and rights to data subjects. The GDPR protects
individuals with regard to the “processing” of “personal data”, including
personal data related to health. The GDPR applies to organizations outside of
the EU, so long as they process data pertaining to EU citizens and the
processing activities relate to the offering of goods or services or the
monitoring of behavior that takes place within the EU. Thus, U.S. entities may
have to comply not only with HIPAA, but also with the GDPR. The GDPR protects
personally identifiable data and pseudonymized data. The law does not cover
“anonymous” data. Pseudonymised data are data that cannot be attributed to a
specific data subject without additional information that is kept separate and
is subject to technical and organizational safeguards to prevent re-
identification of the data subject “Anonymous” data are data that are assumed
to be unlinkable to the data subject. Experts, however, may be skeptical that
any data are truly anonymous in the healthcare sector. In addition, like
HIPAA, the GDPR carves out several exceptions that allow data use absent
patient consent in the contexts of judicial proceedings, legal requirements,
public health measures, and medical treatment.

Thus, despite the existence of the HIPAA and GDPR, patients may continue to be
worried about the privacy of their information when it is curated for research
and AI/ML purposes. They may not understand the concept of de-identification
or they may be aware of the risk of re-identification. Moreover, they may
simply wish to maintain control over their health data in the belief that it
is their property and for the sake of autonomy.

##  4\. Our Vision

To address the critical challenges associated with genomic data privacy, we
developed a comprehensive privacy-preserving solution for integration with the
Lynx.MD platform. This solution leverages advanced privacy-preserving
solutions to ensure robust privacy protections while maintaining the utility
of shared genomic data.

In our broader vision, Lynx.MD serves as a sandbox platform where researchers
can securely upload and share their genomic datasets. Our privacy preserving
algorithms are seamlessly integrated into Lynx.MD, providing users with tools
to analyze and share data without exposing sensitive information.
Additionally, a user-friendly interface and sophisticated visualization tools
will be developed to allow users to easily manage their datasets and
understand the privacy and utility levels of shared data. These tools are
crucial for making informed decisions about data sharing and analysis.

###  4.1. Data Sharing Between Lynx.MD Users

We chose to use a state-of-the-art privacy-preserving genomic dataset sharing
algorithm (Jiang et al., 2023). In this approach, genomic data is encoded into
a binary matrix. Using a previously developed XOR mechanism, controlled noise
is added to the data, effectively masking the original genomic sequences while
retaining essential statistical properties. Following the noise addition, a
novel post-processing technique is employed. This technique uses publicly
known statistical data about the genomic datasets to enhance data quality
without compromising the privacy ensured in the first stage. We implemented
and deployed this privacy-preserving dataset sharing mechanism in a high-
performance computing (HPC) environment to ensure it meets the scalability and
memory requirements necessary for handling large genomic datasets.

In highlevel, researchers upload their genomic datasets to the Lynx.MD
platform, which acts as a controlled environment for data sharing. We have
conducted extensive testing to ensure that our privacy-preserving mechanisms
are correctly applied to each dataset uploaded, safeguarding against potential
privacy breaches. Once datasets are uploaded, the Lynx.MD platform implements
the two-stage privacy-preserving scheme detailed earlier. This implementation
is crucial for maintaining the confidentiality of the data while enabling its
use for research purposes. The platform supports collaborative studies by
allowing secure access to shared datasets and tools necessary for joint
research efforts.

By integrating this functionality on the Lynx.MD platform, we ensure that
genomic data can be utilized for advancing personalized medicine and other
research initiatives without compromising the privacy of the individuals
represented in the datasets. This method aligns with our high-level goal of
fostering collaboration while protecting sensitive information.

###  4.2. Privacy Assurance in ML Models on the Lynx.MD Platform

A fundamental premise of the Lynx.MD platform is that training machine
learning (ML) models on the data available through the platform enables
technologists and scientists to develop innovative technologies and derive
valuable scientific insights. However, as previously discussed (Sec. 3.3),
releasing these models carries significant privacy risks. Adversaries may
leverage access to the models to infer sensitive information about
individuals’ genomic or health data. Thus, it is imperative to ensure that the
models leak minimal to no information about the training data before their
release from the platform. To this end, our vision includes providing
stakeholders (both data owners and users) with an automated evaluation system
that assesses the extent to which models leak information about their training
data. Additionally, stakeholders will receive recommendations for potential
defenses that can be incorporated during training to enhance the protection of
the training data.

Various privacy attacks have been proposed and thoroughly evaluated against ML
models, as previously mentioned (Sec. 3.3). However, genomic datasets exhibit
significant differences compared to the image or text datasets commonly used
in these evaluations (Shokri et al., 2017; Carlini et al., 2022). First,
genomic datasets typically have relatively small sample sizes, as collecting
genomic data remains more expensive than collecting text or image data, which
can be easily gathered from the internet, albeit not always labeled. Second,
while there are millions of genomic features—such as Single Nucleotide
Polymorphisms (SNPs) (Brookes, 1999)—that can be ingested by models, the
number of useful features in a given dataset is often substantially smaller,
with each feature admitting a limited set of values. In contrast, image and
text datasets contain significantly more dimensions (e.g., 3,072–268,203
dimensions for standard image datasets), each admitting 256 different values
(Shokri et al., 2017). These differences can impact the effectiveness of
attacks. For instance, some attacks require auxiliary datasets to train
surrogate models that are later used to infer membership (Carlini et al.,
2022). These may be ineffective against genomic models due to the scarcity of
auxiliary data. Therefore, it may be necessary to develop novel attacks
specifically tailored for genomic data, and health data in general, to
reliably assess ML models’ leakage.

###  4.3. Communicating the Privacy Risk to Users

Once the risk of private information leakage due to data sharing or ML model
release is assessed, it is necessary to communicate the risk to users to
facilitate their decision making regarding whether to share data or models. To
this end, among others, it is crucial to communicate to users the (1)
potential implications of certain risks (e.g., membership inference); (2)
theoretical guarantees of countermeasures applied (e.g., differential privacy
mechanisms); (3) empirical assessment of data leakage (e.g., for potential
success of membership inference); (4) assumptions behind the attacks for which
the risk is estimated (e.g., whether the adversary has auxiliary data); (5)
potential tradeoffs between utility and privacy that can be attained with
various countermeasures. We intend to devise a privacy dashboard that conveys
such information to stakeholders in an usable manner. Primarily, for
usability, it is essential to optimize accessibility and run time. For the
former, we plan to rely on established literature offering ways to describe
the sophisticated theoretical guarantees of certain defenses (Nanayakkara et
al., 2023) and conduct user studies to find adequate means to convey metrics
estimated by ML privacy attacks that would be accessible by stakeholders with
a wide-range of backgrounds and expertise. For the latter, we intend to
explore means to enable prompt assessment of necessary metrics (e.g., by
proposing more efficient attacks).

##  5\. Proof-of-Concept Implementation

We integrated the privacy-preserving genomic data sharing scheme into the
Lynx.MD platform and rigorously assessed both the privacy and utility of the
data using comprehensive metrics. Our analysis employed a variety of tests,
including Average Point Error, Average Sample Error, and Mean and Variance
Error, which together indicate how well the integrity and statistical
properties of the original dataset are preserved post-transformation. The
privacy level of the data was measured using the differential privacy
parameter, $\epsilon$, which provides strong privacy protection at different
levels.

Figure 1. Lynx.MD Platform

###  5.1. Dataset

The dataset utilized for evaluating our privacy-preserving solution comprises
a comprehensive collection of genomic data typical of what might be used in
advanced medical research. This dataset features a broad spectrum of SNP
variations, representing a diverse genetic background to ensure the
generalizability of our results. Prior to the application of our privacy-
preserving mechanisms, genomic data was encoded into a binary matrix format.
This preparatory step was critical for facilitating the subsequent integration
of our novel two-stage privacy-preserving algorithm, which applies controlled
noise and utilizes publicly known statistics to enhance data quality.

We implemented and evaluated our proposed scheme on real-life genomic datasets
from the OpenSNP project (Ope, [n. d.]), which is a public platform that
allows users to share their genetic data, typically derived from consumer
genetic testing services. We selected three phenotypes for our study: lactose
intolerance, hair color, and eye color. The lactose intolerance dataset
includes 9,091 SNPs from 60 individuals with lactose intolerance. The hair
color dataset contains 9,686 SNPs from 60 individuals with dark hair. The eye
color dataset is larger, with 28,396 SNPs among 401 individuals with brown
eyes. Additionally, a handedness dataset is included with 28,396 SNPs among
401 individuals.

We built a reference dataset for each phenotype, aligning the SNPs with the
target dataset to be shared. These reference datasets were constructed from
the remaining data in the OpenSNP project. This thorough dataset selection and
preparation ensured that our evaluation was robust and reflective of real-
world scenarios. Note that we included these datasets on the Lynx.MD platform
as users of the platform and ran the algorithms on the platform using these
datasets as shown in Figure 1.

###  5.2. Utility and Privacy Evaluation

Our results in (Jiang et al., 2023) demonstrate that the proposed scheme
consistently outperforms existing methods in maintaining data utility,
regardless of the privacy budget. This strong performance highlights how
effective our two-stage privacy-enhancing mechanism is. By adding controlled
noise and using a unique post-processing technique, we achieve an excellent
balance between data privacy and usability. This is essential for moving
forward with collaborative genomic research and personalized medicine.

On the other hand, the differential privacy framework safeguards against
inference attacks, such as membership inference attacks, by injecting noise
and protecting SNP value distributions, thereby offering robust privacy
protection and high data utility.

###  5.3. System Performance

The proposed privacy-preserving dataset sharing scheme introduces minimal
overhead and demonstrates lower computational complexity compared to existing
methods. Utilizing an efficient perturbation technique based on the XOR
mechanism, the scheme significantly reduces time complexity by calibrating
noise through column-wise correlation of SNPs, thus expediting the
perturbation process without compromising privacy guarantees. This enhancement
allows the scheme to handle large genomic datasets (Proserpio et al., 2014)
practically within a reasonable timeframe.

The scalability of the proposed dataset-sharing solution, when integrated with
Lynx.MD, is highly promising. Designed to unlock the power of comprehensive
healthcare data, Lynx.MD’s robust infrastructure enables efficient management
and distribution of large-scale genomic datasets. The integration leverages
advanced data processing and security capabilities, ensuring that the solution
can handle increasing volumes of data without performance degradation. The
efficient perturbation and utility restoration mechanisms of the proposed
scheme maintain low computational complexity, enhancing scalability as the
dataset size grows. Consequently, the combined strengths of the platform and
the scalable dataset-sharing solution facilitate effective and secure sharing
of vast genomic data, driving significant advancements in healthcare research.

##  6\. Conclusion

In this paper, we have demonstrated the feasibility and practicality of a
privacy-preserving framework for collaborative genomic research, addressing
the critical need for secure data sharing in the era of personalized medicine.
By leveraging advanced privacy-preserving algorithms, our solution ensures
robust protection of sensitive genomic data while maintaining its utility for
research purposes. The collaboration with Lynx.MD has been instrumental in
validating our approach, showcasing how industry partnerships can enhance the
deployment of privacy-preserving data platforms. Our method effectively
balances the trade-offs between data sharing and privacy, providing
stakeholders with transparent tools to assess privacy risks and make informed
decisions. Our experimental results confirm that the proposed framework
outperforms existing methods in both privacy protection and data utility,
highlighting its potential for broader application in genomic research and
other fields requiring sensitive data handling. The integration of real-time
monitoring and visualization tools further enhances the user experience,
promoting more effective and secure collaboration. Future work will focus on
refining the privacy-preserving techniques and exploring additional
applications in other domains. By continuing to address the unique challenges
posed by genomic data, we aim to foster global collaboration and drive
significant advancements in personalized medicine and public health.

## References

  * (1)
  * Ope ([n. d.]) [n. d.].  opensnp.  https://opensnp.org/.
  * CFR (2017) 2017\.  45 C.F.R. § 164.514(b)(2)(i) (2017).  In addition, information will not be considered de-identified if an entity has “actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information.” Id. at §164.514(b)(2)(ii)..
  * CFR (2020a) 2020a.  45 C.F.R. § 160.103 (2020).  Code of Federal Regulations, Title 45, Section 160.103.
  * CFR (2020b) 2020b.  45 C.F.R. §§ 164.508 - .510 (2020).  Code of Federal Regulations, Title 45, Parts 164.508 to 164.510.
  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep learning with differential privacy. In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_. 308–318.
  * Allen et al. (2010) Hana Lango Allen, Karol Estrada, Guillaume Lettre, Sonja I Berndt, Michael N Weedon, Fernando Rivadeneira, Cristen J Willer, Anne U Jackson, Sailaja Vedantam, Soumya Raychaudhuri, et al. 2010\.  Hundreds of variants clustered in genomic loci and biological pathways affect human height.  _Nature_ 467, 7317 (2010), 832–838.
  * Ayday et al. (2015) Erman Ayday, Emiliano De Cristofaro, J Hubaux, and Gene Tsudik. 2015.  The chills and thrills of whole genome sequencing.  _IEEE Computer Magazine_ (2015).
  * Ayday and Humbert (2017) E. Ayday and M. Humbert. 2017.  Inference Attacks against Kin Genomic Privacy.  _IEEE Security Privacy_ 15, 5 (2017), 29–37.  https://doi.org/10.1109/MSP.2017.3681052
  * Ayday et al. (2013a) E. Ayday, J. L. Raisaro, and J. P. Hubaux. 2013a.  Personal Use of the Genomic Data: Privacy vs. Storage Cost. In _Proceedings of IEEE Global Communications Conference, Exhibition and Industry Forum (Globecom)_.
  * Ayday et al. (2013b) E. Ayday, J. L. Raisaro, and J. P. Hubaux. 2013b.  Privacy-Enhancing Technologies for Medical Tests Using Genomic Data.  _(short paper) in Proceedings of 20th Annual Network and Distributed System Security Symposium (NDSS)_ (2013).
  * Ayday et al. (2013c) Erman Ayday, Jean Louis Raisaro, Jean-Pierre Hubaux, and Jacques Rougemont. 2013c.  Protecting and evaluating genomic privacy in medical tests and personalized medicine. In _Proceedings of the 12th ACM Workshop on Privacy in the Electronic Society_. ACM, 95–106.
  * Ayday et al. (2013d) E. Ayday, J. L. Raisaro, P. J. McLaren, J. Fellay, and J. P. Hubaux. 2013d.  Privacy-Preserving Computation of Disease Risk by Using Genomic, Clinical, and Environmental Data. In _Proceedings of USENIX Security Workshop on Health Information Technologies (HealthTech)_.
  * Baldi et al. (2011) Pierre Baldi, Roberta Baronio, Emiliano De Cristofaro, Paolo Gasti, and Gene Tsudik. 2011.  Countering GATTACA: efficient and secure testing of fully-sequenced human genomes. In _Proceedings of the 18th ACM Conference on Computer and Communications Security_. 691–702.
  * Blanton et al. (2012) Marina Blanton, Mikhail J Atallah, Keith B Frikken, and Qutaibah Malluhi. 2012.  Secure and efficient outsourcing of sequence comparisons. In _Proceedings of European Symposium on Research in Computer Security_. 505–522.
  * Bonawitz et al. (2017) Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017.  Practical secure aggregation for privacy-preserving machine learning. In _proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_. 1175–1191.
  * Brookes (1999) Anthony J. Brookes. 1999.  The essence of SNPs.  _Gene_ 234, 2 (1999), 177–186.
  * Carlini et al. (2022) Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022.  Membership Inference Attacks From First Principles. In _IEEE Symposium on Security and Privacy (SP)_. 1897–1914.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019.  The secret sharer: Evaluating and testing unintended memorization in neural networks. In _28th USENIX security symposium (USENIX security 19)_. 267–284.
  * Claes et al. (2014) Peter Claes, Denise K Liberton, Katleen Daniels, Kerri Matthes Rosana, Ellen E Quillen, Laurel N Pearson, Brian McEvoy, Marc Bauchet, Arslan A Zaidi, Wei Yao, et al. 2014\.  Modeling 3D facial shape from DNA.  _PLoS Genetics_ 10, 3 (2014).
  * Clayton (2010) David Clayton. 2010.  On inferring presence of an individual in a mixture: a Bayesian approach.  _Biostatistics_ 11, 4 (2010), 661–673.
  * Cooke Bailey et al. (2018) J. N. Cooke Bailey, D. C. Crawford, A. Goldenberg, A. Slaven, J. Pencak, M. Schachere, and J. F. O’Toole. 2018.  Willingness to participate in a national precision medicine cohort: Attitudes of chronic kidney disease patients at a Cleveland public hospital.  _Journal of Personalized Medicine_ 8, 3 (Sept. 2018), 21.
  * Dayan et al. (2023) Ortal Dayan, Lior Wolf, Fang Wang, and Yaniv Harel. 2023.  Optimizing AI for Mobile Malware Detection by Self-Built-Dataset GAN Oversampling and LGBM. In _2023 IEEE International Conference on Cyber Security and Resilience (CSR)_. 60–65.  https://doi.org/10.1109/CSR57506.2023.10224927
  * De Cristofaro et al. (2013) Emiliano De Cristofaro, Sky Faber, and Gene Tsudik. 2013.  Secure Genomic Testing with Size- and Position-hiding Private Substring Matching. In _Proceedings of the 12th ACM Workshop on Privacy in the Electronic Society_.
  * Erlich and Narayanan (2014) Yaniv Erlich and Arvind Narayanan. 2014.  Routes for breaching and protecting genetic privacy.  _Nature Reviews Genetics_ 15, 6 (2014), 409–421.
  * Fienberg et al. (2011) Stephen E Fienberg, Aleksandra Slavkovic, and Caroline Uhler. 2011.  Privacy preserving GWAS data sharing. In _IEEE 11th International Conference on Data Mining Workshops (ICDMW)_. 628–635.
  * Fredrikson et al. (2015) Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015.  Model inversion attacks that exploit confidence information and basic countermeasures. In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_. 1322–1333.
  * Gandal et al. (2023) Neil Gandal, Tyler Moore, Michael Riordan, and Noa Barnir. 2023.  Empirically evaluating the effect of security precautions on cyber incidents.  _Computers & Security_ 133 (2023), 103380.  https://doi.org/10.1016/j.cose.2023.103380
  * Gitschier (2009) Jane Gitschier. 2009.  Inferential genotyping of Y chromosomes in Latter-Day Saints founders and comparison to Utah samples in the HapMap project.  _American Journal of Human Genetics_ 84, 2 (2009), 251–258.
  * Gymrek et al. (2013) Melissa Gymrek, Amy L McGuire, David Golan, Eran Halperin, and Yaniv Erlich. 2013.  Identifying personal genomes by surname inference.  _Science_ 339, 6117 (2013), 321–324.
  * Harel et al. (2017) Yaniv Harel, Irad Ben Gal, and Yuval Elovici. 2017.  Cyber Security and the Role of Intelligent Systems in Addressing its Challenges.  _ACM Trans. Intell. Syst. Technol._ 8, 4, Article 49 (may 2017), 12 pages.  https://doi.org/10.1145/3057729
  * Hayden (2013) Erika Check Hayden. 2013.  Privacy protections: The genome hacker.  _Nature_ 497 (2013), 172–174.
  * Hoffman and Podgurski (2012) Sharona Hoffman and Andy Podgurski. 2012.  Balancing Privacy, Autonomy, and Scientific Needs in Electronic Health Records Research.  _Southern Methodist University Law Review_ 65 (April 2012), 85.  Case Legal Studies Research Paper No. 2011-22.
  * Homer et al. (2008) Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. 2008.  Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays.  _PLoS Genetics_ 4, 8 (2008).
  * Humbert et al. (2015) Mathias Humbert, Kévin Huguenin, Joachim Hugonot, Erman Ayday, and Jean-Pierre Hubaux. 2015.  De-anonymizing Genomic Databases Using Phenotypic Traits.  _Proceedings on Privacy Enhancing Technologies_ , 99–114.
  * Im et al. (2012) Hae Kyung Im, Eric R Gamazon, Dan L Nicolae, and Nancy J Cox. 2012.  On sharing quantitative trait GWAS results in an era of multiple-omics data and the limits of genomic privacy.  _American Journal of Human Genetics_ 90, 4 (2012), 591–598.
  * Jha et al. (2008) Somesh Jha, Louis Kruger, and Vitaly Shmatikov. 2008.  Towards practical privacy for genomic computation. In _Proceedings of IEEE Symposium on Security and Privacy_. 216–230.
  * Jiang et al. (2023) Yuzhou Jiang, Tianxi Ji, Pan Li, and Erman Ayday. 2023.  Reproducibility-Oriented and Privacy-Preserving Genomic Dataset Sharing.  arXiv:2209.06327 [cs.CR]
  * Johnson and Shmatikov (2013) Aaron Johnson and Vitaly Shmatikov. 2013.  Privacy-preserving data exploration in genome-wide association studies. In _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 1079–1087.
  * Kale et al. (2017) Gulce Kale, Erman Ayday, and Öznur Tastan. 2017.  A utility maximizing and privacy preserving approach for protecting kinship in genomic databases.  _Bioinformatics_ 34, 2 (2017).
  * Kantarcioglu et al. (2008) M. Kantarcioglu, Wei Jiang, Ying Liu, and B. Malin. 2008.  A Cryptographic Approach to Securely Share and Query Genomic Sequences.  _IEEE Transactions on Information Technology in Biomedicine_ 12, 5 (2008), 606–617.
  * Kayser and de Knijff (2011) Manfred Kayser and Peter de Knijff. 2011.  Improving human forensics through advances in genetics, genomics and molecular biology.  _Nature Reviews Genetics_ 12, 3 (2011), 179–192.
  * Lin et al. (2004) Z. Lin, A. B. Owen, and R. B. Altman. 2004.  Genomic research and human subject privacy.  _Science_ 305, 5681 (Jul 2004), 183.
  * Lippert et al. (2017) Christoph Lippert, Riccardo Sabatini, M. Cyrus Maher, Eun Yong Kang, Seunghak Lee, Okan Arikan, Alena Harley, Axel Bernal, Peter Garst, Victor Lavrenko, Ken Yocum, Theodore Wong, Mingfu Zhu, Wen-Yun Yang, Chris Chang, Tim Lu, Charlie W. H. Lee, Barry Hicks, Smriti Ramakrishnan, Haibao Tang, Chao Xie, Jason Piper, Suzanne Brewerton, Yaron Turpaz, Amalio Telenti, Rhonda K. Roby, Franz J. Och, and J. Craig Venter. 2017.  Identification of individuals by trait prediction using whole-genome sequencing data.  _Proceedings of the National Academy of Sciences_ (2017).  https://doi.org/10.1073/pnas.1711125114
  * Liu et al. (2012) Fan Liu, Fedde van der Lijn, Claudia Schurmann, Gu Zhu, M Mallar Chakravarty, Pirro G Hysi, Andreas Wollstein, Oscar Lao, Marleen de Bruijne, M Arfan Ikram, et al. 2012\.  A genome-wide association study identifies five loci influencing facial morphology in Europeans.  _PLoS Genetics_ 8, 9 (2012).
  * Malin and Sweeney (2004) Bradley A. Malin and Latanya Sweeney. 2004.  How (not) to protect genomic data privacy in a distributed network: using trail re-identification to evaluate and design anonymity protection systems.  _Journal of Biomedical Informatics_ 37, 3 (2004), 179–192.
  * Manning et al. (2012) Alisa K Manning, Marie-France Hivert, Robert A Scott, Jonna L Grimsby, Nabila Bouatia-Naji, Han Chen, Denis Rybin, Ching-Ti Liu, Lawrence F Bielak, Inga Prokopenko, et al. 2012\.  A genome-wide approach accounting for body mass index identifies genetic variants influencing fasting glycemic traits and insulin resistance.  _Nature Genetics_ 44, 6 (2012), 659–669.
  * Nanayakkara et al. (2023) Priyanka Nanayakkara, Mary Anne Smart, Rachel Cummings, Gabriel Kaptchuk, and Elissa M Redmiles. 2023.  What are the chances? explaining the epsilon parameter in differential privacy. In _32nd USENIX Security Symposium (USENIX Security 23)_. 1613–1630.
  * Nasr et al. (2018) Milad Nasr, Reza Shokri, and Amir Houmansadr. 2018.  Machine learning with membership privacy using adversarial regularization. In _Proceedings of the 2018 ACM SIGSAC conference on computer and communications security_. 634–646.
  * National Committee on Vital and Health Statistics (2007) National Committee on Vital and Health Statistics. 2007.  Report to the Secretary of Health and Human Services on Enhanced Protections for Uses of Health Data: A Stewardship Framework for “Secondary Uses” of Electronically Collected and Transmitted Health Data.  36 n.16, available at http://www.ncvhs.hhs.gov/071221lt.pdf.
  * Naveed et al. (2014) Muhammad Naveed, Shashank Agrawal, Manoj Prabhakaran, XiaoFeng Wang, Erman Ayday, Jean-Pierre Hubaux, and Carl Gunter. 2014.  Controlled Functional Encryption. In _Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security_.
  * Naveed et al. (2015) Muhammad Naveed, Erman Ayday, Ellen W Clayton, Jacques Fellay, Carl A Gunter, Jean-Pierre Hubaux, Bradley A Malin, and XiaoFeng Wang. 2015.  Privacy in the genomic era.  _ACM Computing Surveys (CSUR)_ 48, 1 (2015), 6.
  * Ou et al. (2012) Xue-ling Ou, Jun Gao, Huan Wang, Hong-sheng Wang, Hui-ling Lu, and Hong-yu Sun. 2012.  Predicting human age with bloodstains by sjTREC quantification.  _PloS ONE_ 7, 8 (2012).
  * Papernot et al. (2018) Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. 2018.  Scalable private learning with pate.  _arXiv preprint arXiv:1802.08908_ (2018).
  * Proserpio et al. (2014) Davide Proserpio, Sharon Goldberg, and Frank McSherry. 2014.  Calibrating data to sensitivity in private data analysis: a platform for differentially-private analysis of weighted datasets.  _Proceedings of the VLDB Endowment_ 7, 8 (2014), 637–648.
  * Raisaro et al. (2016) Jean L Raisaro, Florian Tramer, Ji Zhanglong, Diyue Bu, Yongan Zhao, Knox Carey, David Lloyd, Heidi Sofia, Dixie Baker, Paul Flicek, Suyash S Shringarpure, Carlos D Bustamante, Suang Wang, Xiaoqian Jiang, Lucila Ohno-Machado, Haixu Tang, XiaoFeng Wang, and Jean-Pierre Hubaux. 2016.  Addressing Beacon Re-Identification Attacks: Quantification and Mitigation of Privacy Risks.  _The Journal of the American Medical Informatics Association_ 24, 4 (2016), 799–805.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models. In _2017 IEEE symposium on security and privacy (SP)_. IEEE, 3–18.
  * Shringarpure and Bustamante (2015) Suyash S Shringarpure and Carlos D Bustamante. 2015.  Privacy risks from genomic data-sharing beacons.  _The American Journal of Human Genetics_ 97, 5 (2015), 631–646.
  * Sweeney et al. (2013) Latanya Sweeney, Akua Abu, and Julia Winn. 2013.  Identifying participants in the personal genome project by name.  _arXiv preprint arXiv:1304.7605_ (2013).
  * Telenti et al. (2014) Amalio Telenti, Erman Ayday, and Jean Pierre Hubaux. 2014.  On genomics, kin, and privacy.  _F1000Research_ (Mar 2014).  https://doi.org/10.12688/f1000research.4089
  * Troncoso-Pastoriza et al. (2007) Juan Ramón Troncoso-Pastoriza, Stefan Katzenbeisser, and Mehmet Celik. 2007.  Privacy preserving error resilient DNA searching through oblivious automata.  _Proceedings of ACM CCS ’07_ (2007).
  * U.S. Department of Health & Human Services (2012) U.S. Department of Health & Human Services. 2012.  Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule.  http://www.hhs.gov/ocr/privacy/hipaa/understanding/coveredentities/De-identification/guidance.html#guidancedetermination Accessed: 2024-07-01.
  * von Thenen et al. (2018) Nora von Thenen, Erman Ayday, and A. Ercument Cicek. 2018.  Re-identification of individuals in genomic data-sharing beacons via allele inference.  _Bioinformatics_ 35, 3 (2018).
  * Walsh et al. (2011) Susan Walsh, Fan Liu, Kaye N Ballantyne, Mannis van Oven, Oscar Lao, and Manfred Kayser. 2011.  IrisPlex: a sensitive DNA tool for accurate prediction of blue and brown eye colour in the absence of ancestry information.  _Forensic Science International: Genetics_ 5, 3 (2011), 170–180.
  * Wang et al. (2009) Rui Wang, Yong Fuga Li, XiaoFeng Wang, Haixu Tang, and Xiaoyong Zhou. 2009.  Learning your identity and disease from research papers: information leaks in genome wide association study. In _Proceedings of the 16th ACM Conference on Computer and Communications Security_. 534–544.
  * Wang et al. (2015) Xiao Shaun Wang, Yan Huang, Yongan Zhao, Haixu Tang, XiaoFeng Wang, and Diyue Bu. 2015.  Efficient genome-wide, privacy-preserving similar patient query based on private edit distance. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_. 492–503.
  * Yu et al. (2014) Fei Yu, Stephen E Fienberg, Aleksandra B Slavković, and Caroline Uhler. 2014.  Scalable privacy-preserving data sharing methodology for genome-wide association studies.  _Journal of Biomedical Informatics_ 50 (2014), 133–141.
  * Zhou et al. (2011) Xiaoyong Zhou, Bo Peng, Yong Fuga Li, Yangyi Chen, Haixu Tang, and XiaoFeng Wang. 2011.  To release or not to release: Evaluating information leaks in aggregate human-genome data, In Computer Security – ESORICS 2011 (Leuven, Belgium).  _Proceedings of ESORICS’11_ , 607–627.
  * Zhu et al. (2019) Ligeng Zhu, Zhijian Liu, and Song Han. 2019.  Deep leakage from gradients. In _Advances in neural information processing systems_ , Vol. 32.
  * Zubakov et al. (2010) Dmitry Zubakov, Fan Liu, MC Van Zelm, J Vermeulen, BA Oostra, CM Van Duijn, GJ Driessen, JJM Van Dongen, Manfred Kayser, and AW Langerak. 2010.  Estimating human age from T-cell DNA rearrangements.  _Current Biology_ 20, 22 (2010), R970–R971.