11institutetext: Ontology Engineering Group, Universidad Politécnica de
Madrid, Boadilla del Monte, Spain 22institutetext: High School of Technical
Industrial and Design Engineering, Universidad Politécnica de Madrid, Madrid,
Spain 33institutetext: Subdirección General de Farmacia y Productos
Sanitarios, Servicio Madrileño de Salud, Madrid, Spain

# Drugs4Covid: Drug-driven Knowledge Exploitation based on Scientific
Publications

Carlos Badenes-Olmedo 11 David Chaves-Fraga 11 María Poveda-Villalón 11 Ana
Iglesias-Molina 11 Pablo Calleja 11 Socorro Bernardos 11 Patricia Martín-
Chozas 11 Alba Fernández-Izquierdo 11 Elvira Amador-Domínguez 11 Paola
Espinoza-Arias 11 Luis Pozo 11 Edna Ruckhaus 11 Esteban González-Guardia 11
Raquel Cedazo 22 Beatriz López-Centeno 33 Oscar Corcho 11

###### Abstract

In the absence of sufficient medication for COVID patients due to the
increased demand, disused drugs have been employed or the doses of those
available were modified by hospital pharmacists. Some evidences for the use of
alternative drugs can be found in the existing scientific literature that
could assist in such decisions. However, exploiting large corpus of documents
in an efficient manner is not easy, since drugs may not appear explicitly
related in the texts and could be mentioned under different brand names.
Drugs4Covid combines word embedding techniques and semantic web technologies
to enable a drug-oriented exploration of large medical literature. Drugs and
diseases are identified according to the ATC classification and MeSH
categories respectively. More than 60K articles and 2M paragraphs have been
processed from the CORD-19 corpus with information of COVID-19, SARS, and
other related coronaviruses. An open catalogue of drugs has been created and
results are publicly available through a drug browser, a keyword-guided text
explorer, and a knowledge graph.

###### Keywords:

Ontology-based technologies NLP Bio-annotations Drugs-catalogue Knowledge
Graph COVID-19.

##  1 Introduction

News on the scarcity of medicines to treat COVID-19 have appeared in many
countries around the world over the past months. Due to the increasing demand
during the peaks of infections worldwide and other problems associated to
logistics, doctors and pharmacists were struggling to treat their patients. We
confirmed this situation after a series of interviews with doctors,
pharmacists and other people from regional Health Services. In the absence of
sufficient medication for such a large amount of patients, practitioners
started applying disused drugs that were available in hospital pharmacies and
modifying the usual doses of those available.

Figure 1: Creation workflow of a search engine and a knowledge graph through
annotations created from the CORD-19 dataset.

However, identifying which drugs can be used as replacements for others to
treat this little-known disease and its associated symptoms is a difficult
challenge. New experiments and results are continually being published, and
people in charge of clinical protocols cannot keep up to date with all of them
[5]. This situation calls for solutions that help health care providers and
researchers easily extract such knowledge from the enormous scientific corpus
that is being created.

Several initiatives have emerged to bring together scientific publications in
this domain. For instance, the COVID-19 Data Portal maintained by the EU, or
those published by Humandata focused on COVID-19 cases around the world.
Similarly, other research groups and universities have also published valuable
datasets, such as the Open COVID-19 Data Working
Group111https://github.com/beoutbreakprepared/nCoV2019, the DisGeNET COVID-19
dataset222https://www.disgenet.org/covid/diseases/summary/ or the data
gathered by John Hopkins
University333https://github.com/CSSEGISandData/COVID-19, among others.

The Allen Institute for Artificial Intelligence created the COVID-19 Open
Research Dataset (CORD-19) [22]. It is a continuously growing corpus with all
publicly available COVID-19 and coronavirus-related research (e.g. SARS, MERS,
etc.) in the last fifty years. This dataset can be used as a source of
information to extract knowledge related to the disease. At the time of this
study, it is composed of 23,428 open access articles from PubMed Central,
35,240 research articles from a corpus maintained by the World Health
Organization (WHO), and 1,945 bioRxiv and medRxiv pre-prints.

Given that volume, a combined used of natural language processing with word
embeddings techniques and knowledge extraction technologies can provide
doctors and medical researchers with tools that make their work easier by
structuring the information contained within the papers. Our goal is to
automatically extract, organize and publish the drug-oriented knowledge from
the medical literature needed to answer common questions posed by domain
experts such as What are the effects of using chloroquine and
hydroxychloroquine to treat COVID-19 patients? Have drugs that combine
immunosuppressive and antimalarial activity with macrolide antibiotics been
used? In which experiments have mefloquine and azithromycin been used and
related to which diseases?

We processed more than 60k medical publications to discover relations amongst
drugs and diseases. The tools created from our workflow help to quickly create
domain-specific search engines and knowledge graphs (KG) (Fig. 1) over any
corpus of scientific documents. Such techniques can be reused for other
similar crisis in the future, and in any other situation where these tools
could be valuable. All the work presented in this paper is available in a
GitHub repository444https://github.com/oeg-upm/drugs4covid19

##  2 Related Work

Due to the fast evolution of the SARS-COV-2, worldwide scientists have been
working as fast as possible to provide tools that help health professionals
treat patients during this pandemic. Most of the ongoing work has been shared
inside research communities (e.g. W3C Semantic Web and W3C Healthcare and Life
Sciences), in hackathons (e.g. the European Commission hosted
EUvsVirus555https://www.euvsvirus.org, VenceAlVirus666https://vencealvirus.org
in the region of Madrid), and in preliminary reports in blogs and archival
services. Multiple datasets777https://data.world/resources/coronavirus/ have
been provided for finding new insights about the novel coronavirus.

Semantic Web technologies have been traditionally used to publish many data
sources in the biomedical domain such as Bio2RDF that provides a network of
linked data for the biomedical domain, DisGeNET RDF that relates diseases and
genes, or SNOMED Clinical Terms that provides a clinical healthcare
terminology. Some of this earlier work has been used by language technologies
to add semantic annotations to scientific literature. The CORD19-Named
Entities Knowledge identified and disambiguated named entities in CORD-19
using DBpedia Spotlight and the NCBO BioPortal annotator. Similarly, a Linked
Data version of the CORD-19 data is provided by the CORD-19-on-FHIR project
introducing semantic annotations on conditions, medications, and procedures.

Some knowledge graphs have also been created such as the
Knowledge4COVID-19888https://devpost.com/software/covid-19-kg that integrates
knowledge from scientific literature and databases, and uses predictive models
to derive interactions among drugs used in the treatment of COVID-19. Others
are mainly focused on a collective generation of knowledge such as KG-COVID-19
Hub999https://github.com/Knowledge-Graph-Hub/kg-covid-19 or the Covid Graph
project101010https://live.yworks.com/covidgraph. Both seek to build
collaboratively a KG hub for COVID-19 so that developers can ingest new
sources into this hub. And there are also initiatives focused on particular
domains such as the SIB COVID-19 Integrated Knowledgebase that is specialized
on genetic information, and the COVID-19 by STKO Lab that combines recorded
cases and suspensions of commercial airline routes.

However, none of these approaches provide sufficient information on how the
knowledge is generated. When two drugs are related through an interaction in
the knowledge graph, the information used to induce that kind of relation is
missing and would be very relevant for its potential users. We believe that
conclusions are as important as the data that led to them during the learning
process. Hence, in this paper we describe a process to create a knowledge
graph that, together with the diseases and drugs identified in the CORD-19
corpus, links to the texts (i.e paragraphs, sentences or even full articles)
that support the inferred knowledge.

##  3 Drug and Disease Annotations

Searching for similar drugs and exploring major diseases covered by different
publications are key activities when browsing medical papers. This manual
know-
ledge-intensive task becomes less tedious and even leads to unexpected
relevant findings when applying unsupervised algorithms to help researchers.

In this work we have addressed the problem of generating drug- and disease-
based annotations for the documents within a large collection of research
articles. We have created vectorial representations of each of them using
state-of-the-art techniques based on word embeddings and probabilistic topic
models. Then, diseases and drugs have been related to each other and suggested
drugs that may be considered as replacements for others.

More than 60k scientific papers were analyzed from the CORD-19 dataset. Around
5M sentences and 2M paragraphs were annotated with the drugs and diseases
mentioned in them. A total of 6,400 diseases and 2,400 drugs were
characterized to enrich the searches on the corpus and the knowledge graph.

###  3.1 Taxonomies used for annotation

Textual searches on a collection like CORD-19 bring accuracy issues since the
terminology associated with drugs and diseases varies between countries. A
search based on an active substance should lead to the drugs that contain it.
Such drugs can be traded under different brand names among countries, hence
appearing with different names in different papers. For example, the active
substance Enalapril/Hydrochlorothiazide is distributed in Spain under the
trade names Co-Renitec, Crinoretic or Dabond Plus, while its trade name is
Renidur in Portugal, and Corodil in Denmark. Thus, we need to unify drugs and
diseases with codes that abstract them from their particular names or textual
representations.

We have used the trade names published by the Spanish Agency of Medicines and
Health Products (AEMPS) and considered the Anatomical Therapeutic Chemical
(ATC) classification system to annotate drugs. The ATC is supported by the
World Health Organization (WHO) and widely used in hospital pharmacies to
identify drug components. It groups active substances according to the organ
or system on which they act and their therapeutic, pharmacological, and
chemical properties. Drugs are classified into groups at five different
levels. The first one corresponds to main groups, the second one to
pharmacological or therapeutic subgroups, the third and the fourth one are
chemical-pharmacological-therapeutic subgroups and the last one is the
chemical substance.

Diseases have been annotated with the Medical Subject Headings (MeSH)
vocabulary, which is used for indexing, cataloging, and searching biomedical
information by the MEDLINE/PubMed article database, the NLM´s books catalog,
and ClinicalTrials.gov among others. This vocabulary is organized in a
hierarchical, numbered tree structure that enables browsing from broader to
narrower topics.

Our goal is not to annotate according to a single classification system, but
to integrate different classification codes suggested by our users to allow
cross-searching from their terminologies.

###  3.2 Annotation techniques

Top-Word | P01BA01 | P01BA02 | J05AR10
---|---|---|---
1 | chloroquine | chloroquine | mer
2 | chikv | hydroxychloroquine | camel
3 | autophagy | glycosylation | cov
4 | endosome | analog | mild
5 | cathepsin | rheumatoid | lopinavir
Table 1: Top5 words related to Chloroquine (P01BA01), Hydroxychloroquine
(P01BA02) and Lopinavir/Ritonavir (J05AR10) according to the PTM created from
the active substances discovered in the CORD-19 corpus.

Common entities identified in medical literature are diseases, drugs and
chemical names among others [20, 10, 19, 6]. These named entities can also be
used to discover relations between them. We created a pipeline to prepare the
textual content from the CORD-19 dataset [22] and index it together with some
additional information in a document-oriented database. It is based on the
librAIry framework [2], which provides a high-performance infrastructure for
text mining tasks. We focused on the following data for this task:

  * •

id: unique identifier for each research article.

  * •

name: publication title.

  * •

abstract: brief summary made by the author.

  * •

text: full textual content of the article.

  * •

url: online resource available from Allen AI repository.

The Named Entity Recognition (NER) task was performed combining different
state-of-the-art methods such as scispaCy [14] and CliNER [4]. After they were
tested to identify disorders, substances, trials and treatments from texts, we
realized the methods did not unify the identification of drugs and diseases as
described in the ATC or MeSH classifications. Sometimes they are identified as
different drugs or diseases, since they are referred by different names, when
in fact they correspond to the same element. To adjust this behavior we have
developed Bio-NLP111111https://github.com/librairy/bio-nlp, a NER service
built on top of scispaCy that uses the MeSH vocabulary and the ATC codes as
gazetteers of terms respectively to identify named entities.

##  4 Drug and Disease Representational Models

In order to establish relations among drugs or diseases, so as to improve
searches on the corpus, we have created representational models capable of
abstracting the entities from their textual forms and relating them to each
other.

###  4.1 Identifying Related Texts

Some text mining algorithms represent documents in a common feature space that
abstracts away from the specific sequence of words used in each of them. That
is, they treat documents as bags-of-words. With appropriate representations,
they ease the analysis of relationships between documents, even when written
using different vocabularies.

Figure 2: Documents described by a PTM from the active substances (ATC-code
level5) in the CORD-19 corpus and organized by similar topic distributions.

  * •

Representational Model: We have used topic models, which are widely used to
uncover the latent semantic structure from text corpora. In particular,
Probabilistic Topic Models (PTM) represent documents as a mixture of topics,
where topics are probability distributions over words. This eases the
exploitation of large document collections, since documents are mapped into a
low-dimensional latent space (i.e they are described by a small number of
topics). It also abstracts the representation of the documents from the words
themselves. Topics emerge as density distributions on the vocabulary to define
the dimensions of the vectors that represent the documents. We used
librAIry[2] to create and fine-tune a PTM that describes the annotated drugs
as probability distributions on the vocabulary of the corpus. Each topic
matches an active substance (see Table 1), avoiding distinctions between
different drug names by country. The model is created and distributed as a
Rest-API to facilitate its reuse.

  * •

Similarity Metric: Due to low storage cost and fast retrieval speed, hashing
is a popular solution for the calculation of approximate nearest neighbors in
the probability simplex space created from topic models. We opted for a
density-based hierarchical hashing method [3] to compare the texts contained
in CORD-19. This approach has proven to obtain high-precision results and can
accommodate additional query restrictions.

  * •

Space Partitioning: Since documents are compared from sets of topics
distributed in levels according to their relevance, the representational space
can be organized in k-d trees. The most discriminating topics and levels
define the branches of the tree and each text is organized among them
according to its topic distributions. We created a web-based document
browser121212https://librairy.github.io/covid19/explorer to navigate through
the corpus using the active substances and content-based relations (Figure 2).

###  4.2 Finding Related Drugs

Drugs were described by the diseases with which they are mentioned in texts
(e.g sentence, paragraph, or full-text). Paragraphs showed an appropriate
balance between generality and particularity to describe drugs, since their
content is more strongly related than the full-text, and less incomplete than
the sentences.

(a) Silhouette analysis

(b) Two-dimensional space

Figure 3: Validation of the 120 clusters of drugs.

  * •

Representational Model: A drugs-diseases matrix was created with the times a
disease appears in the same paragraph than a drug. Then, we used Term
Frequency-Inverse Document Frequency (TFIDF) to transform drugs into numeric
vectors. Two drugs described by TFIDF will be similar if they share rare, but
informative, diseases.

  * •

Similarity Metric: Cosine similarity was adopted as the measure of interest as
it is well-known and frequently-used in vector space models [18].

  * •

Space Partitioning: Similar drugs are organized into the same groups to
suggest their use as replacements. Since the number of groups is unknown, we
created a dendrogram based on simple linkage from their vectorial
representations. As a result we obtained a hierarchical aggregation of drugs
that suggests an optimal number of groups according to the Silhouette
coefficient. Several settings were tested and finally 120 clusters offered the
best performance (Fig. 3). We developed a Rest-API, Bio-
API131313https://librairy.linkeddata.es/bio-
api/replacements?keywords=chloroquine, that makes use of the
Annoy141414https://github.com/spotify/annoy method to index drugs in a kd-
tree. It allows queries like bio-api/replacements?keywords=chloroquine to
retrieve related drugs.

###  4.3 Finding Related Diseases

In order to measure the similarity amongst diseases, a three-step procedure
was followed: 1) Creation of disease-focused word embedding models, 2)
Generation of a terminology representing the whole corpus, used to align the
compared diseases, 3) Disease similarity calculation from the identified key
terms.

CORD-19 top 25 terms
---
T cell | respiratory tract | viral replication | infected cell | health care
public health | infectious disease | viral RNA | cell lines | RNA viruses
amino acid | immune responses | epithelial cell | respiratory syndrome | nucleic acid
immune response | accute respiratory | E. coli | virus infection | acute respiratory syndrome
influenza virus | gene expression | viral infection | immune system | respiratory viruses

Table 2: Main terms discovered in the CORD-19 corpus

  * •

Terminology Generation: The aim of this step was to discover common
terminology amongst diseases. We extracted the most frequent terms from a
representative sample of 100.000 paragraphs from CORD-19. Given the complex
nature of the source texts (full of technical expressions, numbers, figures,
symbols) this task presented a challenge when tested with two terminology
extraction methods: Rake [17] and TBXTools[21]. While the former did not solve
many of the above mentioned hindrances and some extracted terms were
linguistically incorrect, the latter gave good results that needed only some
post-processing work (Table 2).

  * •

Representational Model: Out of the existing state-of-the-art word embedding
models, Word2Vec [13] was selected to generate a representational model for
each disease. To ensure comparability between models, the same initialization
values were used from the pre-trained BioWordVec model [23]. Using a proper
initialization not only enables direct comparability between disease models,
but eases the convergence of the model.

  * •

Similarity Metric: Diseases were represented as vectorial models, where each
word featured in the training corpus found a corresponding embedding. However,
not all words in the resulting vocabulary were equally explanatory of the
disease, and thus comparisons between models should not be established using
the totality of the embedded words. We retrieved the 25 most frequent terms in
CORD-19 with the aim of discovering common terms amongst diseases (Table 2).
They were used as reference points to compare models that visually represent
each disease, considering each term as a node, and the distance between each
pair of nodes as edges. Similarity between diseases is obtained by averaging
individual distances between the representations of the same term on each
disease model. Word Mover’s Distance [12] was used to measure the distance
between terms, as it captures the underlying semantics between the resulting
word representations.

  * •

Space Partitioning: Disease models were initialized using the same values,
meaning that the initial embedding of each term was the same across every
disease. During training, these embeddings evolve, or move throughout the
vectorial space to fit the context constraints from the disease-specific
training corpus. Therefore, while the initial representations were equal, the
final representations differ, as they also embedded specific information about
the disease. However, the resulting representations can still be compared.
Pairwise comparison between diseases can be established by measuring the
Euclidean distance between the embeddings of same terms on each model. Small
distances between term representations mean that they occur in a similar
context, thus their final embeddings remain close. Figure 4 illustrates
pairwise comparison between three diseases. According to the obtained results,
COVID-19 is closer to Malaria than to Conjunctivitis, as the cumulative
distance of all pairwise term distances is lesser in that case. This
functionality was also added to our Bio-API service.

(a) COVID-19 and Malaria

(b) COVID-19 and Conjunctivitis

Figure 4: Comparison of diseases based on distances among common terminology
described in the embedding spaces created for each of them.

##  5 Drugs4Covid Knowledge Graph

All the work described in previous sections allowed us to create the knowledge
to enable the search and navigation over the corpus. To facilitate its
(re)use, we created a graph that contains this knowledge. This section
describes the development of the vocabulary and details the steps followed to
build the Drugs4Covid Knowledge Graph (D4C-KG) by combining the drug and
disease annotations inferred from the CORD-19 corpus with external resources.

###  5.1 The Drugs4Covid vocabulary

The vocabulary defined to support the D4C-KG was developed following the LOT
methodology [16], relying heavily in the communication with users (i.e.
developers generating annotations and KG developers) and domain experts (i.e.
biologists and pharmacists) throughout the ontology development process. The
main goals of the vocabulary were to represent and relate some parts of the
papers analyzed with biomedical concepts, as obtained with the application of
language technologies.

The process started with a meeting where the potential annotations, based on
drugs and diseases, were analyzed. A first set of requirements was extracted:
1) A paper contains paragraphs and sentences; 2) A paragraph and a sentence
can mention chemical substances and disorders; 3) A paper mentions a drug; 4)
A drug has active substances, which are a type of chemical substances; 5) A
drug treats a disorder; 6) A disease has symptoms. The vocabulary requirements
evolved throughout the development of the project.

The vocabulary conceptualization was carried out following the guidelines for
ontology diagrams presented in [9] and splitting the team into the two main
domains involved in the data, namely “publications” and “biomedical” domains.
Figure 5 depicts the vocabulary conceptualization, whose main concepts are:

  * •

Paper: A document of the corpus, subject of annotations. It can contain
paragraphs (one of a series of subsections each usually devoted to one idea)
and sentences (a sequence of words capable of standing alone to make an
assertion, ask a question, or give a command). These concepts can mention
disorders and/or chemical substances.

  * •

Disorder: A disruption to regular bodily structure and function. A disease is
often known to be a medical disorder that is associated with specific
symptoms, while a symptom refers to a physical or mental feature that is
regarded as indicating a condition of disease.

  * •

Chemical substance: A substance produced by or used in a chemical process.
Active substances are chemical substances that are the main ingredients of
drugs used to treat or prevent a disorder. Each active substance matches an
ATC code as described in the Annotation Section.

Figure 5: Main classes, properties and attributes of the Drugs4Covid
vocabulary.

The vocabulary reuses some Dublin-Core Metadata
Terms151515https://dublincore.org/ for bibliographic resources such as
abstract, license, and source. Since we are pursuing a progressively more
complex construction of the knowledge graph, we decided to initially create a
basic model that allows easy alignment with existing ontologies in the
biomedical domain later on. However, some modeling decisions were already
taken in this direction, by aligning the active substance class attributes to
the ATC annotations from
bioontology161616https://bioportal.bioontology.org/ontologies/ATC.

We carried out a twofold evaluation: (1) quality and correctness of the
ontology using OOPS![15], and (2) a data-directed evaluation where we checked
that the data in the indexed corpus can be transformed, annotated and queried
with the vocabulary. Finally, the HTML documentation of the vocabulary was
generated with Widoco [8] and
published171717https://w3id.org/def/DRUGS4COVID19 using OnToology [1].

###  5.2 Knowledge Graph Construction

The D4C-KG contains RDF annotations extracted from the CORD-19 dataset that
are described by the D4C vocabulary. The relations between the source data
(annotations) and the developed ontology were declared using the RML mapping
language [7]. We used YARRRML181818https://rml.io/yarrrml, a human-friendly
RML serialization in YAML format, to define 44 subject and 150 predicateObject
mappings. The rules created by this technique were easy to define and not
time-consuming.

The building process was automated and modular, based on standard W3C
technologies in order to ensure the maintainability of the KG construction
process. The following steps were performed:

  * •

Data Preparation: Annotations from the CORD-19 corpus were exported into CSV
files, cleaned and normalized exploiting CSVW annotations.

  * •

Mapping Definition: RML mapping rules between the data and the ontology were
defined, as well as the rules to link other KGs.

  * •

RDF Generation: The cleaned data and mapping rules were processed with SDM-
RDFizer [11] to generate the corresponding RDF dataset.

  * •

Publication: The resulting RDF, along with the ATC data available in
BioPortal, were published through a Virtuoso SPARQL
endpoint191919https://kg.drugs4covid.oeg-upm.net/sparql. In addition,
federated queries across different datasets can be performed to external
knowledge graphs to enhance the completeness of the provided knowledge.

##  6 Use Cases

Figure 6: Drugs-oriented search interface showing drugs and diseases related
to lopinavir.

As a result of our work, users can quickly locate the paragraphs in the
articles where immunosuppressive and antimalarial activity with macrolide
antibiotics are discussed; or have a list of related diseases to mefloquine
and azithromycin when used to treat the COVID-19.

All this information is available for end users through our search engine, and
for anybody who wants to reuse the results through the restful Bio-API and the
open knowledge graph described above. The main use scenarios are:

  * •

Drug-oriented Search: Our search engine202020https://search.drugs4covid.oeg-
upm.net provides the paragraphs of the articles where the drug is mentioned,
either by its trade name, its generic name or its ATC classification; and a
list of related drugs and related diseases inferred from the representational
models created for each of them (Fig.6).

  * •

Keyword-oriented Search: Our RESTful API212121https://search.drugs4covid.oeg-
upm.net/customsearch browses scientific literature through the relations
inferred among drugs222222https://librairy.linkeddata.es/bio-api/drugs and
diseases232323https://librairy.linkeddata.es/bio-api/diseases. The drugs used
in conjunction with lopinavir (e.g. /bio-api/drugs?keywords=lopinavir), or the
diseases treated with chloroquine (e.g. /bio-
api/diseases?keywords=chloroquine) can be easily explored.

  * •

Pattern-oriented Search: Our SPARQL endpoint242424https://kg.drugs4covid.oeg-
upm.net offers a semantic interface for exploration. The most
representative/descriptive questions asked by practitioners, pharmacists and
general public in natural language were translated into SPARQL. Listing 1
shows one of them. It retrieves the diseases related to drugs that combine
immunosuppressant and antimalarial activities with macrolide antibiotics.

Listing 1: List of drugs by a SPARQL query

⬇

1PREFIX onto: <https://w3id.org/def/DRUGS4COVID19#>

2

3SELECT DISTINCT ?section ?paperTitle ?notation2 ?titleDisease WHERE {

4 ?paragraph a onto:Paragraph .

5 ?paragraph onto:section ?section .

6 ?paper onto:contains ?paragraph .

7 ?paper dc:title ?paperTitle .

8 ?paragraph onto:mentions ?activeSubstance1 .

9 ?paragraph onto:mentions ?activeSubstance2 .

10 ?activeSubstance1 skos:notation "P01BA02"^^xsd:string .

11 ?activeSubstance2 skos:notation ?notation2 .

12 ?paragraph onto:mentions ?disease .

13 ?disease a onto:Disease .

14 ?disease onto:MESHCode ’C000657245’ .

15 ?disease dc:title ?titleDisease .

16 FILTER (STRSTARTS(?notation2,"J01FA"))

17}

##  7 Lessons Learned and Future Work

This paper presents our work to create a knowledge graph and a search engine
to facilitate the exploration of the CORD-19 corpus through the relations
discovered between drugs, diseases and texts. It required an effective
collaboration between language technology and semantic web researchers, and
with domain experts (doctors and pharmacists) who helped us to resolve doubts
and guide functional decisions. The obtained results, software, models and
resources are publicly available for anyone to use.

After the initial release of our services and the creation of a knowledge
graph together with its associated vocabulary, we have received numerous
requests to continue expanding the types of annotations and the information
offered by the knowledge graph. Our results are also being used by third
parties, e.g. to obtain ”therapeutic targets”. The relations sought in this
case arise from the proteins described in articles, with the added difficulty
that they are usually mentioned by acronyms. Their feedback is promising, and
they encourage us to continue incorporating also the management of genes and
biological pathways.

This would lead to an increment of the possibilities offered to users when
browsing the corpus. Moreover, the exploitation of these annotations in the
knowledge graph leads us to increase the reuse of existing biomedical
ontologies and terminologies.

We are also working on a Citizen Science initiative with students of Pharmacy
and Medicine to validate our annotations and help understand the type of
relations that may exist between drugs and diseases. The consensus acquired
from the contributions made by these citizen scientists will serve to measure
the quality of the results and improve the system as a whole. As in many other
Citizen Science projects, a score-based system will be created using
parameters such as user participation, quality of contributions (based on gold
standards and community contributions), etc. And finally, a helpdesk will
allow us receiving feedback from users to continue improving the results that
we provide.

Other more general functionalities that have been already requested by end
users are related to usability and to the creation of alerts when a drug,
protein or combinations of them appear in the corpus.

## References

  * [1] Alobaid, A., Garijo, D., Poveda-Villalón, M., Santana-Perez, I., Fernández-Izquierdo, A., Corcho, O.: Automating ontology engineering support activities with OnToology. Journal of Web Semantics (2018)
  * [2] Badenes-Olmedo, C., Redondo-Garcia, J.L., Corcho, O.: Distributing Text Mining tasks with librAIry. In: Proceedings of the 2017 ACM Symposium on Document Engineering - DocEng ’17. pp. 63–66. ACM Press (2017). https://doi.org/10.1145/3103010.3121040
  * [3] Badenes-Olmedo, C., Redondo-García, J.L., Corcho, O.: Large-scale semantic exploration of scientific literature using topic-based hashing algorithms. Semantic Web pp. 1–16 (2020). https://doi.org/10.3233/SW-200373
  * [4] Boag, W., Sergeeva, E., Kulshreshtha, S., Szolovits, P., Rumshisky, A., Naumann, T.: Cliner 2.0: Accessible and accurate clinical concept extraction. arXiv preprint arXiv:1803.02245 (2018)
  * [5] Brainard, J.: Scientists are drowning in COVID-19 papers. Can new tools keep them afloat? Science (2020), https://www.sciencemag.org/news/2020/05/scientists-are-drowning-covid-19-papers-can-new-tools-keep-them-afloat
  * [6] Cho, H., Lee, H.: Biomedical named entity recognition using deep neural networks with contextual information. BMC bioinformatics 20(1), 735 (2019)
  * [7] Dimou, A., Sande, M.V., Colpaert, P., Verborgh, R., Mannens, E., de Walle, R.V.: RML: A Generic Language for Integrated RDF Mappings of Heterogeneous Data. In: LDOW. CEUR Workshop Proceedings, vol. 1184. CEUR-WS.org (2014)
  * [8] Garijo, D.: WIDOCO: a wizard for documenting ontologies. In: International Semantic Web Conference. pp. 94–102. Springer (2017)
  * [9] Garijo, D., Poveda-Villalón, M.: Best Practices for Implementing FAIR Vocabularies and Ontologies on the Web (2020)
  * [10] Goulart, R.R.V., de Lima, V.L.S., Xavier, C.C.: A systematic review of named entity recognition in biomedical texts. Journal of the Brazilian Computer Society 17(2), 103–116 (2011)
  * [11] Iglesias, E., Jozashoori, S., Chaves-Fraga, D., Collarana, D., Vidal, M.E.: SDM-RDFizer: An RML interpreter for the efficient creation of rdf knowledge graphs. In: ACM Intern. Confer. on Information and Knowledge Management, CIKM (2020)
  * [12] Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From Word Embeddings To Document Distances. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 957–966. PMLR (2015)
  * [13] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed Representations of Words and Phrases and their Compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc. (2013)
  * [14] Neumann, M., King, D., Beltagy, I., Ammar, W.: ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In: Proceedings of the 18th BioNLP Workshop and Shared Task. pp. 319–327. Association for Computational Linguistics (2019)
  * [15] Poveda-Villalón, M., Gómez-Pérez, A., Suárez-Figueroa, M.C.: OOPS! (OntOlogy Pitfall Scanner!): An On-line Tool for Ontology Evaluation. International Journal on Semantic Web and Information Systems (IJSWIS) 10(2), 7–34 (2014)
  * [16] Poveda-Villalón, M., Fernández-Izquierdo, A., García-Castro, R.: Linked Open Terms (LOT) Methodology (2019). https://doi.org/10.5281/zenodo.2539305
  * [17] Rose, S., Engel, D., Cramer, N., Cowley, W.: Automatic keyword extraction from individual documents. Text mining: applications and theory 1, 1–20 (2010)
  * [18] Shahmirzadi, O., Lugowski, A., Younge, K.: Text Similarity in Vector Space Models: A Comparative Study. ArXiv (2018)
  * [19] Song, H.J., Jo, B.C., Park, C.Y., Kim, J.D., Kim, Y.S.: Comparison of named entity recognition methodologies in biomedical documents. Biomedical engineering online 17(2), 158 (2018)
  * [20] Tsai, R.T.H., Wu, S.H., Chou, W.C., Lin, Y.C., He, D., Hsiang, J., Sung, T.Y., Hsu, W.L.: Various criteria in the evaluation of biomedical named entity recognition. BMC bioinformatics 7(1), 92 (2006)
  * [21] Vàzquez, M., Oliver, A.: Improving term candidates selection using terminological tokens. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication 24(1), 122–147 (2018)
  * [22] Wang, L.L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Eide, D., Funk, K., Kinney, R., Liu, Z., Merrill, W., et al.: CORD-19: The Covid-19 Open Research Dataset. arXiv preprint arXiv:2004.10706 (2020)
  * [23] Zhang, Y., Chen, Q., Yang, Z., Lin, H., Lu, Z.: BioWordVec,improving biomedical word embeddings with subword information and MeSH. Scientific Data 6(1), 52 (2019). https://doi.org/10.1038/s41597-019-0055-0