# Explanatory Argument Extraction of Correct Answers in Resident Medical Exams

Iakes Goenaga, Aitziber Atutxa, Koldo Gojenola, Maite Oronoz, Rodrigo Agerri
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
{iakes.goenaga, aitziber.atucha, koldo.gojenola,
<EMAIL_ADDRESS>
###### Abstract

Developing the required technology to assist medical experts in their everyday
activities is currently a hot topic in the Artificial Intelligence research
field. Thus, a number of large language models (LLMs) and automated benchmarks
have recently been proposed with the aim of facilitating information
extraction in Evidence-Based Medicine (EBM) using natural language as a tool
for mediating in human-AI interaction. The most representative benchmarks are
based on a question answering setting, where the objective is for the models
to provide answers to medical questions. Although interesting, these
benchmarks are limited to either multiple-choice or long-form answers and are
available only in English. In order to address these shortcomings, in this
paper we present a new dataset which, unlike previous work: (i) includes not
only explanatory arguments for the correct answer, but also arguments to
reason why the incorrect answers are not correct; (ii) the explanations are
written originally by medical doctors to answer questions from the Spanish
Residency Medical Exams. Furthermore, this new benchmark allows us to setup a
novel extractive task which consists of _identifying the explanation of the
correct answer written by medical doctors_. An additional benefit of our
setting is that we can leverage the extractive QA paradigm to automatically
evaluate performance of LLMs without resorting to costly manual evaluation by
medical experts. Comprehensive experimentation with language models for
Spanish shows that sometimes multilingual models fare better than monolingual
ones, even outperforming models which have been adapted to the medical domain.
Furthermore, results across the monolingual models are mixed, with supposedly
smaller and inferior models performing competitively. In any case, the
obtained results show that our novel dataset and approach can be an effective
technique to help medical practitioners in identifying relevant evidence-based
explanations for medical questions.

###### keywords:

Explainable Artificial Intelligence , Argumentation , Question Answering ,
Resident Medical Exams , Natural Language Processing , Deep Learning

††journal: Artificial Intelligence in Medicine

##  1 Introduction

There is an increasing interest in the medical domain to apply Artificial
Intelligence (AI) techniques to assist medical experts in their everyday
activities. These include automatic information extraction methods to obtain,
from large amounts of unstructured text, relevant and timely information to
facilitate the medical experts’ deliberation and decision processes as part of
the Evidence-Based Medicine (EBM) practice. In other words, the goal is to
help in EBM which is understood as the “conscientious, explicit, and judicious
use of current best evidence” [1] to guide clinical decision-making with
scientific information extracted from systematic reviews of previous work.

However, devising algorithms to help in clinical decision-making, in which the
medical expert needs to identify and diagnose a disease and prescribe a
treatment based on the available evidence, is far from trivial. First, doing
so requires for the systems to automatically identify, access and correctly
leverage the relevant medical knowledge. Second, the acquired medical
knowledge needs to be adequately used to help disambiguate between the variety
of symptoms, each of which may be indicative of multiple diseases. Finally,
the system must interact with the medical experts in a natural manner, ideally
using natural language.

Recently, a number of works have been proposed in this line of research using
natural language as a tool for mediating in AI-human interactions. These
efforts have been focused on: (i) argumentation and explanation-based
approaches to detect, classify and assess the quality of previously given
argumentative structures in medical texts [2]. The general idea is that such
argumentative structures constitute the basis of Evidence-Based Medicine; (ii)
the development of Large Language Models (LLMs) adapted, via a variety of
techniques, to the medical domain. These include encoder models such as
SciBERT, BioBERT or PubMedBERT [3, 4, 5] for discriminative modelling. These
models, although they established new state-of-the-art results in
classification tasks, are typically smaller in scale and scope with respect to
decoder and autoregressive models such as GPT-3 or PALM [6, 7] and their
biomedical counterparts BioGPT [8], SciFive [9], and Med-PaLM [7], designed
for generative language modelling; (iii) the creation of automated benchmarks
with the aim of facilitating information access in EBM. The most popular
benchmarks are based on a question answering (QA) scenario, in which the aim
is for the models to classify or generate answers to medical questions. Among
others, it is worth mentioning MultiMedQA, a multi-task, seven multiple-choice
dataset [7] and TruthfulQA, a dataset to measure the trustworthiness of LLMs
[10].

While interesting, previous work presents the following limitations. First,
the large majority of the proposed benchmarks and LLMs have been developed for
English. Second, most of the medical QA datasets consist of Yes/No or long-
form single answers to multiple-choice questions. Third, despite huge progress
in LLMs, learning to generate answers in the medical domain remains
notoriously difficult, as it requires knowledge about medical concepts and the
capacity to retrieve accurate and reliable information from medical
literature, guidelines and other data sources [7]. This means that medical QA
benchmarks remain extremely challenging due to the complexity and diversity of
medical language and the vast amount of available medical information [7, 10,
9]. Fourth, evaluating generative tasks requires expensive manual effort by
medical experts.

In order to address these challenges, in this work we present a new dataset
which, in contrast to previous work, includes not only explanatory arguments
for the correct answer but also arguments to explain why the remaining
possible answers are incorrect. Additionally, the dataset is entirely written
in Spanish by medical doctors.

This new dataset is leveraged in order to formulate a novel extractive task
which, taking into account the explanations written for both correct and
incorrect answers, consists of _identifying the explanation of the correct
answer_. In other words, the models should respond to medical questions
regarding the correct answer in a multiple-choice setting, by providing the
piece of text in the context which explains why a given answer is correct.
Furthermore, our new benchmark profits from harnessing the extractive QA
paradigm to automatically assess the performance of Language Model-based
Systems (LLMs), effectively eliminating the need for costly manual evaluation
conducted by medical experts.

The data source for our new dataset consists of the Resident Medical Intern
exams, or _Médico Interno Residente_ (MIR) in Spanish, similar to exams for
medical specialists in other countries such as the United States Medical
Licensing Examination (USMLE). The aim of the MIR exam is to assess medical
students’ knowledge by means of a multiple-choice questionnaire contextualized
by a short clinical case. Answering the MIR exam questions requires applying
evidence-based decision making and critical thinking according to the
available evidence present in the case. The process implies forming hypotheses
compatible with the evidence and discarding some of those hypotheses as new
evidence is collected. During this deliberation it is specially relevant to
provide arguments in favour or against each decision, namely, to offer
explanations to support or attack a given hypothesis.

Thus, the MIR-based dataset introduced in this paper includes the clinical
case, the correct answer, the multiple-choice questions and the annotated
explanations result of the deliberation process outlined above. These
explanations have been written by native Spanish medical doctors and offer a
unique resource to set up a novel dataset and task for EBM consisting of
extracting, from the argumentative structure, the explanation which
corresponds to the correct answer. Summarizing, the main contributions of this
work are the following:

  * •

The first dataset for benchmarking LLMs in the medical domain which contains
argumentative structure to provide explanations for both correct and incorrect
answers in a multiple-choice questionnaire.

  * •

A novel task based on extractive QA to _identify the explanations of the
correct answer_ in _commented_ medical exams, thereby helping medical doctors
to automatically retrieve relevant evidence from existing resources.

  * •

The first dataset for medical QA in a language other than English,
facilitating future research on multilingual approaches.

  * •

By formulating this task in an extractive setting, we can objectively and
automatically evaluate the performance of LLMs without requiring any manual
annotation by medical experts.

  * •

Comprehensive experimentation with LLMs for Spanish demonstrates that
sometimes multilingual models fare better than monolingual models, even
outperforming those which have been adapted to the medical domain. We show
that results across the monolingual models are mixed, with supposedly smaller
and inferior models performing competitively.

  * •

Our reported results suggest that our approach can be an effective technique
to help medical practitioners in identifying relevant evidence-based
explanations for medical questions.

  * •

Data and code are publicly available to encourage research on this research
topic and to guarantee reproducibility of results111https://github.com/ixa-
ehu/antidote-casimedicos.

Finally, we believe that our work can be a stepping stone for other
interesting tasks based on generative modelling and explainable AI in
Evicence-Based Medicine, for example, to predict not only the correct answer
but also to generate an explanation associated to it.

In the rest of the paper, we first discuss the related work, while in Section
3 we describe the dataset creation and annotation. Section 4 introduces the
methodology and experimental setup. Results are reported in Section 5 and
discussed and analyzed in Section 6. To wrap up, section 7 offers some
concluding remarks and future work.

##  2 Related work

Due to the unique characteristics of our new dataset and task, there is not
directly comparable work in the literature. However, by casting our new task
in an extractive Question Answering (QA) setting, we can review related work
on QA both from an extractive and abstractive/generative approaches.

In abstractive QA, generative models are not restricted to the input context
to create the answer of the question as they can generate answers word by word
employing the entire vocabulary in an auto-regressive manner [11]. However,
although the answers obtained by the models following this approach are
apparently good, they are not always factually reliable. Thus, the TruthfulQA
benchmark [10] tested large pre-trained generative models in order to find out
whether the answers given to the questions are truthful. The main conclusion
drawn by the authors was that the models generated many false answers that
mimic popular misconceptions and have the potential to deceive humans. In
fact, an interesting result from that paper was that the larger model, the
least truthful it was.

In extractive Question Answering the objective is, given a textual context, to
extract a span from the context which constitutes the answer to the given
question [12, 13]. This type of QA is a very popular setting which has seen a
high activity in the development of both datasets specifically about QA [12,
14, 15, 16], about reading comprehension [17, 18, 19], and on techniques
specific to this task [11, 20, 21, 22, 23, 24].

A number of interesting works on QA for the medical domain can be found in the
literature. Many datasets of various characteristics have been developed [25,
26, 27, 28, 29, 7, 30] and workshops and shared tasks such as the _Question
Answering in the medical domain_ [31] & BioASQ Challenge [32, 33, 34] have
been organized. As it has been been the case for most NLP tasks, Transformer-
based Pre-trained Language Models (PLMs) have facilitate huge improvements in
the state-of-the-art for medical QA [29, 35, 36, 30]. The most popular
approaches are those that use PLMs pre-trained on medical corpora such as
SciBERT [3], BioBERT [4] or PubMedBERT [37] Furthermore, approaches harnessing
general purpose datasets such as the Stanford Question Answering Dataset
(SQuAD) [12] or the Spanish Question Answering Corpus (SQAC) [38] as
intermediate fine-tuning dataset before training on domain-specific medical
datasets have also been shown to be effective [39, 40].

Nevertheless, despite the extensive efforts of the research community,
question answering for medical exams remains a formidable challenge, largely
due to the scarcity of suitable datasets that must be annotated by medical
specialists. Although there are several datasets in which doctors’
explanations of the correct answer to the question are included such as,
MultiMedQA [7], MedQA [30], PubMedQA [25], LiveQA [26] or MedicationQA [31],
most of them consist of Yes/No or long-form single answers to multiple-choice
questions.

However, unlike the MIR-based dataset that we present in this work, none of
them include explanations justifying both the right answer and arguing about
the incorrect ones. Our new dataset not only enables the application of
Extractive QA on explanations given by medical specialists, but it may also
open the door to create clinical argumentative models as they allow to
identify the piece of argument supporting the right answer among arguments
explaining why the remaining answers are incorrect.

##  3 Datasets

As mentioned in the introduction, one of the most important contributions of
this work is the release of a dataset based on MIR exams commented by medical
doctors. We identified two data sources to build our dataset, namely, the
_CasiMedicos_ and the _MIR Asturias_ projects.

| Example of a document from the CasiMedicos Dataset
---|---
C |  A 30-year-old woman comes to your surgery because she notices a recent lump in her right breast. Her grandmother had breast cancer. On examination, a 2.5 cm nodule with regular borders was palpated in the upper outer quadrant. She has no previous imaging tests.
Q |  Mark the correct answer:
P |  (1) You request a breast ultrasound given the patient’s age. (2) You order a mammogram because it is the ”gold standard” diagnostic test. (3) You order an MRI given the patient’s age and family history. (4) Explains that, as she is 30 years old, it is probably a cyst and will opt for a clinical check-up in 6-12 months. If it persists, then he will order an imaging test.
E |  We are presented with a 30 year old female patient presenting with a 2.5 cm nodule with regular borders. The clinical suspicion is of a benign lesion. The imaging test to be performed depends on the age of the patient, the clinical suspicion and the time elapsed since the last examination. In patients <35 years of age it is recommended to start the breast study with ultrasound, adding mammography if there are criteria for malignancy. MRI is reserved for patients with BRCA, breast prostheses, a history of neoadjuvant or conservative surgery. In the case of a palpable lesion of recent appearance and in the absence of any previous imaging test, an ultrasound study is recommended rather than a check-up in 6-12 months.
A |  In patients <35 years of age it is recommended to start the breast study with ultrasound, adding mammography if there are criteria for malignancy.
Table 1: Example of a document in the CasiMedicos dataset with the correct
explanation span manually annotated. C: Clinical Case; Q: Question; E: Medical
Doctor’s Explanations; P: Possible Answers; A: Correct Answer Explanation. The
_Clinical Case_ , _Question_ , and _Possible Answers_ sections are generated
for the MIR exams by the Spanish Ministry of Health. The medical doctors’
explanations (E) and the manually annotated span identifying the explanation
of the correct answer (A) are contributions of this work.

Table 1 shows an example document from our dataset. It should be noted that
the original data is written in Spanish, but for illustrative purposes we
offer a translation into English obtained with
DeepL222https://www.deepl.com/en/translator. This particular example is taken
from the _CasiMedicos_ set, although both resources are structured in the same
manner.

The first three parts of the example document, clinical case (C), question
(Q), possible answers (P), plus the solution (out of 4 options), are made
public every year by the Spanish Ministry of Health. The last two parts,
namely, the comments or medical doctor’s explanations (E) and the annotation
of the span identifying the correct answer explanation (A and in bold in the E
component) are the contribution of this work. More specifically, the
explanatory argumentation (E) has been written by Spanish medical doctors on
top of which we manually annotated the _span corresponding to the explanation
of the correct answer_.

For each of the questions in both data sources, CasiMedicos and MIR Asturias,
we gathered the year, a unique identifier, whether it had been canceled in the
official exam, the specialty related to the question (digestive, surgery,
pediatrics…), the question itself (Q), the possible answers (P), the correct
answer and the explanations written in the commentary given by a medical
doctor indicating the reasons for choosing or excluding the answers (E).

The MIR exams contain three types of questions: (i) those associated with an
image (usually a radiography) which needs to be interpreted in order to answer
the question; (ii) questions including a short clinical case as contextual
information previous to the question; (iii) questions about general medical
knowledge without any specific context.

For this work we selected questions of type (ii), namely, those that include a
clinical case (C in the example) to contextualize the question, possible
answers and the explanations given.

The process of manually annotating the corpus consisted of specifying where
the explanation of the correct answer (A) begins and ends (marked in bold in
the E part). In order to obtain grammatically complete correct answer
explanations, annotating full sentences or subordinate clauses was preferred
over shorter spans. The annotation took the equivalent of a person’s month
work (4 weeks, 160 hours). In the following we provide a description for each
of the data sources used to build our datasets.

CasiMedicos is a community of medical professionals who collaboratively,
voluntarily, and free of charge, publishes written explanations about the
possible answers included in the MIR exams. The aim is to generate a resource
that helps future medical doctors to study towards the MIR examinations. The
commented MIR exams, including the explanations, are published in the
CasiMedicos _Project MIR 2.0_ website333https://www.casimedicos.com/mir-2-0/.

After crawling, cleaning, and organizing the data, we obtained 1,561 commented
questions corresponding to the years 2011-2014, 2016, and 2018-2022. Selecting
those questions which included _clinical cases_ reduced the number of
documents from 1,561 to 575. The next step was to create three splits for
training, development and testing (70%, 15% and 15%, respectively). Table 2
provides some statistics regarding the final dataset where it can be observed
that pediatrics is the most common specialty and that clinical case entries
are longer (185 words wrt to the overal average of 163).

MIR Asturias is a well-known private academy in Spain which offers courses to
prepare for the MIR exams. They produce a number of interesting material,
including textbooks consisting of the commented exams for the MIR exam of the
previous year. We collected 10 books (from 2010 and between 2012-2020) which
were publicly available in their website444https://www.curso-mir.com. They
contained a total of 2,243 questions with associated explanations written by
specialist doctors. Being a professionally produced resource, the explanations
are written following a clear structural and focused pattern.

After cleaning and pre-processing the data, selecting the questions including
clinical cases left a total of 778 documents. Table 2 shows that that the
number of words per entry is slightly higher in MIR Asturias (250 words vs.
185 in _CasiMedicos_) and that the most repeated specialty is, in this case,
Digestive.

CasiMedicos | Full | Clinical Cases
---|---|---
#docs | 1,561 | 575
#words | 254,655 | 108,497
#word avg. in docs | 163 | 185
#docs top speciality | Pediatrics (42) | Pediatrics (41)
MIR Asturias | Full | Clinical Cases
#docs | 1,971 | 778
#words | 450,495 | 199,161
#word avg. in docs | 228 | 258
#docs top speciality | Digestive (157) | Digestive (72)
Table 2: Statistics of the _CasiMedicos_ and _MIR Asturias_ datasets. The
_Clinical Cases_ subset includes our annotations to identify the spans
referring to the explanation of the correct answer.

Some issues worth mentioning which differentiate the datasets obtained from
CasiMedicos and MIR Asturias. First, the writing style is slightly more
spontaneous and heteregeneous in CasiMedicos. This is because, unlike MIR
Asturias, CasiMedicos is the result of a voluntary effort by medical doctors
in Spain to provide the comments to the yearly MIR exams. Thus, the length of
the explanations can vary across questions, as well as the structure of the
explanation itself, which does not obey a clear writing pattern or predefined
methodology. In contrast, the comments given for the MIR Asturias data are
much more structured, often with explicit clues as to which of the possible
answers the explanation is referring to. This may also explain why the average
length of the MIR Asturias documents is higher than in CasiMedicos. Second,
the set of questions included in each of the datasets is different. As a
community-based voluntary effort, and unlike MIR Asturias, CasiMedicos does
not include comments for every question in the MIR exams published every year
by the Spanish Ministry of Health. Third, contrary to the data obtained from
_CasiMedicos_ dataset, we did not manage to obtain permission to share the
_MIR Asturias_ data, not even for research purposes. Hence, only CasiMedicos
is made publicly available555We repeatedly contacted MIR Asturias without any
success..

##  4 Materials and Methods

Since the apparition of Pre-trained Language Models (PLMs) [23], a commonly
used approach to solve many NLP tasks has been to adapt a PLM to the task at
hand by fine-tuning it using a task specific dataset. Pre-trained Language
Models encode information about language syntax and semantics, and the fine-
tuning phase provides the model with additional information specific to the
final target task. By adopting this approach, it is possible to generate
models that are customized for a particular task while simultaneously
leveraging the benefits of an already-existing, task-agnostic Language Model.

Supplementary Training on Intermediate Labeled-data Tasks (STILT) [41] is a
variation of the fine-tuning approach, whereby the pre-trained Language Model
(PLM) undergoes an initial fine-tuning on an intermediate task prior to being
fine-tuned on the final downstream target task. In this case the intermediate
task will be extractive QA where the goal is, given a question, to correctly
extract the textual span corresponding to the right answer.

Figure 1 presents a general overview of the system. We first fine-tune various
PLMs on intermediate labeled data, consisting of general QA Spanish datasets
(SQuaD-es and SQAC). This intermediate step may consist of fine-tuning on
either SQUAD-es or SQAC or on their concatenation. The final step is to then
fine-tune the model on the downstream final task, namely, on either the
_CasiMedicos_ or the _MIR Asturias_ datasets. In the rest of this section we
provide an account of the _fine-tuning datasets used for STILT_ and of the
_PLMs_ used in the experimentation.

Figure 1: General overview of the system

###  4.1 Intermediate Fine-tuning Datasets for Extractive QA

The Intermediate Fine-tuning Datasets are based on the English SQuAD
extractive Question Answering setting. While there are other interesting
datasets [15], SQuAD has been and remains the main benchmark for Reading
Comprehension, which is perhaps the reason for it being translated into
multiple languages, including Spanish, the target language in this paper.

SQuAD [12, 14] is a general domain reading comprehension dataset, consisting
of more than 100,000 questions posed by crowdworkers on a set of Wikipedia
articles, where the answer to every question is either a segment of text, or
span, from the corresponding reading passage, also known as the _context_.
While in SQuADv1 [12] every question had an associated textual span as an
answer, SQuADv2 [14] included an answer for 66% of the questions while the
remaining 33% were unanswerable.

Table 3 presents an example of the most relevant parts of a SQuAD document,
namely, the Question, Context and Answer. It can also be observed that the
answer’s length is 3 words. In fact, the extracted answers in SQuAD are in
general very short spans and quite to the point, averaging 3.2 words per
answer of which 35.9% correspond to named entities, 25% to noun phrases and
16.5% to numerical expressions [15].

Question |  In which R&B group was she the lead singer?
---|---
Context |  Beyoncé Giselle Knowles-Carter (born 4 September 1981) is an American singer, songwriter, producer and actress. Born and raised in Houston, Texas, she performed in several singing and dancing competitions as a child, and rose to fame in the late 1990s as the lead singer of the all-female R&B group Destiny ’s Child. Led by her father, Mathew Knowles, the group became one of the best-selling girl groups of all time. Her hiatus saw the release of Beyoncé’s debut album, Dangerously in Love (2003), which established her as a worldwide solo artist, won five Grammy Awards and featured the Billboard Hot 100 number one single ”Crazy in Love”.
Answer |  text: Destiny ’s Child, answer_start: 306
Table 3: An example of SQuAD-es dataset document translated to English.

As shown by Table 1, in this work we aim to solve the novel task of
extracting, from a context including also explanations about the wrong
answers, the textual span corresponding to the supporting explanatory argument
of the right answer. Thus, although we cast our task in a extractive QA
setting, our final objective is to extract the explanation for the right
answer. Therefore, in addition to the domain of our datasets and SQuAD (based
on Wikipedia text), perhaps the main difference between CasiMedicos and MIR
Asturias with respect SQuAD is the length of the answers.

In fact, the answer spans in SQuAD are typically much shorter than in
CasiMedicos and MIR Asturias. As it can seen in the SQuAD example given by
Table 3, answers in SQuAD are very short, averaging 3.2 words per answer,
compared to 32.02 words average per _explanation of the correct answer_ in
CasiMedicos. This is illustratd by the example in Table 3 where the answer,
_Destiny’s Child_ , a named entity, is much shorter than most of the
explanations to the correct answer annotated in CasiMedicos (e.g., Table 1
which consists of 24 words). Thus, while in SQuAD the answers are mostly named
entities, short noun phrases or numerical expressions, in CasiMedicos the
explanations span usually correspond to one or more full sentences (or at
least a full subordinate clause), making it a much more challenging task than
extracting very short answers, as in SQuAD.

In this paper we use two extractive Spanish QA datasets as intermediate fine-
tuning datasets in a STILT approach, namely, SQuAD-es and SQAC.

The SQuAD-es dataset [42] is the Spanish version of SQuAD [12, 14]. In order
to create the Spanish version of the dataset, Casimiro Pio et al. [42]
developed the Translate Align Retrieve (TAR) method to automatically translate
SQuAD into Spanish.

The Spanish Question Answering Corpus (SQAC) [38] is a general domain corpus
built for extractive QA with only answerable questions, as in SQuAD v1. It was
created from texts extracted from the Spanish Wikipedia, encyclopedic
articles, newswire articles from Wikinews, and the Spanish section of the
AnCora corpus [43]. It consists of 18,817 questions with the annotation of
their answer spans from 6,247 textual contexts, following the guidelines
originally developed to built SQuAD v1.1.

###  4.2 Model Building

For the experimentation various PLMs have been selected following certain
criteria such as: domain (medical vs general), size (base vs large) or
multilinguality (multilingual vs monolingual). The choice of models from the
general general domain, both multilingual and monolingual is based on
previously published results on the evaluation of Spanish language models
[44]. With respect to the PLMs fine-tuned or pre-trained for the medical
domain, we picked all the models available (up to our knowledge). The final
selection of PLMs consists of the following 12 models:

  * •

General Domain Multilingual: IXAmBERT [45], mDeBERTa-v3-base [46], XLM-RoBERTa
(XLM-R) base and large [47].

  * •

Medical Domain Multilingual: XML-R-Galén [48], eriBERTa [49]

  * •

General Domain Monolingual: IXABERTes-v2
(IXAes)666http://www.deeptext.eus/resources/ixabertes-v2.zip, RoBERTa-base-bne
(MarIA-base) and RoBERTa-large-bne (MarIA-large) [38], and BETO [50].

  * •

Medical Domain Monolingual: bsc-bio-ehr-es and bsc-bio-es [51].

Table 4 provides the main characteristics of the PLMs used in this paper,
including the corpus type and size on which they were trained, and technical
pre-training details such as the number of layers, the hidden size, number of
attention heads, the vocabulary and the number of parameters.

With respect to the _general domain PLMs_ , BETO and IXAmBERT are BERT-base
models pre-trained with both Masked Language Modeling (MLM) and Next Sentence
Prediction (NSP) [23]. BETO was trained for 2M steps in two different stages:
900K steps with a batch size of 2048 and maximum sequence length of 128, and
the rest of the training with batch size of 256 and maximum sequence length of
512. IXAmBERT was trained by executing 1M steps with 256 batch size and 512
sequence length.

IXAes and the MarIA models (base and large) are based on the BERT architecture
but following the RoBERTa procedure [52]: (i) trained only on the MLM task,
(ii) on larger batches (iii) on longer sequences and (iv), with dynamic mask
generation. IXAes performed 120.500 steps with 2048 batch size and sequence
length 512. However, the MarIA models opted instead for a batch of 2048 and
512 sequence length, but reducing the training to one epoch without dropout
[53].

Moving on to the multiligual models, XLM-R base and large were trained over
1.5M steps with batch 8192 and sequences of 512 length, while mDeBERTa [46] is
also based on RoBERTa but incorporating disentangled attention, gradient-
disentagled embedding sharing and, most importantly, replacing the MLM task
with replaced token detection [54]; mDeBERTa was trained following the XLM-
RoBERTa procedure but reducing the steps from 1.5M to 500K.

Finally, the _PLMs adapted to the medical domain_ : XLM-R-galén was further
pre-trained following the XLM-R method described above on a corpus of
unlabeled oncology clinical texts [48]. The bsc-bio and bsc-bio-ehr
monolingual models were pre-trained following the RoBERTa method, running the
training for 48 hours using Adam optimizer with a peak learning rate of
0.0005, 10,000 warm-up steps and a batch size of 2,048 sentences.

Thus, while the specific pre-training details and the corpora used to generate
the PLMs are substantially different the fine-tuning performed to evaluate the
models on the intermediate and final downstream task will follow the same
methodology.

Model | corpus | #words | L | H | A | V | #params
---|---|---|---|---|---|---|---
IXAmBERT | Wiki | 0.7B | 12 | 768 | 12 | 119K | 110M
mDeBERTa-v3 | CC-100 | 9.3B | 12 | 768 | 12 | 250K | 198M
XLM-R-base | CC-100 | 9.3B | 12 | 768 | 12 | 250K | 270M
XLM-R-large | CC-100 | 9.3B | 24 | 1024 | 16 | 250K | 550M
XLM-R-Galén | CC-100 + | 9.3B + | 12 | 768 | 12 | 250K | 270M
| EHRSp | 64.4M |  |  |  |  |
eriBERTa | MedSpEn | 900M | 12 | 768 | 12 | 50K | 125M
IXAes | OSCAR | 25B | 12 | 768 | 12 | 50K | 125M
MarIA-base | BNE | 135B | 12 | 768 | 12 | 50K | 125M
MarIA-large | BNE | 135B | 24 | 1024 | 16 | 50K | 350M
BETO | Opus,Wiki | 3B | 12 | 768 | 12 | 30K | 110M
bsc-bio-ehr | EHRMedSp | 1.2B | 12 | 768 | 12 | 50K | 125M
bsc-bio | MedSp | 1.1B | 12 | 768 | 12 | 50K | 125M
Table 4: Language Models used in this paper. L: layer size; H: hidden size; A:
attention heads; V: vocabulary. Wiki: Wikipedia; CC-100: Common Crawl; OSCAR:
Open Super-large Crawled Aggregated coRpus; EHRSp: Spanish Electronic Health
Records (EHR); MedSpEn: Spanish and English Medical Corpus; EHRMedSp: Spanish
EHR and Medical Corpus; MedSp: Spanish Medical Corpus.

###  4.3 Evaluation

For evaluation we use the standard measures of precision, recall, and F1-score
used for the SQuAD benchmark as defined in formulae (1), where TPS =
_correctly identified sections_ , FPS = _incorrectly identified sections_
(marked by system and not present in the annotated gold standard) and FNS =
_false negatives_ , namely, present in the gold standard and not detected by
system.

$\begin{array}[]{lcl}{\scriptstyle Precision}&=&\frac{TPS}{TPS~{}+~{}FPS}\\\
{\scriptstyle Recall}&=&\frac{TPS}{TPS~{}+~{}FNS}\\\ {\scriptstyle
F-score}&=&\frac{2~{}*~{}Precision~{}*~{}Recall}{Precision~{}*~{}Recall}\end{array}$
(1)

In a QA extractive setting such as SQuAD, these metrics are computed over the
individual words in the prediction against those in the correct answer.
Precision corresponds to the ratio of the number of shared words to the total
number of words in the prediction, and recall is the ratio of the number of
shared words to the total number of words in the correct answer.

##  5 Experimental Results

Variant | SQuAD | CasiMedicos
---|---|---
| Question | Clinical Case + Question
| Context | Doctor’s Explanations
CQ#E | Answer | Correct Answer Explanation
| Question | Clinical Case + Question + Possible Answers
| Context | Doctor’s Explanations
CQP#E | Answer | Correct Answer Explanation
| Question | Question
| Context | Clinical Case + Doctor’s Explanations
Q#CE | Answer | Correct Answer Explanation
| Question | Question + Possible Answers
| Context | Clinical Case + Doctor’s Explanations
QP#CE | Answer | Correct Answer Explanation
Table 5: Experimental variants with _CasiMedicos_ and _MIR Asturias_. C:
Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P: Possible
Answers. Fragment before #: generated question; after #: generated context
from which to extract the correct explanation.

As explained in Section 4.2, we have experimented with twelve pre-trained
Transformer-based language models using the SQuAD and SQAC as intermediate
datasets and CasiMedicos and MIR Asturias as the final downstream task. In our
experimental setup, the first experiment involves directly fine-tuning the
PLMs on CasiMedicos and MIR Asturias. Additionally, three other approaches
were explored which involved using STILT prior to the final target task: using
either SQAC or SQuAD-es as intermediate tasks, or the combination of both
datasets. Details of the experimental settings are provided in the next
section.

###  5.1 Experimental Setup

In the SQuAD extractive Question Answering (QA) setting, the input to fine-
tune the PLMs contains at least the following document elements: the
_Question_ , the _Context_ and Answer. In order to cast our _correct answer
explanation extraction_ task as an extractive QA task, we simply experiment by
placing the document parts depicted in Table 1 (_Clinical Case, Question,
Possible Answers and Doctor’s Explanations_) as the _Question_ , _Context_ and
_Answer_ fields used in the SQuAD setting.

| CQ#E
---|---
Question |  An 18-year-old patient consulted for oedema. A complete blood test showed proteinuria of 8 g/day without microhaematuria, hypoalbuminaemia and hypercholesterolaemia with normal renal function. Corticosteroids were administered empirically. After one month, the clinical picture has completely disappeared. What is your diagnostic hypothesis?
| CQP#E
Question |  An 18-year-old patient consulted for oedema. A complete blood test showed proteinuria of 8 g/day without microhaematuria, hypoalbuminaemia and hypercholesterolaemia with normal renal function. Corticosteroids were administered empirically. After one month, the clinical picture has completely disappeared. What is your diagnostic hypothesis? _Amyloidosis. @@ IgA nephropathy or Berger’s disease. @@ Alport syndrome. @@ Nephropathy with minimal changes._
Context |  We describe a young patient with criteria for nephrotic syndrome with excellent response to corticosteroids. Without biopsy, the clinical presentation and evolution suggest a nephropathy with minimal changes, which usually presents with normal or slightly altered renal function together with nephrotic syndrome, and which in 85-90% of cases resolves with steroid treatment. The age is the only data that is a bit of a problem, since although it is the most common cause of idiopathic nephrotic syndrome in children and adolescents, it is usually recommended to perform a biopsy prior to treatment in those over 16 years of age, and in any case there is little discussion.
Answer |  We describe a young patient with criteria for nephrotic syndrome with excellent response to corticosteroids. Without biopsy, the clinical presentation and evolution suggest a nephropathy with minimal changes, which usually presents with normal or slightly altered renal function together with nephrotic syndrome
Table 6: _CasiMedicos_ variants in which the clinical case is included within
the question. The clinical case is marked in bold, the question in small
capitals, and the possible answers are in _italics_.

In this sense, we try 4 different combinations to use the clinical case (C),
question (Q), possible answers (P) and the explanations (E) as Question and
Context (the correct answer explanation A is always the answer) in a SQuAD
sense. These variants are described in Table 5. Thus, in the first variant the
question in the SQuAD scenario consists of the clinical case (C) and the
question (Q). In the second one, we add the possible answers (P), while for
the 4th we remove the clinical case (C); finally, in the third variant we
leave just the question (Q). Regarding the context from which to extract the
explanation of the correct answers, in the first two variants it includes only
the full explanations given in the comment written by the medical doctors (E)
while for the last two we also add the clinical case (C).

For illustration purposes, Table 6 provides an example of the variants CQ#E
and CQP#E in which the clinical case (C) is added together with the question
(Q) for CQ#E and also with the possible answers (P) in CQP#E. In both cases
the context consists of the doctor’s explanations (E) which includes the
_correct answer explanation_ (A) to be extracted.

Putting together the 12 PLMs, 4 datasets (SQuAD-es, SQAC, CasiMedicos and MIR
Asturias) and the 4 experimental variants just presented, we fine-tune the
models using the following procedure. We truncate the maximum total input
sequence length to 384 and the maximum length of an answer that could be
generated to 512. We fine-tuned the models for 20 epochs, set the random seed
for initalization to 42, and kept the default QA values for the other hyper-
parameters [23]. Adding up all the hours needed to train all the models using
a NVIDIA GPU V100 32GB, we get approximately 77 hours, leaving a carbon
footprint of 34.81 (lbs CO2e). The evaluation results are reported in the next
section.

###  5.2 Results

As we use the SQuAD-es and SQAC data to perform STILT for each of the models,
the final number of experimental results was too large ($\sim$400) to present
it here. Thus, in this section we report the results from the best two
multilingual (base and large versions) and monolingual models777There is not a
large version of the models adapted to the medical domain for Spanish. (from
the general and the medical domain, respectively). While Tables 7 and 8
provide the results obtained on the _CasiMedicos_ and MIR Asturias datasets,
respectively, all the results using all twelve models can be found in A.

| CM | SQAC+CM | SQES+CM | ALL | Avg.
---|---|---|---|---|---
IXAes | CQ#E | 68.18 | 67.15 | 65.90 | 69.74 | 67.74
CQP#E | 66.95 | 67.74 | 68.50 | 64.96 | 67.04
Q#CE | 62.34 | 67.93 | 64.53 | 60.43 | 63.80
QP#CE | 60.81 | 64.93 | 60.35 | 62.85 | 62.23
eriBERTa | CQ#E | 67.56 | 70.01 | 70.49 | 67.58 | 68.91
CQP#E | 67.61 | 65.55 | 67.90 | 68.61 | 67.41
Q#CE | 65.53 | 70.46 | 65.58 | 71.39 | 68.24
QP#CE | 67.34 | 64.43 | 64.51 | 70.14 | 66.60
mDeBERTa-v3 | CQ#E | 66.76 | 69.65 | 68.92 | 66.67 | 68.00
CQP#E | 58.16 | 68.32 | 66.33 | 66.10 | 64.72
Q#CE | 65.94 | 66.45 | 68.14 | 69.91 | 67.61
QP#CE | 57.55 | 65.15 | 65.50 | 70.18 | 64.59
XLM-R-large | CQ#E | 71.74 | 74.47 | 72.97 | 71.84 | 72.75
CQP#E | 69.49 | 71.95 | 73.08 | 71.76 | 71.57
Q#CE | 62.39 | 66.55 | 67.86 | 70.18 | 66.74
QP#CE | 57.49 | 69.48 | 73.80 | 70.09 | 67.71

Table 7: F1-scores (partial match) on the _CasiMedicos_ (CM) dataset variants
defined in Table 5. C: Clinical Case; Q: Question; E: Medical Doctor’s
Explanations; P: Possible Answers. Fragment before #: generated question;
after #: generated context from which to extract the correct explanation.
Scores in bold: Best overall result; scores underlined: best result per
dataset used for fine-tuning. SQES: SQUAD-es.

| MA | SQAC+MA | SQES+MA | ALL | Avg.
---|---|---|---|---|---
IXAes | CQ#E | 82.37 | 86.56 | 86.34 | 84.69 | 84.99
CQP#E | 79.98 | 86.85 | 84.39 | 84.80 | 84.00
Q#CE | 81.69 | 83.34 | 81.58 | 79.11 | 81.43
QP#CE | 79.14 | 78.02 | 81.15 | 78.95 | 79.31
eriBERTa | CQ#E | 84.92 | 85.89 | 85.25 | 84.50 | 85.14
CQP#E | 82.66 | 85.45 | 84.33 | 85.42 | 84.46
Q#CE | 82.08 | 83.70 | 84.14 | 87.03 | 84.23
QP#CE | 80.27 | 82.56 | 84.29 | 83.08 | 82.55
mDeBERTa-v3 | CQ#E | 79.36 | 82.57 | 79.40 | 82.23 | 80.89
CQP#E | 77.82 | 81.51 | 77.32 | 80.80 | 79.36
Q#CE | 78.26 | 79.51 | 79.99 | 79.87 | 79.40
QP#CE | 71.49 | 72.19 | 74.39 | 73.72 | 72.94
XLM-R-large | CQ#E | 82.20 | 84.30 | 85.92 | 82.27 | 83.67
CQP#E | 81.25 | 83.63 | 79.68 | 82.78 | 81.83
Q#CE | 81.03 | 80.66 | 82.89 | 80.07 | 81.16
QP#CE | 74.49 | 77.15 | 77.46 | 79.36 | 77.11

Table 8: F1-scores (partial match) on the _MIR Asturias_ (MA) dataset variants
defined in Table 5. C: Clinical Case; Q: Question; E: Medical Doctor’s
Explanations; P: Possible Answers. Fragment before #: generated question;
after #: generated context from which to extract the correct explanation.
Scores in bold: Best overall result; scores underlined: best result per
dataset used for fine-tuning. SQES: SQUAD-es.

By looking at the results on _CasiMedicos_ , reported in Table 7, the first
interesting piece of information is that the best overall performance is
obtained by a multilingual model not adapted to the medical domain, namely,
XLM-RoBERTa-large (almost 4 points F1-score averaged across the 4 fine-tuning
settings over the second best model, eriBERTa, specifically developed for the
medical domain). Second, while eriBERTa is slightly better than IXAes and
mDeBERTa-v3-base, differences are relatively small. Third, the best setting
for this task in _CasiMedicos_ is CQ#E, in which the question includes both
the clinical case and the question (the context corresponds to the
explanations written by the medical doctors). In general, including the
clinical case as part of the question seems to be beneficial. Fourth, there is
not a pattern in the effect of the different intermediate fine-tunings (SQAC,
SQuAD-es or SQAC+SQuAD-es); while using STILT by means of general domain QA
datasets always helps for the downstream task, results are not conclusive as
to which of the datasets provides the largest gains.

The results obtained with _MIR Asturias_ leave a different picture. Thus, by
looking at Table 8, we can see that the model obtaining the best average score
is eriBERTa, although perhaps the most surprising result is the strong
performance of IXAes, a relatively small general domain model for Spanish.
Both eriBERTa and IXAes are slightly better than XLM-RoBERTa-large ($\sim$1.4
points in F-score) while clearly outperforming mDeBERTa-v3-base ($\sim$4
points in F-score).

As for _CasiMedicos_ , the best dataset variant on average remains CQ#E and it
also helps including the clinical case in the question. Finally, as it was
also the case in _CasiMedicos_ , while a previous fine-tune with SQAC, SQuAD-
es or their combination is beneficial, there is not a systematic winner as to
which QA dataset is the best for the majority of the cases. If we were to
choose one, we would probably choose SQAC, as it produces the largest gains
with the best dataset variant (CQ#E) for most of the models.

To conclude, it is worth noting that the results are substantially better for
_MIR Asturias_. Several possible explanations are discussed in the next
section.

##  6 Discussion

In this section we discuss and analyze the main results reported in the
previous section. More specifically, we will first analyze the possible
reasons to explain the large difference in performance when we compare the
results on the _CasiMedicos_ and _MIR Asturias_ datasets. Furthermore, in
Section 6.2 we will discuss the most difficult cases we encounted in the
datasets.

###  6.1 CasiMedicos vs MIR Asturias

As illustrated by the results reported in Section 5, the models we fine-tuned
and evaluated in the same setting obtain substantially better results for _MIR
Asturias_. For example, if we average the results of the 4 best systems
discussed, for _MIR Asturias_ the models obtain an average F-score of 81.40
while for _CasiMedicos_ the score corresponds to 67.23, namely, 14.17 points
lower. After a manual inspection of the _CasiMedicos_ and _MIR Asturias_
training data, we formulate two hypotheses to explain this phenomenon: (1) the
length of the explanations of the correct answer, our task objective, is
longer in the _CasiMedicos_ dataset, making the task more difficult to learn;
(2) the explanations in the _MIR Asturias_ dataset are generated following a
much more structured method, which makes it easier to identify the explanation
of the correct answer.

In order to test the first hypothesis, we calculated the average length of the
explanations of the correct answers for both datasets. Table 9 shows that the
correct answers’ explanations are in fact shorter in _CasiMedicos_ , which
means that this hypothesis is not valid.

Explanation length | CasiMedicos | MIR Asturias
---|---|---
1 | 51.13% | 35.89%
2 | 38.63% | 31.62%
3 | 4.54% | 13.67%
more than 3 | 5.68% | 18.80%
Table 9: Length of explanations of correct answers, in number of sentences,
for the _CasiMedicos_ and _MIR Asturias_ datasets.

With respect to our second hypothesis, that the explanations in the _MIR
Asturias_ dataset might be written in a more structured manner, the analysis
of the two datasets has led us to conclude that this is indeed the case. If we
examine Table 10, which provides different explanations to the same question,
one given in the _MIR Asturias_ and the other one in the _CasiMedicos_
dataset, it is possible to observe that _MIR Asturias_ follows a rather clear
structured pattern where the spans of the explanations are systematically
marked and linked to their corresponding possible answers (expressions marked
in bold in the explanations in Table 10).

CasiMedicos
---
E |  We describe a young patient with criteria for nephrotic syndrome with excellent response to corticosteroids. Without biopsy, the clinical presentation and evolution suggest a nephropathy with minimal changes, which usually presents with normal or slightly altered renal function together with nephrotic syndrome, and which in 85-90% of cases resolves with steroid treatment. The age is the only data that is a bit of a problem, since although it is the most common cause of idiopathic nephrotic syndrome in children and adolescents, it is usually recommended to perform a biopsy prior to treatment in those over 16 years of age, and in any case there is little discussion.
A |  We describe a young patient with criteria for nephrotic syndrome with excellent response to corticosteroids. Without biopsy, the clinical presentation and evolution suggest a nephropathy with minimal changes, which usually presents with normal or slightly altered renal function together with nephrotic syndrome.
MIR Asturias
E |  Patient with a nephrotic syndrome that responds to corticosteroids, characteristic of minimal change disease (answer 4 correct). The only data that can make us doubt is the age of the patient since minimal change disease is characteristic of children from 2 to 6 years old. However, it occurs in other age groups including the elderly. Although amyloidosis produces nephrotic syndrome, it does not respond to corticosteroids (incorrect answer 1), and there is no data in the clinical case that makes us suspect primary amyloidosis (plasma cell dyscrasia) or secondary amyloidosis (uncontrolled chronic inflammatory disease). IgA nephropathy or Alport syndrome are not characterized by producing nephrotic syndrome or responding to treatment with corticosteroids (answer 2 and 3 incorrect).
A |  Patient with a nephrotic syndrome that responds to corticosteroids, characteristic of minimal change disease (answer 4 correct).
Table 10: Comparing the explanations given for the same question (Table 6) in
both the _CasiMedicos_ and MIR Asturias datasets.

In order to find out whether these structural patterns appearing in _MIR
Asturias_ are actually influencing the results obtained by the models for this
dataset, we performed the following experiment. First, we manually removed
every expression such as “(answer 4 correct)” and so on appearing in the _MIR
Asturias_ dataset. Second, we fine-tuned and evaluated the models with this
new _MIR Asturias_ version (_unstructured_) and compared the results with
those reported in Table 8. Thus, in Table 11 we can see the averaged results
of the models for both versions of _MIR Asturias_ and for _CasiMedicos_. It is
quite clear that by removing the patterns from the explanations in the _MIR
Asturias_ data the results obtained are similar to those of _CasiMedicos_.
This suggests that models were somehow being helped by such patterns in order
to learn how to extract the explanation of the correct answer.

| MIR Asturias | MIR Asturias (u) | CasiMedicos
---|---|---|---
IXAes | 84.99 | 66.56 | 67.74
eriBERTa | 85.14 | 70.97 | 68.91
mDeBERTa-v3 | 80.89 | 62.84 | 68.00
XLM-R-large | 83.67 | 67.10 | 72.75

Table 11: Comparison in terms of averaged results (across MA, SQAC+MA,
SQuADES+MA, SQAC+SQuADES+MA) for the CQ#E variant as described in Table 7. MIR
Asturias (u): unstructured.

Motivated by these results, we performed an error analysis and found out that
those language models fine-tuned on the _MIR Asturias - Structured_ data
suffer to extract the correct explanations in the following two cases:

  1. 1.

When pointers to explicit patterns are absent in the context from which to
extract the correct explanation. After all, not every single document in the
_MIR Asturias_ dataset is structured in this manner. For these cases the model
does not provide any answer, it just returns either the entire context or some
random text between brackets.

  2. 2.

When more than one explanation is marked as correct because the question is
about which of the possible answers is FALSE (as illustrated by Table 12), the
models often return the fragment of the explanations in which the word correct
appears.

Q |  A 45-year-old man, previously healthy, develops a fever and acute gastroenteritis with severe liquid diarrhoea. Blood tests show sodium 140 mmol/L, potassium 3.2 mmol/L, chlorine 85 mmol/L and bicarbonate 38 mmol/L. Arterial pH is 7.60 and arterial pCO2 42 mmHg. Arterial pH is 7.60 and arterial pCO2 42 mmHg. _Which of the following statements is FALSE?_
---|---
E |  Given the elevated pH, with elevated bicarbonate, and normal pCO2, the patient has metabolic alkalosis (answer 1 correct). The anion GAP (140 - 85 - 38) is 17 and therefore elevated (answer 3 correct), and the picture is accompanied by low chloride and potassium due to digestive losses from diarrhoea (answer 4 correct). The difficulty with this question is that most patients with diarrhoea have metabolic acidosis and hypokalaemia, yet this patient has metabolic alkalosis. This is because some forms of diarrhoea, especially ionic diarrhoea, produce isotonic losses with chlorine content (chlorine-losing diarrhoea) and this induces metabolic alkalosis through chlorine loss, and through volume loss with secondary aldosterone activation. The loss of chlorine explains why the GAP anion is increased. Apart from this detail, the patient has a normal pCO2 (between 35 and 45) and therefore does not have any form of compensation, let alone an acidosis in which pCO2 should be elevated (answer 2 incorrect).
A |  The loss of chlorine explains why the GAP anion is increased. Apart from this detail, the patient has a normal pCO2 (between 35 and 45) and therefore does not have any form of compensation, let alone an acidosis in which pCO2 should be elevated (answer 2 incorrect).
Table 12: An example of _MIR Asturias_ dataset where the task is to extract
the explanation of the incorrect answer.

These results led us to conclude that learning extractive QA in our setting
using the _unstructured_ version of the datasets is the most interesting
strategy to learn models able to generalize for the _explanation extraction_
task. In other words, the explicit pointers about where the explanation of the
correct answer is located provide spurious clues to the language models, with
the additional undesired effect of artificially inflating the results.

###  6.2 Impact of Intermediate Fine-tuning with QA Datasets from the General
Domain

In a final exercise of error analysis, we wanted to provide a qualitative
explanation of the kind of improvements derived from performing an
intermediate fine-tuning using the general domain QA datasets, namely, SQAC or
SQuAD-es. By analyzing the predictions of the models trained with or without
SQAC before the final fine-tuning step on _MIR Asturias_ , we observed the
following points. If fine-tuned with SQAC (summarized in Table 13):

  1. 1.

IXAes: fewer _empty_ answers and more _exact_ matches.

  2. 2.

eriBERTa: fewer _incomplete_ 888We consider an answer incomplete when it is
too short. For instance: correct), the, or some. answers and more _exact_
matches.

  3. 3.

mDeBERTa-v3-base: fewer _empty_ and _incomplete_ answers.

  4. 4.

XLM-R-large: fewer _empty_ answers and more _exact_ matches.

| Less Empty | Less Incomplete | More Exact
---|---|---|---
IXAes | ✓ |  | ✓
eriBERTa |  | ✓ | ✓
mDeBERTa-v3 | ✓ | ✓ |
XLM-R-large | ✓ |  | ✓
Table 13: Improvements on the MIR Asturias data when SQAC is used for
intermediate fine-tuning for the CQ#E variant.

Summarizing, in order to generalize better in the _correct answer’s
explanation extraction_ task it is better to avoid explicit clues that the
models focus on. In this sense, this turns out to be a good result for
_CasiMedicos_ , as it is written in a more spontaneous and unstructured manner
than MIR Asturias. Furthermore, results show that using SQAC or SQuAD-es in a
STILT setting helps to systematically improve the results, in particular by
making the models better in avoiding empty and incomplete answers and boosting
the number of exact matches.

##  7 Concluding Remarks

In this paper we present a novel dataset based on Spanish MIR exams which
exhibit characteristics not available in any previous work. More specifically,
it includes explanatory arguments for the correct answer and also
argumentative structure to explaning why the remaining answers are incorrect.
Furthermore, while most of the QA-related datasets in the medical domain are
written in English, in our case it has been written in Spanish by medical
doctors.

The MIR-based datasets (_CasiMedicos_ and _MIR Asturias_) introduced in this
paper include the clinical case, the correct answer, the multiple-choice
questions and the annotated explanations result of an evidence-based
deliveration process. We publicly release the annotated _CasiMedicos_ set to
encourage research on this topic and facilitate reproducibility of
results999https://github.com/ixa-ehu/antidote-casimedicos.

The MIR-based datasets allow us to propose a novel extractive task which,
taking into account the explanations written for both correct and incorrect
answers, consists of _identifying the explanation of the correct answer_. In
other words, the models should respond to medical questions regarding the
correct answer in a multiple-choice setting, by providing the piece of text in
the context which explains why a given answer is correct. Experimentally our
new benchmark casts this novel task within the extractive QA, SQuAD-style,
paradigm. By doing so we are able to automatically evaluate the performance of
Pre-trained Language Models while avoiding any costly manual evaluation that
may require medical experts.

Experimental results demonstrate that, compared to SQuAD, our task is more
complex, especially due to the long length of the spans of text to be
extracted (explanations for the correct answer in _CasiMedicos_ averaging
32.02 words length vs 3.2 words per answer in SQuAD). A comprehensive
evaluation of Spanish language models shows that often multilingual non-
specialized models outperform monolingual ones, even those which have been
pre-trained with large amounts of medical texts. Still, reported results show
that our novel dataset and approach can be an effective technique to help
medical practitioners in identifying relevant evidence-based explanations for
medical questions.

Finally, the unique characteristics of the _CasiMedicos_ dataset released with
this work may facilitate a number of research lines involving discriminative
and generative work on argumentation, explanability and truthfulness, to name
but a few. Furthermore, by harnessing recent advances in label projection for
sequence labelling tasks [55, 56] future work may be addressed both from a
monolingual and multilingual point of view.

## Acknowledgements

We thank the CasiMedicos Proyecto MIR 2.0 for their permission to share their
data for research purposes. This work has been partially supported by the HiTZ
center and the Basque Government (Research group funding IT-1805-22). We also
acknowledge the funding from the following
MCIN/AEI/10.13039/501100011033 projects: (i) Antidote (PCI2020-120717-2), and
by European Union NextGenerationEU/PRTR; (ii) DeepKnowledge
(PID2021-127777OB-C21) and ERDF A way of making Europe; (iii) Disargue
(TED2021-130810B-C21) and European Union NextGenerationEU/PRTR. Rodrigo Agerri
currently holds the RYC-2017-23647 fellowship
(MCIN/AEI/10.13039/501100011033 and by ESF Investing in your future).

## References

  * Sackett et al. [1996] D. L. Sackett, W. M. C. Rosenberg, J. A. M. Gray, R. B. Haynes, W. S. Richardson,  Evidence based medicine: what it is and what it isn’t,  BMJ 312 (1996) 71 – 72.
  * Mayer et al. [2021] T. Mayer, S. Marro, E. Cabrio, S. Villata,  Enhancing evidence-based medicine with natural language argumentative analysis of clinical trials,  Artificial Intelligence in Medicine 118 (2021) 102098.
  * Beltagy et al. [2019] I. Beltagy, K. Lo, A. Cohan,  SciBERT: A Pretrained Language Model for Scientific Text,  in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3615–3620.
  * Lee et al. [2020] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang,  BioBERT: a pre-trained biomedical language representation model for biomedical text mining,  Bioinformatics 36 (2020) 1234–1240.
  * Gu et al. [2021] Y. Gu, R. Tinn, H. Cheng, M. R. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon,  Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing,  ACM Transactions on Computing for Healthcare (HEALTH) 3 (2021) 1 – 23\.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.,  Language models are few-shot learners,  Advances in neural information processing systems 33 (2020) 1877–1901.
  * Singhal et al. [2022] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al.,  Large Language Models Encode Clinical Knowledge,  arXiv preprint arXiv:2212.13138 (2022).
  * Luo et al. [2022] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, T.-Y. Liu,  BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining,  Briefings in bioinformatics (2022).
  * Phan et al. [2021] L. Phan, J. T. Anibal, H. T. Tran, S. Chanana, E. Bahadroglu, A. Peltekian, G. Altan-Bonnet,  SciFive: a text-to-text transformer model for biomedical literature,  ArXiv abs/2106.03598 (2021).
  * Lin et al. [2022] S. Lin, J. Hilton, O. Evans,  TruthfulQA: Measuring How Models Mimic Human Falsehoods,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 3214–3252.
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu,  Exploring the limits of transfer learning with a unified text-to-text transformer,  The Journal of Machine Learning Research 21 (2020) 5485–5551.
  * Rajpurkar et al. [2016] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang,  SQuAD: 100,000+ Questions for Machine Comprehension of Text,  in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392.
  * Fisch et al. [2019] A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, D. Chen,  MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension,  in: Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 1–13.
  * Rajpurkar et al. [2018] P. Rajpurkar, R. Jia, P. Liang,  Know what you don’t know: Unanswerable questions for SQuAD,  in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789.
  * Reddy et al. [2019] S. Reddy, D. Chen, C. D. Manning,  CoQA: A Conversational Question Answering Challenge,  Transactions of the Association for Computational Linguistics 7 (2019) 249–266.
  * Kwiatkowski et al. [2019] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al.,  Natural Questions: A Benchmark for Question Answering Research,  Transactions of the Association for Computational Linguistics 7 (2019) 453–466.
  * Yang et al. [2015] Y. Yang, W.-t. Yih, C. Meek,  WIKIQA: A Challenge Dataset for Open-Domain Question Answering,  in: Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2013–2018.
  * Lai et al. [2017] G. Lai, Q. Xie, H. Liu, Y. Yang, E. Hovy,  RACE: Large-scale ReAding Comprehension Dataset From Examinations,  in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 785–794.
  * Zellers et al. [2018] R. Zellers, Y. Bisk, R. Schwartz, Y. Choi,  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference,  in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 93–104.
  * Hovy et al. [2000] E. H. Hovy, L. Gerber, U. Hermjakob, M. Junk, C.-Y. Lin,  Question Answering in Webclopedia,  in: TREC, volume 52, pp. 53–56.
  * Moreda et al. [2011] P. Moreda, H. Llorens, E. Saquete, M. Palomar,  Combining semantic information in question answering systems,  Information Processing & Management 47 (2011) 870–885.
  * Bordes et al. [2014] A. Bordes, S. Chopra, J. Weston,  Question Answering with Subgraph Embeddings,  in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 615–620.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186.
  * Shao et al. [2019] T. Shao, Y. Guo, H. Chen, Z. Hao,  Transformer-Based Neural Network for Answer Selection in Question Answering,  IEEE Access 7 (2019) 26146–26156.
  * Jin et al. [2019] Q. Jin, B. Dhingra, Z. Liu, W. Cohen, X. Lu,  PubMedQA: A Dataset for Biomedical Research Question Answering,  in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 2567–2577.
  * Abacha et al. [2017] A. B. Abacha, E. Agichtein, Y. Pinter, D. Demner-Fushman,  Overview of the medical question answering task at TREC 2017 LiveQA.,  in: TREC, pp. 1–12.
  * Vilares and Gómez-Rodríguez [2019] D. Vilares, C. Gómez-Rodríguez,  HEAD-QA: A Healthcare Dataset for Complex Reasoning,  in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 960–966.
  * Abacha et al. [2019] A. B. Abacha, Y. Mrabet, M. Sharp, T. R. Goodwin, S. E. Shooshan, D. Demner-Fushman,  Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers.,  in: MedInfo, pp. 25–29.
  * Jin et al. [2021] D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, P. Szolovits,  What disease does this patient have? a large-scale open domain question answering dataset from medical exams,  Applied Sciences 11 (2021) 6421.
  * Pal et al. [2022] A. Pal, L. K. Umapathi, M. Sankarasubbu,  MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering,  in: Conference on Health, Inference, and Learning, PMLR, pp. 248–260.
  * Abacha et al. [2019] A. B. Abacha, C. Shivade, D. Demner-Fushman,  Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering,  in: Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 370–379.
  * Nentidis et al. [2020a] A. Nentidis, K. Bougiatiotis, A. Krithara, G. Paliouras,  Results of the seventh edition of the BioASQ Challenge,  in: Machine Learning and Knowledge Discovery in Databases: International Workshops of ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II, Springer, pp. 553–568.
  * Nentidis et al. [2020b] A. Nentidis, A. Krithara, K. Bougiatiotis, M. Krallinger, C. Rodriguez-Penagos, M. Villegas, G. Paliouras,  Overview of BioASQ 2020: The eighth BioASQ challenge on large-scale biomedical semantic indexing and question answering,  in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11, Springer, pp. 194–214.
  * Nentidis et al. [2021] A. Nentidis, G. Katsimpras, E. Vandorou, A. Krithara, L. Gasco, M. Krallinger, G. Paliouras,  Overview of BioASQ 2021: the ninth BioASQ challenge on large-scale biomedical semantic indexing and question answering,  in: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21–24, 2021, Proceedings 12, Springer, pp. 239–263.
  * Ngai et al. [2021] H. Ngai, Y. Park, J. Chen, M. Parsapoor,  Transformer-based models for question answering on COVID19,  arXiv preprint arXiv:2101.11432 (2021).
  * Yoon et al. [2022] W. Yoon, R. Jackson, A. Lagerberg, J. Kang,  Sequence tagging for biomedical extractive question answering,  Bioinformatics 38 (2022) 3794–3801.
  * Gu et al. [2021] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon,  Domain-specific language model pretraining for biomedical natural language processing,  ACM Transactions on Computing for Healthcare (HEALTH) 3 (2021) 1–23.
  * Gutiérrez-Fandiño et al. [2022] A. Gutiérrez-Fandiño, J. Armengol-Estapé, M. Pàmies, J. Llop-Palao, J. Silveira-Ocampo, C. P. Carrino, A. Gonzalez-Agirre, C. Armentano-Oller, C. Rodriguez-Penagos, M. Villegas,  Maria: Spanish language models,  Procesamiento del Lenguaje Natural, Revista nº 68 (2022) 39–60.
  * Máximo [2022] S. Máximo,  Supervised domain adaptation for extractive question answering in Spanish,  in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022), A Coruña, Spain, September 20, 2022.
  * Rosá et al. [2022] A. Rosá, L. Chiruzzo, L. Bouza, A. Dragonetti, S. Castro, M. Etcheverry, S. Góngora, S. Goycoechea, J. Machado, G. Moncecchi, et al.,  Overview of QuALES at IberLEF 2022: Question Answering Learning from Examples in Spanish (2022).
  * Phang et al. [2018] J. Phang, T. Févry, S. R. Bowman,  Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks,  ArXiv abs/1811.01088 (2018).
  * Casimiro Pio et al. [2019] C. Casimiro Pio, C.-j. Marta R., F. Jose A. R.,  Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering,  in: Proceedings of the 12th Conference on Language Resources and Evaluation, pp. 5115–5523.
  * Taulé et al. [2008] M. Taulé, M. A. Martí, M. Recasens,  Ancora: Multilevel annotated corpora for Catalan and Spanish.,  in: Lrec.
  * Agerri and Agirre [2023] R. Agerri, E. Agirre,  Lessons learned from the evaluation of Spanish Language Models,  Proces. del Leng. Natural 70 (2023) 157–170.
  * Otegi et al. [2020] A. Otegi, A. Agirre, J. A. Campos, A. Soroa, E. Agirre,  Conversational Question Answering in Low Resource Scenarios: A Dataset and Case Study for Basque,  in: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 436–442.
  * He et al. [2021] P. He, J. Gao, W. Chen, DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing, 2021\.
  * Conneau et al. [2019] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov,  Unsupervised Cross-lingual Representation Learning at Scale,  CoRR abs/1911.02116 (2019).
  * López-García et al. [2021] G. López-García, J. M. Jerez, N. Ribelles, E. Alba, F. J. Veredas,  Transformers for Clinical Coding in Spanish,  IEEE Access 9 (2021) 72387–72397.
  * de la Iglesia et al. [2023] I. de la Iglesia, A. Atutxa, K. Gojenola, A. Barrena,  Eriberta: A bilingual pre-trained language model for clinical natural language processing,  arXiv 2306.07373 (2023).
  * Cañete et al. [2020] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez,  Spanish Pre-Trained BERT Model and Evaluation Data,  in: PML4DC at ICLR 2020.
  * Carrino et al. [2022] C. P. Carrino, J. Llop, M. Pàmies, A. Gutiérrez-Fandiño, J. Armengol-Estapé, J. Silveira-Ocampo, A. Valencia, A. Gonzalez-Agirre, M. Villegas,  Pretrained Biomedical Language Models for Clinical NLP in Spanish,  in: Proceedings of the 21st Workshop on Biomedical Language Processing, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 193–199.
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  RoBERTa: A Robustly Optimized BERT Pretraining Approach,  in: ArXiv, volume abs/1907.11692.
  * Komatsuzaki [2019] A. Komatsuzaki,  One Epoch Is All You Need,  in: ArXiv.
  * Clark et al. [2020] K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning,  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,  in: ICLR.
  * García-Ferrero et al. [2022] I. García-Ferrero, R. Agerri, G. Rigau,  Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings,  in: Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6403–6416.
  * Yeginbergenova and Agerri [2023] A. Yeginbergenova, R. Agerri,  Cross-lingual Argument Mining in the Medical Domain,  ArXiv abs/2301.10527 (2023).

## Appendix A

In this appendix the results with every pre-trained language model (PLM) are
presented.

| CM | SQAC+CM | SQES+CM | ALL | Avg.
---|---|---|---|---|---
mDeBERTa-v3 | CQ#E | 66.76 | 69.65 | 68.92 | 66.67 | 68.00
CQP#E | 58.16 | 68.32 | 66.33 | 66.10 | 64.72
Q#CE | 65.94 | 66.45 | 68.14 | 69.91 | 67.61
QP#CE | 57.55 | 65.15 | 65.50 | 70.18 | 64.59
XLM-R-base | CQ#E | 63.94 | 71.62 | 65.62 | 67.91 | 67.27
CQP#E | 64.25 | 72.99 | 71.59 | 69.24 | 69.51
Q#CE | 62.39 | 66.55 | 67.86 | 70.18 | 66.74
QP#CE | 56.55 | 68.23 | 65.04 | 64.22 | 63.51
XLM-R-large | CQ#E | 71.74 | 74.47 | 72.97 | 71.84 | 72.75
CQP#E | 69.49 | 71.95 | 73.08 | 71.76 | 71.57
Q#CE | 62.39 | 66.55 | 67.86 | 70.18 | 66.74
QP#CE | 57.49 | 69.48 | 73.80 | 70.09 | 67.71
XLM-R-GALEN | CQ#E | 64.47 | 70.27 | 67.08 | 70.65 | 68.11
CQP#E | 60.37 | 63.49 | 64.79 | 70.22 | 64.71
Q#CE | 56.65 | 52.65 | 59.87 | 60.98 | 57.53
QP#CE | 53.67 | 54.97 | 60.74 | 58.12 | 56.87
eriBERTa | CQ#E | 67.56 | 70.01 | 70.49 | 67.58 | 68.91
CQP#E | 67.61 | 65.55 | 67.90 | 68.61 | 67.41
Q#CE | 65.53 | 70.46 | 65.58 | 71.39 | 68.24
QP#CE | 67.34 | 64.43 | 64.51 | 70.14 | 66.60

Table 14: F1-score results of the multilingual models on different
_CasiMedicos_ dataset variants where CM = Fine-tuning on _CasiMedicos_
dataset, SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es
dataset. C: Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P:
Possible Answers. Fragment before #: generated question; after #: generated
context from which to extract the correct explanation. SQES: SQUAD-es.

| CM | SQAC+CM | SQES+CM | ALL | Avg.
---|---|---|---|---|---
mDeBERTa-v3 | CQ#E | 32.53 | 39.75 | 36.14 | 36.14 | 36.14
CQP#E | 27.58 | 35.63 | 31.03 | 37.93 | 33.04
Q#CE | 28.73 | 27.58 | 29.88 | 35.63 | 30.45
QP#CE | 25.28 | 32.18 | 33.33 | 34.48 | 31.31
XLM-R-base | CQ#E | 25.30 | 42.16 | 28.91 | 33.73 | 32.52
CQP#E | 26.43 | 43.67 | 33.33 | 34.48 | 34.47
Q#CE | 27.58 | 32.18 | 29.88 | 35.63 | 31.31
QP#CE | 28.73 | 33.33 | 29.88 | 32.18 | 31.03
XLM-R-large | CQ#E | 37.34 | 40.96 | 40.96 | 40.96 | 40.05
CQP#E | 32.18 | 39.08 | 41.37 | 37.93 | 37.64
Q#CE | 27.58 | 32.18 | 29.88 | 35.63 | 31.31
QP#CE | 26.43 | 35.63 | 39.08 | 33.33 | 33.61
XLM-R-GALEN | CQ#E | 27.71 | 30.12 | 26.50 | 31.32 | 28.91
CQP#E | 25.28 | 27.58 | 22.98 | 28.73 | 26.14
Q#CE | 18.39 | 17.24 | 21.83 | 26.43 | 20.97
QP#CE | 19.54 | 22.98 | 24.13 | 21.83 | 22.12
eriBERTa | CQ#E | 34.93 | 34.93 | 37.34 | 33.73 | 35.23
CQP#E | 27.58 | 29.88 | 28.73 | 32.18 | 29.59
Q#CE | 29.88 | 39.08 | 28.73 | 41.37 | 34.76
QP#CE | 33.33 | 33.33 | 28.73 | 33.33 | 32.18

Table 15: Exact match results of the multilingual models on different
_CasiMedicos_ dataset variants where CM = Fine-tuning on _CasiMedicos_
dataset, SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es
dataset. C: Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P:
Possible Answers. Fragment before #: generated question; after #: generated
context from which to extract the correct explanation.

| CM | SQAC+CM | SQES+CM | ALL | Avg.
---|---|---|---|---|---
IXAmBERT | CQ#E | 64.47 | 64.89 | 67.40 | 66.11 | 65.71
CQP#E | 64.37 | 67.28 | 63.42 | 67.81 | 65.72
Q#CE | 64.69 | 68.49 | 61.91 | 64.13 | 64.80
QP#CE | 64.88 | 67.69 | 66.57 | 65.39 | 66.13
MarIA-base | CQ#E | 53.51 | 66.11 | 70.25 | 67.47 | 64.33
CQP#E | 53.04 | 67.37 | 73.82 | 65.94 | 65.04
Q#CE | 40.65 | 63.08 | 67.24 | 66.51 | 59.37
QP#CE | 38.80 | 59.51 | 66.68 | 66.82 | 57.95
MarIA-large | CQ#E | 59.18 | 68.75 | 67.69 | 67.21 | 65.70
CQP#E | 58.64 | 68.08 | 64.16 | 70.46 | 65.33
Q#CE | 50.40 | 67.74 | 69.24 | 63.55 | 62.73
QP#CE | 52.59 | 63.13 | 66.71 | 63.40 | 61.45
beto | CQ#E | 67.34 | 66.08 | 69.27 | 68.58 | 67.81
CQP#E | 65.71 | 67.22 | 69.75 | 69.33 | 68.00
Q#CE | 61.36 | 60.71 | 68.85 | 67.26 | 64.54
QP#CE | 62.92 | 62.55 | 67.36 | 66.13 | 64.74
IXAes | CQ#E | 68.18 | 67.15 | 65.90 | 69.74 | 67.74
CQP#E | 66.95 | 67.74 | 68.50 | 64.96 | 67.03
Q#CE | 62.34 | 67.93 | 64.53 | 60.43 | 63.80
QP#CE | 60.81 | 64.93 | 60.35 | 62.85 | 62.23
BSC-BIO-EHR | CQ#E | 61.99 | 66.18 | 72.05 | 69.64 | 67.46
CQP#E | 58.27 | 67.15 | 69.43 | 70.28 | 66.28
Q#CE | 60.55 | 66.11 | 65.42 | 66.29 | 64.59
QP#CE | 61.69 | 63.47 | 66.04 | 68.93 | 65.03
BSC-BIO | CQ#E | 63.87 | 66.66 | 69.84 | 72.30 | 68.16
CQP#E | 64.11 | 66.91 | 66.28 | 71.46 | 67.19
Q#CE | 65.44 | 66.53 | 69.79 | 68.66 | 67.60
QP#CE | 61.22 | 63.37 | 69.82 | 63.99 | 64.60

Table 16: F1-score results of Spanish monolingual models on different
_CasiMedicos_ dataset variants where CM = Fine-tuning on _CasiMedicos_
dataset, SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es
dataset. C: Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P:
Possible Answers. Fragment before #: generated question; after #: generated
context from which to extract the correct explanation.

| CM | SQAC+CM | SQES+CM | ALL | Avg.
---|---|---|---|---|---
IXAmBERT | CQ#E | 27.71 | 30.12 | 34.93 | 33.73 | 31.62
CQP#E | 33.33 | 28.73 | 27.58 | 36.78 | 31.60
Q#CE | 26.43 | 33.33 | 22.98 | 25.28 | 27.00
QP#CE | 27.56 | 32.18 | 29.88 | 31.03 | 30.16
MarIA-base | CQ#E | 26.50 | 26.50 | 37.34 | 34.93 | 31.31
CQP#E | 22.98 | 27.58 | 40.22 | 29.88 | 30.16
Q#CE | 11.49 | 25.28 | 33.33 | 33.33 | 25.85
QP#CE | 14.94 | 19.54 | 28.73 | 32.18 | 23.84
MarIA-large | CQ#E | 24.09 | 32.53 | 32.53 | 33.73 | 30.72
CQP#E | 22.98 | 37.93 | 31.03 | 33.33 | 31.31
Q#CE | 14.94 | 32.18 | 32.18 | 33.33 | 28.15
QP#CE | 20.68 | 31.03 | 34.48 | 28.73 | 28.73
beto | CQ#E | 30.12 | 27.71 | 36.14 | 32.53 | 31.62
CQP#E | 26.43 | 31.03 | 35.63 | 34.48 | 31.89
Q#CE | 22.98 | 25.28 | 32.18 | 32.18 | 28.15
QP#CE | 22.98 | 25.28 | 34.48 | 27.58 | 27.58
IXAes | CQ#E | 28.91 | 31.32 | 28.91 | 33.73 | 30.71
CQP#E | 31.03 | 31.03 | 28.73 | 25.28 | 29.01
Q#CE | 27.58 | 27.58 | 27.58 | 26.43 | 27.29
QP#CE | 28.73 | 29.88 | 24.13 | 26.43 | 27.29
BSC-BIO-EHR | CQ#E | 33.73 | 30.12 | 40.49 | 32.53 | 34.21
CQP#E | 28.73 | 32.18 | 39.08 | 32.18 | 33.04
Q#CE | 29.88 | 32.18 | 31.03 | 29.88 | 31.03
QP#CE | 26.43 | 28.73 | 25.28 | 31.03 | 27.86
BSC-BIO | CQ#E | 31.32 | 31.32 | 33.73 | 38.55 | 33.73
CQP#E | 33.33 | 35.63 | 36.78 | 37.93 | 35.91
Q#CE | 31.03 | 31.03 | 34.48 | 35.63 | 33.04
QP#CE | 27.58 | 29.88 | 32.18 | 31.03 | 30.16

Table 17: Exact match results of Spanish monolingual models on different
_CasiMedicos_ dataset variants where CM = Fine-tuning on _CasiMedicos_
dataset, SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es
dataset. C: Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P:
Possible Answers. Fragment before #: generated question; after #: generated
context from which to extract the correct explanation.

| MA | SQAC+MA | SQES+MA | ALL | Avg.
---|---|---|---|---|---
mDeBERTa-v3 | CQ#E | 79.36 | 82.57 | 79.40 | 82.23 | 80.89
CQP#E | 77.82 | 81.51 | 77.32 | 80.80 | 79.36
Q#CE | 78.26 | 79.51 | 79.99 | 79.87 | 79.40
QP#CE | 71.49 | 72.19 | 74.39 | 73.72 | 72.94
XLM-R-base | CQ#E | 78.39 | 83.52 | 81.98 | 83.83 | 81.93
CQP#E | 82.23 | 81.89 | 82.90 | 77.84 | 81.21
Q#CE | 81.03 | 80.66 | 82.89 | 80.07 | 81.16
QP#CE | 74.25 | 76.39 | 77.73 | 77.95 | 76.58
XLM-R-large | CQ#E | 82.20 | 84.30 | 85.92 | 82.27 | 83.67
CQP#E | 81.25 | 83.63 | 79.68 | 82.78 | 81.83
Q#CE | 81.03 | 80.66 | 82.89 | 80.07 | 81.16
QP#CE | 74.49 | 77.15 | 77.46 | 79.36 | 77.11
XLM-R-GALEN | CQ#E | 74.08 | 72.72 | 79.85 | 79.89 | 76.63
CQP#E | 71.72 | 76.47 | 78.42 | 82.02 | 77.15
Q#CE | 66.88 | 67.55 | 73.42 | 74.84 | 70.67
QP#CE | 70.19 | 68.92 | 74.91 | 71.50 | 71.38
eriBERTa | CQ#E | 84.92 | 85.89 | 85.25 | 84.50 | 85.14
CQP#E | 82.66 | 85.45 | 84.33 | 85.42 | 84.46
Q#CE | 82.08 | 83.70 | 84.14 | 87.03 | 84.23
QP#CE | 80.27 | 82.56 | 84.29 | 83.08 | 82.55

Table 18: F1-score results of the multilingual models on different _MIR
Asturias_ dataset variants where MA = Fine-tuning on _MIR Asturias_ dataset,
SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es dataset. C:
Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P: Possible
Answers. Fragment before #: generated question; after #: generated context
from which to extract the correct explanation.

| MA | SQAC+MA | SQES+MA | ALL | Avg.
---|---|---|---|---|---
mDeBERTa-v3 | CQ#E | 67.52 | 70.08 | 65.81 | 68.37 | 67.94
CQP#E | 65.81 | 67.52 | 65.81 | 65.81 | 66.23
Q#CE | 63.24 | 66.66 | 68.37 | 66.66 | 66.23
QP#CE | 59.82 | 60.68 | 59.82 | 58.11 | 59.60
XLM-R-base | CQ#E | 62.39 | 67.52 | 65.81 | 70.08 | 66.45
CQP#E | 64.95 | 68.37 | 67.52 | 63.24 | 66.02
Q#CE | 64.10 | 66.66 | 67.52 | 65.81 | 66.02
QP#CE | 59.82 | 63.24 | 59.82 | 61.53 | 61.10
XLM-R-large | CQ#E | 68.37 | 70.94 | 71.79 | 69.23 | 70.08
CQP#E | 65.81 | 70.94 | 65.81 | 68.37 | 67.73
Q#CE | 64.10 | 66.66 | 67.52 | 65.81 | 66.02
QP#CE | 59.82 | 63.24 | 62.39 | 62.39 | 61.96
XLM-R-GALEN | CQ#E | 52.13 | 53.84 | 60.68 | 59.82 | 56.61
CQP#E | 47.86 | 54.70 | 58.11 | 63.24 | 55.97
Q#CE | 37.60 | 45.29 | 52.99 | 53.84 | 47.43
QP#CE | 41.02 | 43.58 | 54.70 | 50.42 | 47.43
eriBERTa | CQ#E | 69.23 | 71.79 | 71.79 | 66.66 | 69.86
CQP#E | 69.23 | 68.37 | 69.23 | 70.94 | 69.44
Q#CE | 64.95 | 63.24 | 67.52 | 72.64 | 67.08
QP#CE | 64.95 | 67.52 | 69.23 | 65.81 | 66.87

Table 19: Exact match results of the multilingual models on different _MIR
Asturias_ dataset variants where MA = Fine-tuning on _MIR Asturias_ dataset,
SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es dataset. C:
Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P: Possible
Answers. Fragment before #: generated question; after #: generated context
from which to extract the correct explanation.

| MA | SQAC+MA | SQES+MA | ALL | Avg.
---|---|---|---|---|---
IXAmBERT | CQ#E | 82.66 | 86.14 | 83.86 | 83.54 | 84.05
CQP#E | 82.41 | 86.14 | 82.73 | 83.84 | 83.78
Q#CE | 78.83 | 82.32 | 79.30 | 80.84 | 80.32
QP#CE | 77.56 | 80.18 | 78.89 | 75.03 | 77.91
MarIA-base | CQ#E | 70.85 | 82.53 | 82.71 | 82.01 | 79.52
CQP#E | 75.52 | 83.79 | 81.97 | 81.47 | 80.68
Q#CE | 47.80 | 79.05 | 80.94 | 76.18 | 70.99
QP#CE | 52.95 | 76.09 | 75.87 | 74.93 | 69.96
MarIA-large | CQ#E | 79.98 | 83.66 | 84.80 | 82.26 | 82.67
CQP#E | 80.54 | 82.90 | 83.94 | 80.73 | 82.02
Q#CE | 76.26 | 80.94 | 75.18 | 81.13 | 78.37
QP#CE | 72.55 | 75.77 | 77.59 | 77.11 | 75.75
beto | CQ#E | 82.00 | 82.69 | 84.92 | 82.24 | 82.96
CQP#E | 82.03 | 83.39 | 83.32 | 83.25 | 82.99
Q#CE | 81.89 | 82.18 | 79.15 | 78.83 | 80.51
QP#CE | 76.73 | 80.55 | 79.93 | 80.24 | 79.36
IXAes | CQ#E | 82.37 | 86.56 | 86.34 | 84.69 | 84.99
CQP#E | 79.98 | 86.85 | 84.39 | 84.80 | 84.00
Q#CE | 81.69 | 83.34 | 81.58 | 79.11 | 81.43
QP#CE | 79.14 | 78.02 | 81.15 | 78.95 | 79.31
BSC-BIO-EHR | CQ#E | 79.47 | 83.82 | 83.17 | 81.61 | 82.01
CQP#E | 75.48 | 84.58 | 84.73 | 85.87 | 82.66
Q#CE | 73.99 | 81.17 | 79.18 | 84.37 | 79.67
QP#CE | 74.89 | 80.62 | 80.80 | 82.13 | 79.61
BSC-BIO | CQ#E | 82.74 | 81.56 | 83.56 | 83.47 | 82.83
CQP#E | 82.61 | 83.68 | 82.61 | 84.31 | 83.30
Q#CE | 79.92 | 82.78 | 82.40 | 82.42 | 81.88
QP#CE | 78.57 | 81.61 | 82.46 | 83.26 | 81.47

Table 20: F1-score results of Spanish monolingual models on different _MIR
Asturias_ dataset variants where MA = Fine-tuning on _MIR Asturias_ dataset,
SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es dataset. C:
Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P: Possible
Answers. Fragment before #: generated question; after #: generated context
from which to extract the correct explanation.

| MA | SQAC+MA | SQES+MA | ALL | Avg.
---|---|---|---|---|---
IXAmBERT | CQ#E | 61.53 | 70.08 | 64.95 | 62.39 | 64.73
CQP#E | 65.81 | 67.52 | 64.95 | 67.52 | 66.45
Q#CE | 58.11 | 61.53 | 58.11 | 58.11 | 58.96
QP#CE | 60.68 | 59.82 | 60.68 | 50.42 | 57.90
MarIA-base | CQ#E | 55.55 | 63.24 | 64.10 | 65.81 | 62.17
CQP#E | 58.97 | 64.95 | 61.53 | 62.39 | 61.96
Q#CE | 31.62 | 55.55 | 62.39 | 53.84 | 50.85
QP#CE | 36.75 | 56.41 | 54.70 | 53.84 | 50.42
MarIA-large | CQ#E | 61.53 | 70.08 | 72.64 | 68.37 | 68.15
CQP#E | 60.68 | 65.81 | 65.81 | 66.66 | 64.74
Q#CE | 46.15 | 58.11 | 54.70 | 61.53 | 55.12
QP#CE | 43.58 | 58.11 | 60.68 | 63.24 | 56.40
beto | CQ#E | 66.66 | 64.95 | 68.37 | 65.81 | 66.44
CQP#E | 64.10 | 62.39 | 65.81 | 67.52 | 64.95
Q#CE | 64.95 | 64.95 | 63.24 | 58.97 | 63.02
QP#CE | 59.82 | 64.95 | 63.24 | 58.97 | 61.74
IXAes | CQ#E | 66.66 | 71.79 | 70.94 | 67.52 | 69.22
CQP#E | 58.97 | 69.23 | 69.23 | 67.52 | 66.23
Q#CE | 64.95 | 67.52 | 60.68 | 60.68 | 63.45
QP#CE | 60.68 | 58.97 | 63.24 | 62.39 | 61.32
BSC-BIO-EHR | CQ#E | 61.53 | 67.52 | 69.23 | 60.68 | 64.74
CQP#E | 59.82 | 66.66 | 70.08 | 67.52 | 66.02
Q#CE | 56.41 | 62.39 | 63.24 | 62.39 | 61.10
QP#CE | 58.11 | 61.53 | 62.39 | 60.68 | 60.67
BSC-BIO | CQ#E | 64.95 | 64.10 | 64.95 | 66.66 | 65.16
CQP#E | 68.37 | 64.95 | 66.66 | 67.52 | 66.87
Q#CE | 64.10 | 64.95 | 64.95 | 63.24 | 64.31
QP#CE | 59.82 | 62.39 | 60.68 | 61.53 | 61.10

Table 21: Exact match results of Spanish monolingual models on different _MIR
Asturias_ dataset variants where MA = Fine-tuning on _MIR Asturias_ dataset,
SQAC = Fine-tuning on SQAC dataset, SQES = Fine-tuning on SQuAD-es dataset. C:
Clinical Case; Q: Question; E: Medical Doctor’s Explanations; P: Possible
Answers. Fragment before #: generated question; after #: generated context
from which to extract the correct explanation.