<EMAIL_ADDRESS>Walker Orr TBD 5 June 2023 28 Sept 2023 Orr (2023)
Enhancing Computer Science Education with Pair Programming and Problem Solving
Studios. aseecoed.

# Enhancing Computer Science Education with Pair Programming and Problem
Solving Studios

[  Electrical Engineering and Computer Science, George Fox University,
Newberg, OR, 97132, USA

###### Abstract

This study examines the adaptation of the problem-solving studio to computer
science education by combining it with pair programming. Pair programming is a
software engineering practice in industry, but has seen mixed results in the
classroom. Recent research suggests that pair programming has promise and
potential to be an effective pedagogical tool, however what constitutes good
instructional design and implementation for pair programming in the classroom
is not clear. We developed a framework for instructional design for pair
programming by adapting the problem-solving studio (PSS), a pedagogy
originally from biomedical engineering. PSS involves teams of students solving
open-ended problems with real-time feedback given by the instructor. Notably,
PSS uses problems of adjustable difficulty to keep students of all levels
engaged and functioning within the zone of proximal development. The course
structure has three stages, first starting with demonstration, followed by a
PSS session, then finishing with a debrief. We studied the combination of PSS
and pair programming in a CS1 class over three years. Surveys of the students
report a high level of engagement, learning, and motivation.

###### keywords:

Pair Programming, Problem Solving Studios, Computer Science, Pedagogy, Flipped
Classroom, Active Learning

††articletype: Original††volume: TBD††issue: TBD

##  1 Introduction

Pair programming is an eXtreme programming (XP) methodology (Beck, 2000) that
has seen some use in industry (Hannay et al., 2009). It involves two
programmers working together on a single problem and computer with one
programmer taking the role of a “driver” and the other in the role of
“navigator.” The “driver” operates the keyboard and directly writes the code
while the “navigator” observes and asks questions, critiquing and refining the
code and its design. The “navigator” is not passive, they watch for bugs and
defects, think of alternative designs, and look up related documentation and
resources. Though the effectiveness of pair programming is mixed (Hawlitschek
et al., 2022; Hannay et al., 2009), in some cases it has been shown to produce
higher quality code faster than solo programming (Williams et al., 2000). The
intention is that pair programming will help developers working together catch
mistakes and defects much faster than on their own.

For education, pair programming is compelling because it fits into the
paradigm of apprenticeship and distributed learning, the idea that “Knowledge
is commonly socially constructed, through collaborative efforts toward shared
objectives or by dialogues and challenges brought about by differences in
persons’ perspectives” (Salomon, 1997). Further, it has been shown to increase
student satisfaction, reduce student frustration, improve student’s tendency
to persist, and give students a sense of self-efficacy (Williams and Upchurch,
2001). The ICAP framework describes four modes of student engagement and
behavior, identifying the interactive mode as producing the highest level of
student cognitive engagement. Interactive modes of learning are believed to
produce deep, transferable knowledge (Chi and Wylie, 2014). Pair programming
fits within ICAP’s definition of interactive learning and hence has the
potential to produce robust, transferable, conceptual learning. Recently,
Hawlitschek et al. (2022) conducted a literature review and meta-study of pair
programming in education concluded that pair programming is important and
effective for students, especially beginners, but effective instructional
design was missing. Hence pair programming has been shown to have a lot of
potential as a teaching methodology but the details of how to implement it
correctly in a classroom has yet to be discovered.

We propose that the solution to effective instructional design for pair
programming in the classroom has been found in the Problem Solving Studio
(PSS) learning environment (Le Doux and Waller, 2016). PSS was designed to
teach biomedical engineering students to solve complex problems without having
to resort to rote memorization of procedures and algorithms. Students work in
teams of two to solve ill-defined problems in a public space, enabling
instructors to provide real-time feedback as they progress. A key feature of
PSS is _dynamic scaffolding_ , a targeted adjustment of problem difficulty to
keep students challenged but not discouraged. By increasing or decreasing the
difficulty on a per-team basis in real-time, as many students as possible can
be kept in the zone of proximal development. A lecture-based course will have
a difficult time matching the variety of levels that students are at since the
same lecture content and delivery are communicated to all the students. There
is good evidence that PSS improves students’ conceptual understanding (Le Doux
and Waller, 2016).

PSS and pair programming are a natural fit and the combination of the two
match the objectives and pedagogical needs of CS1 courses. For this reason,
this study specifically addresses the adaptation of PSS in conjunction with
pair programming for CS1 pedagogy. Two key objectives of a CS1 course are to
teach algorithmic problem solving skills and a specific programming language.
One of the challenges for expert instructors is that both problem solving and
language knowledge is so deeply ingrained that it is second-nature to the
instructors. Paradoxically, this high level of understanding means instructors
often have a difficult time communicating this knowledge since it is taken for
granted (Le Doux and Waller, 2016). Further, student ability and background
varies significantly in CS1 courses. However, the synthesis of PSS and pair
programming addresses these challenges and objectives directly, by giving
students a hands-on opportunity to develop problem solving and programming
language skills. PSS with the addition of pair programming, teaches
algorithmic problem solving through a cognitive apprenticeship environment
(Collins et al., 1987). Students learn from each other and are also guided by
the instructor or other teaching assistants. Peer learning is helpful since
students who are at similar levels of ability have recent experience with
similar problems. This means that they are often better at communicating those
solutions since they remember the details and particularities of both what
they found challenging and how they overcame those obstacles. Pair
programming’s ability to give rapid feedback helps students learn the syntax
and semantics of a programming language. Further it promotes pair
collaboration and problem solving. Pair programming has been shown to help
notice programmers solve problems can not handle on their own (Hawlitschek et
al., 2022). The combination of PSS and pair programming creates both an
apprenticeship and peer learning environment in which students develop both
problem solving and programming language skills.

##  2 Literature Review

There has been a significant amount of research into using pair programming
for educational purposes. The popularization of pair programming largely
started by Extreme Programming Explained (Beck, 2000). It promoted pair
programming, among other techniques, as a way of producing high quality code
faster than conventional methods. Williams and Upchurch (2001) combined
professionals with advanced undergraduates and found that “Experimental
results show pair-programming pairs develop better quality code faster with
only a minimal increase in pre-release programmer hours.” Further, they noted
that the programmers generally found that pair programming was more enjoyable
than programming alone.

Hannay et al. (2009) studied the effect of using pair programming in a CS1
course with 600 students. Students did their work with pair programming,
alternating driver and navigator roles. The results were better quality
programs and a significantly higher rate of completion for the course among
those that participated in pair programming.

Williams and Upchurch (2001) had students use pair programming for their work
in a web based programming course. The results of an anonymous survey conclude
that the 74% of students thought that they could solve any problem with their
partner and 84% believed they learned faster because of their partner.

Though there are some mixed results among studies as well. McDowell et al.
(2003) saw no improvement in midterm or programming scores, but did see an
improvement under some “holistic” scores. A large randomized trial of 1,530
undergraduates found that pair programming was virtually unrelated to all
measured outcomes. However, pair programming did appear to hurt the grades,
success, and likelihood of White students and had no effect on any other
demographic (Bowman et al., 2020). A study of persistence of CS students
suggests that pair programming might improve the persistence of women in CS
though the results were not statistically significant Werner et al. (2004).

Hannay et al. (2009) conducted a meta-analysis of 18 studies concluded that
has an overall positive impact on software quality and time, but did require
more total effort by the programmers involved. The studies were on a mix of
educational and professional teams. However, they noted that pair programming
offers a substantial benefit for junior developers, with their code
correctness increasing 73% on regular programs and 149% on more complex
programs.

Recently another meta-study was conducted on all the published articles on
pair programming for higher education from 2010 to 2020. The analysis
contained 61 articles and contained some important conclusions and
observations. First, pair programming is generally beneficial for student
learning, particularly for inexperienced students. Second, pair programming is
generally difficult to apply successfully in the classroom. They noted a lack
of research on effective instructional design for pair programming. Most
research on instructional design is on how to select the pairs, though the
results are mixed. In particular, Hawlitschek et al. (2022) identified that
the problem with pair programming is when two weak students are paired
together without much guidance.

##  3 Approach

PSS is a “flipped classroom” pedagogy in which pairs of students work together
to solve ill-structured, complex problems all while engaging in a critical
dialog with the instructor. This problem solving centered approach is intended
to match real-world problem solving and encourage deep, conceptual thinking,
and partner-based learning. As the pairs work on a problem, they will receive
on the spot feedback called a desk crit. In addition, the instructor may feel
the need to adjust the problem difficulty up or down to match the student’s
ability and progress (Le Doux and Waller, 2016). This real-time adjustment is
called a “dynamic scaffold” and is used to keep the students in their zone of
proximal development (ZPD) (Vygotsky and Cole, 1978). Overall, PSS is a
cognitive apprenticeship model of instruction, in which students learn by
practicing problem solving and receiving personalized feedback as they work.

Our method is to adapt PSS to use the practice of pair programming. In our
adaption, a series of shorter problems was found to be the most effective
approach. The series of problems keep the pairs on-track, focused, and within
their ZPD. This also provides more opportunities to employ the dynamic
scaffolding to adjust problem difficulty.

###  3.1 Problem Structure & Dynamic Scaffolding

One essential aspect to PSS is the problem structure and formulation. Rather
than using typical textbook problems, PSS challenges students by presenting
them with problems that are more ill-structured and complex (Le Doux and
Waller, 2016) according to Jonassen and Hung (2015)’s problem difficulty
scheme . These problems are designed and intended to have multiple legitimate
solutions (Jonassen and Hung, 2015).

In the original PSS for engineering, a single class was typically two hours in
length. During a single class, a student team was expected to finish between
one and three problems (Le Doux and Waller, 2016). This illustrates how the
problems are lengthy and challenging with possibly many “deadends”, sub-
problems, and an overall meandering path to a solution. By comparison, typical
textbook problems are far more formulaic, straightforward, and linear.

In our adaptation of PSS for CS education, the problems are typically
presented to the students with examples of program inputs and desired outputs.
For example, if the problem is to write a program to produce prime numbers,
the prompt for the student is minimal “Work with your partner to write a
program that produces prime numbers.” The prompt is paired with example input
and output, in this case a command-line interface:

    Enter the upper limit for primes: 10
    The primes under 10:
    2
    3
    5
    7

This example is sufficiently ill-structured and complex for a CS1 course
because it is solvable with their knowledge and skill level and allows for
multiple legitimate solutions. First, the pair must identify the problem and
challenges that need to be overcome to produce a solution. The decisions about
which control structures to use and how to use them or which functions would
be helpful to define or use are entirely up to the student pair. Each pair
must create a complete solution i.e. a program for the given problem from
scratch. Further, students may have to do some independent research on the
particular application area, in this example, on the relevant properties of
prime numbers to complete the task. In general, an example of input to a
program and corresponding desired output is essentially a bare minimum
specification for a program. In that sense, the problems are presented in an
ill-structured fashion, since they do not explicitly ask for particular
programming constructs or methodologies, only a desired goal. This gives the
student pairs the opportunity to navigate a large space of solutions while
keeping all the students in the class on the same task.

Another key aspect of PSS problems is complexity. The problem at the center of
a PSS is targeted at the more advanced students. This ensures a sufficiently
complex problem, however it will be too challenging for many of the students.
The way this issue is addressed is through “dynamic scaffolding”, the process
of instructors adjusting problems in real time (Le Doux and Waller, 2016).

If a team is stuck and unable to make any progress on a problem, the
instructor may adjust the problem difficulty down by making the problem less
complex, more structured, or both. This allows as many students as possible to
be within their ZPD (Vygotsky and Cole, 1978). The ZPD means a challenge is
appropriate for student learning, meaning that it is neither too challenging
to be discouraging nor too boring to be uninteresting. Dynamic scaffolding
means students are far more likely to be in their ZPD than traditional
lectures which are typically only suited for a subset of students.

Our application of PSS for CS extends this idea of dynamic scaffolding to the
problem selection and formulation. Rather than selecting a single problem for
the PSS, our adaptation utilizes between one and five problems. The typical
structure of our CS PSS is to have a “ladder” of problems, starting with a
relatively easy problem, then followed by moderately difficult problems, and
then finally ending with an advanced problem. All the problems are centered
around a single concept or methodology being taught, for example, looping
control structures.

We believe this “problem ladder” has some advantages over a single advanced
problem. First it allows students to work at their own pace and naturally fall
into their ZPD. Students will typically have both the experiences of
successfully solving a problem and the experience of being challenged by
another. This way students are encouraged by solving a problem and are also
made aware that there is still more to learn. This is particularly important
in CS1 classes since students typically have a diverse background regarding
prior programming experience. It is important to encourage and motivate the
inexperienced students while challenging the move advanced students. One
important aspect of the “problem ladder” is that instructors should clearly
communicate that students are not expected to solve all the problems. This
will help prevent under-performing students from becoming discouraged by
advanced problems.

Each problem on the “ladder” can be dynamically adjusted as well, which
enables the instructor to have a fine-grained control over problem difficulty.
For example, if a pair spent most of the class period solving the “easy”
problem, it is often more useful for them to revise an adjusted version of the
problem in the remaining time. That way they can spend their time focused on
solving the problem rather than on the contextual switch to another problem.
This dynamic adjustment is useful at both ends of the “ladder”. If the “easy”
problem is too difficult, it can be adjusted down. Likewise, if a pair of
advanced students solve all the problems in the ladder, the last problem can
be adjusted to be more difficult. Returning to the example of computing prime
numbers, one way to apply dynamic scaffolding to the problem is to change the
requirement of producing all the primes under a limit to instead write a
function that determines if a given number is prime or not. The complexity of
the problem is reduced because the initial version of the problem requires two
loops to solve, while the revised version only requires one. Also this
provides additional structure to the problem since the goal is made more
specific.

To ensure student motivation, participation in the PSS is part of the course
grade. Though the requirement is not onerous, all that is required of the
students is their prescience and a good-faith effort. Attendance to PSS
sessions are obviously key to student learning and the requirement to put in
some effort helps encourage them to climb up the problem ladder. Though, the
natural progression of problem difficulty and early success provides intrinsic
motivation. In practice we observed that students enjoyed solving problems and
were naturally motivated generally speaking.

###  3.2 Informal Assessment & Feedback

PSS applies the idea of a desk crit from architecture design studios, where
instructors give “informal formative assessment” to students through a
discussion of their work (Dinham, 1987). In the PSS framework, instructors
provide this feedback and assessment through asking unobtrusive, open-end
questions (Le Doux and Waller, 2016). For example, the dialog can be initiated
by asking questions such as “How are you doing?”, “What are you working on
now?”, or “Are you making progress?” can open a dialog that can provide
specific instruction, assistance, or feedback to the pair. The initial
question enables the instructor to quickly determine the status of the student
pair. From there more intentional and specific questions can be asked to
promote deeper thinking (Raths, 1967). These dialogues are effective
opportunities to discuss how the students solved a problem, what issues they
are stuck on, alternative solutions, get help on practical issues, starting
problem solving, conventions regarding design and communication of solutions,
ways to an improve a solution, clarify conceptual misunderstandings, and so
on. Also, it is natural during these dialogues to enact dynamic scaffolding by
adjusting the problem difficultly up or down based on how the pair is
performing. Further, these dialogues are a good time for individualized help.
Naturally over both the class period and the course of the semester, the need
for support diminishes and the pairs work effectively on their own.
Instructors can also get a sense of which topics or methodologies the entire
class is struggling with versus individual students. This feedback for the
instructor can help them improve their lectures or demonstration sessions as
well as the PSS sessions themselves. This means the course design can be
adjusted on-the-fly or improved for the future. If many pairs are struggling
with the same issues, the instructor can stop the PSS and give some brief
instruction or clarification to the entire class. What this feedback provides
is a means for the instructor to identify “troublesome knowledge” that the
students are struggling to learn. With these insights, the instructor can
provide targeted instruction and clarification to specifically address these
difficult topics (Perkins, 2006).

In our adaptation of PSS for CS, we incorporated the desk crit in two
different ways. First, these dialogues are engaged informally as the student
pairs work on problems. More time is focused on the pairs that were struggling
the most to provide extra support. For the more advanced students, desk crits
are more of a time to suggest improvements or alternative solutions. Secondly,
the desk crits are employed when a pair finishes a problem. This is a good
time to provide extra encouragement, feedback, improvements, etc. Further, it
is a good way to informally assess student progress and ability by tracking
both the number of problems a pair finishes and the amount of time they need.

In general, the desk crit is a good way to provide individualized instruction.
Struggling students can get help with the practical or conceptual problems
they have. More advanced students can be challenged to think more deeply about
the problem or improve the presentation or design of their solution. This kind
of individualized instruction corresponds well to the student’s ZPD.

Naturally an instructor can only provide feedback to one team at a time. To
handle larger class sizes, teaching assistants can be utilized to give
feedback and other guidance to student pairs. This was notable part of the
original PSS design (Le Doux and Waller, 2016), however did not find the use
of teaching assistants necessary for our adaptation.

###  3.3 Pair Programming Dyad

An essential component of PSS is the grouping of students into pairs called
dyads (Le Doux and Waller, 2016). The dyads are important since it gives each
student a partner to better navigate, interpret, and solve the problem
(Hutchins, 2020). This partnership facilitates students sharing knowledge and
expertise as well as providing each other with helpful critique. Moreover, the
instructor’s knowledge and skills, having reached an expert level of ability,
are tacit and difficult to articulate. Students however are at closer levels
of ability to each other and are in the process of learning the material which
can make them more suited to answering each other’s questions.

In the original PSS, students shared a 17” x 22” blotter pad as a publicly
visible problem solving space. One student writes on the pad while the other
partner observes, listens carefully, agrees or critiques the writer. After a
few minutes the pair exchanges the roles of writer and observer. The students
also negotiate the duration of the writer role (Le Doux and Waller, 2016).

Similarly, in computer science education and in industry, pair programming is
a well-established practice of team-based problem solving (Beck, 2000; Hannay
et al., 2009; Hawlitschek et al., 2022). The complementary pair of roles has
been shown to be an effective way of producing high-quality programs and is
generally enjoyable for both partners (Hannay et al., 2009; Williams et al.,
2000). The adaptation of PSS to CS replaces the blotter pad and roles of
writer and observer with pair programming, that is, a single computer and the
roles of driver and navigator. This closely matches this aspect of the design
of the original PSS while contextualizing it to CS education.

A recent large meta-study concluded that the effectiveness of pair programming
in education was mixed, but that was primarily due to a lack of effective
instructional structure and guidance (Hawlitschek et al., 2022). Our
adaptation of PSS seeks to address this problem, specifically with the
structure provided by the use of ill-structured, complex problems, dynamic
scaffolding, problem ladders, and desk crits.

Solving ill-structured, complex problems is arguably the centerpiece of PSS.
For the driver, the benefits of working on these problems is simply the
process of solving and writing a solution to the problem. However, the
navigator is particularly important and beneficial. Since the problems are
ill-structured, by definition there is a lot of research that needs to be
done. As the driver writes the code, the navigator can research possible
important components of a solution, key knowledge about the particular
problem, refer back to the course textbook, consult examples, and explore
alternative solutions. Additionally, the navigator can observe the problem
solving process of the driver. This is especially if the navigator is
relatively inexperienced compared to the driver. For most people, the process
of problem solving is difficult to communicate. The navigator has the
opportunity to learn by example, which directly ties into the apprenticeship
mode of learning that PSS encourages. This means the navigator is learning
from observing their partner, by listening to the feedback from the desk crit,
and by participating and communicating with their partner.

The problem ladder gives direction for each pair and naturally provides
opportunities for the partners to swap roles. Our instructors communicated and
emphasized that pairs exchange roles when they complete a problem. This allows
each member to get the experience and benefits of being both a “driver” and a
“navigator”. Further, the exchange of roles helps prevent either partner from
either dominating or withdrawing. Individuals who tend to dominate and be
assertive will benefit by performing the thoughtful, observation-centered role
of the navigator. Likewise, individuals who tend to be more passive will
benefit by performing the direct, active role of the driver.

One of the criticisms of pair programming for novice programmers is that it is
“the blind leading the blind” (Rosenberg and Stephens, 2008). However, the
combination of the problem ladder, desk crits, and dynamic scaffolding,
student pairs that are struggling can be quickly identified and put back on a
track to success. Within a few minutes of starting the PSS activity, it
becomes clear which pairs will be successful on their own and which need some
help or guidance. Further, if a large number of pairs are stuck, a quick
demonstration or example by the instructor can get the class back on track.
The identification of struggling students is much quicker than a traditional
lecture-based course. Typically the only time when struggling students are
identified is when they turn in their assignments which means it could take
days or weeks. In a PSS, these students can be found and helped within a
single class period.

Both PSS and pair programming are centered on learning through the exchange of
knowledge between peers in the context of a shared problem. The educational
benefit of pair programming is primarily from the verbalization of problem
solving it encourages. The details and roles of pair programming are not
necessarily where the benefits come from but rather the interaction and
discussion they facilitate (Hannay et al., 2009). This is likely the case for
PSS as well, which is based on constructive learning (Ruiz-Primo, 2011), the
benefit of PSS is the facilitated dialog between students and between student
and instructor. This is the central idea of combining PSS and pair
programming, to create dialog, both between students and between student and
instructor, around problem solving for CS. This dialog-centered mode of
learning allows students to construct their own knowledge of CS through
problem solving.

Year | Responses | PSS Useful | PSS Engaging | PSS Challenging | Partner Useful
---|---|---|---|---|---
2020 | 34 | 3.97 | 100% | 97.06% | 94.12%
2021 | 43 | 4.23 | 97.73% | 86.36% | 97.73%
2022 | 39 | 4.18 | 100% | 92.50% | 92.50%
Combined | 116 | 4.16 | 99.15% | 91.53% | 94.92%
Table 1: Survey data for CS1 over three years. “PSS Useful” is on a 1 to 5
Likert scale with 1 indicating not useful and 5 indicating very useful. The
other columns have a binary response and the value reflects the proportion of
with a positive answer. The four corresponding survey questions are in order
“In general, are the in-class activities useful to you?”, “Are the activities
engaging?”, “Are the activities challenging?”, and “Did you appreciate having
a partner?”.

###  3.4 Course Structure

The course we implemented our application of PSS for CS in CS1, CS2, and web
programming courses, but this study is focused on the application to CS1.
These courses followed the typical 3 credit hour format, 50 minute meetings 3
times per week. A similar format was used for each course. The first meeting
for the week is a demonstration session. A new concept or topic is discussed
and applied by the instructor. This typically means that the instructor solves
a problem by writing a program and testing it. For example, for a CS1 course,
a topic could be for-loops. The second meeting of the week is a PSS session.
Students are presented a “ladder” of problems related to that week’s topic. At
a minimum, they are expected to solve the first rung on the ‘ladder’ since all
the problems are related to the topic. The third meeting is either a debrief
of the recent PSS, another PSS, or both. The debrief consists of the
instructor solving the “second rung” problem while explaining each step. This
gives students who only solved the first problem an example of how to extend
their knowledge and skill. For more advanced students, they have the
opportunity to see an expert solve a problem in a way which is likely
different from how they solved it. From week to week (and even from course to
course), some problems are revisited but with new concepts and skills
available. This gives an opportunity to see how the process and solution
changes with the new material. In summary, the weekly structure of the course
is demonstration, PSS, and finally debrief.

PSS is a replacement for the traditional lecture-based pedagogy. Other aspects
of the course such as assignments and exams are independent of the PSS. For
our PSS adaptation, the same homework and exams were utilized. In the CS1
course, there are 10 homework assignments, an online textbooks with built-in
reading and programming assignments, 2 midterms, and a final exam. This is the
same structure of assignments and exams that was employed before the adoption
of PSS. However, the problems presented in the PSS parallel the student’s
homework assignments. The students’ experience with a type of problem proceeds
as: demonstration, PSS, debrief, and finally homework assignment. The homework
is done individually, however the students have the opportunity to learn from
their instructor and peers before engaging with the problem on their own.

##  4 Discussion

Year | Responses | Prefer PSS | Prefer Lecture | Prefer Both | Prefer Neither
---|---|---|---|---|---
2020 | 34 | 35.29% | 5.88% | 58.82% | 0.00%
2021 | 43 | 38.64% | 4.55% | 54.55% | 2.27%
2022 | 39 | 40.00% | 7.50% | 52.50% | 0.00%
Combined | 116 | 38.14% | 5.93% | 55.08% | 0.85%
Table 2: Survey data on which mode of instruction students prefer. The
question presented was “Which do you prefer?” The options for response were
“Lectures,” “In-class exercises and activities,” “Both,” and “Neither.”

Our adaptation of PSS for CS was implemented in a CS1 course and evaluated
over a three year period. In order to evaluate our implementation, voluntary
anonymous surveys were conducted to assess the student’s perception of its
usefulness. Students had two opportunities to fill out the survey over the
course of the 15 semester at about weeks 5 & 10\. The results on how students
viewed the PSS sessions can be found in Table 1. Overall the results of the
surveys are very positive. Averaged over the three years, 99.15% found PSS to
be engaging, 91.53% found it to be challenging, and 94.92% thought their
partners were useful. Furthermore, the average score of 4.16 for PSS
usefulness, on a Likert scale of 1 to 5, strongly supports the claim that the
PSS was helpful for student learning. This is notable considering the diverse
backgrounds of the students in terms of exposure to CS and computer
programming. At our institution, CS1 is a formal requirement or strongly
encouraged by a variety of majors including accounting, biology, finance, and
engineering in addition to computer science. These results appear to strongly
suggest that the combination of the problem ladder and dynamic scaffolding
generally kept students challenged and in their ZPD. Given the overall
positive responses, the PSS must have been effective at matching the
appropriate level of difficulty for the majority of students. It is worth
noting that since the data was collected via voluntary, anonymous surveys,
that this is a caveat to their strong results. Since the students had two
opportunities to fill out the survey and the responses are anonymous, the
statistics reported in Tables 1 & 2 likely aggregate over multiple responses
from a single student. Also, based on the number of responses, there were some
students who completed the courses but did not fill out the survey. However,
the survey results do match the anecdotes from instructors that students are
generally very engaged, asking good questions, completing problems, and are
apparently enjoying the experience.

Since the overall structure of the course included lectures, debriefs, and
demonstrations in addition to PSS, the survey included questions about what
part of the class was preferable. The options of lecture, PSS, both, or
neither were given as choices. The lecture option refers to all the class
periods that were not PSS sessions. The results can be found in Table 2.
Thankfully only one student selected “neither” across the three years. Only
5.93% preferred the lectures, while 39.14% preferred just PSS, and 55.08%
preferred the combination of both. These results suggest the importance of PSS
over simply lecturing but also the need for some combination of the two. The
demonstrations and debrief lectures were important for the students to learn
both essential aspects of the concepts and techniques as well as a chance to
see how an expert would use them. The combination of student application of
knowledge through PSS contrasted with an expert’s demonstration does match the
model of apprenticeship learning better than student application alone. The
importance of the two is reflected in the survey results.

A recent pair programming meta-study concluded, “There is little systematic
knowledge from meta-analyses or literature reviews on effective instructional
design for pair programming, which in fact is a base for effective learning"
(Hawlitschek et al., 2022). The results suggest that the combination of
problem solving plus dynamic scaffolding and the weekly demonstration-PSS-
debrief structure, are an effective instructional design for educational pair
programming. PSS for CS combines both the benefits of active learning along
with enough guidance to maintain a cohesive, effective pair. As the pair
programming meta-study observed, “students – at least, novices – usually need
instructional support to ensure the quality and success of collaborative
learning activities" (Hawlitschek et al., 2022). PSS for CS supplies this
supporting instructional structure.

Our results also indicate that students generally found working with a partner
to be beneficial. On average, 94.92% of the respondents indicated that their
partner was useful. This reflects other studies which found that working in
pairs was more enjoyable than working alone (Williams et al., 2000), enhanced
their learning (Williams and Upchurch, 2001), and lead to more persistence and
success (McDowell et al., 2002; Hannay et al., 2009).

The majority of research on course design for pair programming has focused on
how to form the pairs. Criteria have included student confidence, prior
experience, genders, consistency of mental models, or other personality traits
However the results of these studies were inconclusive to how best form the
student pairs (Hawlitschek et al., 2022). Our use of random assignment for
pair selection for each PSS session appears to have worked well according to
the results of Table 1. Altogether this might suggest that the most effective
pairing strategy is to vary the pairings. This aligns with the notion of
distributed learning, that learning is facilitated by the interactive of a
variety of perspectives.

##  5 Conclusion

Pair programming has long been a promising pedagogical tool but its
application to the classroom has seen mixed results. In particular,
instruction design for pair programming has seen little research. PSS however
is a natural fit for both CS education and for pair programming in particular.

PSS is an active learning pedagogy that involves student pairs solving
problems in class. It uses dynamic scaffolding to adjust the problem
difficulty to match student ability in order to keep them in their ZPD. PSS is
an apprenticeship model of learning that has been successful in engineering
education.

The adaptation of PSS for CS presented and studied here appears to be a good
solution to the problem of applying pair programming to the classroom Both the
“problem ladder” and dynamic scaffolding provide enough guidance and direction
for students of a variety of backgrounds and abilities. The active and
adaptive nature of the learning environment resulted in a large number of
students reporting to be engaged and challenged.

Further, students reported appreciating the weekly structure of the class as
well. By rotating through, demonstration, PSS, and debriefing, students were
able to see and apply new concepts each week. This provides the freedom and
engagement of active learning while avoiding the pitfall of too little
guidance for inexperienced or weak students.

PSS for CS combines a free and active learning environment with a deliberate
structure to keep students on track. This a fruitful middle ground we believe
students find refreshing while being highly educational. The highly positive
results from students, with the overwhelming majority finding PSS for CS
useful, engaging, and challenging, should encourage other educators to adapt
it to their classroom.

##  6 Acknowledgements

We thank Joe Le Doux for teaching us PSS at a KEEN workshop and for his
insights, comments, and edits when preparing this manuscript. We would also
like to thank the reviewers at ASEE for their thorough and helpful feedback.

## References

  * Beck (2000) K. Beck, _Extreme programming explained: embrace change_. addison-wesley professional, 2000.
  * Hannay et al. (2009) J. E. Hannay, T. Dybå, E. Arisholm, and D. I. Sjøberg, “The effectiveness of pair programming: A meta-analysis,” _Information and software technology_ , vol. 51, no. 7, pp. 1110–1122, 2009.
  * Hawlitschek et al. (2022) A. Hawlitschek, S. Berndt, and S. Schulz, “Empirical research on pair programming in higher education: a literature review,” _Computer Science Education_ , pp. 1–29, 2022.
  * Williams et al. (2000) L. Williams, R. R. Kessler, W. Cunningham, and R. Jeffries, “Strengthening the case for pair programming,” _IEEE software_ , vol. 17, no. 4, pp. 19–25, 2000.
  * Salomon (1997) G. Salomon, _Distributed cognitions: Psychological and educational considerations_. Cambridge University Press, 1997.
  * Williams and Upchurch (2001) L. Williams and R. L. Upchurch, “In support of student pair-programming,” _ACM SIGCSE Bulletin_ , vol. 33, no. 1, pp. 327–331, 2001.
  * Chi and Wylie (2014) M. T. Chi and R. Wylie, “The icap framework: Linking cognitive engagement to active learning outcomes,” _Educational psychologist_ , vol. 49, no. 4, pp. 219–243, 2014.
  * Le Doux and Waller (2016) J. M. Le Doux and A. A. Waller, “The problem solving studio: An apprenticeship environment for aspiring engineers.” _Advances in Engineering Education_ , vol. 5, no. 3, p. n3, 2016.
  * Collins et al. (1987) A. Collins _et al._ , “Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. technical report no. 403.” 1987.
  * McDowell et al. (2003) C. McDowell, B. Hanks, and L. Werner, “Experimenting with pair programming in the classroom,” in _Proceedings of the 8th annual conference on Innovation and technology in computer science education_ , 2003, pp. 60–64.
  * Bowman et al. (2020) N. A. Bowman, L. Jarratt, K. Culver, and A. M. Segre, “Pair programming in perspective: Effects on persistence, achievement, and equity in computer science,” _Journal of Research on Educational Effectiveness_ , vol. 13, no. 4, pp. 731–758, 2020.
  * Werner et al. (2004) L. L. Werner, B. Hanks, and C. McDowell, “Pair-programming helps female computer science students,” _Journal on Educational Resources in Computing (JERIC)_ , vol. 4, no. 1, pp. 4–es, 2004.
  * Vygotsky and Cole (1978) L. S. Vygotsky and M. Cole, _Mind in society: Development of higher psychological processes_. Harvard university press, 1978.
  * Jonassen and Hung (2015) D. H. Jonassen and W. Hung, “All problems are not equal: Implications for problem-based learning,” _Essential readings in problem-based learning_ , pp. 17–42, 2015.
  * Dinham (1987) S. M. Dinham, “An ongoing qualitative study of architecture studio teaching: Analyzing teacher-student exchanges. ashe annual meeting paper.” 1987.
  * Raths (1967) L. E. Raths, _Teaching for thinking: Theory, strategies, and activities for the classroom_. Teachers College Press, 1967.
  * Perkins (2006) D. Perkins, “Constructivism and troublesome knowledge,” in _Overcoming barriers to student understanding_. Routledge, 2006, pp. 57–71.
  * Hutchins (2020) E. Hutchins, “The distributed cognition perspective on human interaction,” in _Roots of human sociality_. Routledge, 2020, pp. 375–398.
  * Rosenberg and Stephens (2008) D. Rosenberg and M. Stephens, _Extreme programming refactored: the case against XP_. Apress, 2008.
  * Ruiz-Primo (2011) M. A. Ruiz-Primo, “Informal formative assessment: The role of instructional dialogues in assessing students’ learning,” _Studies in Educational Evaluation_ , vol. 37, no. 1, pp. 15–24, 2011.
  * McDowell et al. (2002) C. McDowell, L. Werner, H. Bullock, and J. Fernald, “The effects of pair-programming on performance in an introductory programming course,” in _Proceedings of the 33rd SIGCSE technical symposium on Computer science education_ , 2002, pp. 38–42.


M¿l¡

# Language Model Knowledge Distillation for Efficient Question Answering in
Spanish

Adrián Bazaga, Pietro Liò & Gos Micklem
University of Cambridge
<EMAIL_ADDRESS>
###### Abstract

Recent advances in the development of pre-trained Spanish language models has
led to significant progress in many Natural Language Processing (NLP) tasks,
such as question answering. However, the lack of efficient models imposes a
barrier for the adoption of such models in resource-constrained environments.
Therefore, smaller distilled models for the Spanish language could be proven
to be highly scalable and facilitate their further adoption on a variety of
tasks and scenarios. In this work, we take one step in this direction by
developing SpanishTinyRoBERTa, a compressed language model based on RoBERTa
for efficient question answering in Spanish. To achieve this, we employ
knowledge distillation from a large model onto a lighter model that allows for
a wider implementation, even in areas with limited computational resources,
whilst attaining negligible performance sacrifice. Our experiments show that
the dense distilled model can still preserve the performance of its larger
counterpart, while significantly increasing inference speedup. This work
serves as a starting point for further research and investigation of model
compression efforts for Spanish language models across various NLP tasks.

##  1 Introduction

In the last years, recent advancements in the field of NLP have panned the way
for progress on a variety of downstream tasks, primarily by fine-tuning pre-
trained language models (PLMs) (Vaswani et al., 2017; Devlin et al., 2019; Liu
et al., 2019; Yang et al., 2019) on large-scale task-specific datasets. These
models are typically trained on large and high-quality annotated corpora,
which are are usually scarce for languages other than English, posing a
significant disadvantage for progressing multilingual NLP research. Therefore,
resources for languages such as Spanish, the fourth most spoken language,
remain underrepresented. Moreover, despite the fact that some language models
are available for the Spanish language (Gutiérrez-Fandiño et al., 2022; Cañete
et al., 2020; de la Rosa et al., 2022; Pérez et al., 2022), operating them in
resource-constrained environments remains an under-explored challenge. To
overcome this limitation, we propose employing knowledge distillation (Jiao et
al., 2020; Boreshban et al., 2021) to transfer the intrinsic knowledge of a
large Spanish RoBERTa language model into a much compact and efficient model,
named as SpanishTinyRoBERTa, while maintaining its performance. In particular,
we focus on the question answering task (Rajpurkar et al., 2016; Carrino et
al., 2019), where the goal of the model is to provide an answer given a
context and a question over that context. Our experiments show that
SpanishTinyRoBERTa attains negligible performance reduction on the question
answering task with respect to the teacher model, whilst increasing
significantly inference speed and reducing resource requirements. The ultimate
goal of our work is to facilitate efforts in the development of efficient
language models for the Spanish language, thereby breaking down computational
barriers and promoting adoption of NLP technologies. Our training
implementation and the source code is available at
https://github.com/AdrianBZG/tinyroberta-distillation-qa-es.

##  2 Methodology

We propose developing a Spanish TinyRoBERTa model using the knowledge
distilled from a Spanish RoBERTa-large model in the SQuAD-es QA task (Carrino
et al., 2019). In the context of language models, knowledge distillation can
be modeled as penalizing the difference of feature representations between the
teacher and the model, therefore aiming to minimize the following objective
function:

$\displaystyle\mathcal{L}_{\text{KD}}=\sum_{x\in\mathcal{M}}L\big{(}f^{S}(x),f^{T}(x)\big{)},$
(1)

where $L(\cdot)$ is a loss function that computes the discrepancy between
teacher and student models, $x$ is the text input and $\mathcal{M}$ denotes
the training dataset. In this work, we employ the knowledge distillation
technique introduced in TinyBERT (Jiao et al., 2020) to reproduce the behavior
of the larger model and leveraging the knowledge transferred from it. During
model training, as the student model (SpanishTinyRoBERTa) contains much less
layers than the teacher, we map layers in the student to the teacher by using
a layer mapping function, so that the student learns from intermittent layers
of the teacher model (more details in Appendix A.1).

##  3 Experimental Setup and Results

To demonstrate the effectiveness of our distillation for Spanish question
answering (QA) tasks, we use the SQuAD-es dataset (Carrino et al., 2019),
which is a Spanish version of SQuAD (Rajpurkar et al., 2016). To address
questions answering as a learning problem, we treat the QA task as the problem
of sequence labeling which predicts the possibility of each token as the start
or end of answer span. We instantiate a tiny RoBERTa student model, with
number of layers $M$=6, the hidden size as $d_{h}$=512, the feedforward size
as $d_{ff}$=3072 and the head number as $h$=16), account for a total of 51.4M
parameters. If not specified, this student model is referred to as the
SpanishTinyRoBERTa. We use as teacher model a Spanish RoBERTa-large model pre-
trained on a corpus from the National Library of Spain (Gutiérrez-Fandiño et
al., 2022). The teacher model has layers $N$=12, a hidden size $d_{h}$=1024,
feed-forward size $d_{ff}$=4096 and $h$=16 heads, accounting for a total of
355M parameters. We use four different models as baselines: the Multilingual
BERT (mBERT), a Spanish BERT-base (Cañete et al., 2020), a Spanish RoBERTa-
base (124M parameters) and a Spanish RoBERTa-large as teacher model. As the
main focus of this work is to improve running time efficiency, we use these
baseline models to measure both the inference speed and performance when
compare with our lighter model. More details on the training hyperparameters
in Appendix A.2.

Model | F1 (%) | EM (%)
---|---|---
Spanish RoBERTa-large (teacher) | 87.50 | 78.30
Multilingual BERT (mBERT) | 77.60 | 61.80
Spanish BERT-base | 82.15 | 73.59
Spanish RoBERTa-base | 81.80 | 72.30
SpanishTinyRoBERTa (ours) | 80.52 | 71.23
Table 1: Comparison of the different models on the SQuAD-es Spanish Question
Answering task using the F1 and Exact Match (EM) metrics.

In terms of results, Table 1 shows the performance of the baseline models and
our distilled model in the SQuAD-es dataset. The SpanishTinyRoBERTa achieved
80.51% and 71.23% for F1 score and Exact Match (EM), respectively, comparable
to the performance attained by the teacher model, 87.50% and 78.30%, for F1
score and EM, respectively. These findings provide evidence that the
SpanishTinyRoBERTa can achieve competitive results on the QA task while
requiring much less computational resources. On this regard, we can observe in
Appendix A.3 that SpanishTinyRoBERTa contains 6.9x less parameters (51M) than
the teacher model (355M), and achieves 4.2x inference speedup (392ms vs 1683ms
per query). This shows the benefit of using the distillation process to
achieve a much lighter, faster and highly performant model.

##  4 Conclusion

In this work we present SpanishTinyRoBERTa, a compressed language model for
efficient question answering in Spanish that meets similar performance results
to its larger counterpart, with a significant reduction in terms of required
computational resources. We showed that the model has the potential to
contribute to the adoption of language model for Spanish question answering
language-related tasks on resource-constrained environments, while preserving
considerable levels of accuracy and robustness. As future work, we aim to
produce compressed models for other NLP downstream tasks.

## References

  * Boreshban et al. (2021) Yasaman Boreshban, Seyed Morteza Mirbostani, Gholamreza Ghassem-Sani, Seyed Abolghasem Mirroshandel, and Shahin Amiriparian.  Improving question answering performance using knowledge distillation and active learning, 2021.
  * Carrino et al. (2019) Casimiro Pio Carrino, Marta R. Costa-jussà, and José A. R. Fonollosa.  Automatic spanish translation of the squad dataset for multilingual question answering, 2019.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at ICLR 2020_ , 2020.
  * de la Rosa et al. (2022) Javier de la Rosa, Eduardo G. Ponferrada, Paulo Villegas, Pablo Gonzalez de Prado Salas, Manu Romero, and Marıa Grandury.  Bertin: Efficient pre-training of a spanish language model using perplexity sampling, 2022.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Gutiérrez-Fandiño et al. (2022) Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquin Silveira-Ocampo, Casimiro Pio Carrino, Carme Armentano-Oller, Carlos Rodriguez-Penagos, Aitor Gonzalez-Agirre, and Marta Villegas.  Maria: Spanish language models.  _Procesamiento del Lenguaje Natural_ , 68(0):39–60, 2022.  ISSN 1989-7553.  URL http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6405.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  TinyBERT: Distilling BERT for natural language understanding.  In Trevor Cohn, Yulan He, and Yang Liu (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4163–4174, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.372.  URL https://aclanthology.org/2020.findings-emnlp.372.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * Pérez et al. (2022) Juan Manuel Pérez, Damián Ariel Furman, Laura Alonso Alemany, and Franco M. Luque.  RoBERTuito: a pre-trained language model for social media text in Spanish.  In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 7235–7243, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.785.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text, 2016.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Huggingface’s transformers: State-of-the-art natural language processing, 2020.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.

##  Appendix A Appendix

###  A.1 Details on the distillation process

In this section, we provide further details on the training distillation
process. As mentioned in Section 2, we utilize the knowledge distillation
process described in (Jiao et al., 2020) with an addition term to account for
task-specific loss. More specifically, we assume that the teacher and student
models have $K$ and $L$ Transformer layers, respectively, where $L$ $\ll$ $K$.
To account for the discrepancy in terms of number of layers for the student
and teacher models, we use a layer index mapping function, $g(l)$ = 3 x $k$,
such that the student learns from every 3 layers of the teacher model.
Therefore, the student acquires knowledge from the teacher by minimizing the
following loss function:

$\displaystyle\\!\\!\\!\mathcal{L}\\!=\\!\sum_{x\in\mathcal{X}}\mathcal{L}_{\text{task}}(f^{S}(x),f^{T}(x))+\sum^{K+1}_{k=0}\mathcal{L}_{\text{layer}}(f^{S}_{k}(x),f^{T}_{g(k)}(x)),$
(2)

where $\mathcal{L}_{\text{layer}}$ is the loss function of a given model layer
(e.g. a Transformer or embedding layer), $f_{k}(x)$ denotes the behavior
function associated with the $k$-th layer of the model and
$\mathcal{L}_{\text{task}}$ is the task-specific distillation loss function
applied to the student and teacher models outputs. The layer loss,
$\mathcal{L}_{\text{layer}}$, is calculated as the sum of two terms,
$\mathcal{L}_{\text{attention}}$ and $\mathcal{L}_{\text{hidden}}$, for the
attention scores and hidden representations, respectively. Specifically,
$\mathcal{L}_{\text{attention}}$ is modelled as:

$\displaystyle\mathcal{L}_{\text{attention}}=\frac{1}{h}\sum\nolimits^{h}_{i=1}\texttt{MSE}(\bm{A}_{i}^{S},\bm{A}_{i}^{T}),$
(3)

where $h$ is the number of attention heads, $\bm{A}_{i}\in\mathbb{R}^{l\times
l}$ is the $i$-th head attention matrix of teacher or student, $l$ is the
input text length and MSE() depicts the mean squared error loss function. On
other side, $\mathcal{L}_{\text{hidden}}$ is defined as:

$\displaystyle\mathcal{L}_{\text{hidden}}=\texttt{MSE}(\bm{H}^{S}\bm{W}_{h},\bm{H}^{T}),$
(4)

where the matrices $\bm{H}^{S}\in\mathbb{R}^{l\times d^{\prime}}$ and
$\bm{H}^{T}\in\mathbb{R}^{l\times d}$ refer to the hidden states of student
and teacher networks respectively. The scalar values $d$ and $d^{\prime}$
denote the hidden sizes of teacher and student models, and $d^{\prime}$ is
often smaller than $d$ to obtain a smaller student network. The matrix
$\bm{W}_{h}\in\mathbb{R}^{d^{\prime}\times d}$ is a learnable linear
transformation, which projects the hidden states of the student network into
the same dimensionality as the teacher feature space. Therefore,
$\mathcal{L}_{\text{layer}}$ is expressed as:

$\displaystyle\mathcal{L}_{\text{layer}}=\mathcal{L}_{\text{attention}}+\mathcal{L}_{\text{hidden}}$
(5)

By following this distillation process, the smaller model is able to mimic the
representation and attention of the teacher over the input sequences, while
maximizing the downstream task performance.

###  A.2 Details on training hyperparameters and environment

Training was done using a single NVIDIA RTX A6000 GPU, with a batch size of
32, a learning rate of 3e-5, and maximum sequence length of 384 running for 20
epochs. For optimization purposes, mixed-precision training is employed, and
gradient clipping with a max gradient norm of 1.0 is applied to improve
training stability. We utilize the HuggingFace library (Wolf et al., 2020) for
our training implementation and the source code is available at
https://github.com/AdrianBZG/tinyroberta-distillation-qa-es.

###  A.3 Experimental results on model efficiency

In Table 2 we show a comparison of model sizes and inference latency between
the baseline models and our SpanishTinyRoBERTa model. The results were
obtained by running on a single NVIDIA RTX A6000 GPU and averaging over 10
different runs.

Model | Layers | Hidden Size | Feed-forward Size | Size | Latency (ms)
---|---|---|---|---|---
Spanish RoBERTa-large (teacher) | 24 | 1024 | 4096 | 355M | 1683
Multilingual BERT (mBERT) | 12 | 768 | 3072 | 179M | 1187
Spanish BERT-base | 12 | 768 | 3072 | 109M | 942
Spanish RoBERTa-base | 12 | 768 | 3072 | 124M | 1015
SpanishTinyRoBERTa (ours) | 6 | 512 | 3072 | 51M | 392
Table 2: Comparison of model sizes and inference latency (in milliseconds for
a single query) between the baselines and our distilled model. The number of
layers does not include the embedding and prediction layers.