# Towards a Success Model for Automated Programming Assessment Systems Used as
a Formative Assessment Tool

Clemens Sauerwein<EMAIL_ADDRESS>0009-0009-9464-5080 University
of InnsbruckTechnikerstraße 21aInnsbruckAustriaA-6020 ,  Tobias Antensteiner
<EMAIL_ADDRESS>0000-0001-5513-1073 University of
InnsbruckTechnikerstraße 21aInnsbruckAustriaA-6020 ,  Stefan Oppl
<EMAIL_ADDRESS>0000-0002-5495-7727 University for Continuing
Education Krems Dr.-Karl-Dorrek-Straße 30KremsAustriaAT-3500 ,  Iris Groher
<EMAIL_ADDRESS>0000-0003-0905-6791 Johannes Kepler University
LinzAltenbergerstrasse 69LinzAustriaAT-4040 ,  Alexander Meschtscherjakov
<EMAIL_ADDRESS>0000-0001-8116-4522 Paris Lodron
University of SalzburgJakob-Haringer-Straße 8SalzburgAustriaAT-5020 ,  Philipp
Zech<EMAIL_ADDRESS>0000-0002-4952-4337 University of
InnsbruckTechnikerstraße 21aInnsbruckAustriaA-6020 and  Ruth Breu
<EMAIL_ADDRESS>0000-0001-7093-4341 University of
InnsbruckTechnikerstraße 21aInnsbruckAustriaA-6020

(2023)

###### Abstract.

The assessment of source code in university education is a central and
important task for lecturers of programming courses. In doing so, educators
are confronted with growing numbers of students having increasingly diverse
prerequisites, a shortage of tutors, and highly dynamic learning objectives.
To support lecturers in meeting these challenges, the use of automated
programming assessment systems (APASs), facilitating formative assessments by
providing timely, objective feedback, is a promising solution. Measuring the
effectiveness and success of these platforms is crucial to understanding how
such platforms should be designed, implemented, and used. However, research
and practice lack a common understanding of aspects influencing the success of
APASs. To address these issues, we have devised a success model for APASs
based on established models from information systems as well as blended
learning research and conducted an online survey with $414$ students using the
same APAS. In addition, we examined the role of mediators intervening between
technology-, system- or self-related factors, respectively, and the users’
satisfaction with APASs. Ultimately, our research has yielded a model of
success comprising seven constructs influencing user satisfaction with an
APAS.

Automated Assessment Tool, Automated Programming Assessment System, Success
Model, User Satisfaction, Empirical Study, Survey, Programming Education

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
2023 Conference on Innovation and Technology in Computer Science Education V.
1; July 8–12, 2023; Turku, Finland††booktitle: Proceedings of the 2023
Conference on Innovation and Technology in Computer Science Education V. 1
(ITiCSE 2023), July 8–12, 2023, Turku, Finland††doi:
10.1145/3587102.3588848††isbn: 979-8-4007-0138-2/23/07††ccs: Social and
professional topics Software engineering education††ccs: Software and its
engineering Software creation and management††ccs: General and reference
Empirical studies

##  1\. Introduction

In the course of the advancing digitization of the economy and society,
programming education at universities is gaining an increasingly important
role. While in the past programming skills played an essential role mainly in
the natural sciences, their relevance is increasing in almost all fields of
science. To meet this increased demand for programming education, higher
education institutions offer a variety of new teaching formats such as
supplementary studies or minor subjects with a strong focus on programming
education. Consequently, the group of students acquiring programming skills is
much larger and has heterogeneous previous knowledge as well as ways of
thinking. The situation is exacerbated by a shortage of teaching staff and
highly dynamic learning content and technologies (mekterovic2017, 25).

Therefore, solutions are needed to efficiently support teachers in managing
courses and assignments for a large number of students from different
backgrounds. These solutions should provide students with automatic and
objective formative feedback or targeted and timely assistance with
programming problems (mekterovic2017, 25). To meet these requirements, various
solutions entered the market in recent years under the umbrella term Automated
Programming Assessment System (APAS) (keuning2018, 17, 26). According to
(mekterovic2020, 26) and (keuning2018, 17), APASs can motivate students,
provide a useful overview of learning progress, improve the quality of
teaching and student contributions, minimize the barrier to entry into
programming, standardize and objectify feedback and reduce dropout rates
(mekterovic2020, 26, 17).

Consequently, recent research has examined the usability of APAS (pettit2015c,
29)total and how they affect students’ learning and lecturers’ teaching
experience (pettit2015b, 30). While current research recommends studies of the
aspects that influence acceptance and satisfaction of users (sauerwein2023,
37), not many attempts have been made to model and evaluate the drivers of
APASs’ success.

To address this research challenge, we devised and validated a comprehensive
success model to determine the effectiveness of APASs from a student
perspective. In doing so, we drew on the works of (delone1992, 5), whose
interdependent and multi-construct view of information systems is widely
accepted and has been applied in a variety of fields (delone2003, 6). In
addition, we derived certain exogenous as well as mediator constructs from
(zhang2020, 45)’s blended learning success model (zhang2020, 45) to ultimately
develop a model measuring the success of APASs in terms of student
satisfaction (wu2010a, 44) utilizing eight constructs. The remainder of the
paper is structured as follows: Section 2 discusses related work. Section 3
outlines the applied research methodology. Section 4 presents the statistical
analysis of our devised success model. Section 5 discusses the key findings
and limitations of our research, as well as directions for future work.
Finally, Section 6 concludes our research and provides an outlook on our
future work.

##  2\. Related Work

Recent research in higher education addresses both the understanding and
support of students learning to program and of teachers providing introductory
programming courses (luxton2018, 23). Inspired by further research shedding
light on the causes of difficulties in learning programming (gomes2007, 8),
methods and tools such as APASs (souza2016, 40) were developed. In this
context, a “process-oriented approach in assessing programming progress”
(villamor2020, 42) was introduced, leading to an iterative learning approach
to programming (pettit2015c, 29).

According to (mekterovic2017, 25), the use of APASs circumvents some of the
hurdles associated with the manual assessment of programming tasks, such as
objectively and efficiently assessing a large number of students as well as
providing timely, individual and helpful feedback (mekterovic2017, 25). Based
on a comprehensive literature review, (pettit2015b, 30) have shown evidence
that using APAS enhances the teacher’s learning experience and increases
students’ learning success (pettit2015b, 30). Subsequent research has examined
how APAS are perceived by users (restrepo2019, 34, 2, 35, 10), how easy they
are to use (pettit2015c, 29), how they affect learning and the teaching
experience (pettit2015b, 30, 32), and what users’ experiences are
(sauerwein2023, 37).

Since e-learning systems are often considered as dedicated instances of
information systems (IS), APASs in particular can be evaluated by using models
that have been developed and validated in the field (mayr2022, 24).
Accordingly, (zhang2020, 45) have developed a thorough blended learning
success model (zhang2020, 45) based on (delone1992, 5)’s revised IS success
model (delone2003, 6) by (i) incorporating, among others, “individual’s
attitudes” (liaw2011, 20) like student’s computer self-efficacy and motivation
as “self-related factors” (zhang2020, 45), and (ii) considering learning
climate (wu2010a, 44), blended learning flexibility (zhang2020, 45), task-
technology fit (goodhue1995, 9) as meditators influencing student’s
satisfaction (sun2008, 41) and intention.

However, recent research has focused primarily on APASs design,
implementation, and use, thus, leaving a few papers to conduct empirical
studies addressing drivers shaping student’s APASs satisfaction. While success
models exist in other fields such as IS and e- or blended learning (wu2010b,
43, 47), a model for gauging the success of APASs is largely lacking
(sauerwein2023, 37). Moreover, the role of mediator constructs intervening
between technology, system or self-related factors and user satisfaction with
APASs has not been examined.

##  3\. Applied Research Methodology

To investigate the drivers of APASs success with respect to student
satisfaction, we (i) defined a research model (cf. Section 3.1) based on
(delone2003, 6)’s and (zhang2020, 45)’s works (delone2003, 6, 45), (ii)
developed a research instrument by adopting approved measurement items and
utilizing an online questionnaire (cf. Section 3.2), and (iii) collected data
from a total of $414$ students (cf. Section 3.3).

###  3.1. Research Model

Our devised success model for APASs builds on the refined (delone2003, 6)
model of IS success (delone2003, 6) as well as (zhang2020, 45)’s blended
learning success model (zhang2020, 45) and consists (cf. Figure 1) of the
following four exogenous constructs:

  * MOT

(Motivation) the APAS’s incentives, encouraging the user to engage in learning
with the system (zhang2020, 45),

  * INQ

(Information Quality) the desirable quality characteristics of the APAS’s
output or content (delone2003, 6),

  * SYQ

(System Quality) the desirable quality characteristics of the APAS itself
(delone2003, 6), and

  * SEQ

(Service Quality) the quality of the support provided by the APAS’s provider
(delone2003, 6);

the following three mediator constructs:

  * LEC

(Learning Climate) the user’s enjoyment and pleasure when using the platform,
as well as the platform’s ability to motivate learning (zhang2020, 45),

  * BLF

(Blended Learning Flexibility) the degree the APAS offers the freedom to learn
at one’s own pace, with effective preparation and feedback (zhang2020, 45),
and

  * TTF

(Task-Technology Fit) the degree to which the services of an APAS meet the
user’s learning goals and requirements (zhang2020, 45);

as well as the endogenous construct:

  * SAT

(Satisfaction) the users’ level of satisfaction with the APAS (delone2003, 6).

Consequently, following (zhang2020, 45)’s approach, we (i) adopted all
endogenous constructs of (delone1992, 5)’s revised IS success model
(delone2003, 6), i.e., INQ, SYQ, and SEQ, (ii) considered MOT as an additional
independent, “self-related factor” (zhang2020, 45), and (iii) introduced the
APAS-related blended learning success model constructs LEC (wu2010a, 44), BLF
(zhang2020, 45), and TTF (goodhue1995, 9), as meditators potentially
influencing the by (delone1992, 5) proposed endogenous latent variable SAT
(sun2008, 41), being explained in our success model for APASs. Therefore, in
accordance with (rungtusanatham2014, 36) transmittal approach
(rungtusanatham2014, 36, 27) and based on (delone1992, 5) as well as
(zhang2020, 45), we hypothesize, each of the aforementioned mediating effects,
i.e., MOT, INQ, SYQ, and SEQ, respectively, have an indirect effect on SAT
through LEC, BLF, and TTF.

To empirically test our theoretical model, we converted it into a structural
equation model (cf. Section 4.1), as depicted in Figure 1, in which, among
others, the hypotheses to be further investigated are represented by arrows,
i.e., directed paths, connecting the respective constructs, denoted by
ellipses.

###  3.2. Research Instrument

Our primary research tool was a self-administered online questionnaire in
LimeSurvey. In order to statistically test the constructs and their
relationships, we defined corresponding questions for each construct. In doing
so, we followed the works of (zhang2020, 45, 6) to formulate appropriate
survey questions. This approach resulted in the following question items:

  * $\texttt{MOT}_{1}$

The APAS motivates me to work on the tasks and the feedback.

  * $\texttt{MOT}_{2}$

Working with the APAS increases my MOT to solve tasks.

  * $\texttt{MOT}_{3}$

The APAS increases my MOT when learning the course content.

  * $\texttt{INQ}_{1}$

The information provided by the APAS is relevant to my learning.

  * $\texttt{INQ}_{2}$

The information provided by the APAS is easy to understand.

  * $\texttt{INQ}_{3}$

The information provided by the APAS is very good.

  * $\texttt{INQ}_{4}$

The information provided by the APAS is up to date.

  * $\texttt{INQ}_{5}$

The information provided by the APAS is complete.

  * $\texttt{INQ}_{6}$

The information provided by the APAS is accurate.

  * $\texttt{SYQ}_{1}$

The APAS offers flexibility in terms of learning time and place.

  * $\texttt{SYQ}_{2}$

The APAS offers interactive feedback on the exercises.

  * $\texttt{SYQ}_{3}$

The APAS offers sufficient functions for my learning process.

  * $\texttt{SYQ}_{4}$

In general, the APAS is reliable.

  * $\texttt{SYQ}_{5}$

In general, the response time of the APAS is reasonable.

  * $\texttt{SEQ}_{1}$

The help and support of the APAS is fast.

  * $\texttt{SEQ}_{2}$

The help and support of the APAS is reliable.

  * $\texttt{SEQ}_{3}$

The help and support of the APAS is accessible.

  * $\texttt{SEQ}_{4}$

The help and support of the APAS is practical.

  * $\texttt{SEQ}_{5}$

The help and support provided by the APAS is satisfactory overall.

  * $\texttt{LEC}_{1}$

I like to use the APAS to support my learning process.

  * $\texttt{LEC}_{2}$

Working with the APAS is fun.

  * $\texttt{LEC}_{3}$

Working with the APAS is very pleasant.

  * $\texttt{LEC}_{4}$

The LEC of the APAS was able to motivate my spontaneous learning.

  * $\texttt{BLF}_{1}$

The APAS gives me more flexibility to learn in this course.

  * $\texttt{BLF}_{2}$

The APAS allows me to complete the exercises at my own pace.

  * $\texttt{BLF}_{3}$

The APAS allows me to prepare more effectively for each lesson.

  * $\texttt{TTF}_{1}$

The services offered by the APAS meet my learning needs.

  * $\texttt{TTF}_{2}$

The services of the APAS are compatible with my learning needs.

  * $\texttt{TTF}_{3}$

The services of the APAS fit my learning needs.

  * $\texttt{TTF}_{4}$

My learning goals and needs are met through the use of the APAS.

  * $\texttt{SAT}_{1}$

Overall, I am satisfied with the APAS.

  * $\texttt{SAT}_{2}$

Overall, I find the APAS enjoyable.

  * $\texttt{SAT}_{3}$

Overall, the APAS meets my learning needs.

All question items were given a five-point Likert scale with $1$ for “strongly
disagree”, $3$ for “neither agree nor disagree” and $5$ for “strongly agree”.
In addition, demographic information such as gender, (academic) degree, number
of semesters studied, and known programming languages were collected through
the survey in order to characterize the sample.

###  3.3. Data Collection

The research model (cf. Section 3.1) was validated in the form of a survey
(cf. Section 3.2) in introductory programming courses at the end of the
semester in February 2021 and 2022. For these introductory programming
courses, we used the APAS ArTeMis (krusche2018, 18) as it provides all the
basic functions, such as course, exercise and test management, learning
analytics, just-in-time teaching, and automated feedback features. In
addition, the same adapted blended learning method was used in each course and
an introduction to programming was taught. To reinforce the learning content
at home, a weekly exercise sheet was made available via ArTeMis. This exercise
sheet consisted of several programming tasks with stored test cases and
formative feedback in ArTeMis. The students had to work on these exercise
sheets independently and were supported by appropriate feedback from ArTeMis.
After successfully completing the exercise sheets, the students had to
participate in a weekly video conference to discuss the results and receive
targeted assistance from the lecturers. Due to the pandemic situation, video
conferences had to be held remotely online instead of face-to-face meetings at
the university.

A total of $414$ students – more than $75\%$ were seeking a Bachelor’s degree,
about $20\%$ were enrolled in a Master’s program and $5\%$ hold a Master’s
degree – participated in our survey (cf. Section 3.2). After removing a total
of $46$ partially completed questionnaires, i.e., responses containing missing
values (MVs) in at least one descriptive statistics data’s or all research
model item’s columns (cf. Section 3.1), $368$ records remained for statistical
analysis. Thus, our dataset contains $5.57\%$ item non-response MVs, which we
can attribute to partially completed questionnaires. However, these missing
values do not affect our data analysis, as the employed estimation method (cf.
Section 4) provides unbiased results up to a MV percentage of $8.89\%$
(grimm2020, 11).

Of the overall $98$ female, $262$ male students, and $8$ with no indication of
gender who participated in our survey, $271$ were in their first semester of
study or had completed it recently. Consequently, about $40\%$, i.e., in total
$156$, were not familiar with any well-known (general-purpose) programming
language such as C, Java or Python. Taken together, $132$ of the students
surveyed stated to know Python, less than one quarter, i.e., overall $88$,
have heard of Java, $77$ were acquainted with C, and $58$ were familiar with
C++ before enrolling in the respective (introductory) course.

##  4\. Data Analysis and Results

In order to conduct a systematic mediation analysis (cf. Section 4.2), which
builds on our theoretically established success model for APASs and
hypothesized relationships (cf. Section 3.1), we estimate and assess the
corresponding structural equation model (cf. Section 4.1) (hair2021b, 14).

[Bootstrapped PLS model using $10\,000$ bootstrap samples.]Bootstrapped PLS
model ($\texttt{nboot}=10\,000$). Coefficients of non-significant paths, i.e.,
dotted lines, are not shown. Items marked in gray refer to eliminated
indicators. Values in braces denote the $R^{2}$ value of the respective
(endogenous) construct, while value pairs in brackets indicate the $95\%$
confidence interval of the corresponding significant path. Stars refer to the
significance level of the respective factor loading or path coefficient, where
three stars (“***”) denotes $p\in[0,.001]$ and two stars (“**”) denotes
$p\in(.001,0.01]$. Furthermore, direct effects used for mediation analysis are
highlighted in gray; thus, both, the ordinary and the gray plotted version of
a respective exogenous construct refer to the same latent variable.

Figure 1. Bootstrapped PLS model using $10\,000$ bootstrap samples.

###  4.1. Model Evaluation

Structural equation modeling (SEM) is a useful technique for evaluating
complex theoretical relationships, especially between multiple latent
variables, i.e., exogenous and endogenous constructs, in particular when
conducting research, such as our study, in social science (hair2022, 12). Two
fundamental SEM methods, covariance-based SEM (CB-SEM) and PLS-SEM have been
proposed and dominate SEM in practice (hair2021a, 15). However, as in our
case, PLS-SEM is particularly useful when the goal of the structural model is
to predict and explain the intended outcomes as “obtained by the in-sample and
out-of-sample metrics” (latan2017, 19). Accordingly, we used the statistical
computing language R version 4.2. and leveraged the SEMinR package (ray2021,
33) version 2.3. to test our research hypotheses (cf. Section 3.1) employing a
PLS-SEM analysis (ray2021, 33, 15).

####  4.1.1. Measurement Model Evaluation

Starting with the reliability and validity evaluation of our reflective
measurement by following (hair2021a, 15), we (i) assess the indicator
reliability using the respective indicator loadings, (ii) examine the internal
consistency reliability leveraging composite reliability ($\rho_{c}$),
Cronbach’s alpha ($\rho_{T}$), as well as the reliability coefficient
($\rho_{A}$), (iii) assess the convergent validity based on the average
variance extracted (AVE), and (iv) check the discriminant validity through the
corresponding Heterotrait-Monotrait (HTMT) ratios (hair2019a, 16).

Regarding (i), all indicator loadings, except $\texttt{INQ}_{4}$,
$\texttt{INQ}_{5}$, $\texttt{SYQ}_{1}$, and $\texttt{SYQ}_{5}$ (cf. Sections
3.2), of the eight (reflectively) measured constructs MOT, INQ, SYQ, SEQ, LEC,
BLF, TTF, and SAT (cf. Sections 3.1) are statistically significant at
$\text{CI}_{\alpha}=.05$ or below as well as higher than the threshold value
of $.708$ (hair2021b, 14), which suggests sufficient levels of indicator
reliability (hair2021a, 15). As according to (hair2021a, 15), “indicators with
loadings between $.40$ and $.708$ should be considered for removal only when
deleting the indicator leads to an increase in the internal consistency
reliability or convergent validity” (cf. (ii) and (iii)), we eliminate
$\texttt{INQ}_{4}$, $\texttt{INQ}_{5}$, $\texttt{SYQ}_{1}$, as well as
$\texttt{SYQ}_{5}$ (cf. Figure 1), as the deletion increases the respective
AVE values (cf. (iii)), but does not affect the respective internal
consistency reliability.

As for (ii), all eight (reflectively) measured constructs show high levels of
internal consistency (hair2021b, 14), as $\rho_{c}$, $\rho_{T}$ and $\rho_{A}$
are all above $.70$ and less than $.95$ (hair2021a, 15). Moreover, concerning
(iii), the respective AVE values are all above the cut-off value of $.50$
(hair2022, 12), implying the measures of the eight (reflectively) measured
constructs MOT, INQ, SYQ, SEQ, LEC, BLF, TTF, and SAT (cf. Sections 3.1) have
high levels of convergent validity (hair2021a, 15).

With respect to (iv), all HTMT ratio values are below the liberal cut-off
value of $.90$ (hair2022, 12) for conceptually similar constructs such as MOT
and LEC or SYQ and TTF and lower than the conservative cut-off value of $.85$
(hair2021a, 15) for conceptually distinct constructs such as SEQ and BLF,
indicating discriminant validity between the eight (reflectively) measured
constructs. Furthermore, as recommended by (hair2021a, 15), the upper bound of
the $95\%$ bootstrap confidence intervals show that the HTMT ratio values are
significantly lower than the liberal threshold value of $.90$.

####  4.1.2. Structural Model Evaluation

Having established the reliability and validity of the constructs, we examine
the structural component of our devised success model for APASs “based on its
ability to predict the outcomes.” (hair2022, 12) Following (hair2022, 12)’s
recommendations, we (i) examine the structural model for collinearity issues
based on the variance inflation factor (VIF), (ii) assess the significance and
relevance of the structural model relationships, i.e, the path coefficients,
using bootstrapping, (iii) evaluate the structural model’s explanatory power
by the coefficient of determination ($R^{2}$) and the effect size ($f^{2}$),
and (iv) examine its out-of-sample predictive power using the PLSpredict
(shmueli2019, 39) method.

As for (i), the VIF results indicate, our model does not show signs of
collinearity among the eight constructs (hair2021a, 15), as the highest VIF
value are close to the conservative threshold of $3.3$ (hair2019b, 13, 12).

Regarding (ii), we assess the significance as well as the relevance of the
structural model paths by bootstrapping the sampling distribution
(preacher2008, 31, 46, 28) with $10\,000$ (hair2021a, 15) resamples to test
the structural model’s relationships coefficients for statistical significance
at $\text{CI}_{\alpha}=.05$, as summarized in Figure 1.

With respect to (iii), the $R^{2}$ values (cf. Figure 1) of the endogenous
constructs LEC and SAT as well as of the endogenous constructs BLF and TTF are
substantial as well as moderate (hair2021a, 15), respectively, indicating a
good in-sample predictive power (hair2021b, 14) of our success model for
APASs. Moreover, the $f^{2}$ values for each of the structural model paths
follow the same rank order as corresponding path coefficient magnitude
(hair2021a, 15).

Concerning (iv), our analysis shows that our bootstrapped PLS model has lower
out-of-sample predictive error compared to the naïve linear regression model
benchmark for the minority of the indicators of the outcome variable
substantially explained by the predictor construct SAT, indicating low
predictive power (hair2021a, 15) and an acceptable claim of external validity
for similar contexts (shmueli2016, 38).

###  4.2. Mediation Analysis

As described by (hair2021a, 15) “mediation occurs when a construct, referred
to as mediator construct, intervenes between two other related constructs.”
(hair2021a, 15, p. 140) Thus, for example, a change in the exogenous construct
MOT may cause a change in the theoretically considered (memon2018, 27)
mediator constructs LEC, BLF, and TTF (zhang2020, 45, 6) (cf. Section 3.1),
respectively, possibly leading to a change in the endogenous construct SAT
(cf. Figure 1), each thereby representing an indirect mediating effect
(hair2021a, 15, 27).

Since all quality criteria of the measurement and structural model are met
(cf. Section 4.1.1 and Section 4.1.2, respectively) and, in particular, as
recommended by (hair2021a, 15), all “reflectively measured mediator constructs
exhibit a high level of reliability” (hair2021a, 15, p. 141), high construct
collinearity is not present and the construct’s discriminant validity are
below the liberal cut-off value of $.90$ (hair2021a, 15, 12), our reflective
measurement model as well as the structural model are sound to perform
mediation analysis.

Following (carrion2017, 3)’s guidelines for multiple mediation analysis in
PLS-SEM (carrion2017, 3) based on (nitzl2016, 28) (nitzl2016, 28), (zhao2010,
46) (zhao2010, 46) as well as (baron1986, 1) (baron1986, 1), we first test for
significance of the indirect effects (carrion2017, 3, 15) from the exogenous
constructs MOT, INQ, SYQ and SEQ via LEC, BLF, and TTF to endogenous
constructs SAT, respectively, (cf. Figure 1) by bootstrapping the sampling
distribution (preacher2008, 31, 46, 28). Since, according to (carrion2017, 3),
“if the significance of each indirect effect cannot be established, there is
no mediating effect” (carrion2017, 3, p. 183), we infer that each of the three
respective indirect effects (nitzl2016, 28) as well as the resulting total
indirect effects (TIE) (carrion2017, 3)
$\text{TIE}_{\texttt{MOT}\xrightarrow{\texttt{LEC},\texttt{BLF},\texttt{TTF}}\texttt{SAT}}\approx.20+.04+.06=.30$,
$\text{TIE}_{\texttt{INQ}\xrightarrow{\texttt{LEC},\texttt{BLF},\texttt{TTF}}\texttt{SAT}}\approx.05+.02+.05=.12$,
and
$\text{TIE}_{\texttt{SYQ}\xrightarrow{\texttt{LEC},\texttt{BLF},\texttt{TTF}}\texttt{SAT}}\approx.07+.04+.08=.19$
are significant at $\text{CI}_{\alpha}=.05$ by inspecting the bootstrap
confidence intervals (CIs) (hair2021a, 15), respectively.

In accordance with (carrion2017, 3) mediation analysis procedure (carrion2017,
3, 28), we secondly assess whether the relevant direct effects (DE) (zhao2010,
46, 15) (cf. Figure 1) are significant for each of the three mediation effects
previously determined. As $\text{DE}_{\texttt{MOT}\rightarrow\texttt{SAT}}$ is
not significant and
$\text{DE}_{\texttt{INQ}\rightarrow\texttt{SAT}}\approx.11$ as well as
$\text{DE}_{\texttt{SYQ}\rightarrow\texttt{SAT}}\approx.16$ are significant at
$\text{CI}_{\alpha}=.05$, respectively, we conclude that (i) the relationship
between exogenous construct MOT and endogenous construct SAT is fully mediated
by LEC, BLF, and TTF, and (ii) the exogenous constructs LEC, BLF, and TTF
partially mediate the effect of INQ respectively SYQ on the endogenous
construct SAT. Moreover, both variance accounted for (VAF) values,
$\text{VAF}_{\texttt{INQ}\xrightarrow{\texttt{LEC},\texttt{BLF},\texttt{TTF}}\texttt{SAT}}\approx
53\%$ and
$\text{VAF}_{\texttt{SYQ}\xrightarrow{\texttt{LEC},\texttt{BLF},\texttt{TTF}}\texttt{SAT}}\approx
54\%$, lie in the interval $[20\%,80\%]$ (hair2021a, 15), thus providing us
with an additional argument to imply partial mediation for the latter two
complementary mediations (nitzl2016, 28, 3).

##  5\. Discussion of Results

While our data analysis has shown that the model developed is valid, we cannot
support all the hypotheses raised (cf. Section 3.1). Therefore, in the
following, we discuss the key findings (cf. Section 5.1) and outline possible
limitations of the present research as well as directions for future work (cf.
Section 5.2).

###  5.1. Key Findings

As shown in Figure 1, most of the model’s hypotheses are supported by our
study. Only the paths from SEQ to LEC and from SEQ to BLF are not
statistically significant and, thus, the corresponding hypotheses are not
supported.

Our investigations showed that MOT significantly influences LEC, BLF and TTF.
In other words, if students are motivated by the platform to work on their
assignments, this has a positive effect on LEC, BLF and TTF of the platform.
Accordingly, MOT of students to use the platform should be kept high by
providing a user-friendly design (sun2008, 41). This could be increased, for
example, through targeted support for editing tasks or additional functions
such as a fully-fledged integrated code editor (sauerwein2023, 37).

The results show that INQ also significantly influence LEC, BLF and TTF. This
means that INQ is a critical success factor, along with the student’s MOT to
use the platform. Therefore, it is clear that the information provided by the
platform, in the form of exercise sheets and feedback, should be taken
seriously and utilized to its full potential. Related work showed that the
preparation of suitable tasks and corresponding feedback for the students is a
major challenge for lecturers (sauerwein2023, 37). To ensure INQ and the
success of the platform, it is essential that teachers create assignments and
test cases with the appropriate feedback. Therefore, an APAS should offer
functions that are capable of aiding lecturers in meeting these challenges
(mekterovic2020, 26).

As discussed, LEC, BLF and TTF are significantly impacted by SYQ. SYQ refers
to the reliability, flexibility, interactive feedback and sufficient functions
offered by the APAS to support the learning process. Recent research has
revealed that while students using an APAS primarily make use of the core
functionalities such as exercise submission and feedback, other services, such
as support functions, are used only marginally (sauerwein2023, 37). Therefore,
it is not surprising that SEQ only affects the TTF, thus playing a subordinate
role. In contrast to SYQ, SEQ refers to the help and support provided by the
APAS. In order to fulfil SEQ it implies that the APAS’s service
functionalities should be fast, reliable, accessible, practical and overall
satisfactory

Consistent with the hypothesis made by (zhang2020, 45) (zhang2020, 45) as well
as in-depth research on blended learning (dang2020, 4), the students’
perceived LEC (wu2010a, 44), BLF (zhang2020, 45), and TTF (goodhue1995, 9)
significantly influences their SAT with the APAS. Accordingly, this points to
the importance (i) of the user’s enjoyment and pleasure when using the APAS,
(ii) “offering the desired level of flexibility in students’ learning process”
(zhang2020, 45), and (iii) the degree to which the services of an APAS meet
the user’s learning needs, goals as well as requirements.

Mediation analysis (cf. Section 4.2) reveals one “indirect-only mediation”
(hair2022, 12) and two “complementary mediations” (hair2022, 12) as the three
hypothesized mediator constructs LEC, BLF, and TTF fully mediate the
relationship between MOT and SAT as well as partial mediate the relationship
between INQ respectively SYQ and SAT (cf. Section 4.2). Furthermore, since
neither SEQ’s direct nor total indirect effect on SAT is significant, the
mediation analysis shows one “no-effect non-mediation” (hair2022, 12) by not
supporting the corresponding hypothesized mediation effect.

As MOT direct effect on SAT is not significant, the important impact of the
mediating effect of LEC, BLF, and TTF emphasis the vital role of the students’
perceptions of the learning climate, task-technology fit, and blended learning
flexibility in positively influencing individual’s attitudes when using the
APAS (sun2008, 41). Moreover, the LEC’s, BLF’s, and TTF’s partial mediation of
the effect of INQ respectively SYQ on SAT entails that factors shaping the
quality characteristics of the APAS, its output and its content, i.e., APAS’s
technology dimension related drivers (dang2020, 4), not only have a
significant direct effect on the APAS success (delone2003, 6, 45), but also
are positively influenced in their impact by (i) students’ overall delight and
happiness in using the APAS (wu2010a, 44), (ii) the degree to which the APAS
provide students with self-paced learning opportunities (zhang2020, 45), and
(iii) the level to which an APAS’s services meet students’ learning concerns,
objectives, and demands (lin2012, 22).

Finally, our research has shown which aspects should be taken into account to
increase users’ SAT with an APAS and thus ensure its successful use. This
statement is reinforced by (lin2008, 21), as the concept of SAT is the
ultimate goal of any product and service (lin2008, 21). Moreover, recent
research has shown (diep2017, 7) that SAT is the most accepted measure of
learning environments’ quality and effectiveness.

###  5.2. Limitations & Directions for Future Work

The research at hand might be limited by a (i) selection bias of participants,
(ii) the used APAS, (iii) a large number of constructs at an abstract level,
and (iv) the used teaching method.

Regarding (i), we asked the students of the respective introductory courses to
voluntarily participate in our survey. Accordingly, only students in computer
science-related studies participated in our study. Due to this, a high
computer affinity of the study participants could be assumed.

The limitation (ii) can be invalidated to a certain extent, as a systematic
analysis of the functional scope of different APASs has shown that the basic
functions offered are similar in all systems (sauerwein2023, 37). Furthermore,
the aim of our work was to identify the overarching success factors of APASs,
rather than assessing individual functions in detail. However, future work is
required to investigate whether our devised success model can be generalised
to other APASs in order to increase the representativeness of our findings.

Concerning (iii), our success model consists of a large number of constructs
at an abstract level (cf. Section 3.1). This was necessary to make the
different constructs comparable on a uniform level of abstraction and analyze
their relationship. For this reason, some constructs, such as MOT,INQ, SYQ, or
TTF, were only considered superficially and should be investigated in more
detail in future work.

Due to the COVID-19 pandemic, the APAS was used in a blended learning format
adapted to distance learning, resulting in students’ face-to-face meetings
with instructors being conducted via videoconferencing. However, limitation iv
could not be avoided due to the pandemic.

##  6\. Conclusion & Outlook

In this paper, by drawing on the work of (delone1992, 5) and (zhang2020, 45),
we introduced a success model for APASs consisting of four exogenous
constructs – MOT, INQ, SYQ and SEQ – and three mediator constructs – LEC, BLF,
and TTF –, influenceing the students’ SAT with the APAS. We estimated the
constructed model using PLS-SEM (hair2021b, 14) based on indicator data from
two online surveys among a total of $414$ students who attended introductory
programming courses using the same APAS. Our investigations have shown that
each of the exogenous constructs, with the exception of SEQ, has a significant
indirect effect on SAT via the hypothesized mediating constructs LEC, BLF, and
TTF. These results emphasize (i) the crucial role of the user’s enjoyment
using the APAS, (ii) the platform’s functionalities in fostering learning
flexibility, and (iii) the extent to which the APAS’s services meet the user’s
learning needs. Furthermore, our findings should enable practitioners to
better understand the key aspects influencing the success of APASs. As part of
our future work, it would be interesting to investigate whether the model can
also be applied to other APASs and to empirically examine important
constructs, e.g., MOT or TTF, in more detail.

## References

  * (1) Reuben M Baron and David A Kenny  “The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.”  In _Journal of personality and social psychology_ 51.6 American Psychological Association, 1986, pp. 1173
  * (2) Enrique Barra et al.  “Automated Assessment in Programming Courses: A Case Study during the COVID-19 Era”  In _Sustainability_ 12.18, 2020, pp. 7451  DOI: 10.3390/su12187451
  * (3) Gabriel Cepeda Carrión, Christian Nitzl and José L Roldán  “Mediation analyses in partial least squares structural equation modeling: Guidelines and empirical examples”  In _Partial least squares path modeling_ Springer, 2017, pp. 173–195
  * (4) Mandy Yan Dang, Gavin Yulei Zhang, Beverly Amer and Kevin Trainor  “Understanding Students Perception Differences on Blended Learning. An Explorative Study among College Students”  In _Academy of Educational Leadership Journal_ 24.1 Jordan Whitney Enterprises, Inc, 2020, pp. 1–15
  * (5) William H DeLone and Ephraim R McLean  “Information systems success: The quest for the dependent variable”  In _Information systems research_ 3.1 INFORMS, 1992, pp. 60–95
  * (6) William H DeLone and Ephraim R McLean  “The DeLone and McLean model of information systems success: a ten-year update”  In _Journal of management information systems_ 19.4 Taylor & Francis, 2003, pp. 9–30
  * (7) Anh-Nguyet Diep, Chang Zhu, Katrien Struyven and Yves Blieck  “Who or what contributes to student satisfaction in different blended learning modalities?”  In _British Journal of Educational Technology_ 48.2 Wiley Online Library, 2017, pp. 473–489
  * (8) Anabela Gomes and António José Mendes  “Learning to program – difficulties and solutions”  In _International Conference on Engineering Education–ICEE_ 7, 2007
  * (9) Dale L Goodhue and Ronald L Thompson  “Task-technology fit and individual performance”  In _MIS quarterly_ JSTOR, 1995, pp. 213–236
  * (10) Aldo Gordillo  “Effect of an instructor-centered tool for automatic assessment of programming assignments on students’ perceptions and performance”  In _Sustainability (Switzerland)_ , 2019  DOI: 10.3390/su11205568
  * (11) Malek Simon Grimm and Ralf Wagner  “The Impact of Missing Values on PLS, ML and FIML Model Fit”  In _Archives of Data Science, Series A_ 6.1, 2020, pp. 4
  * (12) Joseph Hair and Abdullah Alamer  “Partial least squares structural equation modeling (PLS-SEM) in second language and education research: Guidelines using an applied example”  In _Research Methods in Applied Linguistics_ 1.3 Elsevier, 2022, pp. 100027
  * (13) Joseph F Hair et al.  “Partial least squares structural equation modeling-based discrete choice modeling: an illustration in modeling retailer choice”  In _Business Research_ 12.1 Springer, 2019, pp. 115–142
  * (14) Joseph F Hair Jr, G Tomas M Hult, Christian M Ringle and Marko Sarstedt  “A primer on partial least squares structural equation modeling (PLS-SEM)”  Sage publications, 2021
  * (15) Joseph F Hair Jr et al.  “Partial least squares structural equation modeling (PLS-SEM) using R: A workbook”  Springer Nature, 2021
  * (16) Joseph F Hair Jr and Marko Sarstedt  “Factors versus composites: Guidelines for choosing the right structural equation modeling method”  In _Project Management Journal_ 50.6 SAGE Publications Sage CA: Los Angeles, CA, 2019, pp. 619–624
  * (17) Hieke Keuning, Johan Jeuring and Bastiaan Heeren  “A systematic literature review of automated feedback generation for programming exercises”  In _ACM Transactions on Computing Education (TOCE)_ 19.1 ACM New York, NY, USA, 2018, pp. 1–43
  * (18) Stephan Krusche and Andreas Seitz  “ArTEMiS: An automatic assessment management system for interactive learning”  In _Proceedings of the 49th ACM Technical Symposium on Computer Science Education_ , 2018, pp. 284–289
  * (19) Hengky Latan, Richard Noonan and L Matthews  “Partial least squares path modeling”  In _Partial least squares path modeling: basic concepts, methodological issues and applications_ Springer, 2017
  * (20) Shu-Sheng Liaw and Hsiu-Mei Huang  “A study of investigating learners attitudes toward e-learning”  In _5th international conference on distance learning and education_ 12, 2011, pp. 28–32
  * (21) Yi-Mei Lin, Guan-Yu Lin and James M Laffey  “Building a social and motivational framework for understanding satisfaction in online learning”  In _Journal of Educational Computing Research_ 38.1 SAGE Publications Sage CA: Los Angeles, CA, 2008, pp. 1–27
  * (22) Wen-Shan Lin  “Perceived fit and satisfaction on web learning performance: IS continuance intention and task-technology fit perspectives”  In _International Journal of Human-Computer Studies_ 70.7 Elsevier, 2012, pp. 498–507
  * (23) Andrew Luxton-Reilly et al.  “Introductory programming: a systematic literature review”  In _Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education_ , 2018, pp. 55–106
  * (24) Anna Mayr and Stefan Oppl  “Higher education at the margins–success criteria for blended learning systems for marginalized communities”  In _Education and Information Technologies_ Springer, 2022, pp. 1–39
  * (25) Igor Mekterovic and Ljiljana Brkic  “Setting up automated programming assessment system for higher education database course”  In _International Journal of Education and Learning Systems_ 2 International Association of ResearchScience, 2017
  * (26) Igor Mekterović, Ljiljana Brkić, Boris Milašinović and Mirta Baranović  “Building a Comprehensive Automated Programming Assessment System”  In _IEEE Access_ 8, 2020, pp. 81154–81172  DOI: 10.1109/ACCESS.2020.2990980
  * (27) Mumtaz Ali Memon, Hwa Cheah Jun, Hiram Ting and Chin Wei Francis  “Mediation analysis issues and recommendations”  In _Journal of Applied Structural Equation Modeling_ 2.1 Sarawak Research Society, 2018, pp. i–ix
  * (28) Christian Nitzl, Jose L Roldan and Gabriel Cepeda  “Mediation analysis in partial least squares path modeling: Helping researchers discuss more sophisticated models”  In _Industrial management & data systems_ Emerald Group Publishing Limited, 2016
  * (29) Raymond Pettit et al.  “An Empirical Study of Iterative Improvement in Programming Assignments”  In _Proceedings of the 46th ACM Technical Symposium on Computer Science Education_ Kansas City Missouri USA: ACM, 2015, pp. 410–415  DOI: 10.1145/2676723.2677279
  * (30) Raymond Scott Pettit et al.  “Are automated assessment tools helpful in programming courses?”  In _2015 ASEE Annual Conference & Exposition_, 2015, pp. 26–230
  * (31) Kristopher J Preacher and Andrew F Hayes  “Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models”  In _Behavior research methods_ 40.3 Springer, 2008, pp. 879–891
  * (32) Ricardo Queirós, Mário Pinto and Teresa Terroso  “Computer Programming Education in Portuguese Universities”  In _First International Computer Programming Education Conference (ICPEC 2020)_ , 2020  Schloss Dagstuhl-Leibniz-Zentrum für Informatik
  * (33) Soumya Ray, Nicholas Danks and André Calero Valdez  “SEMinR: Domain-specific language for building, estimating, and visualizing structural equation models in R”  In _Estimating, and Visualizing Structural Equation Models in R (August 6, 2021)_ , 2021
  * (34) Felipe Restrepo-Calle, Jhon J. Ramírez Echeverry and Fabio A. González  “Continuous assessment in a computer programming course supported by a software tool”  In _Computer Applications in Engineering Education_ 27.1, 2019, pp. 80–89  DOI: https://doi.org/10.1002/cae.22058
  * (35) Manuel Rubio-Sánchez, Päivi Kinnunen, Cristóbal Pareja-Flores and Ángel Velázquez-Iturbide  “Student perception and usage of an automated programming assessment tool”  In _Computers in Human Behavior_ 31, 2014, pp. 453–460  DOI: https://doi.org/10.1016/j.chb.2013.04.001
  * (36) M Rungtusanatham, JW Miller and Kenneth K Boyer  “Theorizing, testing, and concluding for mediation in SCM research: Tutorial and procedural recommendations”  In _Journal of Operations Management_ 32.3 Elsevier, 2014, pp. 99–113
  * (37) Clemens Sauerwein et al.  “Lecturers’ and Students’ Experiences with an Automated Programming Assessment System”  In _Proceedings of the 56th Hawaii International Conference on System Sciences 2023_ , 2023
  * (38) Galit Shmueli, Soumya Ray, Juan Manuel Velasquez Estrada and Suneel Babu Chatla  “The elephant in the room: Predictive performance of PLS models”  In _Journal of Business Research_ 69.10 Elsevier, 2016, pp. 4552–4564
  * (39) Galit Shmueli et al.  “Predictive model assessment in PLS-SEM: guidelines for using PLSpredict”  In _European journal of marketing_ Emerald Publishing Limited, 2019
  * (40) Draylson M Souza, Katia R Felizardo and Ellen F Barbosa  “A systematic literature review of assessment tools for programming assignments”  In _2016 IEEE 29Th international conference on software engineering education and training (CSEET)_ , 2016, pp. 147–156  IEEE
  * (41) Pei-Chen Sun et al.  “What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction”  In _Computers & education_ 50.4 Elsevier, 2008, pp. 1183–1202
  * (42) Maureen M Villamor  “A review on process-oriented approaches for analyzing novice solutions to programming problems”  In _Research and Practice in Technology Enhanced Learning_ 15.1 SpringerOpen, 2020, pp. 1–23
  * (43) Jen-Her Wu, Robert D Tennyson and Tzyh-Lih Hsia  “A study of student satisfaction in a blended e-learning system environment”  In _Computers & education_ 55.1 Elsevier, 2010, pp. 155–164
  * (44) Wenchieh Wu and Lan-Yin Hwang  “The effectiveness of e-learning for blended courses in colleges: A multi-level empirical study”  In _International Journal of Electronic Business Management_ 8.4 Electronic Business Management Society, Taiwan, 2010, pp. 312
  * (45) Yulei Gavin Zhang and Mandy Yan Dang  “Understanding essential factors in influencing technology-supported learning: A model toward blended learning success”  In _Journal of Information Technology Education. Research_ 19 Informing Science Institute, 2020, pp. 489
  * (46) Xinshu Zhao, John G Lynch Jr and Qimei Chen  “Reconsidering Baron and Kenny: Myths and truths about mediation analysis”  In _Journal of consumer research_ 37.2 The University of Chicago Press, 2010, pp. 197–206
  * (47) Chang Zhu  “Student satisfaction, performance, and knowledge construction in online collaborative learning”  In _Journal of Educational Technology & Society_ 15.1 JSTOR, 2012, pp. 127–136