# PointResNet: Residual Network for 3D Point Cloud Segmentation and
Classification

###### Abstract

Point cloud segmentation and classification are some of the primary tasks in
3D computer vision with applications ranging from augmented reality to
robotics. However, processing point clouds using deep learning-based
algorithms is quite challenging due to the irregular point formats.
Voxelization or 3D grid-based representation are different ways of applying
deep neural networks to this problem. In this paper, we propose PointResNet, a
residual block-based approach. Our model directly processes the 3D points,
using a deep neural network for the segmentation and classification tasks. The
main components of the architecture are: 1) residual blocks and 2) multi-
layered perceptron (MLP). We show that it preserves profound features and
structural information, which are useful for segmentation and classification
tasks. The experimental evaluations demonstrate that the proposed model
produces the best results for segmentation and comparable results for
classification in comparison to the conventional baselines.

Index Terms—  3D point cloud, CNN, Residual Network, Segmentation,
Classification

##  1 INTRODUCTION

3D point cloud has a wide range of applications in various areas such as
robotics, animation, autonomous driving, industrial metrology, augmented
reality, etc. [1, 2]. Point cloud data represents 3D geometry as embedded sets
of data points in continuous space, unlike images that are arranged on regular
pixel grids. Hence, it makes the structural representation of 3D points
different from the images. As a consequence, deep neural networks, such as
convolutional neural networks (CNN), developed for 2D images can not be
directly applied in processing 3D point clouds.

In order to address the issue mentioned above, various methods have been
reported in the literature. Some methods voxelize the 3D points to employ the
3D discrete convolution [3, 4]. However, it increases the computational
complexity and the memory requirement for processing the 3D points. Prior
works like sparse convolutional neural networks can also be employed, but they
operate only on nonempty voxels [5]. Further, some of the approaches directly
operate on 3D points and extract information via soft max-pooling layers [6,
7, 8]. Another family of methods combines the information in the form of 3D
points set into a graph for directive passing [9, 10].

Prior literature [11] has shown that residual blocks-based approaches are
employed as base networks by most state-of-the-art segmentation methods for 2D
images. In this context, we propose a ResNet [12] inspired architecture that
accepts unordered 3D points directly as input and produces per-point segment
or class labels. The network is developed with multi-layer perceptron (MLP)
layers and ResBlocks. MLP layers perform convolution and ResBlocks facilitate
better information flow to produce relevant features for segmentation and
classification tasks. Hence, the main contribution of our network is to employ
ResBlock along with MLP layers to produce features that are well-suited for
segmentation.

The rest of the paper is structured as follows, Section 2 discusses some part
segmentation and object classification related work. Section 3 demonstrates
the proposed approach of 3D point cloud segmentation and classification.
Section 4 shows the experimental study of our model and compares qualitative
and quantitative results with baseline approaches. Section 5 concludes our
work with future directions.

##  2 RELATED WORK

One way to apply deep learning based methods such as CNN on 3D points is to
transform these onto volumetric grid [3, 13, 14, 15, 16]. Further, 3D point
clouds can also be projected onto several image planes and thus mapped into a
2-dimensional representation. The final output is generated by applying 2D CNN
on the representations, and feature fusion [17, 18]. However, these methods
were inefficient in creating dense grids on projection planes due to the
sparse nature of point clouds. These planes can significantly impact
recognition ability, and occlusion in 3D can affect the accuracy.

In the study of 3D point clouds, there has been some recent focus on applying
deep learning for tasks such as segmentation [6, 7, 19], classification [6, 7,
12, 20], denoising [21, 22], etc. In this paper, we address the shape
classification and segmentation problems. Here, we discuss the state-of-the-
art methods related to these problems.

Fig. 1: PointResNet architecture The network takes n input points and applies
transformations, several multi-layer perceptron (MLP) layers with skip
connections, and then aggregates features using max pooling. The
classification network applies fully connected and dropout layers and obtains
k scores for k classes. The segmentation network extends the classification
network by combining local and global features, using convolutional layers,
and finally giving per point scores as output. The connected skip layers
represent ResBlock-n, where n is the number of features of a single block
layer. Numbers in MLP layers represent a number of features.(Zoom in for best
view)

PointNet [6] solves the problem of order invariance caused by simple deep
neural network (DNN) models by directly processing 3D point clouds. The
architecture consists of three main components: max-pooling layers as symmetry
function, shared multi-layer perceptrons (MLPs), and a local-global
information fusion module to classify and segment 3D point clouds. However,
this network cannot describe shape. PointNet++ [7] first partitions points set
into overlapping local regions and then produces higher-level features by
extracting, grouping, and processing the local features.

PointConv [23] and KPConv [24] follow continuous convolution strategy without
quantization. Basically, these methods directly apply convolution to the 3D
point set. It is a density-weighted convolution that can fully estimate the 3D
convolution on any unordered set of 3D points. The weight and density
functions are estimated by MLP networks and kernel, respectively. The work in
[6] cannot capture the local structure, therefore making it difficult to learn
more fine-grained patterns and complex scenes. CurveNet [12] proposes that
continuous descriptors (continuous sequences of point segments) are sufficient
to estimate point cloud geometry. Rotation Transformation Network (RTN) [25]
attempts to extract pose information from 3D objects for analyzing point cloud
under self-supervision.

In deep learning-based models, it is hard to ensure that the finally generated
features are better than the intermediate features. ResNet [12] improves the
scenario by including identity mapping. It enables the researchers to build
deeper network architectures that can learn the difference between the input
and target. It ensures better accuracy in the results. Proceeding in this
direction, we propose to include ResBlocks along with MLP and CNN layers. The
final features of this model are rich in information that enables better
segmentation results while performing not worse than the state-of-the-art
classification results.

##  3 PROPOSED APPROACH

A point cloud is defined as an unordered set of 3D points. Each 3D point
$P_{i}$ is a vector that contains information about the $x,y,$ and $z$
coordinates of that point, along with extra details such as color and normal.
These vectors of point clouds are directly used as the input to our approach.

###  3.1 Network architecture

In our proposed method, feature transformation network is similar to the
PointNet [6] model. PointNet can not capture local context at different
scales, and hence by including different residual layers and additional mlp
layers, we try to extract the missing information from the input data. While
taking 3D point clouds as input, we need to consider the three main properties
of 3D point clouds that differentiate them from 2D images: 1) unordered set,
2) neighboring points can form meaningful interactions, and 3) transformation
invariance.

In this scenario, the network architecture (shown in Figure 1) is designed to
consider these unordered point sets into the model directly. Further, we only
require the cartesian coordinates of 3D points [6]. Hence, we ignore the
excess feature channels by mapping the unordered point set ${\\{x_{i}\\}}$
${\in}$ ${R_{d}}$ to a vector $f_{i}$ by equation 1.

$\displaystyle f_{i}=g(\\{h(\\{x_{i}\\}).$ (1)

Here, the function $h$ depicts an MLP layer, and $g$ is a function of single
variable along with maxpooling. The transformed vector $f_{i}$ is passed
through three MLP layers. MLP layer maps each $n$ input points to 64
dimensions. The introduction of residual layers in the MLP concatenates the
input features to output features. This enables a smoother flow of information
from the input to the output.

$\displaystyle\mathcal{B}(y)=\mathcal{F}(y)+y.$ (2)

The identity connection comes from the feature $y$. It learns the residual
$\mathcal{F}(y)$. The $\mathcal{B}(y)$ is further transformed using Eq. (1) to
pass the relevant features. The output feature is then passed through an MLP
layer, followed by a residual connection. The resultant features are then
mapped to the dimension of 1024 via two ResBlocks. Each of the blocks consists
of several MLP layers with a residual connection. The 1024-dimensional vector
is fed to the segmentation network for the segmentation task. The same output
vector is fed to the classification network for classification purposes.

A symmetric function (max pooling) is used to create a global feature vector
to aggregate information from all the higher dimensional points. This vector
ensures that the output is independent of the different input permutations. In
the end, a five-layer fully-connected network assigns classification scores to
the $n$ classes. For the segmentation task, we concatenate the local
64-dimensional feature with global features. In the end, we use MLPs to lower
the dimension from 1088 to $m$ and give an output array of $n\times m$. Each
row of the output matrix corresponds to each of the $n$ points, and the
columns are related to the $m$ sub-categories. The local and global feature
aggregation is quite essential to predict per-point quantities based on local
and global-level semantics. Thus our architecture considers explicitly all the
essential properties of the point cloud that require special attention.

Our architecture solves the problem of transformation invariance by
introducing transformations such that output for a particular object remains
the same even after rotational or translation transforms. The invariance
problem is addressed by input and feature transformations sub-networks that
normalize the pose for the input point cloud. The network applies
transformations on the input point cloud similar to spatial transformers [26]
to achieve pose normalization. A geometric transformation is equivalent to
multiplying each point with a transformation matrix.

Thus during input transformation, a regression network predicts a $3\times 3$
transformation matrix, which is multiplied by the $n\times 3$ input.
Similarly, feature transformation predicts a $64\times 64$ transformation
matrix, which gets multiplied with the $n\times 64$ input.

The proposed architecture performs the segmentation and classification of 3D
point clouds from a directly sampled point cloud. Our network adds a
regularization term with the cross-entropy loss function to avoid over-fitting
caused by the increased training parameters and instability during training.

Fig. 2: Visualization of part segmentation results on ShapeNetPart dataset.

###  3.2 Loss function

We use the softmax function on the output feature to calculate a probability
score $\sigma(x_{j})$ as shown in equation 3.

$\displaystyle\sigma(x_{j})=\frac{e^{x_{j}}}{\Sigma_{i}e^{x_{i}}}$ (3)

which is the used to compute the cross-entropy loss in equation 4.

$\displaystyle H(p,q)=-\Sigma_{x}p(x)\log q(x).$ (4)

Here, $p(x)$ = labels, and $q(x)=\sigma(x_{j})$ or $1-\sigma(x_{j})$ depending
on whether $j$ is the correct ground truth or not.

In addition to the cross-entropy loss, we include a regularization term to
stabilize the training, which is given by equation 5.

$\displaystyle L_{reg}=||I-AA^{T}||^{2}.$ (5)

Here, $A$ is the predicted feature alignment matrix. The feature
transformation matrix is constrained to be closed to the orthogonal matrix.
Hence, orthogonal transformation will have all the information of the input.
Thus, the final loss function becomes as in equation 6.

$\displaystyle\text{Loss}=H+\alpha\times L_{reg},$ (6)

where $\alpha$ is the assigned weight to the regularization term.

##  4 EXPERIMENTS

We evaluate our architecture on three standard datasets; ModelNet40 [27],
ModelNet10 [27], and ShapeNetPart [28]. Further, we compare our results with
the baselines and the existing methods in segmentation and classification
tasks, which demonstrates the effectiveness of our model. The advantage of our
architecture is further demonstrated by analyzing the effect of various
parameters along with different architectures.

We implement our network on on a GeForce RTX 2080 Ti GPU using Tensorflow [29]
environment. For training, we use 0.001 as learning rate that gets reduced
with a decay parameter of 0.8. We choose 0.5 as the decay rate for batch
normalization with a batch size of 32. For classification, we train our model
for 250 epochs, and for segmentation, we train our model for 200 epochs with
the adam optimizer.

|  Eval acc $\uparrow$
---|---
PointNet [6] | 90.6
PointNet++ [7] | 88.4
ECC [20] | 90.8
RTN [25] | 92.6
PointResNet (Ours) | 94.79
Table 1: Quantitative comparison of part segmentation accuracy on ShapeNetPart
dataset with the existing state-of-the-art methods.

###  4.1 Datasets

ShapeNetPart: ShapeNet is a large-scale dataset of 3D shapes containing 16
different single object classes. Each object includes many parts that can be
segmented.

ModelNet: The Princeton ModelNet project created the ModelNet40 dataset that
includes 12,311 pre-aligned shapes from 40 categories. ModelNet10 is a subset
of the ModelNet40 dataset that consists of 4,899 shapes from 10 categories.

The train-test split for both datasets is 80%-20%.

Fig. 3: Qualitative comparison of part segmentation results on ShapeNetPart
dataset.

###  4.2 Results

####  4.2.1 Part segmentation on ShapeNetPart dataset

Table 1 demonstrates the quantitative comparison with our baselines and the
state-of-the-art methods. It can be observed from Table 1 that our proposed
model outperforms the existing PointNet [6], PointNet++ [7], ECC [20], and RTN
[25] methods for segmented parts. The reason for improvements can be observed
visually in Figure 2, where segmented results produced by our model are very
close to the ground-truths. Further, we also compare our results qualitatively
with PointNet [6] in Figure 3. One can note that our segmented part results
are better than this method (see the marked area). Having achieved the desired
results on segmentation using our proposed approach, we now validate our
method for classification.

####  4.2.2 Classification on ModelNet dataset

Table 2 shows the comparison of our method with a representative set of the
previously proposed state-of-the-art: PointNet [6], PointNet++ [7], ECC [20],
RTN [25] and ResNet-50 [30]. We follow similar criteria to run our method on
ModelNet datasets. To compare our results with these methods, we take the
accuracy from the publicly available articles. In Table 2, it can be noted
that our proposed model produces better or comparable results for
classification of the ModelNet datasets.

| ModelNet10 | ModelNet40
---|---|---
|  Eval acc $\uparrow$ |  | Eval avg
---
class acc $\uparrow$
Eval acc $\uparrow$ |  | Eval avg
---
class acc $\uparrow$
PointNet [6] | 92.52 | 92.08 | 89.2 | 86.2
PointNet++ [7] | - | - | 91.9 | -
PointConv [23] | - | - | 92.5 | -
ECC [20] | 90.0 | - | 83.2 | -
RTN [25] | - | - | 90.2 | 86.5
ResNet-50 [30] | - | - | 66.3 | -
PointResNet (Ours) | 92.86 | 92.29 | 88.76 | 85.58
Table 2: Quantitative comparison of classification accuracy on ModelNet10 and
ModelNet40 with the state-of-the-art methods.

###  4.3 Ablation study

This section shows the importance of choosing 10 MLP layers and 4 residual
blocks for our proposed model. For this task, we implement three more
architecture PointResNet 11, 15, and 16, as mentioned in Table 3. In
PointResNet 11 model, we increase the sampling points from 1024 to 2048 with
one MLP layer before transform-net. Further, in comparison to PointResNet 11
model, the PointResNet 15 network consists of additional MLP layer of size 64,
three MLP layers of size 512, and one MLP layer of size 1024.

PointResNet 16 comprises of an encoder and decoder of UNet architecture [31].
Hence, we add 128 and 64 dimensional convolution transpose layers in the
decoder part of the network. For segmentation task, PointResNet 11 and 16
models are evaluated on ShapeNetPart. The obtained accuracies are tabulated in
Table3 (right column). Both accuracies (93.67% and 94.73%) are inferior than
the proposed model’s result. In Table 3 (second and third columns), we show
the classification results on ModelNet 40 (third column) and ModelNet 10
(second column) datasets for various models. One can note that the performance
of our PointResNet 10 is better as compared to other models.

| Classification |  | Part
---
Segmentation
|  | ModelNet10
---
Eval acc $\uparrow$
| ModelNet40
---
Eval acc $\uparrow$
| ShapeNetPart
---
Eval acc $\uparrow$
PointResNet 15 | 91.62 | - | -
PointResNet 11 | 92.29 | 88.67 | 93.67
| PointResNet 16
---
\- UNet [31]
- | 88.75 | 94.73
PointResNet 10 | 92.86 | 88.76 | 94.79
Table 3: Quantitative comparison of evaluated accuracy with proposed and our
three newly implemented models by using the ModelNet40 and 10 dataset for
classification and ShapeNetPart dataset for part segmentation.

##  5 CONCLUSION

We have proposed a deep neural network based on residual blocks and MLPs for
the 3D point cloud segmentation and classification. Our proposed architecture
directly operates on the 3D point cloud data instead of using any grid-based
representation. The residual connection has shown an essential role in
preserving the deeper feature information of 3D points essential for
segmentation. The experimental results have depicted that our proposed unified
framework is capable of performing outstanding segmentation and classifying as
good as the baselines and state-of-the-art.

We have implemented our network for part segmentation and object
classification, and we plan to extend to segment semantic scenes by simply
considering point labels as semantic object classes. In future, we plan to
improve the classification of 3D points using the data-augmentation technique.

## References

  * [1] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun,  “Point transformer,” 2020.
  * [2] Seema Kumari, Ranjeet Ranjhan Jha, Arnav V. Bhavsar, and Aditya Nigam,  “Autodepth: Single image depth map estimation via residual cnn encoder-decoder and stacked hourglass,”  2019 IEEE International Conference on Image Processing (ICIP), pp. 340–344, 2019.
  * [3] Daniel Maturana and Sebastian Scherer,  “Voxnet: A 3d convolutional neural network for real-time object recognition,”  in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015, pp. 922–928.
  * [4] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser,  “Semantic scene completion from a single depth image,”  arXiv preprint arXiv:1611.08974, 2016.
  * [5] Chris Choy, JunYoung Gwak, and Silvio Savarese,  “4d spatio-temporal convnets: Minkowski convolutional neural networks,”  06 2019, pp. 3070–3079.
  * [6] R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas,  “Pointnet: Deep learning on point sets for 3d classification and segmentation,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 77–85.
  * [7] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas,  “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,”  arXiv preprint arXiv:1706.02413, 2017.
  * [8] Seema Kumari and Shanmuganathan Raman,  “3d point cloud completion using stacked auto-encoder for structure preservation,”  in 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 3123–3127.
  * [9] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon,  “Dynamic graph cnn for learning on point clouds,”  ACM Transactions on Graphics (TOG), 2019.
  * [10] Guohao Li, Matthias Müller, Guocheng Qian, Itzel Carolina Delgadillo Perez, Abdulellah Abualshour, Ali Kassem Thabet, and Bernard Ghanem,  “Deepgcns: Making gcns go as deep as cnns,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [11] Fei Liu, Jing Liu, Jun Fu, and Hanqing LU,  “Improving residual block for semantic image segmentation,”  in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), 2018, pp. 1–5.
  * [12] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai,  “Walk in the cloud: Learning curves for point clouds shape analysis,”  arXiv preprint arXiv:2105.01288, 2021.
  * [13] Jing Huang and Suya You,  “Point cloud labeling using 3d convolutional neural network,”  in 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, pp. 2670–2675.
  * [14] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao,  “3d shapenets: A deep representation for volumetric shapes,”  in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1912–1920.
  * [15] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger,  “Octnet: Learning deep 3d representations at high resolutions,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3577–3586.
  * [16] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong,  “O-cnn: Octree-based convolutional neural networks for 3d shape analysis,”  ACM Transactions On Graphics (TOG), vol. 36, no. 4, pp. 1–11, 2017\.
  * [17] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas,  “Volumetric and multi-view cnns for object classification on 3d data,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5648–5656.
  * [18] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller,  “Multi-view convolutional neural networks for 3d shape recognition,”  in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 945–953.
  * [19] Kangcheng Liu, Zhi Gao, Feng Lin, and Ben M Chen,  “Fg-net: Fast large-scale lidar point cloudsunderstanding network leveraging correlatedfeature mining and geometric-aware modelling,”  arXiv preprint arXiv:2012.09439, 2020.
  * [20] Martin Simonovsky and Nikos Komodakis,  “Dynamic edge-conditioned filters in convolutional neural networks on graphs,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3693–3702.
  * [21] Marie-Julie Rakotosaona, Vittorio La Barbera, Paul Guerrero, Niloy J Mitra, and Maks Ovsjanikov,  “Pointcleannet: Learning to denoise and remove outliers from dense point clouds,”  in Computer Graphics Forum. Wiley Online Library, 2020, vol. 39, pp. 185–203.
  * [22] Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski,  “Total denoising: Unsupervised learning of 3d point cloud cleaning,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 52–60.
  * [23] Wenxuan Wu, Zhongang Qi, and Li Fuxin,  “Pointconv: Deep convolutional networks on 3d point clouds,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9621–9630.
  * [24] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas,  “Kpconv: Flexible and deformable convolution for point clouds,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6411–6420.
  * [25] Shuang Deng, Bo Liu, Qiulei Dong, and Zhanyi Hu,  “Rotation transformation network: Learning view-invariant point cloud for classification and segmentation,”  in 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2021, pp. 1–6.
  * [26] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.,  “Spatial transformer networks,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [27] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao,  “3d shapenets: A deep representation for volumetric shapes,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1912–1920.
  * [28] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, and Silvio et al. Savarese,  “ShapeNet: An Information-Rich 3D Model Repository,”  Tech. Rep., Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [29] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.,  “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,”  arXiv preprint arXiv:1603.04467, 2016.
  * [30] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira,  “Perceiver: General perception with iterative attention,”  in International Conference on Machine Learning. PMLR, 2021, pp. 4651–4664.
  * [31] O Ronneberger, P Fischer, and T Brox,  “U-net: Convolutional networks for biomedical image segmentation. arxiv,”  Lecture Notes in Computer Science, vol. 2015, 2015.


11institutetext: LMU Munich, Germany 11email:
<EMAIL_ADDRESS>
Accepted to the International Journal of Engineering Pedagogy (iJEP; eISSN:
2192-4880).
This is the author’s version.

# AI-enhanced Auto-correction of Programming Exercises: How Effective is
GPT-3.5?

Imen Azaiz  0009-0005-6458-4169 Oliver Deckarm  Sven Strickroth
0000-0002-9647-300X

###### Abstract

Timely formative feedback is considered as one of the most important drivers
for effective learning. Delivering timely and individualized feedback is
particularly challenging in large classes in higher education. Recently Large
Language Models such as GPT-3 became available to the public that showed
promising results on various tasks such as code generation and code
explanation. This paper investigates the potential of AI in providing
personalized code correction and generating feedback. Based on existing
student submissions of two different real-world assignments, the correctness
of the AI-aided e-assessment as well as the characteristics such as fault
localization, correctness of hints, and code style suggestions of the
generated feedback are investigated. The results show that 73 % of the
submissions were correctly identified as either correct or incorrect. In 59 %
of these cases, GPT-3.5 also successfully generated effective and high-quality
feedback. Additionally, GPT-3.5 exhibited weaknesses in its evaluation,
including localization of errors that were not the actual errors, or even
hallucinated errors. Implications and potential new usage scenarios are
discussed.

###### Keywords:

E-Assessement Personalized Feedback GPT-3.5 Large Language Model Programming
Education Formative Assessment

##  1 Introduction

Learning programming is often experienced as hard by students [1]. It involves
various aspects like mastering the formal programming language syntax,
problem-solving skills, computational thinking, and testing/debugging
abilities. Hence, learning programming requires practice, commonly in the form
of weekly homework assignments in traditional programming courses. Timely
feedback is a key driver of learning [2]. However, introductory courses are
frequently characterized by large class sizes with several hundred students,
making manual teacher corrections for timely feedback impractical due to
resource limitations [3]. Crowdsourcing feedback through peer review has
limitations, relying on the motivation and expertise of participating students
[4, 5, 6]. Many e-assessment systems have been developed to
(semi-)automatically assess students’ submissions and provide feedback [7].
However, a common issue is that these systems often focus solely on functional
correctness, through employ standard compilers and test cases. Moreover,
developing sophisticated tests is very laborious.

Recently, powerful and accessible AI technologies such as GPT-3.5 have
emerged, demonstrating their ability to solve programming assignments and pass
exams [8, 9, 10, 11]. Hence, the idea was born to investigate whether the
Large Language Model (LLM) GPT-3.5 can be used to assess students’ solutions
and deliver automated feedback [12]. This study explores the advantages of an
AI-assisted approach in introductory programming education, where assignment
instructions and student solutions are sent to GPT-3.5 to obtain tailored
feedback at scale. The main research question is whether GPT-3.5 can provide
“good” personalized feedback, including code style suggestions, for
programming tasks. The following research questions are investigated:

  * •

RQ1: What is the quality of GPT-3.5 in determining the submissions’
correctness?

  * •

RQ2: How can the GPT-3.5-generated feedback be characterized regarding length,
suggestion hints, compliance to assignment instruction, code correction, error
localization, and personalization?

  * •

RQ3 What are the strengths and weaknesses of GPT-3.5 in delivering
personalized feedback and code corrections?

By addressing these questions, we contribute to the understanding of how AI
can be effectively utilized to provide personalized and accurate support in
beginner programming education. Our findings provide insights into
implications and potential limitations of the AI-assisted approach, benefiting
educators and developers aiming to improve programming instruction and
support. The paper’s structure includes a presentation of related research in
programming education and e-assessment, a description of our empirical study’s
methodology, the presentation of study results in the evaluation section, and
concludes with a discussion and conclusions.

##  2 Related Research

LLMs such as GPT and ChatGPT became available recently and have already been
investigated in the domain of programming to assist developers, educators, and
learners to explore capabilities and limitations: Tian et al. explored the
potential of ChatGPT as an assistant bot for programmers and found ChatGPT to
be effective in tasks such as code generation, program repair, and code
summarization [9]. Moreover, its summary explanations of incorrect code offer
valuable insights into the original intentions of developers. Surameery &
Shakor [13] show that ChatGPT can provide effective debugging assistance by
predicting and explaining programming errors. In the context of automatic
program repair, ChatGPT has shown to have a similar performance on a standard
benchmark set as existing deep-learning program repair techniques and
outperforms standard program repair approaches [14].

In the domain of education, one of the pioneering papers used OpenAI Codex
model and showed that it can solve CS1 programming exercises with a quality
comparable to students, but sometimes struggles with output formatting [15].
In a follow up study, Codex outscored most students for CS2 assignments and
exams [11]. Becker et al. [16] discuss the rapid advancement of AI-driven code
generation tools and emphasize the need for educators to address their
potential impact on teaching. Danny et al. [17] explore the impact of LLMs’ on
computing education, summarizing recent studies on their performance and
pedagogical influence, covering responsible tool use, societal implications,
and readiness for dynamic environments.

LLMs have already been successfully used to generate programming assignments
[18] and to explain syntax errors and compiler error messages in an actionable
way that can be used as feedback for students [19, 20]. Furthermore, LLMs have
been used to generate good code explanations [18] and a study by Macneil et
al. [21] showed that automatic generated code explanations by LLMs in an
e-book were appreciated by students . In a study by Leinonen et al. [22],
students rated generated code explanations as better than those of their
fellow peers. This research underlines that LLMs can be used to support
students in learning programming and students do not reject their usage.
Common is that all authors argue that instructors need to inspect the
generated results.

The capabilities of LLMs for providing feedback was already investigated (e.
g., [23, 24]). Kiesler et al. [23] investigated ChatGPT’s responses to
students seeking assistance with introductory programming tasks. They also
assessed ChatGPT’s performance in providing formative feedback to students on
programming tasks, demonstrating its effectiveness in areas like compilation
error detection, correction, and textual explanations. However, they only
provided the code written by the student to ChatGPT and not the assignment
specification. While doing this research, Balse et al. [24] published a paper
investigating the feasibility of GPT-3 to check, critique and suggest changes
to code written by students in an online programming exam of an undergraduate
Python programming course. They found a high variability in the accuracy
(between 57 and 79 %) and received incorrect and inconsistent feedback. This
research focuses on Java programming assignments, analyzes the performance of
GPT-3.5 based on the correctness of the submission, and systematically
analyses qualitative aspects of the feedback content such as (correct) fault
localization, coding style suggestions, and type of the feedback (contains
some code, no code, only code).

In summary, prior research demonstrates the effectiveness of LLMs in assisting
developers, learners, and educators in programming (education). However, there
is a research gap in assessing GPT-3.5’s ability to correct student code
submissions and providing feedback while maintaining assignment requirements
and characteristics and quality of the generated feedback. LLMs offer an
advantage by eliminating the need for test case development compared to
traditional e-assessment systems. Our research aims to assess GPT-3.5’s
ability to provide personalized, assignment-compliant feedback for enhanced
auto-correction, ensuring effective and valuable corrections that match the
submitted code and adhere to the assignment’s specifications.

##  3 Methodology and Approach

Our primary goal is to assess the quality of GPT-3.5-generated feedback for
student programming assignment submissions. We aim to ascertain whether
GPT-3.5 can efficiently provide automated formative feedback for programming
assignments. An empirical study involving two programming assignments selected
from an introductory course at Ludwig Maximilian University Munich, Germany
was conducted. Approximately 900 students, majoring or minoring in computer
science, attended this course, which included ten weekly homework assignments.
Students uploaded their solutions to the e-assessment system GATE [25] to
receive instant feedback. Additionally, a weekly peer review was conducted for
one of the assignments (cf. [6]). Participation in both, the assignments and
peer review, was voluntary. Teaching assistants held weekly exercise sessions
to discuss homework assignment solutions. We selected a less complex
assignment from the third assignment sheet and a more challenging one from the
seventh for this study. We randomly sampled approx. 10 % of student
submissions for these two assignments for an in-depth analysis. For privacy
and ethical compliance, we exclusively analyzed submissions from students who
explicitly consented to their use for research purposes (695 out of 900
students). There are 33 sampled student submissions for the first selected
assignment. The instruction is (translated to English): “Write a Java
application named SimpleWhileLoop that prints all odd numbers from 1 to 10
using a while loop, and prints ‘Boom!’ (without quotation marks) afterwards.”
The instruction of the second selected assignment is as follows (translated to
English, 16 sampled submissions): “This task refers to the abstract data types
singly linked list and queue introduced in the lecture. Implement the (given)
Queue interface based on the specifications (in the interface) for a queue
using the QueueImpl class, using a singly linked list.” Additionally, the
interface for “Queue” was provided as a Java file which specified the methods
to implement as well as a description of their semantics as JavaDoc. The Queue
interface defines several methods, including void append(), which adds a value
to the end of the queue; boolean isEmpty(), which checks if the queue is
empty; void remove(), which removes the first element from the queue; int
peek(), which returns the value of the first element in the queue (or the
given constant EMPTY_VALUE if the queue is empty); and int[] toArray(), which
returns an array containing the values of the queue. The students were
expected to use a static inner class for the node class.

Submissions were processed by the GPT-3.5 API (davinci-003 model) on June 12,
2023, three times in randomized order, resulting in 99 feedbacks for the first
assignment and 48 for the second. We experimented with various prompts to suit
our research question and settled on the following prompt structure:

> ASSIGNMENT INSTRUCTION
>
> Find all kinds of errors, including logical ones, and provide hints for
> their correction or improvement, including suggestions for code style.
>
> STUDENT SUBMISSION

To ensure a thorough evaluation, two researchers with extensive expertise in
correction and providing feedback for programming assignments meticulously
reviewed all submissions and the generated feedback. They assessed the
submissions’ correctness (both syntactic and functional) and analyzed 147
generated feedback texts for the SimpleWhileLoop and Queue tasks. The feedback
analysis covered an analysis of the classification quality (RQ1) and various
characteristics, including length, content (code-only, text with code, style
hints), hint accuracy, references to the specific submissions, and error
detection comprehensiveness (RQ2). The applied methods are described in the
respective sections before the results are presented.

##  4 Results

In this section, we present the results of the manual evaluation of
submissions and the analysis of GPT-3.5’s generated feedback. We first assess
the feedback length. Next, we report on the correctness and accuracy of
GPT-3.5’s assessment of the submissions. Finally, the evaluation of the
characteristics of the generated feedback is presented.

We examined 49 submissions in total for our investigation across the two
assignments. For the SimpleWhileLoop assignment, there are 33 submissions, of
which 19 are “correct” and 14 are “incorrect”. For the Queue assignment, there
are 16 submissions of which 4 are “correct” and 12 are “incorrect”. “Correct”
submissions met the assignment requirements accurately, while “incorrect”
submissions had syntactic or functional errors, did not fulfill the
requirements, or were incomplete (e. g., missing files or empty classes with
no solution attempt).

###  4.1 Length of the Generated Feedback

The feedback length was determined by tokenizing the feedback string using
white space (`"\s+"`) and counting the resulting tokens. The overall median
length is 82 words ($\bar{x}=85$). For the SimpleWhileLoop, the median word
count is 65 words ($\bar{x}=72$, min=21, max=156). For the Queue assignment,
the median word count is 112 words ($\bar{x}=111$, min=19, max=178). Tab. 1
shows the number of words for both assignments and for all three runs.
According to the Mann-Whitney U-Test the difference between the two
assignments is statistically significant ($U=3783$, $p=.000$, two-sided).

| SimpleWhileLoop | Queue
---|---|---
# words in feedback | 1st | 2nd | 3rd | OA | 1st | 2nd | 3rd | OA
Mean word count | 72 | 68 | 75 | 72 | 121 | 103 | 109 | 111
Median word count | 65 | 67 | 65 | 65 | 120 | 106 | 108 | 112
Min word count | 28 | 21 | 28 | 21 | 44 | 19 | 28 | 19
Max word count | 156 | 116 | 156 | 156 | 174 | 178 | 165 | 178
Table 1: Overview of Word Counts in GPT-3.5’s Generated Feedback

###  4.2 Correctness of the GPT-3.5 Evaluation

We assessed the correctness of the feedback generated for the 49 submissions
over 3 iterations, for a total of 147 feedback instances. Since GPT-3.5 does
not always explicitly label submissions as “correct” or “incorrect”, the
researchers made this determination: If the feedback solely contains
suggestions without identified errors or functional changes, we classified it
as being labeled “correct” by GPT-3.5. Conversely, if GPT-3.5 identified
errors, proposed modifications that address critical issues, or resembled a
sample solution without errors or specific hints, we interpreted it as labeled
“incorrect”. Out of the 147 submissions, 69 incorrect submissions were
correctly identified as “incorrect” by the GPT-3.5 (TN, 47 %) and 39 correct
submissions were identified as “correct” (TP, 27 %). Additionally, 30 correct
submissions were identified as “incorrect” by the LLM (FN, 20 %), and 9
incorrect submissions were identified as “correct” (FP, 6 %). Overall, a total
of 108 submissions out of 147 were correctly identified (accuracy: 73 %). Tab.
2 presents an in-depth evaluation of GPT-3.5’s correctness based on different
evaluation metrics for the three runs (1st, 2nd, 3rd) as well as the overall
average (OA) for each task and both tasks combined. The metrics include
accuracy ((TP+TN)/(TP+TN+FP+FN)) that presents the proportion of correct
classifications made by GPT-3.5, precision (TP/(TP+FP)) which represents
GPT-3.5’s ability to correctly identify correct submissions and minimize false
positives, recall (TP/(TP+FN)) which indicates GPT-3.5’s ability to correctly
identify correct submissions from the total number of actual correct
submissions, specificity (TN/(TN+FP)) which reflects GPT-3.5’s ability to
correctly identify incorrect submissions and minimize false negatives, and
finally the F1 score (2$*$TP/(2*TP+FP+FN)) [26]. The F1 score (Dice
Coefficient) is the harmonic mean of precision and recall, providing a
balanced measure of GPT-3.5’s performance. Unlike accuracy alone, the F1 score
considers both the true positive rate (recall) and the positive predictive
value (precision). It assesses the ability to correctly identify positive
instances and avoid false positives, offering a more comprehensive evaluation
of GPT-3.5’s effectiveness and robustness.

| SimpleWhileLoop | Queue | OA
---|---|---|---
Metric | 1st | 2nd | 3rd | OA | 1st | 2nd | 3rd | OA
Accuracy | 75 % | 72 % | 57 % | 70 % | 81 % | 75 % | 81 % | 81 % | 73 %
Precision | 92 % | 91 % | 71 % | 85 % | 67 % | 50 % | 67 % | 60 % | 80 %
Recall | 63 % | 61 % | 52 % | 58 % | 50 % | 50 % | 50 % | 50 % | 57 %
Specificity | 92 % | 92 % | 71 % | 86 % | 92 % | 83 % | 92 % | 88 % | 88 %
F1 score | 75 % | 70 % | 60 % | 69 % | 57 % | 50 % | 57 % | 57 % | 67 %
Table 2: Evaluation Metrics Comparison of GPT-3.5’s Classification Performance
Across the Three Runs for the Two Assignments

For the first assignment (SimpleWhileLoop), the results show an overall
accuracy of 70 %, with an overall precision of 85 %. However, the recall
values hover around 58 %. The F1 score, which reflects the balance between
precision and recall shows a similar pattern, with an overall average of 69 %.
Although, the specificity values show a wide distribution between 71 % and 92
% across the runs, with an overall of 86 %. GPT-3.5 shows a slightly better
performance overall for the second assignment (Queue): The accuracy ranges
from 75 % to 81 % across the runs, with an overall accuracy of 81 %. The
precision shows some variation but still maintains an overall value of 60 %.
The recall remains consistent at 50 % for all 3 runs, indicating that GPT-3.5
consistently identifies only half of the correct submissions correctly. The
specificity ranges from 83 % to 92 %.

The overall accuracy for both tasks (all submissions, and all six runs) is 73
%. A precision of 80 % and specificity of 88 % was achieved by GPT-3.5.
However, GPT-3.5 achieved a recall of only 57 %. The overall F1 score for
GPT-3.5’s performance in correcting the two programming assignments is 67 %.

###  4.3 Evaluation of the Generated Feedback Text

Only looking at the submission classification quality regarding correctness by
GPT-3.5 does not capture the feedback quality. Hence, we qualitatively
assessed the feedback content, considering external traits like appearance and
superficial content, as well as internal traits, such as content quality (cf.
Table 3 and 4, RQ2). Internal traits categorize the feedback into three
groups: feedback without code (lacking Java programming language keywords or
variable/method names), feedback with code, and feedback containing only code.
The feedback is identified as personalized if it references parts or
(variable) names from the student’s submission. If the feedback contains
suggestions/corrections and nothing wrong was stated, which means that fixing
all errors mentioned and implementing the suggestions results in working code
or at least does not break the code further, it is categorized as only correct
corrections/suggestions. Partially correct is assigned when only some feedback
parts are correct, while others introduce new issues. Finally, false
correction indicates that the feedback only contains non-existent errors or
suggestions resulting in broken code. These explanations show that correctness
does not necessarily imply compliance with the assignment. Feedback was deemed
compliant with the assignment if the suggestions align with the given
instructions. This means that both correct and incorrect corrections can be
compliant, and for the feedback to be considered compliant, the code does not
have to meet all specifications after implementing the suggestions, which may
happen if GPT-3.5 misses some errors. Completely correct correction signifies
that applying the feedback results in a fully correct submission. Finally, we
also tracked whether feedback identified and localized mentioned bugs and if
these bugs were present in those locations.

The analysis of erroneous feedback revealed patterns, which were then used to
group similar feedback. We categorized submissions into three cases:
syntactically and functionally correct (SCFC), syntactically correct but not
functionally correct (SCFI, meaning they do not meet task requirements), and
syntactically and functionally incorrect (SIFI). We separately analyzed the
generated feedback for these three cases to gain a deeper understanding of
GPT-3.5’s behavior and the feedback effectiveness.

####  4.3.1 Analyzing the SimpleWhileLoop Assignment

SimpleWhileLoop | SCFC, n=19 | SCFI, n=11 | SIFI, n=3 | All, n=99
---|---|---|---|---
Characteristics of feedback | 1st | 2nd | 3rd | 1st | 2nd | 3rd | 1st | 2nd | 3rd | Sum |  %
Feedback without code | 1 | 1 | 2 | 1 | 1 | 2 | 0 | 1 | 0 | 9 | 9
Feedback text with code | 17 | 17 | 16 | 10 | 9 | 8 | 3 | 2 | 2 | 84 | 85
Feedback containing just code | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 6 | 6
Personalized feedback | 16 | 18 | 18 | 11 | 11 | 10 | 3 | 3 | 3 | 88 | 89
Code style suggestion | 16 | 11 | 11 | 9 | 7 | 6 | 0 | 0 | 0 | 60 | 61
Only correct correction/suggestion | 10 | 10 | 9 | 6 | 9 | 6 | 2 | 2 | 3 | 57 | 58
Partially correct correction/suggestion | 4 | 1 | 2 | 3 | 1 | 2 | 0 | 0 | 0 | 13 | 13
Only false correction/suggestion | 5 | 7 | 8 | 2 | 1 | 3 | 1 | 1 | 0 | 28 | 28
Compliance with assignment | 15 | 12 | 13 | 8 | 10 | 6 | 1 | 2 | 3 | 70 | 71
Completely correct correction | 4 | 5 | 9 | 4 | 5 | 4 | 2 | 2 | 2 | 37 | 37
(Fault) localization | 11 | 5 | 6 | 8 | 7 | 7 | 3 | 3 | 2 | 43 | 43
(Fault) localization correct | 9 | 5 | 6 | 1 | 3 | 4 | 2 | 2 | 0 | 32 | 32
Table 3: Characteristics of the Generated Feedback for the SimpleWhileLoop
Assignment (SCFC: syntactically and functionally correct; SCFI: syntactically
correct and functionally incorrect; SIFI: syntactically and functionally
incorrect)

Tab. 3 presents the results of the qualitative evaluation of the feedback
generated by GPT-3.5. It includes a total of 99 feedback instances for 33
submissions across 3 runs for the SimpleWhileLoop assignment. For this
assignment, 89 % of the generated feedback was personalized, providing
specific guidance based on the individual submissions. The remaining feedback
consisted of general suggestions or simple statements such as “no errors were
found”. Among all the feedback, 61 % included code style suggestions,
excluding those related to the SIFI case. Furthermore, 37 % of the feedback
provided completely correct corrections, accurately identifying and addressing
all the issues in the submissions. Fault localization was provided for 43 % of
all the feedback; however, only 32 % of them were correct. We clustered the
faulty feedback into patterns, such as, finding false errors, presenting the
submitted code as the solution, incorrect fault localization, ignoring
instructions, and ignoring subtle requirements

Starting with the case of SCFC, which refers to correct submissions, GPT-3.5
consistently provided 4 completely correct feedback instances over three runs.
The localization for suggestions for improvement was also accurate. Most of
these suggestions were related to code style, such as using more descriptive
variable names and providing comments with examples. Interesting was the
feedback, where GPT-3.5 said that there are no logical errors in the code and
provided a suggestion to enhance the code readability by commenting where
nearly every line of the submission was commented (such as `i++; // Increment
the loop variable`). Other suggestions included to use more white space, to
use `i+=2` instead of `i=i+2` (or adding 1 twice). Improvement suggestions
were not limited to code style recommendations, but also included functional
alternatives as a possible solution approach for the problem, e. g. “use an if
statement inside the loop to only print the odd numbers” followed by the
presentation of an alternative solution. Throughout all 3 runs, there were 7
to 9 false negative feedback. Although the submissions were correct, the
provided feedback included false suggestions and corrections, indicating a
finding false errors pattern. For instance, suggestions like using a for loop
instead of a while loop or recommending to “consider using a for loop and
printing the range of odd numbers instead of printing individual odd numbers”
that made no sense. These suggestions were not compliant with the assignment
requirements or contained incorrect statements, such as “i is initialized to
0, which is an even number, so the loop never reaches the print statement”.
There is also a main pattern in most of the feedback, in finding false errors
that does not exist, e. g. GPT-3.5 suggested incrementing a variable by two
instead of adding 1 twice where the code was actually already doing exactly
that. An example where GPT-3.5 did not correctly identify a semantically
correct solution printing the odd numbers from 1 to 10 is `while (i < 6)
{System.out.println(2*i - 1); i++;}` where GPT-3.5 suggested to change `while
(i < 6)` to `while (i <= 10)`. The pattern presenting the submitted code as
the solution occurred three times.

For the SCFI cases, out of 11 submissions, in the first two runs, 10 were
correctly identified as incorrect by GPT-3.5, except for one submission where
GPT-3.5 incorrectly explicitly said that it is correct in all three runs. This
submission was quite remarkable, because all numbers are printed out and, in
the second run, GPT-3.5 recognized that only odd numbers should be printed and
suggested to introduce a condition variable “isOdd” to improve readability. In
the last run, three additional false positive cases occurred, where either the
statement “No errors were found” was provided or a code correction was
proposed. All feedback provided was personalized. The only case when a sample
solution was suggested without further hints or explanations was for a
submission that contains only a class name and an empty while loop body. The
quality of the feedback texts remained consistently low across the runs,
particularly in terms of being completely correct, with a maximum of 5 out of
11 submissions. The number of feedback instances that demonstrated compliance
also decreased from 10 to 6 instances.

In the SIFI case, there are 3 submissions that are syntactically and,
therefore, also functionally incorrect. GPT-3.5 correctly classified all 3
submissions as “incorrect” in all three runs. There were no false positives.
Personalized feedback was generated for each of these submissions throughout
the three runs. No feedback included code style suggestions. In the first run,
the feedback exclusively consisted of hints with code sections. However, in
the subsequent two runs, the feedback varied and included hints with and
without, or solely corrected code. However, out of these submissions, only 6
feedbacks were found to be compliant with the task requirements. Fault
localization was provided in 8 feedbacks, but only 4 had their errors
correctly localized, along with proper correction suggestions. Examples of
feedback with correct fault localization include the following (quote from
GPT-3.5): “// Error: The code does not count from 1 but from 0. // Hint: The
while loop should start with the variable ’i’ set to 1 instead of 0.”

An overall recurring pattern was, however, that GPT-3.5 failed to distinguish
the difference in capitalization of the word “Boom” in the print statement, as
explicitly stated in the assignment instructions, in all 99 feedback
instances. Submissions with “BOOM” or “boom” were handled exactly the same way
by GPT-3.5 (ignoring instructions or ignoring subtle requirements patterns),
and other correction suggestions used the exact same capitalization.

####  4.3.2 Analyzing the Queue Assignment

The qualitative evaluation of the Queue assignment is based on 16 submissions,
each processed 3 times by GPT-3.5, resulting in 48 generated answers providing
feedback. As before we examine the three different cases separately. All
received feedback was personalized, as they referenced the related
submissions, when pointing out errors and suggestions. 28 of the 48 answers
contained suggestions to improve the code style. With 8 out of 12 answers
(SCFC), 3 out of 6 (SCFI), and 17 out of 30 (SIFI), the feedback comprising
code style suggestions is equally distributed through all three cases.
However, these suggestions consisted merely of general statements, such as the
need for comments throughout the code to explain the logic and proper
indentation for better readability.

Queue | SCFC, n=4 | SCFI, n=2 | SIFI, n=10 | All, n=48
---|---|---|---|---
Characteristics of feedback | 1st | 2nd | 3rd | 1st | 2nd | 3rd | 1st | 2nd | 3rd | Sum |  %
Feedback without code | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Feedback text with code | 4 | 4 | 4 | 2 | 2 | 2 | 10 | 8 | 9 | 45 | 94
Feedback containing just code | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1 | 3 | 6
Personalized feedback | 4 | 4 | 4 | 2 | 2 | 2 | 10 | 10 | 10 | 48 | 100
Code style suggestion | 4 | 2 | 2 | 1 | 1 | 1 | 6 | 5 | 6 | 28 | 58
Only correct correction/suggestion | 1 | 0 | 0 | 0 | 0 | 0 | 3 | 5 | 3 | 12 | 25
Partially correct correction/suggestion | 0 | 2 | 1 | 2 | 2 | 2 | 5 | 4 | 4 | 22 | 46
Only false correction/suggestion | 3 | 2 | 3 | 0 | 0 | 0 | 2 | 1 | 3 | 14 | 29
Compliance with assignment | 2 | 2 | 2 | 0 | 0 | 0 | 6 | 6 | 7 | 25 | 52
Completely correct correction | 1 | 0 | 1 | 0 | 0 | 0 | 3 | 3 | 1 | 9 | 19
(Fault) localization | 3 | 2 | 3 | 2 | 2 | 2 | 7 | 8 | 6 | 35 | 73
(Fault) localization correct | 0 | 0 | 0 | 1 | 1 | 1 | 6 | 7 | 3 | 19 | 40
Table 4: Characteristics of the generated feedback for the Queue assignment

There are 4 submissions and 12 feedback instances belonging to the SCFC case
(cf. Tab. 4). All feedback was given in the form of text with code, as lists
of errors and suggestions. In 50 % of the feedback, the student submission was
identified incorrectly as an incorrect submission. Noticeable is, that only
one feedback is completely correct, but three partially correct and eight
simply false. This is accompanied by an error localization in eight
submissions with not even one being correct. Only 52 % of the feedback
complies with the instructions. Furthermore, we observed three patterns
present in the feedback. The first and most prominent pattern is evident in
all feedback, wherein GPT-3.5 references the code originally found in the
student submission and presents it as a solution to a non-existent error. This
is closely associated with GPT-3.5, assuming that the submitted code contains
an error in the student’s code that requires correction. For instance, one
piece of feedback states, “The ‘remove’ method should check to see if the
queue is empty before trying to remove elements, otherwise there would be an
error.” while the respective method already performs this exact check. The
second pattern is present in the feedback for 3 submissions. Here, GPT-3.5 is
simply ignoring the given instructions. It suggests to change names, like
renaming `isEmpty()` to `isListEmpty()`, or the return types of methods, in
contradiction to the given interface twice. The third pattern is seen in
feedback for two submissions where GPT-3.5 mistakenly identifies non-existent
code errors. Unlike the first pattern, there’s no submitted involved. Instead,
GPT-3.5 seems to struggle in understanding the code and attempts to fix what
it perceives as issues. For instance, it suggests adding an unnecessary “else”
statement in the “peek()” method, stating: “Error: `peek()` does not include
an else case to return `EMPTY_VALUE`. Hint: In the `peek()` method, add an
else statement to return `EMPTY_VALUE` when the queue is empty.” with GPT-3.5
apparently not recognizing, that an else case is not needed for this method to
work:

        public int peek() {
            if(!this.isEmpty()) {
                return head.getValue();
            }
            return EMPTY_VALUE;
        }

For the SCFI case, involving two submissions and six GPT-3.5 feedback
instances, one incorrectly classified a submission as correct (cf. Tab. 4),
while another provided a mix of text and code. While none of the corrections
was entirely correct, none was entirely wrong; they were all partially
correct. Each feedback included error localization, which was accurate for
half of the submissions (3). However, none of the feedback fully complied with
the provided instructions, exhibiting two recurring patterns. The pattern of
finding false errors (only false correction/suggestion) occurred in two
feedbacks, and the incorrect error localization observed in the 2nd and 3rd
run of the SCFI case can be linked to this pattern. The pattern ignoring the
instructions was evident in all six corrections (cf. Tab. 4). The first of
these two patterns can be traced back to the usage of the ternary condition
operator inside a return statement, the second pattern shows up, because all
submissions do not use the required singly linked list, but a doubly linked
list or the LinkedList from the Java collection framework.

For the third case (SIFI), we examined the 30 feedback instances from GPT-3.5
based on ten student submissions, from which two classified the associated
submission incorrectly as correct. Three responses consisted of both text and
code, and three further responses consisted of code only. The correction of 11
feedback instances was completely correct, 13 corrections were partially
correct and six corrections false. A total of 21 corrections has fault
localization included, and 16 of these were correct. Moreover, 19 responses of
GPT-3.5 conform to the given instructions. All three patterns are also present
in the feedback for this case. Interestingly the presenting the submitted code
as the solution pattern is present in 9 corrections, but 8 of these
corrections have no semantic error but only a syntactical error based on a
missing semicolon. The finding false errors pattern can be found five times.
These always correspond to some code where a connection to some other part of
code has to be made such as GPT-3.5 mentioning missing null checks but
actually this check is done through a method call. Similar false errors are
found for if-else conditions or for loops in a slightly more complicated
fashion. Also, code using the ternary condition operator gets tagged again as
incorrect. The feedback for 16 submissions contains the ignoring instructions
pattern. In 18 submissions an implementation of the singly linked list itself
was missing (the node class), with GPT-3.5 not correcting these in 15 cases.
Furthermore, one correction ignored the given interface and suggested renaming
a method. A new observed pattern unique to the third case is that if a
submission had significant errors, including syntax and logic errors leading
to exceptions, GPT-3.5 provided code solutions.

##  5 Discussion

###  5.1 Quality of GPT-3.5 in Determining the Submissions’ Correctness

For RQ1, we investigated the quality of GPT-3.5 in determining the
correctness. On one hand, the overall accuracy shows that 73 % of all
classifications made by GPT-3.5 are correct. Also, when GPT-3.5 identifies a
submission as correct, it is indeed correct in 80 % of the cases (precision).
However, the recall value is significantly lower, indicating that GPT-3.5
could only identify 58 % of all correct submissions in the data set. This
implies that a portion of the correct submissions remain unnoticed, which
could be discouraging for students whose submissions are correct but still get
evaluated as incorrect. The specificity of 88 % highlights GPT-3.5’s ability
to correctly identify and classify incorrect submissions. On the other hand, a
comparison between the two assignments shows significant differences for the
precision (SimpleWhileLoop: 8 % vs. Queue: 60 %). A reason for this might also
lie in the different frequency of correct submissions in the dataset.
Furthermore, there are runs in which the accuracy is as low as 57 %, the
precision as low as 50 %, and the specificity as low as 71 % (cf. Tab. 2).
Hence, the quality of GPT-3.5’s feedback seems to be dependent on the specific
assignment and also on hidden factors related to the GPT-3.5 model. This point
of uncertainty has been highlighted in related studies as well (e. g., [12]).
The overall accuracy of GPT-3.5 (73 %) seems to be in line with comparable
research on Python tasks (57 to 79 %) [24] and to be comparable with student
peer reviews where partial incorrect feedback in approx. 14 % of the cases [5]
or an accuracy of 64 % [6] have been reported.

###  5.2 Characterization of the Generated Feedback by GPT-3.5

An in-depth analysis of the generated feedback (RQ2) reveals for the
SimpleWhileLoop assignment that only 37 % of the feedback were completely
correct (cf. Tab. 3) and for the Queue 25 % (cf. Tab. 4). Here, however, needs
to be considered whether is pedagogically better to provide a list of all
errors at once or only the major ones first (cf. discussion in [27]). Hence,
the completely correct characterization might be too strict, and the focus
should be put on the only correct correction/suggestion and partially correct
correction/suggestion characterizations. In 28 % of the cases in which GPT-3.5
provided hints on hallucinated errors, suggested alternative solution paths
such as using a for loop instead of a while loop (which is not compliant with
the instruction), or incorporating an if condition to test for odd numbers
instead of using i+2, where the submissions were already correct. Only 58 % of
the feedback contained only correct correction/suggestion. From these results,
it can be concluded that currently GPT-3.5 does not guarantee reliability in
providing completely accurate feedback. It may be important to further
investigate how students respond to partially correct feedback or feedback
that only provides false suggestions for improvement or what other scenarios
are beneficial here such as providing the GPT-3.5 feedback to teaching
assistants as a draft for semi-automatic feedback provisioning. This also
opens up the debate between formative and summative assessment. For summative
assessment an accurate diagnosis is fundamental. Formative assessment,
however, should help learners learn and reflect on their submission, so it may
be more forgiving if occasional errors or incorrect suggestions are included
if they foster reflection and exploration. Still, incorrect feedback should be
avoided, if possible, to not confuse novices. For peer reviews, often very
short feedback is reported (e. g., median 13–14 words [5, 6]), here GPT-3.5
provides significantly longer feedback with a median of 82 words. However,
students do not get the advantage of seeing other solution strategies and
learning to critique code.

One significant advantage of GPT-3.5’s feedback is that it is personalized in
all cases for the Queue assignment and in 89 % of cases for the
SimpleWhileLoop. Another advantage of GPT-3.5 is error localization “for
free”. For instance, 53 % of errors were successfully localized, and among
these localized errors, 35 % were correctly identified overall for both
assignments. These advantages of personalized feedback and error localization
sets GPT-3.5 apart from conventional e-assessment systems that require the
laborious task of writing functional tests for all anticipated errors to
provide fine grained feedback and error localization. With GPT-3.5, the
generation of personalized feedback can be automated, providing a scalable and
efficient solution for delivering targeted guidance to students. GPT-3.5’s
ability to better identify incorrect submissions as incorrect to identify
correct submissions as correct may be the prompt used that specifically asked
for finding errors. Hence, prompt engineering or using different prompts for
different cases might allow to optimize the situations where the use of
GPT-3.5 is most appropriate: In the SCFC (syntactically and functionally
correct) case, it would be beneficial to let GPT-3.5 suggest code style
improvements, as the evaluation showed that GPT-3.5 provided many good coding
style suggestions and alternative solutions. This can stimulate creativity and
critical thinking among students. In the SCFI (syntactically correct but
functionally incorrect) case, the focus would be on querying GPT-3.5
specifically for semantic errors in the submissions and error localization.
Finally, in the SIFI (syntactically and functionally incorrect) case, GPT-3.5
showed the best results compared to the other cases. Hence, it is crucial the
prompt GPT-3.5 to be more precise about the types of errors, as it has already
demonstrated good performance in fault localization. To better detect the
different cases, it might be helpful to combine traditional e-assessment
systems (with test cases) with GPT-3.5 to enhance the benefits of both
systems.

There are several submissions for which GPT-3.5 provided code-only feedback.
It seemed as if this only happened when there were too many errors or no real
solution attempted by the students. Here it is questionable whether such a
model solution without explanation helps struggling students or might
encourage gaming the system approaches (cf. [28]). It would be interesting to
compare this with the feedback provided by real teaching assistants or peer
reviews.

###  5.3 Strengths and Weaknesses of GPT-3.5 in Delivering Feedback

To answer RQ3, GPT-3.5’s strengths lie in its ability to provide personalized
feedback and suitable code corrections that may enhance the learning
experience and helping address coding weaknesses effectively at scale quickly.
Particularly interesting is that for the SimpleWhileLoop GPT-3.5 did not honor
the casing of the string to be printed out but focused on the overall
correctness of the algorithm (the more difficult part of the assignment; this
was also partly the case for the Queue assignment; or submissions where the
algorithm was basically correct but contained a syntax error). This aligns
with findings of a related study that the LLM Codex struggles with output
formatting on code generation [15]. Although weaknesses include occasional
inaccuracies in error identification (particularly for subtle requirements
such as the casing of strings), provisioning of model solutions without
explanations, suggesting changes violating the assignment instructions, too
generic feedback, and limitations with complex tasks.

GPT-3.5 is just one example of an LLM that was chosen here due to its
availability and public API. In the future, there might be different LLMs or
even more specialized LLMs for this specific task that show a better
performance. Further research is needed.

###  5.4 Limitations

It needs to be noted that GPT is under active development. Hence, future
versions might yield different results. Additionally, the response highly
depends on the questions asked. Hence, using different questions might also
have an impact on the results. We experimented with different (slightly)
prompts, and all showed comparable results on our tests. As GPT-3.5 is a LLM
and is trained to predict subsequent tokens based on preceding ones, its
responses can be different even if the very same question is asked. To address
this threat, every submission was sent to GPT-3.5 three times and the results
are aggregated. Moreover, the results might vary depending on aspects such as
the programming language used, and the description of the task. Also, only two
assignments were analyzed. The tasks were selected from a real course and,
therefore, allow insights into how GPT-3.5 performs on real input. Finally,
the manual classifications used here to categorize the responses from GPT-3.5
might be subjective. To mitigate this threat, the classification was conducted
by two researchers with an intensive exchange.

##  6 Conclusions and Outlook

In this paper, we explored how effectively the LLM GPT-3.5 can provide
personalized feedback, including code correction and style suggestions, for
programming tasks. We evaluated GPT-3.5 using student submissions from two
real-world assignments in a first-semester course. The results indicate a 73 %
accuracy rate in correctly identifying submissions as either correct or
incorrect. Furthermore, GPT-3.5 provided adequate feedback in 47,% of cases.
Overall, it performed better in identifying incorrect submissions than in
recognizing correct ones. An in-depth analysis revealed GPT-3.5’s capability
to offer personalized feedback by detecting syntax and functional errors in
student submissions and providing suggestions for code improvement. However,
GPT-3.5 occasionally made errors or suggested changes that did not align with
assignment instructions. Due to identified reliability and consistency issues,
we advise caution when using LLMs and particularly GPT-3.5 for fully automatic
student feedback on programming tasks at this time. Currently, it is not good
enough for summative grading. Nonetheless, the current version can be a
valuable tool for teaching assistants in pre-assessing large-scale
assignments. It can be used to facilitate quick error identification and
generating drafts for personalized feedback, reducing the need for writing
unit tests. We plan further research to address current weaknesses and
determine optimal use cases for LLMs. This includes exploring how to integrate
them in e-assessment systems with traditional tests, allowing for different
prompts in syntactically correct submissions. An alternative approach might be
to use a LLM to repair syntactically incorrect submissions and then use the
repaired version for partial scoring of the functional correctness. Finally,
various feedback characteristics, as described in Section 4.2, could be
automatically measured to determine whether feedback needs regeneration or
optimization with a different prompt and learner data. Future versions of GPT
and other LLMs will likely see improvements, expanding their potential
applications in education as discussed in this paper. Hence, we plan to
compare our results with other LLMs.

##  7 Acknowledgments

This research is part of the project AIM@LMU funded by the German Federal
Ministry of Education and Research (BMBF) under the grant number 16DHBKI013.
The responsibility for the content of this publication lies with the authors.

The authors thank the students of the lecture Einführung in die Programmierung
of winter semester 2021/22 who allowed us to use their submissions for this
research.

## References

  * [1] A. Luxton-Reilly, Simon, I. Albluwi, B. A. Becker, M. Giannakos, A. N. Kumar, L. Ott, J. Paterson, M. J. Scott, J. Sheard, and C. Szabo, “Introductory Programming: A Systematic Literature Review,” in _Proc. ITiCSE_ , 2018, pp. 55–106. [Online]. Available: https://doi.org/10.1145/3293881.3295779
  * [2] J. Hattie and H. Timperley, “The power of feedback,” _Review of Educational Research_ , vol. 77, no. 1, pp. 81–112, mar 2007.
  * [3] S. Strickroth and F. Bry, “The Future of Higher Education is Social and Personalized! Experience Report and Perspectives,” in _Proc. CSEDU_ , vol. 1, 2022, pp. 389–396. [Online]. Available: https://doi.org/10.5220/0011087700003182
  * [4] T. D. Indriasari, A. Luxton-Reilly, and P. Denny, “A Review of Peer Code Review in Higher Education,” _TOCE_ , vol. 20, no. 3, pp. 1–25, sep 2020\.
  * [5] N. Heller and F. Bry, “Organizing Peer Correction in Tertiary STEM Education: An Approach and its Evaluation,” _iJEP_ , vol. 9, no. 4, pp. 16–32, 2019\.
  * [6] S. Strickroth, “Does Peer Code Review Change My Mind on My Submission?” in _Proc. ITiCSE_ , 2023, pp. 498–504. [Online]. Available: https://doi.org/10.1145/3587102.3588802
  * [7] S. Strickroth and M. Striewe, “Building a Corpus of Task-Based Grading and Feedback Systems for Learning and Teaching Programming,” _iJEP_ , vol. 12, no. 5, p. 26–41, 2022. [Online]. Available: https://doi.org/10.3991/ijep.v12i5.31283
  * [8] C. Geng, Y. Zhang, B. Pientka, and X. Si, “Can ChatGPT Pass An Introductory Level Functional Language Programming Course?” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.02230
  * [9] H. Tian, W. Lu, T. O. Li, X. Tang, S.-C. Cheung, J. Klein, and T. F. Bissyandé, “Is ChatGPT the Ultimate Programming Assistant – How far is it?” 2023.
  * [10] J. Savelka, A. Agarwal, M. An, C. Bogart, and M. Sakr, “Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses,” in _Proc. CER_ , aug 2023. [Online]. Available: https://doi.org/10.1145/3568813.3600142
  * [11] J. Finnie-Ansley, P. Denny, A. Luxton-Reilly, E. A. Santos, J. Prather, and B. A. Becker, “My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises,” in _Proc. ACE_ , 2023, p. 97–104. [Online]. Available: https://doi.org/10.1145/3576123.3576134
  * [12] A. Hellas, J. Leinonen, S. Sarsa, C. Koutcheme, L. Kujanpää, and J. Sorva, “Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests,” in _Proc. CER_ , aug 2023. [Online]. Available: https://doi.org/10.1145/3568813.3600139
  * [13] N. M. S. Surameery and M. Y. Shakor, “Use ChatGPT to Solve Programming Bugs,” _IJITC_ , vol. 3, no. 01, p. 17–22, 2023. [Online]. Available: http://doi.org/10.55529/ijitc.31.17.22
  * [14] D. Sobania, M. Briesch, C. Hanna, and J. Petke, “An Analysis of the Automatic Bug Fixing Performance of ChatGPT,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.08653
  * [15] J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather, “The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming,” in _Proc. ACE_ , 2022, p. 10–19. [Online]. Available: https://doi.org/10.1145/3511861.3511863
  * [16] B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, “Programming Is Hard – Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation,” in _Proc. SICSE TS_ , 2023, p. 500–506. [Online]. Available: https://doi.org/10.1145/3545945.3569759
  * [17] P. Denny, J. Prather, B. A. Becker, J. Finnie-Ansley, A. Hellas, J. Leinonen, A. Luxton-Reilly, B. N. Reeves, E. A. Santos, and S. Sarsa, “Computing Education in the Era of Generative AI,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2306.02608
  * [18] S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models,” in _Proc. CER_ , aug 2022. [Online]. Available: https://doi.org/10.1145/3501385.3543957
  * [19] T. Phung, J. Cambronero, S. Gulwani, T. Kohn, R. Majumdar, A. Singla, and G. Soares, “Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.04662
  * [20] J. Leinonen, A. Hellas, S. Sarsa, B. Reeves, P. Denny, J. Prather, and B. A. Becker, “Using Large Language Models to Enhance Programming Error Messages,” in _Proc. SIGCSE TS_ , 2023, pp. 563–569. [Online]. Available: https://doi.org/10.1145/3545945.3569770
  * [21] S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen, “Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book,” in _Proc. SIGCSE TS_ , 2023, p. 931–937. [Online]. Available: https://doi.org/10.1145/3545945.3569785
  * [22] J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas, “Comparing Code Explanations Created by Students and Large Language Models,” in _Proc. ITiCSE_ , 2023, pp. 124–130. [Online]. Available: https://doi.org/10.1145/3587102.3588785
  * [23] N. Kiesler, D. Lohr, and H. Keuning, “Exploring the Potential of Large Language Models to Generate Formative Programming Feedback,” in _Proc. FiE_ , 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.00029
  * [24] R. Balse, B. Valaboju, S. Singhal, J. M. Warriem, and P. Prasad, “Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments,” in _Proc. ITiCSE_ , 2023, p. 292–298. [Online]. Available: https://doi.org/10.1145/3587102.3588852
  * [25] S. Strickroth, H. Olivier, and N. Pinkwart, “Das GATE-System: Qualitätssteigerung durch Selbsttests für Studenten bei der Onlineabgabe von Übungsaufgaben?” in _Proc. DeLFI_. Bonn: Gesellschaft für Informatik e.V., 2011, pp. 115–126. [Online]. Available: https://dl.gi.de/handle/20.500.12116/4740
  * [26] A. Larner, _The 2x2 Matrix Contingency, Confusion and the Metrics of Binary Classification_. Springer, 2021\. [Online]. Available: https://doi.org/10.1007/978-3-030-74920-0
  * [27] J. Jeuring, H. Keuning, S. Marwan, D. Bouvier, C. Izu, N. Kiesler, T. Lehtinen, D. Lohr, A. Peterson, and S. Sarsa, “Towards Giving Timely Formative Feedback and Hints to Novice Programmers,” in _ITiCSE-WGR_ , 2022, p. 95–115. [Online]. Available: https://doi.org/10.1145/3571785.3574124
  * [28] R. S. Baker, A. De Carvalho, J. Raspat, V. Aleven, A. T. Corbett, and K. R. Koedinger, “Educational software features that encourage and discourage “gaming the system”,” in _Proc. AIED_ , 2009, pp. 475–482. [Online]. Available: https://doi.org/10.3233/978-1-60750-028-5-475