# Weakly-Supervised Optical Flow Estimation for Time-of-Flight

Michael Schelling, Pedro Hermosilla, Timo Ropinski
Ulm University, Germany
https://github.com/schellmi42/WFlowToF

###### Abstract

Indirect Time-of-Flight (iToF) cameras are a widespread type of 3D sensor,
which perform multiple captures to obtain depth values of the captured scene.
While recent approaches to correct iToF depths achieve high performance when
removing multi-path-interference and sensor noise, little research has been
done to tackle motion artifacts. In this work we propose a training algorithm,
which allows to supervise Optical Flow (OF) networks directly on the
reconstructed depth, without the need of having ground truth flows. We
demonstrate that this approach enables the training of OF networks to align
raw iToF measurements and compensate motion artifacts in the iToF depth
images. The approach is evaluated for both single- and multi-frequency sensors
as well as multi-tap sensors, and is able to outperform other motion
compensation techniques.

Figure 1: Illustration of the flow estimation. Given iToF measurements at
subsequent time steps, a network is used to predict optical flows, in order to
align the images to the reference image (bottom row). From the warped
measurements a ToF depth image can be reconstructed. We propose to supervise
the training directly on this ToF depth, and propagate gradients through the
ToF depth computation. This figure shows the single frequency, single-tap case
with four measurements. Note the modality change in the input due to different
phase shifts $\theta$.

##  1 Introduction

Time-of-Flight (ToF) cameras are sensors that aim to capture depth images by
measuring the time the light needs to travel from a light source on the camera
to an object and back to the camera sensor. Apart from direct ToF cameras,
such as LiDAR, which register the time of incoming reflections of a light
pulse at a high temporal resolution, another common and cost-efficient
approach are indirect ToF (iToF) cameras, which do not require as precise
measuring devices. One realization of iToF devices are Amplitude Modulated
Continuous Wave (AMCW) ToF sensors, as for example used in the Kinect system.
These sensors continuously illuminate the scene with a periodically modulated
light signal and aim to retrieve the phase offset between the emitted and the
retrieved signal, which gives information about the travel time of the signal
[12]. In order to retrieve the phase offset it is necessary to perform
multiple captures, which makes this approach sensible to movements of both,
the camera and the objects in the illuminated scene. As the measurements are
taken with differing sensor settings, so called multi modality, standard
Optical Flow (OF) algorithms achieve only low performance, and hence require
adaptation. While there are works that investigate the compensation of motion
using OF, they are only applicable to specific sensor types [20, 13] or
require carefully designed datasets [10] to train OF networks. Hence, it is
still a common approach to merely detect motion artifacts and mask the
affected pixels in the final depth image, as is for example realized by the
LF2-algorithm [31] for the Kinect sensor.

In this work, we propose a training algorithm for OF networks which allows to
supervise the flow prediction using the ToF depth image, without the need to
directly supervise the predicted flow, see Fig. 1. To this end, we analyze the
ToF depth computation to provide reliable and stable gradients during
training. Further, we introduce a set of regulatory losses, which guide the
network towards predicting flows, that are consistent with the underlying
images.

##  2 Technical Background

In this section, we briefly describe iToF cameras.

##### ToF Working Principle.

An AMCW iToF camera emits a modulated light signal $s(t)$, which is correlated
at the sensor with a phase shifted version of the emitted signal $s(t+\theta)$
during the exposure time. The resulting measurement $m$ is repeated
sequentially for different phase shifts $\theta$, from which the distance $d$
is retrieved indirectly by estimating the phase shift $\Delta\varphi$ of the
signal $s$ when arriving at the sensor. In the common case of four
measurements $m_{0},\dots,m_{3}$ at $\theta\in\\{0,\pi/2,\pi,3\pi/2\\}$, the
distance $d$ is retrieved as

$\displaystyle\Delta\varphi$
$\displaystyle=\arctan\left(\frac{m_{3}-m_{1}}{m_{0}-m_{2}}\right),$ (1)
$\displaystyle d_{ToF}$ $\displaystyle=\frac{c\cdot\Delta\varphi}{4\pi f},$
(2)

where $c$ is the speed of light, and $f$ is the modulation frequency of the
signal $s$ [12]. Due to the periodic nature of Eq. (1), the reconstructed
$d_{ToF}$ is only unambiguous up to a maximum distance of

$\displaystyle d_{max}=c/(2f),$ (3)

specifically, $d_{ToF}=d\mod d_{max}$, where the distance $d$ is referred to
as _depth_ , as is common practice in the area of ToF imaging. The so called
phase wrapping of $d_{ToF}$ is typically resolved by using additional
measurements at different frequencies $f$ [12].

However Eq. (2) is based on the assumptions that, (a) only the direct
reflection $s(t+\Delta\varphi)$ is captured and (b) the scene is static
between the different captures. While (a) has been dealt with to a large
extent in recent work on correcting iToF depths [1, 22, 26], only little
research has been done to reduce motion artifacts stemming from (b).

##### Multi-Tap Sensors.

A realization of iToF sensors are so called _multi-tap_ sensors, which are
able to capture multiple measurements of $m_{\theta}$ in parallel. The most
widespread approach are two-tap sensors, which allow the capture of
$m_{A,i}=m_{i}$ and $m_{B,i}=m_{i+2}$ at the same time, by sorting the
electrons generated by incoming photons into two quantum wells using a
modulated electric field [27]. Internally, these two measurements are used to
compensate for hardware inaccuracies and reduce noise [13] by computing:

$\displaystyle m_{i}$ $\displaystyle=m_{A,i}-m_{B,i}.$ (4)

In order to make direct use of $m_{A},m_{B}$ in Eq. (1), it is necessary to
calibrate the differences in the photo responses [27]

$\displaystyle m_{A,i}$ $\displaystyle=r_{\theta}(m_{B,i+2}),$ (5)

which doubles the effective frame rate, and reduce, but not eliminate, motion-
artifacts. Recently also prototypes for four-tap sensors have been developed
[6, 16], which in the future might eliminate motion artifacts in single-
frequency captures, but not in multi-frequency sensors.

##  3 Related Work

This section briefly summarizes previous work on related fields.

##### ToF Motion Artifact Correction.

Early methods on motion compensation used detect-and-repair approaches [27,
12], _e.g_. by performing bilateral filtering [21]. One of the first methods
to resolve movement artifacts using optical flow was introduced by Lindner _et
al_. [20] who aim to tackle the cross modality through a correction scheme to
compute intensity images from two-tap captures, which can be used as input to
a standard OF algorithm. Based on this method, Hoegg _et al_. [13] derived
optimizations for the OF prediction algorithm by incorporating motion
detection and refining the spatial consistency to achieve real-time
performance. The performance of these approaches was further improved with the
calibration of Gottfried _et al_. [9]. In contrast we integrate the entire
computational flow, from raw iToF measurement to depth reconstruction into our
optimization pipeline.

The first learned approach was presented by Guo _et al_. [10], who provide
methods to correct errors for the Kinect2 sensor, including an encoder-decoder
network for OF prediction. To enable the supervised learning of motion
compensation, a specific dataset is generated, which allows for simulating
linear movements in the image domain, while separating the motion of
foreground and background. Contrarily, we propose a weakly supervised
training, which does not require flow labels, and instead uses ToF depths for
supervision, which are available in existing iToF datasets.

##### Optical Flow.

Recent works on OF regression rely on neural networks, which have proven to
outperform traditional approaches [29]. The typical design, using shared image
encoders and a latent cost volume, was first introduced Dosovitskiy _et al_.
[8] in their FlowNetC architecture, alongside the FlowNetS network, which uses
a encoder decoder architecture. Subsequent, a large literature on various
applications [33, 19] and formulations [2, 14] in the field of motion
estimation emerged. In order to reduce the computational costs, Sun _et al_.
[29] introduced a hierarchical architecture with coarse-to-fine warping in
their Pyramid-Warping-Cost-volume (PWC) network. This design was further
refined by Kong _et al_. [18] in their FastFlowNet (FFN) architecture, which
reduced the computational complexity and achieves fast inference times.

To overcome the need of generating ground truth flows for a supervised
training, unsupervised approaches [15, 25, 32, 14] optimize the photometric
consistency between images and apply regularizations to refine the flow
prediction.

##### ToF Correction.

The occurrence of Multi-Path-Interference (MPI) is the main source of errors
in iToF depth reconstructions. Consequently, existing works on correcting iToF
data focus on removing MPI artifacts. As with OF prediction, 2D neural
networks have proven to achieve high noise removal performance [1, 22, 28, 10,
7]. However, also other learned approaches have been investigated recently,
such as reconstructing the transient response [4, 11] or using 3D point
networks [26].

##  4 Method

Figure 2: Overview over the loss functions used in this work. Our main loss is
the ToF-loss $\mathcal{L}_{ToF}$ (right), which is computed on the
reconstructed ToF depth using a differentiable operation, and is adapted to
provide phase unwrapped gradients. To constrain the flow prediction the loss
$\mathcal{L}_{smooth}$ (top) is used to regularize the flow, and an additional
regularization on the warped image $m_{i}$ is given through the loss
$\mathcal{L}_{edge}$ (center). Finally, the loss $\mathcal{L}_{sim}$ aims to
create consistency between the latent representations inside the network.
Note: The losses $\mathcal{L}_{ToF}$ and $\mathcal{L}_{sim}$ are computed over
all $i$. This figure shows the single-tap case, where only one measurement is
taken per time step.

In this work we propose a weak supervision of an OF network using the ToF-
depth $d_{ToF}$ as label, without providing ground truth flow vector fields.
In order to enable training using depth labels, the phase wrapping
discontinuities in Eq. (1) of the $\arctan$ function require consideration,
and regularizations on the flow prediction need to be established to predict
consistent flows without direct supervision.

We consider an OF network
$g:(\\{m_{i}\\}_{i=0}^{N-1},m_{N})\rightarrow\\{V_{i}\\}_{i=0}^{N-1}$, which
predicts a set of optical flows $V_{i}$ for a set of measurements $m_{i}$, in
order to align them to a measurement $m_{N}$ taken at the reference time step.
The standard photometric loss in this setting would be given as

$\displaystyle\hat{m}_{i}$ $\displaystyle=warp(m_{i},V_{i})$ (6)
$\displaystyle\mathcal{L}_{photo}$
$\displaystyle=\sum_{i}\|\hat{m}_{i}-m_{i}^{GT}\|_{1},$ (7)

where $m_{i}^{GT}$ is taken at the same time step as $m_{N}$.

Instead, we propose to supervise the network $g$ indirectly on the
reconstructed depth using the ToF depth $d_{ToF}$ without motion as target. To
increase the numerical stability we formulate the reconstructed depth
$\hat{d}$ as

$\displaystyle s$ $\displaystyle=\text{sign}(\hat{m}_{0}-\hat{m}_{2})$ (8)
$\displaystyle\hat{d}$ $\displaystyle=\frac{c}{4\pi
f}\arctan\left(\frac{\hat{m}_{3}-\hat{m}_{1}}{\hat{m}_{0}-\hat{m}_{2}+s\cdot\epsilon}\right),$
(9) $\displaystyle\mathcal{L}_{ToF}$ $\displaystyle=\|\hat{d}-d_{ToF}\|_{1},$
(10)

which avoids singularities as the denominator in Eq. (24) is strictly positive
for $\epsilon>0$. The implementation of Eq. 25 on commonly used learning
packages with auto-differentiable features, such as Pytorch [24] or JAX [3],
allows to train the flow network $g$ in a weakly-supervised fashion.

###  4.1 Phase Unwrapping

The phase wrapping in the above formulation can be tackled by generating
multiple candidate depths $\hat{d}_{k}=\hat{d}+k\cdot d_{max}$ and using the
one closest to the label as prediction

$\displaystyle\hat{d}_{k}$ $\displaystyle=\hat{d}+k\cdot d_{max}$ (11)
$\displaystyle\mathcal{L}_{ToF,PU}$
$\displaystyle=\min\\{\|\hat{d}_{k}-d_{ToF}\|_{1}\ \big{|}\
k\in\mathbb{Z}\\}.$ (12)

As both $\hat{d}$ and $d_{ToF}$ are in the range of $[0,d_{max})$, the
candidate space is reduced to $k\in\\{-1,0,1\\}$ and the minimization in Eq.
(12) can be realized by a simple lookup table

$\displaystyle\hat{d}-d_{ToF}$ $\displaystyle\in(-d_{max},d_{max}/2]:$
$\displaystyle k=-1,$ (13) $\displaystyle\hat{d}-d_{ToF}$
$\displaystyle\in(-d_{max}/2,d_{max}/2]:$ $\displaystyle k=0,$ (14)
$\displaystyle\hat{d}-d_{ToF}$ $\displaystyle\in(d_{max}/2,d_{max}):$
$\displaystyle k=1.$ (15)

However, during training only the gradients of $\mathcal{L}_{ToF,PU}$ are
relevant, which can be derived from the lookup table as

$\displaystyle\nabla\mathcal{L}_{ToF,PU}=\left\\{\begin{aligned}
\nabla\mathcal{L}_{ToF},&\qquad 0\leq\mathcal{L}_{ToF}<{d_{max}/2},\\\
-\nabla\mathcal{L}_{ToF},&\qquad\mathcal{L}_{ToF}\geq{d_{max}/2},\\\
\end{aligned}\right.$ (16)

and can thus be directly computed from Eq. (25). This allows a computational
cheap and elegant implementation of the phase unwrapping, by only adjusting
the gradients of $\mathcal{L}_{ToF}$, Eq. (25), based on the conditions in Eq.
(29) in the backpropagation step during the training of $g$.

###  4.2 Regularization

By regularizing the predictions, additional constraints for the predicted
flows $V_{i}$ are established, which enables the network to produce coherent
predictions without using flow labels. We use two additional regularization
losses, a smoothing loss $\mathcal{L}_{smooth}$ and an edge-aware loss
$\mathcal{L}_{edge}$.

For smoothing we adapt the formulation of Jonschkowski _et al_. [15] to our
setting

$\displaystyle\mathcal{L}_{smooth}=\sum_{i,j}\exp\left(-\lambda\left|\frac{\partial
m_{i}}{\partial x_{j}}\right|\right)\cdot\left|\frac{\partial V_{i}}{\partial
x_{j}}\right|,$ (17)

where $\lambda$ is an edge weighting factor and $x_{0},x_{1}$ are the two
image dimensions. This loss penalizes high gradients on $V_{i}$ in homogeneous
regions of $m_{i}$, _i.e_. regions where $m_{i}$ has small gradients. The
intuition of $\mathcal{L}_{smooth}$ is that homogeneous regions are expected
to move in the same direction.

To further regularize the network to predict correctly aligned object
boundaries, we introduce an edge-aware loss

$\displaystyle\mathcal{L}_{edge}=\sum_{i,j}\exp\left(\frac{-1}{\epsilon+\left|\frac{\partial
m_{N}}{\partial
x_{j}}\right|}\right)\cdot\frac{1}{\left|\frac{\partial\hat{m}_{i}}{\partial
x_{j}}\right|+s},$ (18)

where $\epsilon$ is a small constant for numerical stability and the shift $s$
is used to provide an upper bound on the gradients of $\mathcal{L}_{edge}$.
This loss penalizes small gradients in the warped measurements $\hat{m}_{i}$
in regions where $m_{N}$ has large gradients, _i.e_. regions where $m_{N}$ has
edges. The intuition of $\mathcal{L}_{edge}$ is that boundaries of objects can
be expected to create edges in the measurements independent of the modality.

Note that $\mathcal{L}_{smooth}$ acts on the flows $V_{i}$ whereas
$\mathcal{L}_{edge}$ is computed on the warped measurements $\hat{m}_{i}$, see
Fig. 2.

###  4.3 Cross Modality

To guide the network towards learning latent representations $F_{i}$, see Fig.
2, that are robust to the input modality, we make use of a latent similarity
loss on the column vectors $F_{i}(k,l)$ of the latent representation in
$g(m_{i})$, inspired by the formulation of contrastive learning

$\displaystyle\mathcal{L}_{sim}=\sum_{i\neq
j}\sum_{k,l}L\bigg{(}F_{i}(k,l),F_{i}(k,l)\bigg{)},$ (19)

where $L$ is a similarity loss, _e.g_. L1, L2, the cosine-similarity or a cost
function.

During training we optimize the similarity loss on static scenes, without
motion. An overview of all losses and their integration in the computational
flow are shown in Fig. 2.

Figure 3: Motion compensation results in the single frequency single tap case.
Both pre-trained networks and our method resolve the motion artifacts, however
our method improves performance over the pre-trained networks. Moreover, the
UFlow method is not able to correct the motion artifacts. However, while the
camera is static and only the center object is moving, all methods have some
tendency to move the background, which introduces additional artifacts. (Empty
regions after warping are shown in black.)

###  4.4 Network Architecture

As OF backbone we investigate two networks with different architectures, the
Motion Module (MOM), which was introduced by Guo _et al_. [10] for ToF motion
correction, and the FFN of Kong _et al_. [18] which is a lightweight network
with on-par performance to State-of-the-Art OF networks. The MOM network is an
encoder-decoder network based on FlowNetS [8], while the FFN integrates a
latent cost volume and is based of the PWC network. Both networks allow for
fast evaluation times and low memory consumption which enables us to predict
multiple flows.

While the flow prediction of the MOM network is rather straightforward, _i.e_.
it takes the set $\\{m_{i}\\}_{i=0}^{N}$ as input and predicts all flows
$\\{V_{i}\\}_{i=0}^{N}$ at once, we will briefly describe how we execute the
FFN in the following. Please note, that the computations of FFN are realized
on a hierarchical feature pyramid, but for compact notation we neglect the
hierarchy levels in the following description.

The FFN consists of the common building blocks, an image encoder $E$, a cost
volume computation $C$ and a flow prediction decoder $D$. Given the
measurements $\\{m_{i}\\}_{i=0}^{N}$, we encode each measurement $m_{i}$ into
a latent vector $F_{i}=E(m_{i})$. The latent vectors are then used to compute
cost volumes for each pairing with the last measurement $m_{N}$, _i.e_.
$c_{i}=C(F_{i},F_{N})$ for $i=1,\dots,N-1$. The decoder then predicts the
flows using pairs of cost volumes and latent vectors as input
$V_{i}=D(F_{i},c_{i})$, the process for a single image pair is also shown on
the left of Fig. 2. After warping the measurement $m_{i}$, parts of the image
might remain empty, as no pixels were warped to this region, these regions are
referred to as masked.

In this formulation the network only considers the two measurements
$m_{i},m_{N}$ to compute $V_{i}$. Although the other measurements contain
additional information about the movement, the above formulation allows to
share the encoder and decoder networks for all measurements and does not
increase the number of parameters.

We further apply an instance normalization to the input of the network, as
also used in the ToF error correction approach of Su _et al_. [28], which does
not affect the depth reconstruction in Eq. (2), as it is invariant to uniform
scaling and translation of the measurements.

In case of multi-tap sensors we change the input dimension of the encoder $E$
such that it receives all measurements captured at the same time step as
input.

##  5 Experiments

| Method | $\mathcal{L}_{photo}$ | $\mathcal{L}_{ToF}$ | mask
---|---|---|---|---
SF 1Tap | Input | 50.09 | 16.87 | -
FFN | 54.21 | 14.63 | 12.40%
PWC | 49.16 | 13.70 | 4.12%
UFlow | 58.71 | 12.76 | 3.24%
Ours(MOM) | 34.64 | 7.64 | 0.97%
Ours(FFN) | 23.27 | 5.81 | 1.60%
SF 2Tap | Input | 34.45 | 5.93 | -
FFN | 29.83 | 5.44 | 6.18%
PWC | 19.77 | 4.03 | 3.55%
UFlow | 38.22 | 4.90 | 2.07%
Lindner (FFN) | 21.01 | 4.22 | 2.35%
Lindner (PWC) | 18.11 | 3.85 | 2.12%
Ours(MOM) | 24.67 | 3.25 | 0.73%
Ours(FFN) | 17.22 | 3.66 | 0.56%
Table 1: Results for single frequency single-tap (SF 1Tap) and two tap (SF
2Tap). The pre-trained networks, FFN and PWC, and the unsupervised UFlow
method achieve only low correction rates in most cases. The Lindner method
reduces the error notably, especially when using the larger PWC as backbone,
still it is outperformed by our proposed method on smaller backbones.  Figure
4: Motion compensation results in the multi frequency four-tap case, for a
scene with moving camera. Our method achieves the best motion compensation,
followed by Lindner’s method on the more powerful backbone PWC, although
Lindner’s method introduces more additional errors. Both the pre-trained PWC
and the UFlow method fail in this case. (Empty regions after warping are shown
in black.)

In our experiments we train instances of both FFN and MOM using the loss
functions described in Sec. 4. In the case of the MOM network we do not use
the similarity loss $\mathcal{L}_{sim}$, as the network does not produce
latent vectors $F_{i}$ due to its different architectural design. We compare
against using pre-trained instances on RGB data of FFN and and also the larger
PWC [29], which needs $\approx$8 times the compute [18]. In the case of multi-
tap sensors we additionally compare against the Lindner method [20] in
combination with the pre-trained instances of FFN and PWC. Further, we compare
against the UFlow method [15], which is a method to train OF networks in an
unsupervised fashion, and uses the PWC as backbone. We train the UFlow method
on the same dataset as our method.

##### Dataset.

We conduct the experiments on the CB-dataset of Schelling _et al_. [26], as it
contains raw measurements $m_{i}$ for three different frequencies. It consists
of 143 scenes each rendered from 50 viewpoints along a camera trajectory,
which allows to simulate real movements that change the point of view. As the
CB-Dataset only incorporates static scene geometries we generated 14
additional scenes with moving objects using the same data simulation pipeline,
to increase the variation of movements in the dataset. We divide the dataset
using the original training, validation and testing split, and further divide
the additional scenes into 10 training scenes, and 2 each for testing and
validation, whereby we use the 20MHz measurements.

###  5.1 Single Frequency Motion Compensation

For the single frequency experiment we also use the 20MHz measurements of the
datasets. In the case of single-tap we take the four measurements from four
subsequent time steps, in the case two-tap we take the pairs $(m_{0},m_{2})$
and $(m_{1},m_{3})$ from two times steps. We measure $\mathcal{L}_{ToF}$, the
photometric loss $\mathcal{L}_{photo}$ and the percentage of masked pixels
after warping, and report results on the test set in Tab. 1. We find that the
networks trained with our method achieve better results than the pre-trained
OF networks and the UFlow method. Results for the single tap case can be seen
in Fig. 3. The results of Lindner’s method come close to our method, but only
when using the larger backbone network PWC. On the same backbone FFN the gap
in performance is larger. Additionally, in the simple setting of two-taps, and
thus also two time steps, the simple MOM backbone results in better
performance than the more complex FFN backbone, both trained with our method.

Further, we observe that the UFlow method increases the photometric loss,
which we attribute to the fact that the method aims to minimize the
photometric loss between the images of different modalities. Additionally,
UFlow has a tendency to mask out areas affected by motion, as is shown in Fig.
4, which leads to a reduced ToF loss, without correcting the errors.

###  5.2 Multi Frequency Motion Compensation

| Method | $\mathcal{L}_{photo}$ | $\mathcal{L}_{ToF}$ | mask
---|---|---|---|---
MF 1Tap | Input | 113.73 | 19.68 | -
FFN | 124.88 | 25.06 | 10.76%
PWC | 83.15 | 16.01 | 8.91%
UFlow | 136.55 | 13.86 | 7.76%
Ours(MOM) | 65.91 | 11.92 | 1.43%
Ours(FFN) | 80.43 | 13.77 | 0.34%
MF 2Tap | Input | 69.06 | 8.17 | -
FFN | 78.33 | 9.71 | 5.90%
PWC | 49.23 | 7.51 | 4.02%
UFlow | 81.45 | 5.95 | 4.82%
Lindner (FFN) | 40.26 | 5.60 | 2.55%
Lindner (PWC) | 35.24 | 5.16 | 1.80%
Ours(MOM) | 44.68 | 4.98 | 0.64%
Ours(FFN) | 30.71 | 4.43 | 0.32%
MF 4Tap | Input | 40.42 | 5.26 | -
FFN | 57.54 | 6.93 | 0.06%
PWC | 31.09 | 5.41 | 0.06%
UFlow | 51.10 | 4.17 | 1.96%
Lindner (FFN) | 27.52 | 3.94 | 0.06%
Lindner (PWC) | 22.17 | 3.49 | 0.06%
Ours(MOM) | 29.64 | 3.11 | 0.48%
Ours(FFN) | 27.14 | 3.03 | 0.08%
Table 2: Results for multi frequency single-tap (MF 1Tap), two-tap (MF 2Tap)
and four-tap (MF 4Tap). In this setting with stronger modality changes pre-
trained networks fail in most cases. Our method is again closely followed by
Lindner on the larger PWC.

For the multi frequency experiment we use the three frequencies 20MHz, 50MHz
and 70MHz of the datasets. In the case of single-tap, we take the twelve
measurements from twelve subsequent time steps. In the two-tap case, we take
pairs $(m_{0},m_{2})$ and $(m_{1},m_{3})$ from six time steps. Lastly, in the
case of four-tap, we use three time steps, one per frequency. The results on
the test set for both $\mathcal{L}_{ToF}$ and the photometric loss
$\mathcal{L}_{photo}$ are reported in Tab. 2, and are shown for the four-tap
case in Fig. 4.

The findings from the single frequency experiment can also be observed in this
setting, with our approach achieving the best performance followed by
Lindner’s method. Further, the FFN trained with our method, while still
outperforming the other methods, achieves rather low performance in the single
tap setting, which is arguably the hardest case with the highest number of
time steps, and thus the largest motion, and additionally the lowest input
dimensionality of only one tap, which might make it harder for the encoder $E$
to extract modality invariant features.

Additionally, the pre-trained OF networks have a tendency to fail in these
settings, especially the FFN, which might come from the larger modality gap of
measurements taken at different frequencies, as can also be seen in Fig. 4.

###  5.3 Motion Compensation and Error Correction

| Method | MAE | Rel. Error
---|---|---|---
SF 1Tap | Input | 39.49 | 100.00%
CFN | 19.39 | 49.10%
CFN + Ours(FFN) | 11.47 | 29.05%
DeepToF | 16.65 | 42.17%
DeepToF + Ours(FFN) | 15.11 | 38.26%
MF 2Tap | Input | 10.65 | 100.00%
CFN | 6.71 | 63.01%
CFN + Ours(FFN) | 5.54 | 52.02%
E2E | 10.44 | 98.03%
E2E + Ours(FFN) | 8.27 | 77.65%
RADU | 11.21 | 105.26%
RADU + Ours(FFN) | 8.00 | 75.12%
Table 3: Results of motion, multi-path-interference and sensor noise
compensation, for the single frequency single tap (SF 1Tap) and the multi
frequency two-tap (MF 2Tap) case. All methods benefit from the motion
correction using our method.

To measure the influence on downstream error compensation techniques, we train
instances of ToF correction networks on the output of our model. For this
experiment, the single frequency single tap case and the multi frequency two-
tap case are considered. We use the single frequency approaches DeepToF [22]
and an adapted CFN [1] in the single frequency case, and the multi-frequency
approaches CFN, E2E [28] and RADU [26] in the multi frequency case. For
comparison we also train instances of the networks without performing motion
compensation, and report results on the test set in Tab. 3

We observe, that all methods benefit from motion compensation in their input.
We further observe that the 2D networks that frame the task as denoising
handle motion artifacts quite well, see Fig. 5, whereas the more complex
approaches E2E, which formulates a generative image translation task, and
RADU, which operates on 3D point clouds, struggle in this setting. It is to be
remarked, that none of the approaches were designed to correct motion
artifacts.

Figure 5: Results of combined motion and MPI correction using the CFN network.
Without additional motion compensation, the motion artifacts are only
partially corrected. In combination with method they are restricted to the
object boundaries.

###  5.4 Ablations

This section provides ablations on the loss components.

####  5.4.1 Component Ablation

Method |  |  |  | $\mathcal{L}_{photo}$ | $\mathcal{L}_{ToF}$
---|---|---|---|---|---
Input |  |  |  | 70.39 | 23.71
$\mathcal{L}_{photo}$ + |  $\mathcal{L}_{smooth}$ + |  $\mathcal{L}_{edge}$ + | $\mathcal{L}_{sim}$ | 38.65 | 12.43
$\mathcal{L}_{ToF}$ |  |  |  | 38.42 | 10.17
$\mathcal{L}_{ToF}$ + |  |  $\mathcal{L}_{edge}$ + | $\mathcal{L}_{sim}$ | 35.94 | 9.67
$\mathcal{L}_{ToF}$ + |  $\mathcal{L}_{smooth}$ + |  | $\mathcal{L}_{sim}$ | 34.65 | 8.54
$\mathcal{L}_{ToF}$ + |  $\mathcal{L}_{smooth}$ + | $\mathcal{L}_{edge}$ |  | 32.57 | 7.87
$\mathcal{L}_{ToF}$ + |  $\mathcal{L}_{smooth}$ + |  $\mathcal{L}_{edge}$ + | $\mathcal{L}_{sim}$ | 28.76 | 7.21
Table 4: Ablation on the loss components in the single frequency single-tap
case, using the FFN as OF backbone.

To investigate the influence of each loss component separately, we train
instances of the FFN network while disabling individual components. Further,
we replace the ToF loss $\mathcal{L}_{ToF}$ with the photometric loss
$\mathcal{L}_{photo}$ and additionally train an instance using only the ToF
loss as baselines. The results on the validation set are reported in Tab. 4

From the results it can be seen that the combination of all losses achieves
the best performance, and that each component reduces the loss. Out of the
regulatory losses the smoothing loss $\mathcal{L}_{smooth}$ has the highest
impact, followed by the edge-aware loss $\mathcal{L}_{edge}$ and finally the
latent similarity loss $\mathcal{L}_{sim}$. Further, the ToF loss yields a
large performance gain compared to the photometric loss, and even without
regularizations achieves a better performance.

####  5.4.2 Similarity Loss Function

As the definition of the latent similarity loss $\mathcal{L}_{sim}$ in Eq.
(19) was kept general, it allows for the usage of different similarity
measures $L$. We investigate the standard L1 and L2 distances, the cost
function that is used in the cost volume computation and the cosine similarity

$\displaystyle\text{L}_{p}:$
$\displaystyle\quad\|F_{i}(k,l)-F_{j}(k,l)\|_{p},\qquad p=1,2$ (20) Cost:
$\displaystyle\quad-F_{i}(k,l)\cdot F_{j}(k,l),$ (21)
$\displaystyle\text{Cosine}:$ $\displaystyle\quad\frac{-F_{i}(k,l)\cdot
F_{j}(k,l)}{\|F_{i}(k,l)\|_{2}\|F_{j}(k,l)\|_{2}},$ (22)

where $\cdot$ denotes the scalar product. We consider the single frequency
single tap and the multi frequency two-tap case in this ablation, and train
instances of the FFN using the above similarity measures, together with all
other loss components. Further, we train an instance using no similarity loss
as a baseline, and, in the case of two taps, compare to using Lindner’s
features as input instead of a similarity measure. From the results, which can
be seen in Tab. 5, we find that the cosine similarity achieves the best
performance in both cases. Additionally, in the multi frequency two-tap case,
the cosine similarity is the only measure that improves over not using a
similarity loss at all, including Lindner’s method. Consequently, both the use
and the choice of the similarity measure needs careful consideration.

##  6 Limitations

Figure 6: Example of an object, where our regularizations fail. The high
frequency pattern prevents $\mathcal{L}_{smooth}$ from enforcing a consistent
flow for the object. Due to the repetitive pattern, the network matches the
yellow region in the input image with the cyan region in the reference image,
and the object gets distorted.

Although, both backbone OF networks achieve good results, we experience cases
that escape our regularization losses. For example, the smoothing loss
$\mathcal{L}_{smooth}$ ensures a continuous flow for an object, however
objects are detected based on their homogeneous appearance, which can fail on
high frequency details. While the edge loss $\mathcal{L}_{edge}$ can resolve
most of the cases, still sometimes wrong parts of the images are matched,
especially when nearby image patches have a similar appearance, see Fig. 6. We
attribute this to the fact that without access to ground truth flows, such
cases present a local minima during training.

Moreover, while we demonstrated our method on the largest available iToF
dataset [26], this work is restricted to a synthetic setting as no real world
data set containing raw iToF measurements is currently available. Lastly, the
choice of the backbone network impacts the performance in different settings,
_i.e_. MOM clearly outperforms the FFN backbone in the multi frequency single
tap setting. Additionally, as our contribution is a training algorithm, the
execution time is given by the execution time of the underlying OF network,
while it is almost constant in the different settings for the MOM network, it
grows linearly with the number of predicted flows for the FFN. As a
consequence it would be desirable to have a OF network for iToF motion
correction with a constant high performance in this multi-modality multi-frame
flow prediction problem.

| SF 1Tap | MF 2Tap
---|---|---
Method | $\mathcal{L}_{photo}$ | $\mathcal{L}_{ToF}$ | $\mathcal{L}_{photo}$ | $\mathcal{L}_{ToF}$
Input | 70.39 | 23.71 | 93.17 | 11.98
Lindner | - | - | 45.59 | 7.48
None | 32.57 | 7.87 | 48.13 | 7.36
L1 | 32.15 | 7.97 | 54.27 | 7.88
L2 | 34.37 | 7.61 | 54.32 | 7.78
Cost | 41.88 | 10.73 | 53.98 | 7.87
Cosine | 28.76 | 7.21 | 45.49 | 6.67
Table 5: Ablation on different loss function for $\mathcal{L}_{sim}$, using
FFN as backbone. On validation set.

##  7 Conclusion

In this work, we presented a training method for OF networks to align iToF
measurements in order to reduce the motion artifacts in the reconstructed
depth images. To this end we enable the weakly supervised training on the ToF
loss $\mathcal{L}_{ToF}$ using a phase unwrapping scheme for gradient
correction. In combination with the regularizing losses $\mathcal{L}_{smooth}$
and $\mathcal{L}_{edge}$ which regulate the flow predictions, and the
similarity loss $\mathcal{L}_{sim}$ to resolve the multi-modality, our method
enables training without the need of ground truth flow labels. The experiments
indicate that our method is able to compensate motion artifacts for both
single and multi frequency settings as well as single and multi tap sensors.
Further, our training method was demonstrated for two backbone OF networks,
with different architectures, and was able to outperform existing methods.

##  8 Acknowledgements

This project was financed by the Baden-Württemberg Stiftung gGmbH.

## References

  * [1] Gianluca Agresti and Pietro Zanuttigh.  Deep learning for multi-path error removal in ToF sensors.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [2] Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia.  Learning optical flow from still images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15201–15211, 2021.
  * [3] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.
  * [4] Enrico Buratto, Adriano Simonetto, Gianluca Agresti, Henrik Schäfer, and Pietro Zanuttigh.  Deep learning for transient image reconstruction from ToF data.  Sensors, 21(6):1962, 2021.
  * [5] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black.  A naturalistic open source movie for optical flow evaluation.  In European conference on computer vision, pages 611–625. Springer, 2012.
  * [6] Faquan Chen, Rendong Ying, Jianwei Xue, Fei Wen, and Peilin Liu.  A configurable and real-time multi-frequency 3D image signal processor for indirect time-of-flight sensors.  IEEE Sensors Journal, 22(8):7834–7845, 2022.
  * [7] Guanting Dong, Yueyi Zhang, and Zhiwei Xiong.  Spatial hierarchy aware residual pyramid network for time-of-flight depth denoising.  In European Conference on Computer Vision, pages 35–50. Springer, 2020.
  * [8] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  * [9] Jens-Malte Gottfried, Rahul Nair, Stephan Meister, Christoph S Garbe, and Daniel Kondermann.  Time of flight motion compensation revisited.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 5861–5865. IEEE, 2014.
  * [10] Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, and Jan Kautz.  Tackling 3D ToF artifacts through learning and the FLAT dataset.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 368–383, 2018.
  * [11] Felipe Gutierrez-Barragan, Huaijin Chen, Mohit Gupta, Andreas Velten, and Jinwei Gu.  iToF2dToF: A robust and flexible representation for data-driven time-of-flight imaging.  IEEE Transactions on Computational Imaging, 7:1205–1214, 2021.
  * [12] Miles Hansard, Seungkyu Lee, Ouk Choi, and Radu Patrice Horaud.  Time-of-flight cameras: principles, methods and applications.  Springer Science & Business Media, 2012.
  * [13] Thomas Hoegg, Damien Lefloch, and Andreas Kolb.  Real-time motion artifact compensation for PMD-ToF images.  In Time-of-flight and depth imaging. Sensors, algorithms, and applications, pages 273–288. Springer, 2013.
  * [14] Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger.  Unsupervised learning of multi-frame optical flow with occlusions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 690–706, 2018.
  * [15] Rico Jonschkowski, Austin Stone, Jonathan T Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova.  What matters in unsupervised optical flow.  arXiv preprint arXiv:2006.04902, 2020.
  * [16] Min-Sun Keel, Young-Gu Jin, Youngchan Kim, Daeyun Kim, Yeomyung Kim, Myunghan Bae, Bumsik Chung, Sooho Son, Hogyun Kim, Taemin An, et al.  A VGA indirect time-of-flight CMOS image sensor with 4-tap 7-$\mu$ m global-shutter pixel and fixed-pattern phase noise self-compensation.  IEEE Journal of Solid-State Circuits, 55(4):889–897, 2019.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [18] Lingtong Kong, Chunhua Shen, and Jie Yang.  FastFlowNet: A lightweight network for fast optical flow estimation.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10310–10316. IEEE, 2021.
  * [19] Ruoteng Li, Robby T Tan, Loong-Fah Cheong, Angelica I Aviles-Rivero, Qingnan Fan, and Carola-Bibiane Schonlieb.  Rainflow: Optical flow under rain streaks and rain veiling effect.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7304–7313, 2019.
  * [20] Marvin Lindner and Andreas Kolb.  Compensation of motion artifacts for time-of-flight cameras.  In Workshop on Dynamic 3D Imaging, pages 16–27. Springer, 2009\.
  * [21] Oliver Lottner, Arnd Sluiter, Klaus Hartmann, and Wolfgang Weihs.  Movement artefacts in range images of time-of-flight cameras.  In 2007 International Symposium on Signals, Circuits and Systems, volume 1, pages 1–4. IEEE, 2007.
  * [22] Julio Marco, Quercus Hernandez, Adolfo Munoz, Yue Dong, Adrian Jarabo, Min H Kim, Xin Tong, and Diego Gutierrez.  DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging.  ACM Transactions on Graphics (ToG), 36(6):1–12, 2017.
  * [23] Henrique Morimitsu.  Pytorch lightning optical flow.  https://github.com/hmorimitsu/ptlflow, 2021.
  * [24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [25] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha.  Unsupervised deep learning for optical flow estimation.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * [26] Michael Schelling, Pedro Hermosilla, and Timo Ropinski.  RADU: Ray-aligned depth update convolutions for ToF data denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 671–680, 2022.
  * [27] Mirko Schmidt.  Analysis, modeling and dynamic optimization of 3D time-of-flight imaging systems.  PhD thesis, 2011.
  * [28] Shuochen Su, Felix Heide, Gordon Wetzstein, and Wolfgang Heidrich.  Deep end-to-end time-of-flight imaging.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6383–6392, 2018.
  * [29] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943, 2018.
  * [30] Z. Teed and J. Deng.  RAFT: Recurrent all-pairs field transforms for optical flow.  In Proc. of ECCV, 2020.
  * [31] Lingzhu Xiang, Florian Echtler, Christian Kerl, Thiemo Wiedemeyer, Lars, hanyazou, Ryan Gordon, Francisco Facioni, laborer2008, Rich Wareham, Matthias Goldhoorn, alberth, gaborpapp, Steffen Fuchs, jmtatsch, Joshua Blake, Federico, Henning Jungkurth, Yuan Mingze, vinouz, Dave Coleman, Brendan Burns, Rahul Rawat, Serguei Mokhov, Paul Reynolds, P.E. Viau, Matthieu Fraissinet-Tachet, Ludique, James Billingham, and Alistair.  libfreenect2: Release 0.2, Apr. 2016.
  * [32] Jason J Yu, Adam W Harley, and Konstantinos G Derpanis.  Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness.  In European Conference on Computer Vision, pages 3–10. Springer, 2016.
  * [33] Yinqiang Zheng, Mingfang Zhang, and Feng Lu.  Optical flow in the dark.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6749–6757, 2020.

##  Appendix A Contents

This supplementary material provides additional information about our method
in Sec. B and Sec. C and further details on how the experiments were conducted
in Sec. D. Finally, we show additional qualitative results in Sec. E.

Our code, trained networks and the additional scenes to expand the CB-dataset
[26], containing moving objects, are available at
https://github.com/schellmi42/WFlowToF.

##  Appendix B Phase Unwrapping of the ToF Loss Function

In this section we provide more information on the phase unwrapping of the
gradients of the ToF loss function $\mathcal{L}_{ToF}$, which is given through

$\displaystyle s$ $\displaystyle=\text{sign}(\hat{m}_{0}-\hat{m}_{2})$ (23)
$\displaystyle\hat{d}$ $\displaystyle=\frac{c}{4\pi
f}\arctan\left(\frac{\hat{m}_{3}-\hat{m}_{1}}{\hat{m}_{0}-\hat{m}_{2}+s\cdot\epsilon}\right),$
(24) $\displaystyle\mathcal{L}_{ToF}$ $\displaystyle=\|\hat{d}-d_{ToF}\|_{1},$
(25)

where $\hat{m}_{i}$ are the iToF measurements after warping, and $\epsilon$ is
a small positive constant. While the standard $\arctan$ function has a range
limited to a semi-circle $(-\pi/2,\pi/2)$, the sign of the numerator and the
denominator in Eq. (24) can be used to extend the range to a full circle
$(-\pi,\pi]$. This method is commonly referred to as the
$\operatorname{arctan2}$ function

$\displaystyle x$ $\displaystyle=\hat{m}_{0}-\hat{m}_{2},$ (26) $\displaystyle
y$ $\displaystyle=\hat{m}_{3}-\hat{m}_{1},$ (27) $\displaystyle\hat{d}$
$\displaystyle=\frac{c}{4\pi f}\operatorname{arctan2}(y,x+s\cdot\epsilon).$
(28)

As a consequence, the $\operatorname{arctan2}$ function has multiple branches,
corresponding to the sign of its arguments, as can be seen in Fig. 7.

The figure also illustrates the difference of $d_{max}/2$ between the two
branches in this case. As a result if the target (red point) is on the
different branch than the current depth estimate (green point), the direction
of the optimization needs to be inverted in order to move the estimate through
the phase wrapping of the $\arctan 2$ function and change to the correct
branch. This is realized by our proposed gradient correction presented in the
main paper

$\displaystyle\nabla\mathcal{L}_{ToF,PU}=\left\\{\begin{aligned}
\nabla\mathcal{L}_{ToF},&\qquad 0\leq\mathcal{L}_{ToF}<{d_{max}/2},\\\
-\nabla\mathcal{L}_{ToF},&\qquad\mathcal{L}_{ToF}\geq{d_{max}/2}.\\\
\end{aligned}\right.$ (29)

Figure 7: Reconstructed depth $\hat{d}$ dependent on the measurement
$\hat{m}_{0}$ in the case of a positive denominator. The
$\operatorname{arctan2}$ function changes branches at
$\hat{m}_{3}=\hat{m}_{1}$, which introduces a discontinuity. As a result, if
the prediction (red point) and the target value (green point) are on separate
branches, the gradient points in the wrong direction (red arrow), and is
corrected by our method (green arrow). The branches are separated by
$d_{max}/2$, in line with Eq. (29). Figure 8: Reconstruction of measurement
$\hat{m}_{3}$ by minimizing the ToF loss. Without phase unwrapping (PU) only
partial reconstruction is possible. After correcting the gradients with our
method, the phase wrapping is successively resolved by the optimization
(right).

To show the influence on the optimization we conduct a toy experiment, in
which we formulate a simple reconstruction task. We assume $m_{0},m_{1},m_{2}$
and $d_{ToF}$ are given and the task is to reconstruct the measurement $m_{3}$
by minimizing the ToF loss $\mathcal{L}_{ToF}$

$\displaystyle\underset{\hat{m}_{3}}{\min}\ \mathcal{L}_{ToF}$ (30)
$\displaystyle=$
$\displaystyle\underset{\hat{m}_{3}}{\min}\left\|\frac{c}{4\pi
f}\operatorname{arctan2}\left(\hat{m_{3}}-m_{1},m_{0}-m_{2}\right)-d_{ToF}\right\|_{1}.$
(31)

We initialize $\hat{m}_{0}=0$ and optimize it using simple gradient descent,
with and without applying our gradient correction method. Without gradient
correction only parts initialized on the correct branch are reconstructed,
wheres after gradient correction all parts can be reconstructed, as is shown
in Fig. 8.

##  Appendix C Regularization Losses

This section provides more insights on the effect of the regularization
losses.

Without regularizations the problem of supervising the four measurements with
a single depth is under-determined, _e.g_. in Eq. (28) the ratio of $y/x$ is
the determining factor, but not the individual values. While the search space
is already limited when predicting optical flows, as not arbitrary values are
allowed, but only values from a local neighborhood can be warped to a certain
position, still regularization is necessary to further restrict the network
predictions. Moreover, unsupervised Optical Flow (OF) networks require such
regularizations in general to achieve competitive performance [15].

In our work we introduced two main regularizations which measure consistency
in the image space. The smoothing loss $\mathcal{L}_{smooth}$ measures region
consistency between the predicted flow and the input image, and is applied
before warping. The edge-aware loss $\mathcal{L}_{edge}$ measures edge
consistency between the warped image and the target image, and is applied
after warping.

The impact of these losses on the warped images can be seen in Fig. 9

Figure 9: Effect of the regularization losses on the warped image. The
smoothing loss $\mathcal{L}_{smooth}$ ensures that pixels that belong to a
visually similar region are moved in the same direction (top row). With the
edge aware loss $\mathcal{L}_{edge}$ the edges of the warped image align with
the target image, preserving object boundaries and details. (bottom row)

##  Appendix D Experiments

In this section we provide detailed information about the hyperparameters used
for training the networks.

###  D.1 Implementation

All custom implementations were done in `PyTorch 1.10.+cu102` [24] and `Python
3.6`. For the OF networks FFN [18] and PWC [29], and the warping operation, we
use implementations provided in the PyTorch library `ptlflow 0.2.5` [23].

###  D.2 Motion Compensation

##### Our Method.

Both OF backbone networks FFN and MOM [10] are trained with the ADAM [17]
optimizer using a learning rate scheduler which decays the learning rate by a
factor $0.5$ when the ToF loss on the validation set did not decrease for 50
epochs.

We augment the input data by simulating shot noise on the iToF measurements,
following the noise model described by Schelling _et al_. [26]. Additionally,
we use random image rotations by
$0^{\circ},90^{\circ},180^{\circ},270^{\circ}$, random mirroring along the
image axes, and crop random $512\times 512$ image patches during training.

We train the FFN network with the combination of all losses

$\displaystyle\mathcal{L}_{ToF}+\lambda_{smoooth}\cdot\mathcal{L}_{smooth}+\lambda_{edge}\cdot\mathcal{L}_{edge}+\lambda_{sim}\cdot\mathcal{L}_{sim}.$
(32)

We compared the following values to select the hyperparameters: weights
$\lambda_{i}$ from {1, 1e-1, 1-e2, 1e-3, 1e-4}, and the shift parameter $s$ in
the edge-aware loss from {1e-2, 1-e1, 0, 1e1, 1e2, 1e3}. The results reported
in the main paper were achieved with
$\lambda_{smooth}=1,\lambda_{edge}=\text{1e-1},\lambda_{sim}=\text{1e-2},s=\text{1e2}$
in the single frequency case, and in the multi frequency case only the
similarity weight was changed to $\lambda_{sim}=1e-2$. In all experiments the
cosine similarity was used in $\mathcal{L}_{sim}$. In the single frequency
experiments we train with a batch size of 8, and in the multi-frequency
experiment with a batch size of 4. The initial learning rate is set to 1e-3.

For the MOM network we do not use the similarity loss, as the encoder decoder
architecture does not have latent features for a cost volume computation, thus
we set $\lambda_{sim}=0$. We perform the same hyperparameter tuning as for the
FFN network. The results in the main paper were achieved
with$\lambda_{smooth}=1,\lambda_{edge}=1,s=\text{1e3}$ in both the single and
the multi-frequency experiments. We train with a batch size of 1 and an
initial learning rate of 1e-5, as recommended by the authors of MOM [10],

##### Pre-Trained Networks.

The pretrained networks, FFN and PWC were trained on RGB data and hence
require three input channels. In our experiments we normalize the iToF
measurements to a range of [0, 255] and repeat the the scalar image three
times to match the RBG input. We use weights pre-trained on the Sintel dataset
[5].

##### UFlow.

The UFlow [15] method is trained on the same data as our method, using the
TensorFlow2 implementation provided by the authors. We use the hyperparameters
recommended in the documentation for custom datasets, which correspond to the
settings for the Flying Chairs dataset [8] in the UFlow paper.

##### Lindner Method.

For the Lindner method, we match the input dimensionality of the pre-trained
networks using the same scheme as above on the intensity computed with
Lindner’s method.

##### Comparison to RAFT (SotA)

As the task involves the prediction of multiple optical flows at once, only
lightweight OF networks, such as FFN, can be trained in this setting. More
advanced architectures, such as RAFT [30], which is currently the State-of-
the-Art (SotA) in supervised RGB OF prediction, increase the computational
cost and we were unable to train it as a backbone network. While a pre-trained
RAFT model achieves a better performance than the simpler PWC (see Tab 6), the
inference times are much slower (see Tab. 7). Still the pre-trained RAFT model
is outperformed by our method with a much smaller FFN backbone (see Tab. 6).

| SF 1T | SF 2T | MF 1T | MF 2T | MF 4T
---|---|---|---|---|---
PWC (PT) | 13.70 | 4.03 | 16.01 | 7.51 | 5.41
RAFT (PT) | 8.48 | 4.08 | 14.72 | 6.55 | 4.63
Our (FFN) | 5.81 | 3.66 | 13.77 | 4.43 | 3.03
Table 6: Resulting $\mathcal{L}_{ToF}$ of a pre-trained (PT) RAFT in
comparison to a pre-trained PWC and our method using FFN as backbone. Results
on the test set.

###  D.3 Inference time

As our approach is a training algorithm, it does not affect the evaluation
times of the backbone OF networks. As stated in the main paper, the encoder
decoder architecture of MOM allows fast execution times almost independent of
the number of predicted flows. In contrast, the runtime of networks using
derivatives of the more advanced cost-volume architecture [8] grows linearly
corresponding to the number of predicted flows. Prediction times for the
networks used in this work and a comparison to the SotA RAFT network are shown
in Tab. 7.

| SF 1T | SF 2T | MF 1T | MF 2T | MF 4T
---|---|---|---|---|---
MOM | 0.002 | 0.002 | 0.002 | 0.002 | 0.002
FFN | 0.067 | 0.025 | 0.230 | 0.107 | 0.047
PWC | 0.092 | 0.054 | 0.342 | 0.154 | 0.062
RAFT | 0.342 | 0.110 | 1.275 | 0.578 | 0.228
Table 7: Inference times in _s_ of the OF backbone networks, averaged over the
test set, on a GTX 1080 GPU.

Figure 10: Results of our method for single frequency single-tap (SF 1Tap,
left) and multi frequency two-tap (MF 2Tap, right). The scenes contain moving
objects. First row shows ToF depths, second row shows error maps. Please note
that the ToF depths are not phase unwrapped.

###  D.4 Motion Compensation and Error Correction

We implement the CFN [1] in PyTorch and also adapt it to the single frequency
case by reducing the input dimension to one. For the other approaches DeepToF
[22], E2E [28] and RADU [26] we use the TensorFlow2 implementations by
Schelling  _et al_. [26]. We train all networks using the respective
hyperparameters reported by Schelling  _et al_. for their CB-Dataset.

###  D.5 Ablation: Similarity Loss Function

We train the FFN network with the different similarity measures in the
similarity loss function $\mathcal{L}_{sim}$ and optimize their weight
$\lambda_{sim}$ from {1e1, 1, 1e-1, 1e-2, 1e-3, 1e-4} for each measure. When
training with input generated by Lindner’s method in the multi frequency two-
tap case, we reduce the network input dimension to one. The other
hyperparameters, including the weights of the other losses, are set as in the
main experiments. The results in the main paper were achieved with the
following weights:
SF 1Tap: L1: 1e-2, L2: 1e-1, Cost: 1e-2, Cosine: 1e-3
MF 2Tap: L1: 1e-3, L2: 1e-2, Cost: 1, Cosine: 1e-3

##  Appendix E Qualitative Results

Results of our method using the FFN and the MOM network can be seen in Fig.
10, 11, 12 and 13. The figures show one frame per scene from the test set. To
cover both single and multi frequency and single and multi-tap the two cases
single frequency single tap and multi frequency two tap are shown.

In Fig. 14 we show additional results for the combined correction of motion
artifacts and Multi-Path-Interference (MPI) using the CFN as error correction
network.

Figure 11: Results of our method for single frequency single-tap (SF 1Tap,
left) and multi frequency two-tap (MF 2Tap, right). First row shows ToF
depths, second row shows error maps. ToF depths are not phase unwrapped.

Figure 12: Results of our method for single frequency single-tap (SF 1Tap,
left) and multi frequency two-tap (MF 2Tap, right). First row shows ToF
depths, second row shows error maps. Please note that the ToF depths are not
phase unwrapped.

Figure 13: Results of our method for single frequency single-tap (SF 1Tap,
left) and multi frequency two-tap (MF 2Tap). First row shows ToF depths,
second row shows error maps. Please note that the ToF depths are not phase
unwrapped.

Figure 14: Results of combined motion and MPI correction using the CFN
network. First row shows depths, second row shows error maps. Input in the MF
2Tap case is shown at highest frequency. First scene contains a moving object.
Please note that CFN receives input from all frequencies, which can result in
additional motion artifacts in the prediction compared to the high frequency
input ToF depth.


11institutetext: University of South Eastern Norway, Bø i Telemark, Norway
11email: {angu, tor.lonnestad, ingrid.sundbo, marius.johannessen,
<EMAIL_ADDRESS>22institutetext: University of Agder,
Kristiansand, Norway
22email<EMAIL_ADDRESS>
# Generative AI in Undergraduate Information Technology Education - Insights
from nine courses

Anh Nguyen Duc 11 Tor Lønnestad 11 Ingrid Sundbø 11 Marius Rohde Johannessen
11 Veralia Gabriela 11 Salah Uddin Ahmed and Rania El-Gazzar 1122

###### Abstract

The increasing use of digital teaching and emerging technologies, particularly
AI-based tools, such as ChatGPT, is presenting an inevitable and significant
impact on higher education. The capability of processing and generating text
could bring change to several areas, such as learning assessments or learning
experiences. Besides the negative impact, i.e exam cheating, we also see a
positive side that ChatGPT can bring to education. This research article aims
to contribute to the current debate on ChatGPT by systematic reflection and
experience reported from nine bachelor IT courses at a Norwegian university.
We conducted inductive empirical research with reflective notes and focused
groups of lecturers from nine different IT courses. The findings were
thematically organized with numerous use cases in teaching IT subjects. Our
discussion highlights the disruptive implications of AI assistant usage in
higher education and emphasizes the need for educators to shape this
transformation

###### Keywords:

large language models prompt patterns prompt engineering software engineering
ChatGPT

##  1 Introduction

In the era of modern education, technology has ushered in a transformative
wave that has redefined the way we teach and learn. Among these innovations,
i.e. gamification, virtual reality and distant learning, Generative Artificial
Intelligence (AI) has emerged as a promising tool that transcends the
boundaries of traditional pedagogy. Generative Artificial intelligence (GenAI)
generates different types of content, such as pictures, text, audio, and 3D
models, based on the input they receive. Famous examples of GenAI include
ChatGPT and Google’s Bard (Google). ChatGPT, a large language model based on
GPT-3, is an AI chatbot designed to understand and respond to a wide variety
of questions and topics, which has the potential to revolutionize our
interaction with technology [8]. While some view ChatGPT as a threats to
traditional systems that should be prohibited [6, 7, 16, 15, 24], we cannot
ignore the potential of efficiency and productivity the tool could bring to
our society. For instance, it could be used for voice user interfaces to
overcome issues with response behavior or response quality [12]. AI-based
tools can support the creation of new ideas or help in terms of automating
tasks.

In the education domain, the emergence of AI has an already direct impact on
higher education, including unknown handling by students, more time-consuming
assessments, and the need to evaluate exams differently due to established
control structures such as plagiarism checkers and AI detection tools not
being able to distinguish between AI-generated and human-generated text. As a
result, we will have a change in the way we conduct and evaluate exams. These
examples will not be the only changes. Equally important is a change at the
program level toward how student-centered style, activities, and approaches
can be supported by this tool.

We believe this will be a revolutionary shift in IT education. Thus, it is
important to have discussion and guidance on how to deal with such emerging
technologies to actively shape this transformation. We contribute to the
discussion about ChatGPT in education by systematically examining the adoption
of ChatGPT in several courses in Information Technology bachelor programs at
the University of South Eastern Norway (USN). Our discussion is associated
with different types of courses, from theory courses, programming courses and
project-based courses. Besides reflective notes from course lecturers, we also
gathered experiences from courses where ChatGPT has been used for teaching,
supervising and assessment. We present these reflections under four
categories: strengths, weaknesses, threats, and opportunities. We enriched the
discussion with evidence from related literature.

The paper is organized as follows: Section 2 is background and related work.
Section 3 presents our research method. Section 4 briefly describes the
setting of our courses. Section 5 presents our findings. Section 6 discusses
and concludes the paper.

##  2 Background

###  2.1 Generative AI and ChatGPT

Large Language Models (LLMs) are artificial intelligence systems trained to
understand and process natural language. They are a type of machine learning
model that uses very deep artificial neural networks and can process vast
amounts of language data [17]. An LLM can be trained on a very large dataset
comprising millions of sentences and words. At the moment, the most popular
LLMs include BERT, T5 and GPT-3 [16, 15]. Generative modeling is an artificial
intelligence (AI) technique that generates synthetic artifacts by analyzing
training examples, learning their patterns and distribution [10]. Generative
Pre-trained Transformer (GPT) models generate text in different languages and
can create human-sounding words, sentences, and paragraphs on almost any topic
and writing style [11]. One recent example is ChatGPT, a large language model
trained by OpenAI [4]. It is designed to generate human-like text based on a
given prompt or context. It can be used for various natural language
processing tasks, such as text completion, conversation generation, and
language translation [22]. When given a prompt or context, the model processes
the input and generates a response. The response is generated one word at a
time, with the model predicting the next word based on the input and the words
it has generated so far. After the training, the model can be fine-tuned on a
specific task, such as question answering or dialogue generation, by providing
it with task-specific examples and fine-tuning the model on this data. It can
also generate text in multiple languages by fine-tuning the model on the
multilingual dataset or by providing the model with the language code (ChatGPT
2023). This research discusses ChatGPT 3 due to its current availability in
Norway, which has made it a focal point of interest. ChatGPT-3 uses a deep
neural network architecture known as Transformer. The model is trained on a
massive dataset of text from the internet, which allows it to generate human-
like responses to a wide range of input prompts. At a high level, the model
works by taking in a sequence of input tokens (usually words or sub-words) and
using them to predict the next token in the sequence. For an incomplete
sentence, for example: “The book is on the”, these models use training data to
produce a probability distribution to determine the most likely next word,
e.g., ”table” or ”bookshelf”.

###  2.2 Some recent work on GenAI in education

Much education research has been done on topics like Artificial Intelligence
in general, personalized learning and chatbots within the context of higher
education. For instance, Wartman and Combs argued that education is evolving
in lockstep with changes in the professional sector, demanding the use of
Artificial Intelligence (AI) in teaching and learning [25]. Alam and Mohanty
conducted a systematic literature review to summarize ethical considerations,
challenges, and potential threats linked to the application of AI in higher
education while also exploring potential uses of AI [2]. Their findings were
grouped into four categories: intelligent tutoring systems, personalization
and adaptive systems, evaluation and assessment, and prediction and profiling.
Their research highlights a shortage of critical thinking regarding the
challenges and potential threats of using AI in higher education. A chatbot is
a computer program that is capable of maintaining a conversation with a user
in natural language, understanding their intent, and replying based on preset
rules and data [1]. Okonkwo and colleagues conducted a systematic literature
review of 53 articles about chatbot applications in education [19]. Among
common use cases, chatbot systems were reported to deliver course content to
students via an online platform as a conversational agent capable of providing
accurate information to users and getting individualized help.

Since the release of ChatGPT in November 2022, there have already been several
publications on AI-assisted education. For example, Rudolph et al. have
examined potential use cases in higher education and recommend an engaged
approach where educators figure out ways of incorporating AI assistant
technology in teaching and examinations [23]. Kung has performed experiments
using ChatGPT to answer medical exams and suggests the technology has the
potential to assist in medical education [13]. Pavlik has done a similar
experiment for media and journalism education and found that AI can be a
decent co-author. His paper was written with ChatGPT as a “co-author” [20].
Baidoo-Anu et al. discuss how ChatGPT can assist in the learning process and
also some of the potential pitfalls, such as wrong factual information,
incomplete data sets, or bias in data training [3]. Most recently, Nguyen-Duc
et al. conducted a comprehensive survey of more than 200 recent works on GenAI
in software engineering and education, and presented some future directions
for GenAI in software engineering education [18]

##  3 Research method

Lecturers from nine IT courses were invited to participate in the study.
Before group meetings, each of them wrote a reflection note about the
application of ChatGPT in their courses. They were given a list of questions
to guide their reflections (Table 1). The results were a written collection of
the participant’s responses. We followed up with focus groups, as it allows
researchers to collect a large amount of data from a substantial group of
people in a relatively short amount of time, particularly for exploring how
people perceive, feel about, or view a certain domain [26]. Two focus groups
were conducted online during March-April 2023. The focus groups were conducted
as discussion sessions, each lasting 60 to 90 minutes [26]. The implementation
was conducted by involving nine lecturers, including the moderator in the
research group. The first session focused on generating and synthesizing the
themes regarding the research question. The second session adopted the
interpretation approach in the context of self-understanding, where they tried
to formulate in a condensed form what the participants themselves meant and
understood (Spencer et al., 2003)

To guide the analysis step, we used the SWOT framework [14]. SWOT presents
Strengths, Weaknesses, Opportunities, and Threats related to a plan,
situation, business competition, or project planning. It is also called
situational assessment or situational analysis. Strengths and weaknesses are
internal to the organization and can be directly managed by it, while the
opportunities and threats are external and can only anticipate and react to
them. The framework is also commonly used in academics for analyzing teaching
and learning situations.

Table 1: Reflective questions used in SWOT regarding the adoption of ChatGPT Section |  Questions
---|---
Strengths |  1\. What are the advantages of adopting ChatGPT in your course?
2\. What unique features does ChatGPT offer that can enhance the teaching and
learning experience in your course?
3\. What are the potential benefits for students, in your course?
Weaknesses |  4\. What are the disadvantages of adopting ChatGPT in your course?
5\. What are the potential limitations of ChatGPT in terms of domain knowledge
in your course?
6\. What are the potential negative impacts on student-teacher interaction in
your course?
7\. What are other potential technical challenges?
Threats |  8\. What are the potential negative impacts on the quality of the course?
9\. What are the potential negative perceptions or resistance from students,
teachers, or parents towards using ChatGPT in your course?
Opportunities |  10\. What are the potential opportunities to customize ChatGPT to meet the specific needs of students and teachers in your course?
11\. What are the untapped areas in education where ChatGPT can be applied,
such as language teaching or special needs education?
12\. How can ChatGPT be integrated into existing pedagogical approaches to
create new teaching and learning experiences that were not previously
possible?

##  4 Study context

The Bachelor in IT and Information Systems offered by the University of South-
Eastern Norway provides students with a comprehensive education in various
areas of information technology. This program is designed to equip students
with practical skills in program and system development, web-based solutions,
user support, and IT management. With a focus on both theory and practical
application, students will gain a deep understanding of the latest
technological developments in the field while also developing the problem-
solving skills required to excel in a constantly changing industry. By the end
of the program, graduates will have the knowledge and expertise to pursue a
successful career in various sectors of the IT industry. The bachelor’s
program is in Norwegian. We categorize the courses into three categories:

  * •

Programming courses: Web development and Human-Computer Interaction (WEB1000)
and Database (DAT1000). WEB1000 teaches students basics of HTML, CSS, and
JavaScript. DAT1000 teaches the student the basics of database system’s
concepts, terminology, database management systems, structured query language
(SQL) and relational models (ER).

  * •

Theory courses: Information System (INF1000), Artifical Intelligence for
Business Applications (AI3000), and Digital Strategy (DIG3000). INF1000
provides students with an introduction to the field of Information Systems and
current research in the field. AI3000 focuses on the impact of Artificial
Intelligence (AI) on organizations and society in general. DIG3000 provides
students with an introduction to the strategic use of IT in organizations and
how organizations can use technology to support new and existing business
processes and develop new business models.

  * •

Project-based courses: Practical project work (PRO1000), Business Intelligence
and Data Warehouse (BID3000), and System Development (SYS1000). PRO1000
focuses on software project work and project development methodology, with an
emphasis on Agile development. In BID3000, students learn about the ETL-
process (extraction - transformation - lasting), the data cube, and how to
plan the introduction of a BI system. SYS1000 is an introductory course on
software development covering project management, processes, models, methods,
techniques, and tools.

##  5 Results

The result is organized into four aspects of adopting ChatGPT – strength
(Section 5.1), weakness (section 5.2), opportunities (section 5.3) and threat
(Section 5.4).

###  5.1 Strength

ChatGPT can be a powerful tool for teaching IT subjects to students for a
variety of reasons. ChatGPT can support the formulation of structured
theoretical elements, providing examples to demonstrate for cases, concepts
and phenomena. Another strength is its ability to provide quick and accurate
responses to a wide range of questions, which can be particularly helpful in
programming courses where students may encounter abstract and inter-related
topics. Moreover, ChatGPT can adapt its responses to the specific needs and
learning styles of individual students, providing a personalized learning
experience. It is common that some students are fast to learn and some others
are slower. A ChatGPT-based virtual assistant can be accessed from anywhere at
any time, making it a convenient resource for students who need assistance
outside of class hours or who are studying remotely.

As a multi-language chatbot, ChatGPT can remove language barriers for non-
native English-speaking students, making IT education more accessible to a
wider range of learners. For programming courses, learning materials,
including technique description, concept explanation, code comment, etc.,
originally in English, can be easily offered in a local language. It can also
be helpful in generating ideas and starting points for a lecture, assisting
with problem-solving, and even acting as a conversational partner for students
who feel lonely in classes. The tool can be programmed to offer encouragement
and positive reinforcement, helping to boost their confidence and morale. If a
student is struggling with a particular concept, ChatGPT can suggest
additional resources or provide examples to help them better understand the
topic. In a project-based course, ChatGPT can be a valuable resource for
project management and development. It can assist with finding reading
materials for a subject and explain general domain or contexts. For example,
if a student is working on a project related to e-commerce, ChatGPT can
explain the basics of e-commerce, including different models, challenges, and
opportunities. ChatGPT can assist in generating ideas for team and task
assignments. For example, if a team is working on a project related to
building a mobile app, ChatGPT can suggest typical architectural models,
features and functionalities that can be included in the app. If a team
practices Agile methodology, ChatGPT can act as a source of information about
Agile principles, techniques, best practices and describe how to implement it
effectively.

Figure 1: Strengths of ChatGPT in assisting IT education

###  5.2 Weaknesses

The weaknesses of ChatGPT in our education are described in four categories:
common weaknesses, programming courses, project-based courses and theory
courses. One common weakness of ChatGPT in education is that its responses are
limited to the information provided in the question. This means that if a
question is poorly formulated or lacks relevant details, the response from
ChatGPT may not be satisfactory. Additionally, the model that ChatGPT is based
on is trained on pre-existing data up to 2021, which means that it may not
have up-to-date knowledge on certain topics. Answers from ChatGPT may also
need to be verified by experts to ensure accuracy and prevent the spread of
misinformation. Another weakness is the requirement for awareness and critical
thinking of what the tool can provide, which students might not have. This
could lead to false confidence about certain topics or subjects. There is also
a repeatability issue, as different answers can be generated for the same
question, which could lead to confusion for students. When data is limited,
responses from ChatGPT can be shallow or incorrect. Finally, ChatGPT has
limited domain knowledge, so questions that require a lot of context or
background information may not receive meaningful answers without sufficient
input from the user. Questions such as “how do I implement system X in
organization Y?” would require feeding chatGPT with a lot of contexts in order
to receive meaningful answers.

Regarding programming courses, tools like ChatGPT cannot provide hands-on
experiences, such as coding challenges or real-world projects. Firstly, while
it can provide information and guidance on programming concepts, students
cannot develop their practical skills solely by using ChatGPT. For example,
for a student who is learning about Machine Learning models, and needs to
build the model, ChatGPT cannot provide it. Secondly, with lectures with a lot
of visual representations, ChatGPT relies on natural language processing and
may struggle to interpret visual information. For example, in a lecture about
3D modeling, ChatGPT may not be able to explain a particular aspect of the
modeling process that is best understood through visual demonstration.
Thirdly, unlike teachers, ChatGPT cannot tell industry-specific programming
practices. While ChatGPT can provide information on general programming
concepts, it may not be able to provide insight into specific practices used
in a particular industry. For example, a student studying game development may
need to learn about specific optimization techniques used in the gaming
industry, which ChatGPT may not be familiar with.

Regarding project-based learning, where project contexts can change
frequently, it is necessary to update the context information for the tool to
catch up with project progress. For example, if a project in a software
engineering class changes its requirements, students may need to make changes
to their code. ChatGPT may not be able to keep up with these changes and may
not be able to provide relevant advice to students. This is also because real-
world settings can not be fully described as input for the tool. ChatGPT is
based on pre-existing data and may not be able to provide practical advice for
unique situations that occur in real-world settings. For example, a business
student may need advice on how to handle a specific client situation that is
not covered by the pre-existing data used to train ChatGPT.

Figure 2: Weaknesses of ChatGPT in assisting IT education

###  5.3 Opportunities

The potential use cases for ChatGPT in education are diverse and beneficial
for teachers and students. In theory and programming classes, ChatGPT can
assist in structuring thoughts and learning activities. Students can utilize
the chatbot to organize their ideas, identify knowledge gaps, and create mind
maps or outlines of course reading materials.

Teachers can rely on ChatGPT to design curricula based on existing materials
and receive suggestions for new content and topics. The chatbot can also aid
in planning lecture schedules to ensure comprehensive coverage of necessary
subjects. Additionally, ChatGPT can be employed as an educational tool to
teach students about the advantages of using AI tools in their learning
process. Students can actively engage with the chatbot to support their own
learning endeavors.

Furthermore, ChatGPT is capable of organizing and reorganizing curriculum
components based on personal requests, assisting students in structuring
assignments, and providing outlines and content suggestions for reports. It
can also educate students with little or no IT background by offering simple
explanations and examples in a user-friendly manner. The chatbot can provide
personalized advice on the best courses of study tailored to individual skills
and experiences.

By combining ChatGPT with an active learning or flipped classroom approach,
the tool can adapt to student’s needs and facilitate a personalized and
responsive learning experience. It can generate new assignments for the class
by incorporating contextual data such as current events or student interests.
Moreover, students can work at their own pace and receive feedback on their
progress, making learning more flexible. Lastly, ChatGPT can effortlessly
generate customized examples and code snippets to enhance students’
understanding of CSS or JavaScript concepts.

In a programming course, the opportunities for assistance with source code
include code explanation, bug fix, learning about concepts and code
generation. ChatGPT can also be used as a personal assistant for students. It
can help students manage their time, schedule, and assignments. Code
explanation is one of the opportunities that ChatGPT can provide in education.
ChatGPT can help students understand code snippets or programming concepts
that they may have difficulty with. For instance, if a student is having
difficulty understanding a specific code syntax or function, they can use
ChatGPT to get an explanation of that code snippet or concept. ChatGPT can
provide step-by-step instructions, examples, and explanations to help students
understand programming concepts. Bug-fixing is another opportunity that
ChatGPT can provide in education. ChatGPT can help students debug code or find
errors in their programming assignments. For instance, if a student has an
issue with their code and is unable to identify the source of the problem,
they can use ChatGPT to get help in debugging their code. ChatGPT can analyze
the code, identify the problem, and suggest a solution to the student.

Figure 3: Opportunities of ChatGPT in assisting IT education

###  5.4 Threats

To realize the potential of chatbots, there are potential threats to consider.
Figure 6 describes common threats and threats for programming courses and
theory courses. Student may lose their ability to develop critical thinking.
By relying too heavily on ChatGPT to generate ideas or provide answers,
students may lose their ability to think independently and critically evaluate
information. This can happen when students use the chatbot to access vast
amounts of data without the required accountability, which can undermine
academic integrity and personal growth. One common threat is the potential for
cheating with the tool, as students may use ChatGPT to access a vast amount of
data and information without the required accountability. Additionally,
potential technical issues, such as server downtime, connectivity problems, or
software bugs, could disrupt the learning process. Another potential threat is
insufficient training with the tool, which may lead to misuse, such as
students relying on code generation without understanding it.

Figure 4: Threats of ChatGPT in assisting IT education

In programming courses, this threat is especially significant, as students may
misuse ChatGPT to generate code without understanding it. For example, in a
computer science course, students may use ChatGPT to generate code for a
programming assignment without understanding the code, resulting in a lack of
mastery of programming concepts. Similarly, in a literature review task,
students may use ChatGPT to generate summaries without engaging with the text,
resulting in a superficial understanding of the material. Moreover, there is a
risk of oversimplification of complex concepts, as ChatGPT may not always be
able to provide in-depth explanations of complex programming concepts,
resulting in a shallow understanding of the material. Misinterpretation of
code is also a potential threat.

In theory courses, ChatGPT can help prepare students for written exams by
creating summaries and mind maps. Hence, it uncritically would lead to a
shallower understanding and students not reaching the learning outcomes
related to reflection, understanding and discussion. ChatGPT can help prepare
for the written exam by creating summaries, mind maps and similar, but unless
the student engages with the material, actual learning may suffer.

##  6 Discussions and Conclusion

This section presents our implications to learners and educators, the possible
changes to higher education and our future work.

###  6.1 Implications to learners and educators

With time and the advancement of technology, it is assumed that the usage of
ChatGPT will be increased on both leaners’ and educators’ sides. This will
incur some changes in the education settings if it has not already disrupted
the course [21, 5, 9]. First, the teacher needs to be aware of the impacts,
effects, and usage of ChatGPT in his/her course. S/he needs to assess the
scopes and possibilities, and strengths, weaknesses, opportunities, and
threats of using ChatGPT in different teaching, learning, and assessing
activities for his/her course. For adopting successfully, the strengths and
opportunities of using ChatGPT, learning activities and assessment methods
might need to be revised and changed. Besides, threats are particularly
important to address as they may jeopardize the overall learning outcome and
the credibility of assessment in the course. Assessment methods and process
needs to be revised and changed as well as exam question patterns and content
and the focus of the test might need reassessed and changed in order to ensure
that they hold values and integrity. Courses may also need to include
instructions for students about how to use ChatGPT, where to use and where not
to use, including the correct way of referencing ChatGPT work to avoid
cheating and dishonesty.

###  6.2 Possible disruptive change in higher education

ChatGPT can improve education, as presented in the benefits in the result
section [21, 5, 9]. ChatGTP is used almost every day by students and several
teachers are incorporating it into their teaching. ChatGPT has appeared with
many benefits and challenges, just as search engines like Google did back in
1998. The launch of ChatGPT in November 2022 did not allow teachers to prepare
for this technology. This prompt appearance of ChatGPT and fast-growing
popularity among students has forced teachers to research, use and be up to
date with the potential impact that this technology might have on their
courses and teachings. As shown in nine IT courses at USN, several teachers
have already implemented ChatGPT in their teaching, while other teachers have
encouraged students to use it for learning and assignments. Teachers of these
courses have expressed that the use of ChatGPT is unavoidable. As such, it is
better to adapt to this new technology and incorporate it in different ways
during teaching. Nevertheless, ChatGPT has led teachers to wonder how they can
trust their students. Incorporating ChatGPT into a sustainable model has
become a goal for several teachers. ChatGTP is re-inventing the teaching
process, it’s transforming the way teachers deliver assignments and
assessments. How to avoid plagiarism and cheating in projects and home exams
has become an ongoing discussion among the teachers at USN. Since ChatGPT is
unavoidable, it is paramount to create a participatory culture where students
and teachers work together to examine the outcomes and implementation of this
disruptive technology. In addition, the university should establish guidelines
on the use of ChatGPT to ensure transparency, trustworthiness, and create a
safe teaching environment.

###  6.3 Future work

One potential avenue of exploration is to investigate the ethical implications
of using ChatGPT as a teaching tool. As previously discussed, ChatGPT has
several ethical issues, such as discrimination, no attribution, weak and
arrogant character, and consent and privacy concerns. Future research work
could examine how these ethical issues could impact the use of ChatGPT in IT
education and develop strategies to address them. Additionally, the research
could investigate the effectiveness of ChatGPT as a teaching tool in IT
education, exploring its ability to engage students, provide personalized
learning experiences, and enhance their understanding of complex topics.
Another area of future research work in adopting ChatGPT in IT education is
exploring its potential impact on the job market. With the increasing demand
for IT skills, there is a need for effective and innovative ways of teaching
and training the future workforce. ChatGPT could offer a new approach to IT
education that is more accessible, personalized, and engaging. Future research
could explore how the adoption of ChatGPT in IT education could impact the job
market, such as whether it could lead to a shift in the skills and knowledge
required by employers or whether it could lead to an increase in demand for
IT-related jobs. Additionally, the research could examine the potential
benefits and drawbacks of ChatGPT in IT education, such as its impact on the
quality of education, the role of teachers, and the cost-effectiveness of
implementing this technology.

## References

  * [1] Adamopoulou, E., Moussiades, L.: An overview of chatbot technology. In: Artificial Intelligence Applications and Innovations. vol. 584, pp. 373–383 (2020)
  * [2] Alam, A., Mohanty, A.: Foundation for the future of higher education or ‘misplaced optimism’? being human in the age of artificial intelligence. In: Panda, M., et al. (eds.) Innovations in Intelligent Computing and Communication, pp. 17–29. Springer International Publishing (2022)
  * [3] Baidoo-Anu, D., Owusu Ansah, L.: Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Available at SSRN 4337484 (2023)
  * [4] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., et al.: Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. pp. 1877–1901 (2020)
  * [5] Daun, M., Brings, J.: How ChatGPT will change software engineering education. In: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. pp. 110–116. ITiCSE 2023, Association for Computing Machinery. https://doi.org/10.1145/3587102.3588815, https://doi.org/10.1145/3587102.3588815
  * [6] Dwivedi, Y.K., Kshetri, N., Hughes, L., Slade, E.L., Jeyaraj, A., Kar, A.K., Baabdullah, A.M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M.A., Al-Busaidi, A.S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., et al.: “so what if chatgpt wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational ai for research, practice and policy. International Journal of Information Management 71, 102642 (2023)
  * [7] Hamdan, M.: The impact of chatgpt on students: Positive effects and potential negative effects. Modern Diplomacy (2023), https://moderndiplomacy.eu/2023/03/14/the-impact-of-chatgpt-on-students-positive-effects-and-potential-negative-effects/
  * [8] Haque, M.U., Dharmadasa, I., Sworna, Z.T., Rajapakse, R.N., Ahmad, H.: “i think this is the most disruptive technology”: Exploring sentiments of chatgpt early adopters using twitter data. arXiv:2212.05856 (2022), https://doi.org/10.48550/arXiv.2212.05856
  * [9] Jalil, S., Rafi, S., LaToza, T.D., Moran, K., Lam, W.: ChatGPT and software testing education: Promises & perils. In: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). pp. 4130–4137. https://doi.org/10.1109/ICSTW58534.2023.00078, ISSN: 2159-4848
  * [10] Jovanović, M., Campbell, M.: Generative artificial intelligence: Trends and prospects. Computer 55(10), 107–112 (2022)
  * [11] Karpathy, A., Johnson, J., Fei-Fei, L.: Visualizing and understanding recurrent networks. arXiv 1506.02078 (2015), https://doi.org/10.48550/arXiv.1506.02078
  * [12] Klein, A.M., Rauschenberger, M., Thomaschewski, J., Escalona, M.J.: Comparing voice assistant risks and potential with technology-based users: A study from germany and spain. Journal of Web Engineering pp. 1991–2016 (2021)
  * [13] Kung, T.H., Cheatham, M., et al.: Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digital Health 2(2), e0000198 (2023)
  * [14] Leigh, D.: SWOT analysis. In: Handbook of Improving Performance in the Workplace: Volumes 1-3, pp. 115–140. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470592663.ch24, https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470592663.ch24, section: 5 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470592663.ch24
  * [15] Lund, B., Ting, W.: Chatting about chatgpt: How may ai and gpt impact academia and libraries? (2023), https://doi.org/10.2139/ssrn.4333415
  * [16] Lund, B.D., Wang, T.: Chatting about chatgpt: How may ai and gpt impact academia and libraries? Library Hi Tech News (2023), https://doi.org/10.1108/LHTN-01-2023-0009
  * [17] MacNeil, S., Tran, A., et al.: Generating diverse code explanations using the gpt-3 large language model. In: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2. pp. 37–39 (2022)
  * [18] Nguyen-Duc, A., Cabrero-Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.K., Saari, M., Zhang, Z., Le, H., Quan, T., Abrahamsson, P.: Generative artificial intelligence for software engineering – a research agenda. https://doi.org/10.48550/arXiv.2310.18648, http://arxiv.org/abs/2310.18648
  * [19] Okonkwo, C.W., Ade-Ibijola, A.: Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence 2, 100033 (2021)
  * [20] Pavlik, J.V.: Collaborating with chatgpt: Considering the implications of generative artificial intelligence for journalism and media education. Journalism & Mass Communication Educator (2023)
  * [21] Qadir, J.: Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. In: 2023 IEEE Global Engineering Education Conference (EDUCON). pp. 1–9. https://doi.org/10.1109/EDUCON54358.2023.10125121, https://ieeexplore.ieee.org/document/10125121, ISSN: 2165-9567
  * [22] Radford, A., Narasimhan, K.: Improving language understanding by generative pre-training (2018), https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035
  * [23] Rudolph, J., Tan, S., Tan, S.: Chatgpt: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching 6(1) (2023)
  * [24] Ventayen, R.J.M.: Chatgpt by openai: Students’ viewpoint on cheating using artificial intelligence-based application (2023), https://doi.org/10.2139/ssrn.4361548
  * [25] Wartman, S.A., Combs, C.D.: Medical education must move from the information age to the age of artificial intelligence. Academic Medicine 93(8), 1107–1109 (2018)
  * [26] Wilson, V.: Research methods: Focus groups. Evidence Based Library and Information Practice 7(1), 129–131 (2012)