# GPR-based Model Reconstruction System for Underground Utilities Using GPRNet

Jinglun Feng1, Liang Yang1, Ejup Hoxha1, Stanislav Sotnikov1, Diar Sanakov1,
Jizhong Xiao*1 1 Electrical Engineering Department, The City College of New
York, New York, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>the corresponding author is
<EMAIL_ADDRESS>
###### Abstract

Ground Penetrating Radar (GPR) is one of the most important non-destructive
evaluation (NDE) instruments to detect and locate underground objects (i.e.
rebars, utility pipes). Many of the previous researches focus on GPR image-
based feature detection only, and none can process sparse GPR measurements to
successfully reconstruct a very fine and detailed 3D model of underground
objects for better visualization. To address this problem, this paper presents
a novel robotic system to collect GPR data, localize the underground
utilities, and reconstruct the underground objects’ dense point cloud model.
This system is composed of three modules: 1) visual-inertial-based GPR data
collection module which tags the GPR measurements with positioning information
provided by an omnidirectional robot; 2) a deep neural network (DNN) migration
module to interpret the raw GPR B-scan image into a cross-section of object
model; 3) a DNN-based 3D reconstruction module, i.e., GPRNet, to generate
underground utility model with the fine 3D point cloud. The experiments show
that our method can generate a dense and complete point cloud model of pipe-
shaped utilities based on a sparse input, i.e., GPR raw data, with various
levels of incompleteness and noise. The experiment results on synthetic data
as well as field test data verified the effectiveness of our method.

##  I Introduction

(a) The omnidirectional robot for visual-based GPR data collection, where a
GPR antenna is installed on the bottom of the robot chassis.

(b) Two different view angles of a concrete slab with two utility pipes buried
in.

(c) The left part shows the GPR data collection results with B-scan images
while the right part shows the GPR data interpretation results which indicate
the cross-section of utility pipes.

(d) The input indicates the coarse point cloud which represents the GPR
interpretation results, while the output is a coarse-to-fine format by
implementing our proposed GPR model reconstruction approach.

Figure 1: Our proposed GPR-based point cloud model reconstruction system,
where (a) demonstrates our robotics GPR data collection platform (b) shows the
layout of underground utilities. GPR data is firstly collected with a loosely
spacing distance. Then, by implementing a GPR interpretation method, we can
convert GPR raw data into a cross-section image which matches the utilities,
as depicts in (c). At last, by representing interpretation results into 3D
point cloud format, (d) indicates that our proposed method could complete the
3D coarse point cloud input into a dense representation.

Ground penetrating radar (GPR) has been widely used as a remote sensing
technique in geophysical and civil engineering applications to provide
effective radargram image and detection of underground structures. Among
various types of none-destructive evaluation (NDE) techniques, such as
acoustic, infrared, thermography [1], GPR gains its fame due to the high
resolution capability and applicability in different fields detection.
Specifically, GPR could not only evaluate the location and condition of
underground utilities such as rebars, pipes and power cables, but also reveal
the relative size of the subsurface objects.

Thanks to the advent of Deep Neural Networks applications in the GPR area,
learning-based architectures are capable of directly operating on GPR data.
DepthNet [2] introduces the first deep-learning-based network for dielectric
prediction of GPR data. Moreover, with the development in learning-based shape
completion task, such as L-GAN [3], PCN [4], FoldingNet [5] and PF-Net [6], it
allows us to reconstruct 3D point cloud representation for underground
objects.

However, in the practical GPR applications, the challenge of such a technique
mainly lies in the following factors: data collection, GPR image
interpretation, and GPR data visualization. Specifically, current GPR data
collection requires inspector moves GPR device along grid lines without any
rotations, which is hard to implement by regular ground mobile robots which
cannot move sideway. As for the interpretation of the complex GPR data, it
requires experienced professionals to interpret the GPR B-scan hyperbolic
features with trained eyes. In addition, the reconstruction model produced by
conventional migration algorithm contains too much noise and not easy to
understand.

To address these challenges, we propose a novel 3D model reconstruction system
based on GPR data collected by an omnidirectional robot to visualize the
underground utilities, which is illustrated in Fig.1. The key difference
between our method and existing ones are that only a small amount of GPR data
is required for 3D model reconstruction, in the meanwhile, it still achieves a
better visualization result modelled by 3D point clouds.

The main contributions of this work are:

  * •

an automated GPR data collection solution by using an omnidirectional robot,
which holds the GPR antenna on the bottom and move forward, backforward and
sideward without any rotations to collect GPR data.

  * •

a DNN model that operates and interprets the GPR B-scan image into a cross-
section of underground objects model.

  * •

a novel 3D reconstruction model that takes sparse GPR measurement as input and
complete it with a dense point cloud which is able to present the fine details
of the object structure.

##  II Related Works

Our work is motivated by a number of related works which aim to interpret GPR
radargram data and the coarse-to-dense 3D point clouds completion.

Mathematical Methods on GPR Data Interpretation The mathematical-based
technique which serves as the traditional GPR imaging tools, enabling to
convert the unfocused raw B-scan radargram data into a focused target, is
called _migration_. Migration methods can be roughly categorized into
Kirchhoff migration, the the phase-shift migration, the finite-difference
method and back-projection algorithm.

Kirchhoff migration method [7] is first introduced in 1970s, [8] tests
Kirchhoff migration method by using both synthetic and real GPR data, the
results show that though the localization of the buried targets could be
achieved, the information about target shape and extension are not provided by
implementing this method. In the meanwhile, [9] represents that Kirchhoff
migration method is capable to focus on the target position, however, it’s
processing speed is slower than the rest of migration approaches.

Similar to Kirchhoff migration, phase-shift migration is first proposed and
applied in 1978[10], this is also a mathematical-based method which utilizes
the ESM concept [11]. For comparable accuracy, this approach to migration is
computationally more efficient than finite‐difference [12] approach, which is
designed for the geometry of a single radargram source with a line of surface
receivers.

Unlike the above conventional migration methods, the back-projection algorithm
(BPA) back projects the emitted signal based on Electromagnetic waves travel
path and the associated travel time, analogous to the traditional back-
projection methods in computer aided tomography [13]. By implementing BPA,
[14] conceals object detection through Millimeter-wave 2D imaging methods and
works well both on metallic and dielectric types. BPA is also exploited for
imaging of water leaks from buried pipes [1].

Machine Learning Methods on GPR Data Interpretation In addition to the many
researches that focus on the mathematical-based GPR migration methods, machine
learning based methods are also widely studied to interpret GPR data, which
could be catergorized into Hough transform method, SVM method and deep neural
network (DNN) methods.

In 2000s, Hough transform method was first applied for underground targets
detection [15], however, this method is not effective on complex curves and
diverse signals[16]. Furthermore, SVM applications play a important role in
GPR studies on GPR imaging classification [17, 18] for underground materials.

Compared with conventional machine learning related methods, more and more
DNN-based implementations are applied in GPR data interpretation due to its
high accuracy and efficiency capability, especially in detection, localization
and classification of hyperbolic features in GPR radargram image [19, 20, 21,
2].

Deep Learning on 3D Point Cloud Completion Point cloud has been widely used in
3D vision related researches, which become one of the hottest topics in deep
learning field. Specifically, PointNet and its following works [22, 23, 24] is
the pioneer in this area and still be the state-of-the-art until now. It
solves the variance of permutation and robustness exists in point clouds, by
proposing a symmetric aggregation function and pointwise multilayer
perceptrons.

By taking advantage of the above works and [5], [4] proposes a point cloud
based completion network, which combines the merits of PointNet and
FoldingNet, to complete point cloud in a coarse-to-fine fashion. In addition,
[6] presents a multi-stage completion loss to make a detailed completion in
pointwise 3D shape. Moreover, a encoder-decoder architecture is designed in
[25] to achieve the shape completion task. These methods all achieve a good
result on both completion accuracy and noise robustness.

##  III Methodology

In this section, we discuss the proposed GPR-based 3D model reconstruction
system. Firstly, the vision-based robotic GPR data collection module provides
a 6-DOF pose information of the GPR data, and it allows robot moves forward,
backward and sideward without rotation during GPR data collection. Secondly,
we interpret the GPR B-scan images into the cross-section of underground
object model, and representing them as point cloud. Thirdly, a coarse 3D map
is generated by registering the cross-section point cloud into the 3D space,
where the 3D positioning information is obtained from the GPR data collection
module. At last, we propose learning-based method to complete the sparse point
cloud, thus provide dense model for better visualization of the underground
utilities.

###  III-A Visual-Inertial Robotic GPR Data Collection Module

To automate the GPR data collection, we developed an omnidirectional robot for
the inspection of underground utilities. Our robot uses triangle-shaped
chassis and three Mecanum wheels to move in any direction, it avoids rotation
motion because it may cause the failure of vision-based positioning. As
depicts in Fig.1(a), our robot motion meets the following relation:

$\begin{bmatrix}v_{w1\\_drv}\\\ v_{w2\\_drv}\\\ v_{w3\\_drv}\\\
\end{bmatrix}=\begin{bmatrix}1&0&-d\\\
\cos{\frac{2\pi}{3}}&\sin{\frac{2\pi}{3}}&-d\\\
\cos{\frac{-2\pi}{3}}&\sin{\frac{-2\pi}{3}}&-d\\\
\end{bmatrix}\begin{bmatrix}v_{x}\\\ v_{y}\\\ \omega\\\ \end{bmatrix}$ (1)

where $v_{w1\\_drv}$, $v_{w2\\_drv}$, $v_{w3\\_drv}$ represents the linear
velocity of each wheel, $d$ indicates the distance between the center of a
wheel and the center of the robot body. $v_{x}$, $v_{y}$ and $\omega$
represent the linear velocity and angular velocity of the robot body
respectively.

In order to provide pose information for GPR B-scan data, the robot also
carries an Intel NUC on-board computer and an Intel D435i RGB-D camera, which
has a 6-axis IMU embedded in, to provide accurate and robust pose estimation
for GPR sensor. We further fuse the IMU and the visual odometry data to
improve the positioning accuracy by using a loosely-coupled approach, which
was introduced in our previous works [2, 26].

###  III-B GPR Data Analysis

GPR antenna transmits a pulse of high-frequency electromagnetic (EM) wave in
the ground [27] and waits for the EM waves echos. The signal emitted from GPR
antenna is affected by EM characteristics, such as the dielectric of the
subsurface objects or materials.

Figure 2: The CAD model in the left indicates a slab with three utilities
buried in while the dot line on the top of the slab represents the motion
direction of GPR device for data collection, which generates a B-scan image
shows as in the right figure. The round shape features in the B-scan image
indicates the cross-section of the utilities in the 3D model.

As depicted in Fig.2, the dotted line on the top of the slab which has three
utilities embedded in, indicates the trajectory of a GPR device for data
collection. Therefore, when GPR antenna scans along the perpendicular
direction of the pipes, the composition of received signals can be identified
as the hyperbolic feature, as shown in the right side of the Fig.3, due to the
different dielectric between underground material and utilities. The image
generated by a set of the transmit pulses is called _B-scan_ , where the
around blue feature on the B-scan image indicates the cross-section of the
utilities.

###  III-C GPR Data Interpretation

In this section, we introduce a learning model, that is, MigrationNet, to
detect subsurface objects. Compared with the existing learning-based method of
GPR, our approach does not focus on the detection of hyperbola feature in the
raw B-scan image, which is not helpful enough to GPR data interpretation task.
MigrationNet aims at inferring the migration result by interpreting B-scan
measurement.

Figure 3: Framework of the MigrationNet. The input is the raw GPR B-scan
image. Then, the features are extracted through the multiple resolution
encoder and further concatenated into 1536 channels. The encoder consists of
several de-convolutional groups, the global feature is produced by
concatenating local features from MRE through skip-connection operation
indicated by $\oplus$ while the numbers 1-9 indicate the layers which are used
for concatenation, and finally decoded into a binary migration image. Figure
4: GPRNet framework. The input is a sparse point cloud which represents the
cross-section of utilities, the encoder abstracts the input as a global
feature and the decoder recovers the global feature to a dense point cloud.

It illustrated in Fig.3, the input of MigrationNet is raw GPR B-scan data and
it connects to our proposed Multiple Resolution Encoder (MRE).

The proposed encoder inherits the context capture ability by combining two
convolution layers and one max-pooling layer. We follow the down-sampling
group to encode the input data into a feature map $f$ with the same size,
where size = [$M$ x $N$ x $512$]. In details, to get the same size of output
feature, our input data is extracted into the different resolution by .
Specifically, at the top layer of , the input data follows a down-sampling
group where the kernel size of max-pooling layer is 8. In the middle layer,
the kernel size of the first max-pooling layer is 4 while the rest of pooling
layers’ kernel size are all equal to 2. At the bottom layer, all the kernel
size of max-pooling layers in the down-sampling groups are 2 and it allows the
final output feature map $f$ has the same size in each input. At last, all
three feature maps are then concatenated together as $F$, where size = [$M$ x
$N$ x $1536$]. This design brings the combined latent feature to contain more
detailed information of the input data.

The decoder takes global feature map $F$ as input to predict a [$M$ x $N$ x
$1$] binary image, with the white indicates the pipe and the black indicates
the background. In details, our decoder consists of 5 up-sampling group, and
each group contains two convolutional layers and one deconvolutional layer.
Besides, we also take the advantage of skip connections as illustrated in
Fig.3.

###  III-D GPRNet for Underground Utilities Reconstruction

Section.III-C facilitates to interpret raw B-scan into a cross-section image,
however, MigrationNet only provides the slices of utilities’ structure. One
could only get a sparse point cloud of the subsurface objects by concatenating
the interpreted slices into the 3D space.Thus, we need to further complete the
sparse point cloud map to recover the whole structure of utilities.

Inspired by the 3D point cloud completion works [5, 4, 25, 6] in computer
vision area, we proposed a model, i.e. GPRNet, to address the challenge
mentioned above. We firstly represent the interpreted image, which indicates
the cross-section of the underground utilities, as the 3D point cloud data
format. Then, our encoder-decoder based network would complete the sparse
input to generate a smooth and fine 3D point cloud reconstruction map.

####  III-D1 Encoder Design

Our proposed encoder is an extended version of PCN [4], it takes charge of
representing the geometric information in the input point cloud as the global
feature vector $\mathbf{v}\in\mathbb{R}^{n}$ where $n=896$. In addition, our
encoder inherits the feature extraction ability by implementing PointNet
layer, which is a combination of the convolutional multi-layer perceptron
(MLP) layer and the max-pooling layer. Compared with the previous methods [22,
23], our encoder could extract multiple resolution information of the input
data, which leads a better performance on small structure completion.

Specifically, by passing through three MLP layers which have the different
dimensions, our input $m\times 3$ point cloud data, where $m$ is the number of
the points and set to $1500$, is firstly encoded into three point feature
vectors $\mathbf{f}_{i}$, where size $\mathbf{f}_{i}:=$ $m\times 64$, $m\times
128$, $m\times 256$ for $i=1,2,3$ respectively. Then, a max-pooling layer is
performed on each extracted feature to obtain three intermediate features
$\mathbf{g}_{j=1,2,3}$ with multiple dimensions $[64-128-256]$. Furthermore,
we firstly concatenate each point feature vector $\mathbf{f}_{i}$ and each
intermediate feature $\mathbf{g}_{j}$ together to obtain a expanded feature,
which includes the feature information at different level. In addition, each
$\mathbf{g}_{j}$ is concatenated to each $\mathbf{f}_{i}$ to obtain the
feature matrix $\mathbf{F}$. At last, $\mathbf{F}$ is passed through another
PointNet layer to generate the global feature vector $\mathbf{v}$.

####  III-D2 Decoder Design

Fully-connected decoder [3] is good at predicting the global geometry of point
cloud, however, it ignores the local features. FoldingNet decoder [5] is good
at generating a smooth local feature. Therefore, by combining the above
decoders, we design our decoder as a hierarchical structure [6].

In the primary decoder layer, the global feature vector is extracted into
three feature vectors (size:=256,128,64 neurons respectively) by passing
through the fully-connected layers, which is responsible for generating point
cloud in different resolutions. Then, these three vectors are expanded into
three local feature matrices (size:= $256\times 3$,$128\times 3$,$64\times
3$). In the meanwhile, in order to get a better structure sense from global
feature, the global vector is reshaped into the same size of the local feature
matrices through the MLP layers, as it is shown in Fig.4. Then, the global
feature and local feature are concatenated together as a $896\times 3$ matrix.
At last, by taking advantages of folding operation, a patch of $9$ points is
generated at each point in the point set generated from last step. Thus, we
can obtain the detailed output consisting of $896*9$ points. A dense point
cloud output is then generated from our multi-resolution decoder, based on the
fully-connected and folding operations.

Thanks to this multi-resolution architecture, high-level features will affect
the expression of low-level features, and low resolution features can
contribute to form the global feature, which provide a sense of local geometry
of the shape. Our experiments show that the prediction of our proposed decoder
has fewer distortions and better detailed geometry of the shape.

####  III-D3 Loss Design

To constrain and compare the difference between the output point cloud S and
the ground truth point cloud $\textit{S}_{gt}$, an ideal loss must be
differentiable with respect to point locations and invariant to the
permutation of point cloud. In this paper, we use Chamfer Distance (CD), which
is proposed by Fan _et al._. It calculates the average closest point distance
between S and $\textit{S}_{gt}$, that meets the requirement of the above
conditions:

$d_{CD}(\textit{S},\textit{S}_{gt})=\frac{1}{\textit{S}}\sum_{x\in\textit{S}}\min\limits_{y\in\textit{S}_{gt}}\|x-y\|_{2}+\frac{1}{\textit{S}_{gt}}\sum_{y\in\textit{S}_{gt}}\min\limits_{x\in\textit{S}}\|y-x\|_{2}$
(2)

The Chamfer Distance finds the nearest neighbor in the ground truth point set,
thus it can force output point clouds to lie close to the ground truth and be
piecewise smooth.

##  IV Experimental Study

To evaluate our approach, we perform $725$ tests on NDT-GPR dataset [2] while
$120$ automated field tests using GPR-Cart. The field tests of robotic GPR
inspection system was conducted on a concrete slab at River Edge, NJ, USA. In
this section, the effectiveness and robustness of our 3D reconstruction method
as well as the results of on-site tests are discussed in details.

###  IV-A Model Training and Evaluation

Dataset In this paper, we use synthetic GPR dataset proposed in [2]. This
dataset builds a synthetic testing environment which simulates the real NDT
condition, where rebars, utilities and PVC pipes are buried underneath the
surface. This simulated environment mimics this property and involves pipe-
shaped objects with different location as well as the size. Noticed all of the
simulated objects have a round cross section. Specifically, this dataset
contains $1628$ B-scan data which could convert into images, as well as their
cross section images as the ground truth. After register each cross-section
image into the 3D point cloud, each point set contains $1500$ points while
each ground truth point set contains $8064$ points.

Network Training We train our model on a server with Intel Core i9-9900K
3.2GHz CPU, GeForce RTX 2080 Ti GPU, and 32GB RAM. Among all the $1628$ model
of GPR data, we reserve $100$ models for validation and $150$ models for
testing, the rest is used for training. Notice that our models are trained for
$100$ epochs with an Adam optimizer. The initial learning rate is set to
0.00005 while the batch size is $16$. The weight decay is $0.7$ for every
50000 iterations.

###  IV-B Model Reconstruction Study

Baseline Comparison In order to assess the effectiveness of our proposed
method, we compare our baseline against PCN and PF-Net. Since related point
cloud completion works are trained with different datasets, we use the GPR
B-scan dataset [2] which matches our testing condition so that we can evaluate
the above methods quantificationally. Specifically, we use two evaluation
metrics to compare the methods mentioned above, which are the average squared
distance [3] and the $L_{1}$ distance from each point between output point
cloud and the ground truth. It need to be noted that the number of points in
the predicted output is set to $8192$ while the input number of points are
$1500$.

Figure 5: The comparison of completion results between other methods and our network. From left to right: the slab CAD model, input data, PCN[4], PF-Net[6], our method and the ground truth. Based on the results, our method could reconstruct a better 3D model for visualization. TABLE I: Evaluation Performance Comparison with Different Baselines. $CD$: average squared distance between two points, $L_{1}$: norm distance between two points | GPRNet | PCN | PF-Net
---|---|---|---
CD | 6.214 | 6.965 | 7.005
_L_ 1 | 22.97 | 24.90 | 33.68

The results in Table. I shows that our proposed method outperforms other
methods in different GPR-based inspection environments, note that the Chamfer
distance is reported multiplied by $10^{3}$ while $L_{1}$ distance is scaled
by $100$. In addition, we could also visualize the output utilities point
cloud generated by all the methods. Compared to other methods, the prediction
of our method outperforms the other methods in spatial continuity and shape
accuracy level.

Figure 6: Noise robustness comparison result. Pictures (a)-(c) indicate the model, raw input and ground truth respectively. The second line represents the Gaussian-white-noised input, where noise variance are $0.05$, $0.1$, $0.2$, $0.1$ respectively, while pictures (h)-(k) in the last line demonstrate the predicted results of the noised input (d)-(g). TABLE II: Noise Robustness Evaluation on GPRNet with two metrics. | CD distance |  _L1_ distance
---|---|---
Variance & Noise density = 0.05 | 6.350 | 23.029
Variance & Noise density = 0.1 | 6.715 | 24.851
Variance & Noise density = 0.2 | 7.374 | 25.639
Variance & Noise density = 0.5 | 7.765 | 26.589

(a) The design details and ground truth of the concrete slab.

(b) 3D reconstruction result of the field test.

Figure 7: A field test on a concrete slab which has multiple utilities
embedded in, as demonstrated in (a). Our robot collected 10 groups of GPR data
along the straight lines perpendicular to the pipes. As shown in (b), our
input data is coarse and represents the cross-section of the utilities while
the predicted result indicates the 3D model of the embedded pipes.

Noise Robustness To evaluate the effectiveness of our method under different
sensor noise level, we perturbed the input sparse point cloud with multiple
Gaussian white noise levels, where the standard deviations are 0.01, 0.05, 0.1
and 0.2 respectively. For each data set, we further compare average squared
distance and $L_{1}$ distance between the noised data input and ground truth.
As illustrated in Table.II and Fig.6, we could find our proposed method
achieved higher robustness against the noise.

###  IV-C Dataset Test and Field Test

In this section, we evaluate the robotic GPR inspection system with both
synthetic dataset and field test. The performance of underground utilities
reconstruction is illustrated in Fig.5, where the model of the synthetic
slabs, coarse input point cloud data, predicted output data and the ground
truth are provided. The field test result is also demonstrated in Fig.7, where
pipes with different size are colored respectively. In this test, we drive our
robot with a sparse line spacing, that is, 11 tests in total along the
horizontal direction, all the utilities are not parallel from the data
collection direction are detected excepted for one pipe which is parallel with
data collection direction. By comparing the reconstructed 3D map with the
ground truth of the slab in the test pit, the effectiveness of proposed system
is well verified.

##  V Conclusion

This paper presents a novel GPR-based 3D model reconstruction system for
underground utilities by using an omnidirectional robot. Our omnidirectional
robot allows GPR device move forward, backward and sideward without rotations.
The proposed system is able to reconstruct underground utilities model
represented as 3D point cloud. According to the experiments, our approach
could obtain a fine 3D model for a better visualization while we could find
out our method has a higher robustness against noise.

## References

  * [1] S. Demirci, E. Yigit, I. H. Eskidemir, and C. Ozdemir, “Ground penetrating radar imaging of water leaks from buried pipes based on back-projection method,” _Ndt & E International_, vol. 47, pp. 35–42, 2012.
  * [2] J. Feng, L. Yang, H. Wang, Y. Song, and J. Xiao, “Gpr-based subsurface object detection and reconstruction using random motion and depthnet,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 7035–7041.
  * [3] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3d point clouds,” in _International conference on machine learning_. PMLR, 2018, pp. 40–49.
  * [4] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, “Pcn: Point completion network,” in _2018 International Conference on 3D Vision (3DV)_. IEEE, 2018, pp. 728–737.
  * [5] Y. Yang, C. Feng, Y. Shen, and D. Tian, “Foldingnet: Point cloud auto-encoder via deep grid deformation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 206–215.
  * [6] Z. Huang, Y. Yu, J. Xu, F. Ni, and X. Le, “Pf-net: Point fractal network for 3d point cloud completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7662–7670.
  * [7] W. A. Schneider, “Integral formulation for migration in two and three dimensions,” _Geophysics_ , vol. 43, no. 1, pp. 49–76, 1978.
  * [8] X. Liu, M. Serhir, A. Kameni, M. Lambert, and L. Pichon, “Ground penetrating radar data imaging via kirchhoff migration method,” in _2017 International Applied Computational Electromagnetics Society Symposium-Italy (ACES)_. IEEE, 2017, pp. 1–2.
  * [9] N. Smitha, D. U. Bharadwaj, S. Abilash, S. Sridhara, and V. Singh, “Kirchhoff and fk migration to focus ground penetrating radar images,” _International Journal of Geo-Engineering_ , vol. 7, no. 1, p. 4, 2016.
  * [10] J. Gazdag, “Wave equation migration with the phase-shift method,” _Geophysics_ , vol. 43, no. 7, pp. 1342–1351, 1978.
  * [11] I. Lecomte, S.-E. Hamran, and L.-J. Gelius, “Improving kirchhoff migration with repeated local plane-wave imaging? a sar-inspired signal-processing approach in prestack depth imaging,” _Geophysical Prospecting_ , vol. 53, no. 6, pp. 767–785, 2005.
  * [12] J. F. Claerbout and S. M. Doherty, “Downward continuation of moveout-corrected seismograms,” _Geophysics_ , vol. 37, no. 5, pp. 741–768, 1972.
  * [13] M. Pereira, Y. Zhang, D. Orfeo, D. Burns, D. Huston, and T. Xia, “3d tomography for multistatic gpr subsurface sensing,” in _Radar Sensor Technology XXII_ , vol. 10633. International Society for Optics and Photonics, 2018, p. 1063302.
  * [14] S. Demirci, H. Cetinkaya, E. Yigit, C. Ozdemir, and A. A. Vertiy, “A study on millimeter-wave imaging of concealed objects: Application using back-projection algorithm,” _Progress In Electromagnetics Research_ , vol. 128, pp. 457–477, 2012.
  * [15] W. Al-Nuaimy, Y. Huang, M. Nakhkash, M. Fang, V. Nguyen, and A. Eriksen, “Automatic detection of buried utilities and solid objects with gpr using neural networks and pattern recognition,” _Journal of applied Geophysics_ , vol. 43, no. 2-4, pp. 157–165, 2000.
  * [16] J. Illingworth and J. Kittler, “A survey of the hough transform,” _Computer vision, graphics, and image processing_ , vol. 44, no. 1, pp. 87–116, 1988.
  * [17] M. S. El-Mahallawy and M. Hashim, “Material classification of underground utilities from gpr images using dct-based svm approach,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 10, no. 6, pp. 1542–1546, 2013.
  * [18] U. Ozkaya, F. Melgani, M. B. Bejiga, L. Seyfi, and M. Donelli, “Gpr b scan image analysis with deep learning methods,” _Measurement_ , p. 107770, 2020\.
  * [19] H. Liu, C. Lin, J. Cui, L. Fan, X. Xie, and B. F. Spencer, “Detection and localization of rebar in concrete by deep learning using ground penetrating radar,” _Automation in Construction_ , vol. 118, p. 103279, 2020.
  * [20] J. Gao, D. Yuan, Z. Tong, J. Yang, and D. Yu, “Autonomous pavement distress detection using ground penetrating radar and region-based deep learning,” _Measurement_ , p. 108077, 2020.
  * [21] S. Khudoyarov, N. Kim, and J.-J. Lee, “Three-dimensional convolutional neural network–based underground object classification using three-dimensional ground penetrating radar data,” _Structural Health Monitoring_ , p. 1475921720902700, 2020.
  * [22] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [23] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in _Advances in neural information processing systems_ , 2017, pp. 5099–5108.
  * [24] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d object detection from rgb-d data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 918–927.
  * [25] L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, “Topnet: Structural point cloud decoder,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 383–392.
  * [26] L. Yang, G. Yang, Z. Liu, Y. Chang, B. Jiang, Y. Awad, and J. Xiao, “Wall-climbing robot for visual and gpr inspection,” in _2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA)_. IEEE, 2018, pp. 1004–1009.
  * [27] M.-S. Kang, N. Kim, J. J. Lee, and Y.-K. An, “Deep learning-based automated underground cavity detection using three-dimensional ground penetrating radar,” _Structural Health Monitoring_ , vol. 19, no. 1, pp. 173–185, 2020\.


# Active Learning-Based Optimization of Scientific Experimental Design

Ruoyu Wang
School of Computer Science & Engineering
South China University of Technology
Guangzhou, Guangdong, 510006, China
<EMAIL_ADDRESS>
###### Abstract

Active learning (AL) is a machine learning algorithm that can achieve greater
accuracy with fewer labeled training instances, for having the ability to ask
oracles to label the most valuable unlabeled data chosen iteratively and
heuristically by query strategies. Scientific experiments nowadays, though
becoming increasingly automated, are still suffering from human involvement in
the designing process and the exhaustive search in the experimental space.
This article performs a retrospective study on a drug response dataset using
the proposed AL scheme comprised of the matrix factorization method of
alternating least square (ALS) and deep neural networks (DNN). This article
also proposes an AL query strategy based on expected loss minimization. As a
result, the retrospective study demonstrates that scientific experimental
design, instead of being manually set, can be optimized by AL, and the
proposed query strategy ELM sampling shows better experimental performance
than other ones such as random sampling and uncertainty sampling.

Keywords— Machine Learning, Active Learning, Scientific Experiment Design,
Alternating Least Square, Deep Neural Network

##  1 Introduction

Scientific experiments have become progressively reliant on automated machines
overtime to cope with large datasets and high concurrency. For instance,
Hafner et al.[1] propose a mostly automated pipeline for drug response
experiments implementation and results quantification. However, the remained
human involvement in the process of experimental design has a high tendency to
generate errors that are difficult to tackle. The most conventional
replacement for manual experimental design is an exhaustive one. But without
heuristic sampling methods, the exhaustive exploration in the experiment space
tends to lead to excessive consumption of experimental resources and manpower.

Some non-exhaustive methods were proposed to achieve experimental design
automation, such as random sampling and orthogonal experimental design[2].
However, since the two methods are based on stochastic distribution and
predetermined orthogonal table respectively, both of them fail to take the
properties of the experimental subjects into consideration and therefore their
performance varies in real-life experimental practices.

Accordingly, an active method is needed to implement automated experimental
design adaptively based on the properties of the experimental subjects. And
active learning (AL) is a machine learning algorithm that can achieve greater
accuracy with fewer labeled training instances, for having the ability to ask
oracles to label the most valuable unlabeled data chosen iteratively by query
strategies[3].

In the regard of above, this article performs a retrospective study on a
dataset produced by the mentioned pipeline and discusses the feasibility of
adaptive design of scientific experiments using AL strategy. Insights and
inspiration can be found in this article for scientists to harness the
strength of AL to avoid human involvement in experimental design,
heuristically achieve experimental design optimization and train satisfying
predictive models with relatively fewer physical experiments.

##  2 Dataset

###  2.1 Dataset Description

The dataset used in this article is Breast Cancer Profiling Project, Drug
Sensitivity 1[4]. This dataset, produced from the mentioned drug response
experiment pipeline[1] of Hafner’s as shown in Figure 2.1, contains the
exhausted experimental results of measuring the sensitivities of 35 breast
cancer cell lines to 34 small molecule perturbagens. Every combination of cell
and molecule is treated with 9 doses of different concentrations of small
molecule perturbagens respectively. Therefore, the dataset contains 10710
instances in total.

Figure 1: Hafner’s Pipeline for experimental design and analysis

The independent and dependent variables of the dataset are described as
follows. The independents variables are Cell HMS LINCS ID, Cell Name, Small
Molecule HMS LINCS ID, Small Molecule Name, Small Mol Concentration (uM),
Primary Target and Pathway. The dependent variables are two measurements of
sensitivity that are Mean Normalized Growth Rate Inhibition Value (GR) and
Increased Fraction Dead (IFD).

$GR=2^{\frac{\log_{2}\left(\frac{x(c)}{x_{0}}\right)}{\log_{2}\left(\frac{x_{ctrl}}{x_{0}}\right)}}\cite[cite]{[\@@bibref{}{hafner2016growth}{}{}]}$

Where $x(c)$, $x_{0}$ and $x_{ctrl}$ is the mean of the measured live cell
counts after a given treatment, from the day 0 untreated plate grown in
parallel until the time of treatment, and of the control wells for all
technical replicates respectively.

$IFD=fd(c)-fd_{ctrl}$

Where $fd(c)$ is the mean fraction of dead cells in the wells from a given
treatment and $fd_{ctrl}$ is the mean fraction of dead cells in the control
wells across all technical replicates.

###  2.2 Properties of Variables

High interdependencies exist among the independent variables. For example,
Cell HMS LINCS ID and Cell Name have a one-to-one mapping relationship, and
Primary Target and Pathway are both determined by each certain combination of
cell and molecule. Thus, the independent variables of the dataset can be
restated as Cell Type, Molecule Type and Molecule Concentration (uM).

Notably, the explicit physical or chemical properties of cells and molecules
are not given in the dataset.

The independent variables measuring sensitivity are continuous. Thus, a
regression model with regression metrics, such as rooted mean square error, is
needed to perform prediction work.

Scientists are also interested in the classification accuracy that whether
certain molecule accelerates or restrains the growth of certain cell.
Obviously, a GR value greater than 1 or a negative IFD value indicates a
positive effect of certain molecule on the growth of certain cell. On the
contrary, a GR value less than 1 or a positive IFD value indicates a negative
one. In the following experiments, the GR values are typically reduced by 1 so
that both the independent variables have a binary classification boundary of
0.

###  2.3 Dataset Redefinition

As previously discussed, every combination of cell and molecule are treated
with 9 different concentrations of molecule respectively. However, the 9
concentrations are not entirely the same as each other. For the convenience of
doing retrospective AL experiments (i.e. to make sure that the sensitivity
result is known for every possible combination of cell, molecule and
concentration of molecule), the problem space will be restricted to a subset
of all concentration values in which every concentration value has
correspondent results of all combinations of cell and molecule. As it turns
out, this subset of concentration values has a size of 4.

Therefore, the problem space will be redefined to one containing 4760 samples.
Although the size of the dataset is halved, it is still worth studying since
one single experiment takes three days to complete in the physical world in a
serialized manner[4], and a better experimental design can avoid excessive
consumption of resources and can also produce more accurate predictive models.

In the following experiments, all the proposed methods will be performed on
the 4 subdatasets of different concentration values separately. Particularly,
the training curves of the subdataset with a molecule concentration of 0.01uM
are shown to illustrate the traits of the algorithms. And the arithmetic mean
of all the 4 subdatasets’ results is calculated when referring to average.

Figure 2: Heatmap visualization of the redefined dataset

##  3 Predictive Models

The difficulty of building predictive models for the dataset lies in that the
independent variables do not represent the explicit features of cells and
molecules, therefore cannot be used to train predictive models directly. In
this section, a matrix factorization method called Alternating Least Square
(ALS) will be used to produce the latent features of cells and molecules, then
an ALS improvement method with the combination of Deep Neural Network (DNN)
will be proposed.

###  3.1 Metrics

The metrics for the predictive models are loss and accuracy. Rooted mean
square error (RMSE) is used to measure the loss. And accuracy is calculated as
the percentage of predicted sensitivity values that are on the same side of
decision boundary (0 in this article) as the ground truth sensitivity values.

$RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(\hat{y}_{i}-y_{i}\right)^{2}}$

$n$ is the number of samples, $\hat{y}_{i}$ is the predicted value and $y_{i}$
is the ground truth for sample $i$.

###  3.2 Alternating Least Square

####  3.2.1 ALS Prerequisites

ALS[6] is a collaborative filtering algorithm in the field of recommender
systems using matrix factorization. Specifically, given a sparse user-item
real matrix recording user ratings on the items, the aim is to decompose the
user-item matrix into a dot product of a user embedding matrix and an item
embedding matrix such that the squared error between the dot product and the
user-item matrix is minimized. The loss function is shown below where $i$,
$j$, $d$, $x$, $w$, $y$ and $r$ is the $i^{th}$ of $m$ users, $j^{th}$ of $n$
items, number of dimensions of embedding vector, $m\times{}d$ user embedding
matrix, $d\times{}n$ item embedding matrix, $m\times{}n$ user-item matrix and
$m\times{}n$ 0-1 matrix recording rated positions respectively.

$loss=\frac{1}{2}\sum_{(i,j)\in
r(i,j)=1}\left(\sum_{l=1}^{d}x_{il}w_{lj}-y_{ij}\right)^{2}$

One can then deduce the partial derivatives of loss with respect to $x$ and
$w$.

$\frac{\partial loss}{\partial x_{ik}}=\sum_{j\in
r(i,j)=1}\left(\sum_{l=1}^{d}x_{il}w_{lj}-y_{ij}\right)w_{kj}$

$\frac{\partial loss}{\partial w_{kj}}=\sum_{i\in
r(i,j)=1}\left(\sum_{l=1}^{d}x_{il}w_{lj}-y_{ij}\right)x_{ik}$

Finally, the gradient decent method can be used to train the model with
learning rate $\alpha$.

$x=x-\alpha\Delta x=x-\alpha r\cdot\left(xw-y\right)w^{T}$

$w=w-\alpha\Delta w=w-\alpha x^{T}\left[\left(xw-y\right)\cdot r\right]$

This method is called alternating because the user matrix and the item matrix
are trained alternately while the other one is fixed.

####  3.2.2 ALS Experiment Settings

The number of dimensions of the embedding vector $d$ is intuitively set to 5.
The learning rate $\alpha$ is 0.01. An ALS model is trained for every cell-
molecule matrix and a 10-fold cross-validation is performed with 400 epochs of
training for every ALS model.

####  3.2.3 ALS Results Analysis

Figure 3: ALS training curve of GR with molecule concentration of 0.01uM

Figure 4: ALS training curve of IFD with molecule concentration of 0.01uM

The ALS training curves of GR and IFD with molecule concentration of 0.01 uM
are shown in Figure 3.2.3 and Figure 3.2.3 and the average test loss and
average test accuracy are shown in Table 3.3.3. From Figure 3.2.3 and Figure
3.2.3, it is revealed that:

The training loss, RMSE, converges at a relatively satisfying value of around
0.15 for GR and around 0.05 for IFD after 100 to 200 epochs of training.

The phenomenon of overfitting is quite obvious especially as shown in Figure
3.2.3, where the test loss only converges at around 0.2 and lacks behind the
training loss by roughly 0.1.

The model reduces loss at the expense of accuracy. After 100 to 200 epochs of
training, the test loss drops at an extremely slow rate while the test
accuracy is stable or even reduced. This reduction of test accuracy can
significantly impair the practical performance of the predictive model and
thus should be addressed carefully.

###  3.3 Alternating Least Square with Deep Learning (ALSDL)

####  3.3.1 ALS Improvement with DL

An ALS improvement using Deep Learning (DL) method is proposed to solve the
problems above. After several training epochs of the ALS algorithm, the latent
features of cell and molecule under certain concentration can be achieved by
user (cell) and item (molecule) matrixes. Then the aligned features of cell
and molecule are fed to a Fully Connected Neural Network (FCNN) to predict the
sensitivity value.

A new loss function is proposed with RMSE reducing a penalty term for
incorrect classification, where $\beta$ is a weight parameter and n is the
size of the training set, so that the model has the tendency to correctly
classify the sensitivity value. Specifically when the RMSE is fixed, the loss
increases as more sensitivity values are incorrectly classified thus producing
a negative product of prediction and ground truth, vice versa.

$loss=RMSE-\beta\frac{1}{n}\sum_{i=1}^{n}sign\left(\hat{y}_{i},y_{i}\right)$

$sign(x)=\begin{cases}1,&x>0\\\ 0,&x=0\\\ -1,&x<0\end{cases}$

A more general form of the proposed loss function is shown below, where the
predictive model is tackling a k-class classification problem with
classification boundaries $c_{1},c_{2},......,c_{k}$. As a brief explanation,
only a prediction falls in the same interval between two adjacent
classification boundaries as the ground truth will produce a positive penalty
term, otherwise, negative.

$loss=RMSE-\beta\frac{1}{n}\sum_{i=1}^{n}sign\left(\sum_{j=1}^{k}sign\left(\left(\hat{y}_{i}-c_{j}\right)\left(y_{i}-c_{j}\right)-k+1\right)\right)$

####  3.3.2 ALSDL Experiment Settings

The parameters of ALS in ALSDL are the same as in the former experiment. The
FCNN has three hidden layers containing 20, 10, 5 neural cells respectively
and an output layer of 1 neural cell. The activation functions between hidden
layers are all tanh and the optimizer for back propagation is rmsprop. The
weight parameter $\beta$ of the loss function is 0.1. An ALSDL model is
trained for every cell-molecule matrix and a 10-fold cross-validation is
performed with 200 epochs of ALS training and another 200 epochs for FCNN
training for every ALSDL model.

####  3.3.3 ALSDL Result Analysis

Figure 5: ALSDL training curve of GR with molecule concentration of 0.01uM

Figure 6: ALSDL training curve of IFD with molecule concentration of 0.01uM

The ALSDL training curves of GR and IFD with molecule concentration of 0.01 uM
are shown in Figure 3.3.3 and Figure 3.3.3 and the results are promising in
three aspects comparing with ALS. The loss, RMSE, continues to converge at a
lower value of around 0.1 for GR and around 0.02 for IFD with a significant
improvement compared with ALS convergence value at 200 epochs of training.

The phenomenon of overfitting is less intense as the curves of training loss
and test loss almost overlap during the 200 training epochs of DL.

With the restriction of the penalty term for incorrect classification, the
accuracy does not decrease as loss reduces at the DL training stage. Instead,
for both GR and IFD, the test accuracy is not significantly worse if not
better than the training accuracy.

| Average Test Loss | Average Test Accuracy
---|---|---
ALS | ALSDL | ALS | ALSDL
GR | 0.250860068 | 0.160126785 | 0.854432773 | 0.872478992
IFD | 0.0097235223 | 0.054959413 | 0.748781513 | 0.818277311

Table 1: ALS and ALSDL average test loss and average test accuracy

Table 3.3.3 shows that, for both GR and IFD, the average test loss is reduced
and the average test accuracy is increased by ALSDL compared with ALS.

##  4 Active Learning Implementation

With the predictive model ALSDL proposed and its effectiveness examined, the
AL algorithm is to be implemented on the model to form the AL scheme as a
whole. However, incompatibility occurs when common classification-based query
strategies, such as the uncertainty sampling, are performed on the regression
problem. In this section, a query strategy based on expected loss minimization
and suitable for regression problem will be proposed followed by comparative
experiment and result analysis.

###  4.1 Active Learning Prerequisites

Active learning (AL) is a machine learning algorithm that can train models to
achieve greater accuracy or less error with relatively fewer labeled training
instances. Figure 4.1 illustrates the procedure of pool-based AL where all the
unlabeled data are maintained as a pool and are queried by query strategies
till the pool is empty. Carefully designed query strategies will actively
choose the most valuable unlabeled data for oracles to label, thus a more
representative training set and a better-trained model can be obtained. The
most common query strategies include uncertainty sampling, least confident and
margin sampling[7].

Figure 7: Pool-Based Active Learning

###  4.2 Query Strategy based on Expected Loss Minimization

To implement an AL algorithm on the predictive model, a query strategy based
on the minimization of the expected loss (ELM) of the model is proposed.

The core idea of ELM query strategy is to choose the unlabeled sample
$\left(x^{+},y^{+}\right)$ from the unlabeled dataset $U$ such that when
$\left(x^{+},y^{+}\right)$ is removed from $U$ and added to the current model
$M$’s training set $D$ to train a new model $M^{+}$, the new model produces
the least test loss. The test loss is measured with the combination of $D$ and
$U$, i.e. the whole dataset.

$x_{ELM}=\mathop{\arg\min}\limits_{x^{+}\in U}loss\left(M^{+},D\cup U\right)$

However, since the ground truth of $y^{+}$ and $U$’s label values are unknown,
the expected new model $\hat{M}^{+}$ is trained with the current model $M$’s
prediction of $\left(x^{+},\hat{y}^{+}\right)$ and training set $D$, then the
expected test loss of $\hat{M}^{+}$ is evaluated with $M$’s predictions of
unlabeled data $\hat{U}$ in substitution of $U$.

$x_{ELM}=\mathop{\arg\min}\limits_{x^{+}\in
U}loss\left(\hat{M}^{+},D\cup\hat{U}\right)$

Multiple samples can also be obtained in one query by sorting the unlabeled
samples in ascending order according to their expected loss then selecting the
top ones.

###  4.3 Proposed Active Learning Scheme and Experiment Settings

The procedures of the proposed AL scheme are as follows.

Initialization: Randomly choose $n_{init}$ samples to be labeled as the
training set $D$. $n_{query}=0$.

Training: Train a new ALSDL predictive model on $D$.

Query: If $n_{query}>n_{max_{q}uery}$, terminate the scheme. Otherwise, select
$n_{per_{q}uery}$ samples to be added to $D$ from the unlabeled dataset using
ELM query strategy, $n_{query}=n_{query}+1$, then return to procedure
Training.

Both $n_{init}$ and $n_{per_{q}uery}$ are set to 40 and $n_{max_{q}uery}$ is
8. The ALSDL model has the same settings as in the former experiment. The RMSE
loss and accuracy of the ALSDL model evaluated on the full dataset are
recorded after every training procedure. The RMSE loss is used for new model
evaluation in the ELM query strategy and is evaluated after 200 epochs of ALS
training to speed up the query procedure.

In comparison, three other sampling methods (query strategies) are implemented
to substitute ELM query strategy in the scheme. Orderly sampling resembles a
manual experimental design that queries user-item matrix in a row or column
order. Random sampling queries the unlabeled pool stochastically. Uncertainty
sampling chooses the unseen samples with sensitivity values closest to the
classification boundary.

###  4.4 Active Learning Result Analysis

The proposed AL scheme training curves of GR and IFD with molecule
concentration of 0.01uM are shown respectively as Figure 4.4 and Figure 4.4.
After 8 times of queries, with a total of 360 examples learned, the proposed
ELM sampling method reaches the highest accuracy and the least loss for both
GR and IFD. Compared with other query strategies, the proposed expected loss
minimization query strategy constructs better predictive models more
efficiently for the following aspects.

Figure 8: Active learning training curve of GR with molecule concentration of
0.01uM

Figure 9: Active learning training curve of IFD with molecule concentration of
0.01uM

The rate of convergence of ELM sampling is more rapid. With little percent of
total experiment space covered, the performance of ELM sampling falls behind
other sampling methods in the beginning. Then ELM soon catches up and
outperforms other sampling methods as the expected test loss becoming more and
more accurate. ELM sampling, whose loss function always encourages the
training of a more accurate model, is more heuristically designed than orderly
and random sampling. It also prevails over uncertainty sampling because the
latter depends on the classification boundary that is constantly shifting with
the result of the ALS model’s training. With the same amount of examples
queried, ELM sampling results in a predictive model of less loss and higher
accuracy because the queried samples are among the most valuable ones.
Therefore, the optimized design of the experiment of cells’ sensitivity to
molecules can be achieved by the proposed AL scheme with the query strategy of
expected loss minimization.

##  5 Conclusion

This article proposed an AL scheme that combines ALS, DNN and a query strategy
variant for conducting scientific experiments design without prior knowledge
of the properties of experimental subjects. After performing a retrospective
study on a dataset of scientific experiment results, it was demonstrated that
the optimization of experimental design that builds better predictive models
more efficiently can be automatically reached by the AL scheme and the
proposed ELM query strategy outperforms several others.

Due to the constraint of computational power, all the hyperparameters in this
article are not heuristically tuned but manually set based on the optima of
several trials. Similarly, the FCNN setting used in this article is not
necessarily optimal and it requires further investigation. However, the
performance should not vary significantly as the hyperparameters shift since
the whole scheme is built on mature and robust algorithms.

The work in this article can be an inspiration for optimizing the design of
automated scientific experiments, especially whose consumption of physical
experimental resources is tremendous and the knowledge of experimental
subjects’ properties is limited, such as drug response experiments in the
field of biomedicine. In the future, more datasets would be tested and more
experiments would be conducted on the proposed AL scheme to further improve
its generality.

## Acknowledgement

This article would not have been finished without the valuable reference
materials and lectures that I received from the professor, Dr. Robert F.
Murphy of Carnegie Mellon University, whose insightful guidance and
enthusiastic encouragement gave me much help in the course when I was shaping
this article hence I want to show my deepest gratitude to him.

I would also avail myself of this opportunity to extend my sincere thanks to
the teachers from whose teaching and instruction I obtained during my
undergraduate education in the School of Computer Science & Engineering, South
China University of Technology.

Last but not least, I am much indebted to my family, without whose
unconditional affection and consistent support this article could not have
appeared in its final form.

## References

  * [1] Marc Hafner, Mario Niepel, Kartik Subramanian, and Peter K Sorger.  Designing drug-response experiments and quantifying their results.  Current protocols in chemical biology, 9(2):96–116, 2017.
  * [2] Liu Ruijiang, Zhang Yewang, Wen Chongwei, and Tang Jian.  Study on the design and analysis methods of orthogonal experiment.  Experimental Technology and Management, 9:52–55, 2010.
  * [3] Burr Settles.  Active learning literature survey.  2009\.
  * [4] Peter K. Sorger.  Breast cancer profiling project, drug sensitivity 1: Fixed-cell gr measures of 35 breast cell lines to 34 small molecule perturbagens. dataset 1 of 2: Normalized growth rate inhibition values., 2018.
  * [5] Marc Hafner, Mario Niepel, Mirra Chung, and Peter K Sorger.  Growth rate inhibition metrics correct for confounders in measuring sensitivity to cancer drugs.  Nature methods, 13(6):521–527, 2016.
  * [6] Yifan Hu, Yehuda Koren, and Chris Volinsky.  Collaborative filtering for implicit feedback datasets.  In 2008 Eighth IEEE International Conference on Data Mining, pages 263–272. Ieee, 2008.
  * [7] Burr Settles and Ronald J Brachman.  Active learning. morgan claypool.  Synthesis Lectures on AI and ML, 2012.