# Towards robust music source separation
on loud commercial music

###### Abstract

Nowadays, commercial music has extreme loudness and heavily compressed dynamic
range compared to the past. Yet, in music source separation, these
characteristics have not been thoroughly considered, resulting in the domain
mismatch between the laboratory and the real world. In this paper, we
confirmed that this domain mismatch negatively affect the performance of the
music source separation networks. To this end, we first created the out-of-
domain evaluation datasets, musdb-L and XL, by mimicking the music mastering
process. Then, we quantitatively verify that the performance of the state-of-
the-art algorithms significantly deteriorated in our datasets. Lastly, we
proposed LimitAug data augmentation method to reduce the domain mismatch,
which utilizes an online limiter during the training data sampling process. We
confirmed that it not only alleviates the performance degradation on our out-
of-domain datasets, but also results in higher performance on in-domain data.

##  1 Introduction

Recent commercial music has extreme loudness compared to the past [1, 2].
Since many artists and producers want their music to be perceptually louder,
it has become so trendy that it rather harms the quality of music [3] and even
there exists the expression ‘loudness war’ [4].

A dynamic range compressor [5] is used to increase the loudness of music while
keeping the digital level under 0 decibels relative to full scale (dBFS),
which is the maximum possible level in the digital domain. It is a time-
varying non-linear processor that adjusts the level of the signal when the
level exceeds the threshold. Especially, a limiter refers to a dynamic range
compressor that strongly compresses the signal with a high ratio parameter
above 1:10 and increases the gain of the signal by the headroom obtained
through compression.

Figure 1: The short example of a limiter applied music source in our musdb-XL
dataset. Recent commercial music has this loud volume and distorted signal
characteristics. Figure 2: The boxplot representing the loudness distributions
of different evaluation datasets, recent commercial music, and training
examples generated from the official implementation of Open-unmix [6].

In the last stage of music production, so-called mastering, engineers are
often asked to increase the overall loudness of music. A limiter is a core
tool for achieving it; it is used to make small parts of music to be louder
and loud parts of music fit under 0 dBFS [7, 8]. In this process, original
signals are distorted and become louder. The example of a limiter usage is
depicted in Figure 1.

However, these characteristics have not yet been thoroughly considered in the
music source separation. Although many data augmentation techniques have been
proposed, such as random gain scaling and mixing of different instrumental
stem sources [9, 10], the training examples have small overall loudness that
is far from real-world commercial music111Based on the investigation of 50
songs in ‘Pop Life playlist’ created by Tidal, May. 2022., as can be seen in
the blue line of Figure 2. This causes a huge domain shift between train and
real-world domains. The term LUFS in Figure 2 is an abbreviate for loudness
unit relative to full scale, which is a measure of music loudness [11, 12].

Furthermore, since the standard benchmark datasets for music source
separation, musdb [13] and musdb-hq [14], do not reflect the characteristics
of loud commercial music as shown in the red lines of Figure 2, they are
unable to show performance degradation comes from the domain shift. Therefore,
we conjecture that even good-performing networks based on musdb test subset
may exhibit worse performance on real world. This problem is no exception to
other datasets [15, 16, 17] for music source separation, since they do not
consider the music mastering process.

Based on this question, we investigated how heavily compressed music and its
loudness affect the degradation of music source separation networks. To this
end, we manually built new evaluation datasets by applying a limiter to the
musdb-hq test subset [14] to get loud and compressed music imitating the
modern music mastering process. One is the musdb-L dataset, with increased
loudness to an appropriate degree for pleasant sound following [3]. The other
is the musdb-XL dataset, which raised the loudness to an excessive degree, as
often found in recent popular music. Using our datasets, we confirmed that
more dynamic range compression in test domains results in more performance
degradation of the networks.

Moreover, we conducted various experiments on the training data creating
methods to reduce the domain shift between train and real world data from the
perspective of music loudness and heavy compression. We trained music source
separation networks using the training examples constructed by (1) linear gain
increasing, (2) the proposed LimitAug method which utilizes an online limiter
during the sampling process of training examples, (3) simple input loudness
normalization, and (4) the loudness normalization after applying the LimitAug.
We confirmed that all of the methods not only showed robust performance on
out-of-domain musdb-L and musdb-XL data, but also showed better performance on
in-domain musdb-hq test dataset than the baseline.

To summarize, our contributions are three-fold.

  * •

We built musdb-L and XL datasets222https://github.com/jeonchangbin49/musdb-XL,
which have comparable overall loudness to commercial music, for evaluation of
music source separation algorithms.

  * •

Using musdb-L and XL, we quantitatively confirmed that the domain shift causes
performance degradation of the state-of-the-art networks that were trained
without considering loud and compressed music characteristics.

  * •

We proposed LimitAug333https://github.com/jeonchangbin49/LimitAug data
augmentation method and experimentally confirmed that it is beneficial to
alleviate the domain shift between train data and the musdb-L or XL.

##  2 Musdb-L and Musdb-XL

musdb [13] and musdb-hq [14] have been the standard benchmark datasets on
music source separation since their presentation in Signal Separation and
Evaluation Challenge (SiSEC) 2018 [18]. They consist of various genres of 150
professionally produced songs — 100 songs for train, 50 songs for test — from
folk, indie to electronic, rock genres. Each song has its constituting 4
stems, vocals, drums, bass, and the remaining as other. The musdb-hq dataset
is the uncompressed version of musdb with the extended frequency bandwidth
from 16kHz to 22.05kHz, which is a full bandwidth of 44.1kHz sample rate. We
used musdb-hq for high-quality dataset construction.

Since the test subset of musdb-hq has small overall loudness compared to
commercial mastering-finished music, we conjectured that it could not fully
reflect the performance degradation related to heavy dynamic range
compression, which prevails in recent commercial music. In addition, to the
best of our knowledge, there is no dataset for music source separation that
reflects these characteristics or considers the music mastering process.
Therefore, we built musdb-L and musdb-XL datasets, which have loud and
compressed characteristics similar to commercial music. L and XL each stands
for Loud and eXtremely Loud.

###  2.1 Dataset construction

To reflect the characteristics of commercial music, we created datasets by
imitating the music mastering process. Both datasets were made by manually
applying the commercial digital limiter, iZotope Ozone 9
Maximizer444https://www.izotope.com/en/products/ozone.html, to the musdb-hq
test subset. For each musdb-L and musdb-XL, we controlled the threshold
parameter of a limiter so that compression is applied about 3-4 dB and 6-7 dB
in loud parts of mixture tracks of the musdb-hq. As shown in Table 1, musdb-L
and musdb-XL are about 5 and 7 LUFS louder than the original musdb-hq dataset
on average, respectively. musdb-L has insufficient loudness compared to loud
commercial music but less distorted sources. musdb-XL has comparable loudness
to commercial music and more distorted sound than musdb-L.

To make the ground truth stem tracks of each mixture, we calculated the
sample-wise ratio between the limiter applied mixture and the original
mixture, then multiply it to the individual stems to make the ground truth
stems for musdb-L and XL.

dataset | Loudness [LUFS]
---|---
min | max | median | mean (std)
musdb-hq | -18.84 | -13.52 | -16.02 | -15.92 (1.27)
musdb-L | -14.39 | -8.61 | -10.61 | -10.89 (1.19)
musdb-XL | -11.93 | -6.99 | -8.41 | -8.61 (1.17)
commercial | -10.75 | -6.10 | -7.96 | -8.05 (1.06)

Table 1: Loudness statistics of musdb-hq, musdb-L, musdb-XL datasets, and the
investigated commercial music.

##  3 Methods

In general, a limiter is used as the last signal processor in music mastering
of commercial music [7, 8]. Since it tends to be used excessively in modern
commercial music [1, 2, 4], it should be considered in music source separation
for the development of robust application.

We assumed that the key differences between the real-world mastering-finished
music, i.e. a limiter applied music, and the standard training examples for
music source separation are i) the overall amplitude scale, and ii) the signal
distortion caused by a limiter.

Here we introduce two ways to avoid each domain mismatch problem.

###  3.1 Input loudness normalization

First of all, the simple loudness normalization, which is a linear gain
adjustment of network inputs to the pre-defined reference level, is the
easiest technique to avoid the domain shift caused by i) overall amplitude
scale mismatch. This can be categorized into two, (i) input loudness
normalization during both the training and evaluation stages of networks, and
(ii) normalization only at the evaluation stage.

The method (i) is already used in various studies with different ways. For
example, the network such as Demucs v3 [19], uses input standardization in
time domain based on the mean and standard deviation of the training data. In
Open-unmix [6], the network implicitly normalizes the input by utilizing
trainable input scaling parameters that works in time-frequency domain.
However, we assumed that explicitly adjusting the gain of the inputs based on
the waveform, thereby minimizing the overall amplitude scale mismatch between
train and test domain, can greatly reduce the performance degradation comes
from the domain shift.

The method (ii), normalization only at the evaluation stage, can be considered
as a readily applicable workaround for the models that were trained without
considering music loudness. This method is a simple idea to reduce the overall
amplitude scale difference between train and real-world domain. Since the
music source separation networks are non-linear systems, we hypothesized that
linear scale difference of the inputs might affect the quality of final
outputs. That is, input normalization only at the evaluation stage can be a
simple, yet effective trick for models that were already trained without
considering the domain mismatch.

Though these methods can mitigate the domain mismatch related to i) amplitude
scale, there needs to be another strategies that can reflect ii) the signal
distortion caused by a limiter.

###  3.2 LimitAug

The best way to consider the characteristics of the limiter applied source is
simply using the limiter during the network training. Therefore, we propose
the LimitAug data augmentation method, which utilizes an online limiter during
the training examples construction process, to reduce the domain shift between
the training examples and real-world commercial music. The LimitAug can be
considered as creating train examples that forcibly reflect the distortion
that comes from a limiter, which cannot be reflected by the simple input
loudness normalization technique.

The pseudocode for LimitAug is shown in Figure 3. First, calculate the LUFS of
the mixture created by the data sampling process. Second, adjust the gain of
the input mixture source targeting the randomly sampled LUFS value. Then, it
is followed by the online limiter to fit the waveform under 0 dBFS. Lastly,
calculate the sample-wise (A sample refers to an each waveform value) ratio
between the limiter applied mixture source and the original mixture source,
then multiply the ratio to the original target source to get the ground truth
target signal of the limiter applied mixture source.

When adjusting the gain and applying the limiter to the input mixture, for
example, if the input mixture had -15 LUFS and the randomly sampled target
LUFS was -10, note that the gain-scaled mixture by +5 dB does not have exact
-10 LUFS due to the compression of a limiter and the nature of LUFS
calculation, which considers frequency weighting [11, 12].

The proposed LimitAug can be used with other data augmentation methods; in our
study, random gain scaling, channel swap and mixing of different instrumental
stem sources [9, 10] were used. Also, additional loudness normalization can be
applied after the LimitAug as depicted in conditional statement of Figure 3,
so that the overall amplitude scale mismatch between training and evaluation
stages be minimized.

Figure 3: numpy-like pseudocode of the proposed LimitAug method.

network |  extra train data |  test data | SDR median (mean) [dB]
---|---|---|---
vocals | bass | drums | other | avg
Open-unmix [6] | - | hq | 6.16 (2.54) | 5.03 (2.67) | 6.00 (5.46) | 4.22 (3.46) | 5.35 (3.53)
L | 6.33 (1.63) | 4.81 (2.71) | 5.82 (5.38) | 4.11 (3.42) | 5.27 (3.28)
XL | 5.98 (0.89) | 4.76 (2.59) | 4.97 (4.89) | 4.04 (3.29) | 4.94 (2.92)
TFC-TDF -U-net [20] | - | hq | 7.18 (4.26) | 5.59 (3.35) | 5.76 (5.30) | 4.04 (3.18) | 5.64 (4.02)
L | 7.03 (3.65) | 5.41 (3.08) | 5.52 (5.09) | 3.67 (3.00) | 5.41 (3.71)
XL | 6.95 (3.14) | 5.48 (2.90) | 5.11 (4.68) | 3.55 (2.82) | 5.27 (3.39)
Demucs v3-A [19] | - | hq |  8.11 (5.22) |  9.34 (6.21) |  8.57 (8.01) |  5.51 (5.03) |  7.88 (6.12)
L | 7.54 (5.15) | 9.32 (6.22) | 8.26 (7.65) | 5.51 (5.01) | 7.66 (6.01)
XL | 7.30 (4.86) | 9.19 (6.14) | 7.62 (6.78) | 5.37 (4.97) | 7.37 (5.69)
Open-unmix [6] | ✓ | hq | 7.02 (4.93) | 5.91 (4.06) | 7.18 (6.91) | 4.94 (4.76) | 6.26 (5.17)
L | 6.83 (5.12) | 6.23 (4.09) | 7.07 (6.92) | 4.94 (4.78) | 6.27 (5.23)
XL | 6.70 (4.77) | 6.16 (3.87) | 6.80 (6.48) | 4.89 (4.61) | 6.14 (4.93)
Spleeter [21] | 25000+ | hq | 6.51 (4.42) | 4.77 (3.57) | 6.00 (6.09) | 4.22 (4.12) | 5.38 (4.55)
L | 6.18 (3.90) | 4.73 (3.34) | 5.67 (5.94) | 4.37 (4.03) | 5.24 (4.30)
XL | 6.03 (3.38) | 4.80 (3.13) | 5.55 (5.52) | 4.24 (3.91) | 5.15 (3.98)
Demucs v3-B [19] |  200+ including musdb-hq test set | hq |  9.24 (7.05) | 11.65 (9.58) | 11.73 (11.34) | 7.83 (8.03) | 10.11 (9.00)
L | 9.05 (6.91) | 11.61 (9.55) | 11.05 (10.27) | 7.83 (7.91) | 9.88 (8.66)
XL | 8.76 (6.41) | 11.56 (9.29) | 9.22 (8.78) | 7.52 (7.51) | 9.26 (8.00)

Table 2: Performance of various music source separation networks trained with
musdb-hq [14] on the test using musdb-hq, musdb-L and musdb-XL.

##  4 Experiments

Here we briefly summarize our following experiments.

In Section 5.1, we quantitatively evaluated various music source separation
networks that were trained without considering the loudness and heavy dynamic
range compression, and confirmed the performance degradation on musdb-L and
musdb-XL, compared to the origianl musdb-hq evaluation dataset.

In Section 5.2, we simply normalized the network inputs in the evaluation
stage, to observe the performance degradation caused by the signal distortion
caused by a limiter, not an overall amplitude mismatch between the training
and the evaluation data. Then, we carefully analyzed the results on Demucs v3
[19], which took the 1st place in the 2021 Sony Music Demixing (MDX) Challenge
leaderboard A and the 2nd place in the leaderboard B [22]. The leaderboard A
was the competition that only allowed training on the only musdb-hq train
subset, and the leaderboard B allowed the extra training data.

In Section 5.3, we conducted a comparative study on various training data
construction strategies. Unfortunately, we were unable to train the current
state-of-the-art Demucs v3, due to the constraint of our GPU experimental
environments. Considering the training efficiency and reasonable performance,
we used TFC-TDF-U-Net [20] — a backbone architecture of KUIELab-MDX-Net [23]
with slight modifications, which took the 2nd place in the 2021 MDX Challenge
leaderboard A — in this experiment.

###  4.1 Training

In Section 5.1 and Table 2, we used official pre-trained weights of each model
except TFC-TDF-U-Net since there were no official weights for the 4 stems of
musdb-hq. For fair comparison in Section 5.3, we trained TFC-TDF-U-Net for 300
epochs with early stopping, based on official training framework of Open-unmix
[6] and official network implementation of TFC-TDF-U-net. We used default
network hyper-parameters introduced in its webpage555https://github.com/ws-
choi/ISMIR2020_U_Nets_SVS.

For the LimitAug, we used the limiter implemented in pedalboard [24]. The
threshold parameter was set to 0 dBFS, and the release parameter was randomly
sampled from uniform distribution of (30, 200) millisecond in data sampling
process. Also, we used pyloudnorm [25] for loudness calculation.

###  4.2 Evaluation

In all of the evaluations, Signal-to-Distortion Ratio (SDR) [26] was
calculated using museval python library [18]. Also, in following experimental
results, both median and mean SDR scores were presented for the detailed
comparison. Note that we only used the musdb-hq train subset for training the
networks. musdb-L and XL are only for evaluation.

In the original musdb-hq test subset, as stated in the official
webpage666https://sigsep.github.io/datasets/musdb, ‘PR - Oh No’ track’s
mixture signal is panned to the right channel, which causes the inconsistency
between linear summation of stems and mixture source. Since this causes the
limiter to be operated in un-musical way while the construction of musdb-L and
XL, which also can be hardly found in popular music, we used the linear
summation of stems as a mixture only for this track. This may results in the
slight difference of SDR scores between the Table 2 and the official scores of
each network.

network |  extra train data |  test data | SDR median (mean) [dB]
---|---|---|---
vocals | bass | drums | other | avg
Demucs v3-A [19] | - | hq |  8.11 (5.22) | 9.34 (6.21) | 8.57 (8.01) | 5.51 (5.03) | 7.88 (6.12)
L | 8.05 (5.23) | 9.25 (6.20) | 8.47 (7.92) |  5.53 (5.02) | 7.82 (6.09)
XL | 7.93 (5.03) | 9.27 (5.92) | 7.74 (7.44) | 5.55 (4.91) | 7.62 (5.82)
Demucs v3-B [19] |  200+ including musdb-hq test set | hq | 9.24 (7.05) | 11.65 (9.58) | 11.73 (11.34) | 7.83 (8.03) | 10.11 (9.00)
L | 9.19 (7.04) | 11.64 (9.55) | 11.68 (11.21) | 7.82 (8.02) | 10.08 (8.95)
XL | 9.13 (6.90) | 11.56 (9.33) | 11.32 (10.75) | 7.74 (7.95) | 9.94 (8.73)

Table 3: Performance of Demucs v3 [19] trained with musdb-hq [14]. On the test
using musdb-L and musdb-XL, each of the input sources were loudness normalized
targeting the LUFS of its original musdb-hq sources.

network |  extra train data | SDR median [dB]
---|---|---
hq | L | XL
Open-unmix [6] | - | 5.35 | 5.32 | 5.25
TFC-TDF-U-Net [20] | - | 5.64 | 5.62 | 5.51
Demucs v3-A [19] | - | 7.88 | 7.82 | 7.62
Open-unmix [6] | ✓ | 6.26 | 6.25 | 6.18
Spleeter [21] | ✓ | 5.38 | 5.33 | 5.21
Demucs v3-B [19] | ✓ | 10.11 | 10.08 | 9.94

Table 4: Performance of networks with loudness normalized input at the
evaluation stage. Each score represents the average score across the 4 stems.
Note that the networks were trained without considering music loudness or
dynamic range compression explicitly.

##  5 Results

###  5.1 Performance degradation on musdb-L and XL

Here we quantitatively evaluated the performance of state-of-the-art networks
on musdb-hq [14], L and XL datasets and confirmed that the domain shift in
perspective of music loudness and compression negatively affect the
performance, indeed. As shown in the Table 2, all networks showed significant
performance degradation on the evaluations with musdb-L and musdb-XL datasets.
The amount of decrease on Demucs v3 [19] was slightly larger than the others.
Overall, we concluded that every networks are somewhat overfitted to the
musdb-hq data, making the networks vulnerable to loud and heavily compressed
music. Therefore, it is highly required to consider these characteristics for
the robust music source separation. Note that we stated the scores on
different test datasets in each block to emphasize the performance degradation
between the domains.

####  5.1.1 Extra training data

Though the extra training data was of help, it did not work as the fundamental
solution to the domain shift. Open-unmix [6] with extra training data showed
more robust performance than that of the network without extra data, but
performance degradation on Demucs v3-B was more significant than that of
Demucs v3-A. Since Demucs v3-B was trained with extra training data including
musdb-hq test subset, it is reasonable to guess that the network is highly
overfitted to the musdb-hq data. Nevertheless, the amount of performance
decrease especially on drums comparing the scores between musdb-hq and XL, 2.5
dB, is somewhat high. Considering Demucs v3 uses an input standardization
technique based on mean and standard deviation of waveform values both at
training and evaluation stages, this implies that there needs to be another
solution for the robust music source separation.

####  5.1.2 Performance degradation on drums and vocals

It is noteworthy that the degradation on drums and vocals were more
significant than the others. Due to the percussive nature of drums, in
general, they have the biggest momentary energy in music. Also, since vocals
are important ingredients in modern commercial music, they usually consist of
not only a single singing voice but also doubling, harmony and chorus.
Therefore, we assumed drums and vocals are most affected by the limiter, which
is activated when loud input sources that are above the threshold are given.
To quantitatively confirm the signal distortion by the limiter, we calculated
Scale-invariant Signal-to-Distortion Ratio (SI-SDR) [27] between musdb-hq and
musdb-XL for each stem. As a result, drums and vocals scored each 19.97 and
23.69 dB, on the other hand, bass and other scored each 25.12 and 25.48 dB on
average. That is, the signal distortion caused by a limiter is more
significant on drums and vocals.

Unfortunately, since the networks in the Table 2 have never seen these kinds
of distorted drums or vocals as training examples, we assumed that the
degradation on drums and vocals were significant compared to the rest. This
result strongly supports the necessity of considering the heavy dynamic range
compression from the training stage, i.e. the LimitAug, which forcibly makes
the distorted and compressed training examples for training purposes, thereby
minimizing the domain shift.

network | methods |  linear gain increase | LimitAug |  input loud-norm | target LUFS | SDR median (mean) [dB]
---|---|---|---|---|---|---
hq | L | XL | avg
TFC-TDF -U-Net [20] | baseline | - | - | - | - | 5.64 (4.02) | 5.41 (3.71) | 5.27 (3.39) | 5.44 (3.71)
(1) | ✓ | - | - | $\mathcal{N}(\mu_{L},\sigma_{L}^{2})$ |  5.90 (4.31) | 5.86 (4.33) | 5.73 (4.15) | 5.83 (4.26)
$\mathcal{N}(\mu_{XL},\sigma_{XL}^{2})$ | 5.32 (3.43) | 5.36 (3.62) | 5.28 (3.49) | 5.32 (3.51)
(2) | - | ✓ | - | $\mathcal{N}(\mu_{L},\sigma_{L}^{2})$ | 5.79 (4.30) |  5.90 (4.41) | 5.74 (4.25) | 5.81 (4.32)
$\mathcal{N}(\mu_{XL},\sigma_{XL}^{2})$ | 5.69 (3.93) | 5.72 (4.22) | 5.57 (4.15) | 5.66 (4.10)
(3) | - | - | ✓ | -14 | 5.89 (4.38) | 5.87 (4.35) | 5.82 (4.25) | 5.86 (4.33)
(4) | - | ✓ | ✓ |  $\mathcal{N}(\mu_{L},\sigma_{L}^{2})$, -14 | 5.87 (4.25) | 5.85 (4.21) | 5.76 (4.16) | 5.83 (4.21)
$\mathcal{N}(\mu_{XL},\sigma_{XL}^{2})$, -14 | 5.78 (4.27) | 5.78 (4.26) | 5.73 (4.20) | 5.76 (4.24)

Table 5: Performance of TFC-TDF-U-Net [20] trained with various training data
construction strategies. Each score represents the average score across the 4
stems.

network | methods |  target LUFS |  test data | SDR median (mean) [dB]
---|---|---|---|---
vocals | bass | drums | other | avg
TFC-TDF -U-Net [20] | baseline | - | hq | 7.18 (4.26) | 5.59 (3.35) | 5.76 (5.30) | 4.04 (3.18) | 5.64 (4.02)
L | 7.03 (3.65) | 5.41 (3.08) | 5.52 (5.09) | 3.67 (3.00) | 5.41 (3.71)
XL | 6.95 (3.14) | 5.48 (2.90) | 5.11 (4.68) | 3.55 (2.82) | 5.27 (3.39)
(3) loud-norm | -14 | hq | 7.35 (4.76) |  5.93 (3.61) |  5.91 (5.37) | 4.39 (3.79) |  5.89 (4.38)
L | 7.32 (4.72) | 5.91 (3.61) | 5.85 (5.29) | 4.39 (3.78) | 5.87 (4.35)
XL | 7.26 (4.64) | 5.91 (3.62) | 5.68 (4.99) | 4.42 (3.78) | 5.82 (4.25)
(4) LimitAug, loud-norm |  $\mathcal{N}(\mu_{L},\sigma_{L}^{2})$, -14 | hq |  7.59 (4.64) | 5.75 (3.25) | 5.63 (5.28) | 4.50 (3.82) | 5.87 (4.25)
L | 7.58 (4.61) | 5.69 (3.21) | 5.62 (5.22) | 4.50 (3.82) | 5.85 (4.21)
XL | 7.48 (4.55) | 5.67 (3.29) | 5.36 (4.99) |  4.51 (3.82) | 5.76 (4.16)

Table 6: Stem-wise performance of TFC-TDF-U-Net [20] trained with the method
(3) input loudness normalization, and (4) input loudness normalization after
the proposed LimitAug.

###  5.2 Analysis on the input normalization at the evaluation stage

If the key differences between the real-world music and the training examples
of music source separation networks are i) overall amplitude scale, and ii)
the signal distortion caused by a limiter, as stated in Section 3, then what
if we give the loudness normalized musdb-L or XL data as inputs to the
networks in Table 2? Due to the non-linear nature of deep neural networks, we
assumed that simply normalizing the amplitude scale of the networks on the
evaluation stage may affect the performance.

The inference was conducted with following procedures, (i) reducing the
loudness of musdb-L or XL input so that its loudness becomes same with that of
the corresponding original musdb-hq data, (ii) inference with loudness
normalized input, and (iii) increasing the scale of output as much as reduced
in step (i).

Comparing the scores between Table 2 and Table 4, we confirmed that the
performance degradation was greatly alleviated by just the simple loudness
normalization of the inputs only at the evaluation stage. Note that the
networks were not trained with loudness normalized inputs. This result shows
that the input loudness normalization at the evaluation stage can be a quick
and easy solution to get robust results from the pre-trained music source
separation networks.

Especially, on drums of Demucs v3-B, it should be noted that the amount of
decrease on median SDR score between the test using musdb-hq and XL was
sharply reduced from 2.5 dB in Table 2 to 0.4 dB in Table 3. This implies that
the network is overfitted not only to the contents of the signal, but also to
the scale or loudness, especially on drums. Note that there was no distinction
between the given input sources to the networks except the linear gain
difference.

Although the performance degradation was reduced by the input loudness
normalization, still there exists the performance degradation due to the
signal distortion caused by a limiter. Similar to Section 5.1.2, this result
also supports the necessity of the LimitAug for robust music source
separation.

###  5.3 Analysis on various training strategies

In this section, we trained the TFC-TDF-U-net [20] with various training data
creating methods; (1) linear gain increasing, (2) the proposed LimitAug, (3)
input loudness normalization, and (4) input loudness normalization after the
LimitAug. Of course, the methods (3) and (4) includes the input loudness
normalization at the evaluation stage for the consistency between train and
test domains. We compared the results to check which one is the most powerful
way for training robust music source separation networks. For the input
normalization, we chose the target reference LUFS value as -14.

In Table 5, we confirmed that all of the methods were effective for robust
music source separation; every methods showed relatively robust performance on
musdb-L and XL, compared to the baseline. Especially, except the method (1)
targeting LUFS of a normal distribution following statistics of musdb-XL,
$\mathcal{N}(\mu_{XL},\sigma_{XL}^{2})$, all methods showed greater
performance on musdb-hq than the baseline. This result implies that these
methods are useful not only for the domain shift, but also for the standard
benchmark data.

Furthermore, the methods (3) and (4), which prevent the domain shift caused by
overall amplitude scale mismatch by input loudness normalization, showed
slightly better performances than others. In the stem-wise analysis as
presented in Table 6, though we expected that the LimitAug would be of help
for vocals and drums, the method (4) was better at vocals and other than the
method (3).

Overall, it seems obvious that considering the music loudness and heavy
dynamic range compression from the training stage is beneficial for robust
music source separation. For real world applications, it is highly recommended
that not just using the single method we experimented, but using various
training methods on different stems or using a bag of models trained with
various methods.

##  6 Conclusions

In this study, we questioned the domain shift between the research and the
real-world data for music source separation, from the viewpoint of music
loudness and heavy dynamic range compression. To answer this, We first built
new evaluation datasets, musdb-L and musdb-XL, which reflect dynamic range
compressed music characteristics and heavy loudness. Then, we confirmed the
significant performance degradation of state-of-the-art networks on our
datasets. To alleviate this, we conducted various experiments on training data
construction strategies, including the proposed LimitAug method, and confirmed
that the methods using the input loudness normalization only or with the
LimitAug greatly improved the robustness. We hope that our proposed methods
and evaluation datasets could contribute to future music source separation
research and application.

##  7 Acknowledgements

We appreciate Zafar Rafii, the author of musdb, for allowing us to reprocess
the original musdb data. We thank Antoine Liutkus, also the author of musdb,
for giving the creative suggestion on the distribution of our proposed
datasets. We are grateful to Ben Sangbae Chon, Keunwoo Choi, and Hyeongi Moon
from GaudioLab for their fruitful discussions on our proposed methods. Last
but not least, we thank Donmoon Lee, Juheon Lee, Jaejun Lee, Junghyun Koo, and
Sungho Lee for helpful feedbacks.

## References

  * [1] S. Dredge, “Pop music is louder, less acoustic and more energetic than in the 1950s,” _The Guardian_ , 25 Nov. 2013.
  * [2] G. Milner, “They really don’t make music like they used to,” _The New York Times_ , 7 Feb. 2019.
  * [3] N. B. Croghan, K. H. Arehart, and J. M. Kates, “Quality and loudness judgments for music subjected to compression limiting,” _The Journal of the Acoustical Society of America_ , vol. 132, no. 2, pp. 1177–1188, 2012.
  * [4] E. Vickers, “The loudness war: Background, speculation, and recommendations,” in _Audio Engineering Society Convention 129_. Audio Engineering Society, 2010.
  * [5] E. F. Stikvoort, “Digital dynamic range compressor for audio,” _Journal of the Audio Engineering Society_ , vol. 34, no. 1/2, pp. 3–9, 1986.
  * [6] F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji, “Open-unmix - a reference implementation for music source separation,” _Journal of Open Source Software_ , 2019. [Online]. Available: https://doi.org/10.21105/joss.01667
  * [7] B. Katz and R. A. Katz, _Mastering audio: the art and the science_. Butterworth-Heinemann, 2003.
  * [8] L. Bregitzer, _Secrets of recording: Professional tips, tools & techniques_. Routledge, 2008.
  * [9] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji, “Improving music source separation based on deep neural networks through data augmentation and network blending,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 261–265.
  * [10] L. Prétet, R. Hennequin, J. Royo-Letelier, and A. Vaglio, “Singing voice separation: A study on training data,” in _ICASSP 2019-2019 ieee international conference on acoustics, speech and signal processing (icassp)_. IEEE, 2019, pp. 506–510.
  * [11] R. ITU-R, “Itu-r bs. 1770-2, algorithms to measure audio programme loudness and true-peak audio level,” _International Telecommunications Union, Geneva_ , 2011.
  * [12] R. EBU-Recommendation, “Loudness normalisation and permitted maximum level of audio signals,” _European Broadcasting Union_ , 2011.
  * [13] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “The MUSDB18 corpus for music separation,” Dec. 2017. [Online]. Available: https://doi.org/10.5281/zenodo.1117372
  * [14] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “Musdb18-hq - an uncompressed version of musdb18,” Aug. 2019. [Online]. Available: https://doi.org/10.5281/zenodo.3338373
  * [15] E. Manilow, G. Wichern, P. Seetharaman, and J. Le Roux, “Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity,” in _2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2019, pp. 45–49.
  * [16] A. Liutkus, F.-R. Stöter, Z. Rafii, D. Kitamura, B. Rivet, N. Ito, N. Ono, and J. Fontecave, “The 2016 signal separation evaluation campaign,” in _International conference on latent variable analysis and signal separation_. Springer, 2017, pp. 323–332.
  * [17] R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “Medleydb: A multitrack dataset for annotation-intensive mir research.” in _ISMIR_ , vol. 14, 2014, pp. 155–160.
  * [18] F.-R. Stöter, A. Liutkus, and N. Ito, “The 2018 signal separation evaluation campaign,” in _Latent Variable Analysis and Signal Separation: 14th International Conference, LVA/ICA 2018, Surrey, UK_ , 2018, pp. 293–305.
  * [19] A. Défossez, “Hybrid spectrogram and waveform source separation,” _arXiv preprint arXiv:2111.03600_ , 2021.
  * [20] W. Choi, M. Kim, J. Chung, D. Lee, and S. Jung, “Investigating u-nets with various intermediate blocks for spectrogram-based singing voice separation,” in _21th International Society for Music Information Retrieval Conference, ISMIR, Ed_ , 2020.
  * [21] R. Hennequin, A. Khlif, F. Voituret, and M. Moussallam, “Spleeter: a fast and efficient music source separation tool with pre-trained models,” _Journal of Open Source Software_ , vol. 5, no. 50, p. 2154, 2020.
  * [22] Y. Mitsufuji, G. Fabbro, S. Uhlich, and F.-R. Stöter, “Music demixing challenge 2021,” _arXiv preprint arXiv:2108.13559_ , 2021.
  * [23] M. Kim, W. Choi, J. Chung, D. Lee, and S. Jung, “Kuielab-mdx-net: A two-stream neural network for music demixing,” _arXiv preprint arXiv:2111.12203_ , 2021\.
  * [24] Spotify, “pedalboard : A python library for manipulating audio.” 2022. [Online]. Available: https://github.com/spotify/pedalboard
  * [25] C. J. Steinmetz and J. D. Reiss, “pyloudnorm: A simple yet flexible loudness meter in python,” in _150th AES Convention_ , 2021.
  * [26] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” _IEEE transactions on audio, speech, and language processing_ , vol. 14, no. 4, pp. 1462–1469, 2006.
  * [27] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 626–630.


# Development of muon scattering tomography for a detection of reinforcement
in concrete

Magdalena Dobrowolska1111corresponding author<EMAIL_ADDRESS>1
School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall
Avenue, BS8 1TL, Bristol, United Kingdom  Jaap Velthuis1,2,3 1 School of
Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue,
BS8 1TL, Bristol, United Kingdom  Anna Kopp1 1 School of Physics, HH Wills
Physics Laboratory, University of Bristol, Tyndall Avenue, BS8 1TL, Bristol,
United Kingdom  Chiara De Sio1 1 School of Physics, HH Wills Physics
Laboratory, University of Bristol, Tyndall Avenue, BS8 1TL, Bristol, United
Kingdom  Ruaridh Milne4 1 School of Physics, HH Wills Physics Laboratory,
University of Bristol, Tyndall Avenue, BS8 1TL, Bristol, United Kingdom
Philip Pearson4 1 School of Physics, HH Wills Physics Laboratory, University
of Bristol, Tyndall Avenue, BS8 1TL, Bristol, United Kingdom

###### Abstract

Inspection of ageing, reinforced concrete structures is a world-wide
challenge. Existing non-destructive evaluation techniques in civil and
structural engineering have limited penetration depth and don’t allow to
precisely ascertain the configuration of reinforcement within large concrete
objects. The big challenge for critical infrastructure (bridges, dams, dry
docks, nuclear bioshields etc.) is understanding the internal condition of the
concrete and steel, not just the location of the reinforcement. In most new
constructions the location should be known and recorded in the as-built
drawings, where these might not exist due to poor record keeping for older
structures.

Muon scattering tomography is a non-destructive and non-invasive technique
which shows great promise for high-depth 3D concrete imaging. Previously, we
have demonstrated that individual bars with a diameter of 33.7 $\pm$ 7.3 mm at
50 cm depth can be located using muon scattering tomography. Here we present
an improved method that exploits the periodicity of bar structures. With this
new method, reinforcement with bars down to 6 mm thickness can be detected and
imaged.

keywords:  3D imaging, concrete imaging, reinforcement location, bar location,
muon scattering tomography, NDE technique

##  1 Introduction

Old reinforced concrete structures may need to be inspected or replaced.
Knowing the location of the steel is the first step towards determining the
condition of the reinforcement. The key for assessing and substantiating the
structure for life extensions is being able to state the design was built as
planned (location and size of reinforcement) to a high quality (no voids from
construction) and that the internal condition is satisfying (not degraded -
reinforcement corrosion, cracking - beyond a critical value). Current non-
destructive evaluation (NDE) scanning technology is based on the detection of
reflected or transmitted electromagnetic, thermal or acoustic waves generated
by a local source. The most used NDE techniques are magnetic imaging and
ground penetrating radar (GPR) [1, 2, 3, 4], which can image bars with the
diameters of 10-20 mm at depths of 100-500 mm. Low depth imaging, for depths
at $<$ 20 cm, can be performed with infrared thermographics [5, 6] and
ultrasonics [7, 8]. These techniques are suitable for assessing element
thickness and bar location, but precise estimation of bar size is still an
unsolved problem [2]. Furthermore, a detailed testing below the first or
second reinforcement layer is often beyond the bounds of possibility because
errors greatly increase with penetration depth and number of bars [4]. Other
NDE methods currently used are x-ray and neutron radiography [9, 10], which
enable a high resolution and a high depth scanning. However, the use of active
sources of radiation is a serious threat to human health.

Muon tomography is being investigated for many different challenges, relying
on both Monte Carlo simulation studies and experiments. A traditional
application of muon tomography is the characterization of nuclear waste drums
and related security applications, where contents of concrete or bitumen
filled waste drums are studied. Key issues here include the potential presence
of gas bubbles in the matrix of the waste drum [11] and identification of the
material inside the drums [12, 13, 14]. Security applications have been mainly
focused on detection of lumps of high-Z material in cargo containers [15, 16],
but work on the detection of explosives is ongoing as well [17]. Examples
include experimental studies of imaging of concrete blocks [18, 19, 20].

Previously, we published a novel approach exploiting muon scattering
tomography (MST) to detect the presence and location of reinforcement bars
[21]. This work has shown that a 100 cm long, singular bar with a diameter of
33.7 $\pm$ 7.3 mm can be detected using three weeks of data taking at sea
level. It was also shown that the signal has a monotonic dependence on the
volume of the bar contained in the concrete drum. Moreover, the volume of the
inclusion can be measured with a resolution of 5.4 $\pm$ 0.3%, and relative
uncertainty below 10%, for bar volumes above 2 500 cm3. This outcome does not
depend on the location of the bar. Bars as close as 6 cm apart can be detected
as two individual items. However, the separation starts to be observable at a
4 cm distance. The approach also demonstrated to be suitable for imaging
purposes, such as depicting bar structures in reinforced concrete.
Differentiation between single and double layers of grids with bars diameters
of 30 mm was also possible.

Here we present a new method that exploits the periodicity of bar structures,
and is able to detect much smaller bar sizes within shorter time of data
collection.

For most reinforced concrete structures, bars with diameters between 8 mm and
40 mm are used [22]. The thinnest bars in use are 6 mm in diameter, whereas
for walls and bridges much thicker bars ($\geq$10 mm) are used. The yield
strength of the concrete depends strongly on the regular placement of the
bars. Precise measurements are important for structural re-assessment to
define a structural capacity or longerity of a concrete element or building
structure. The spacing on most bar products is 10 or 20 cm [23]. Therefore, we
have performed our studies with bars of a minimum diameter of 6 mm in a
perfect grid of 7.5, 10, 15 and 20 cm.

##  2 Muon scattering tomography (MST)

Muon scattering tomography is a non-invasive method which shows great
potential to generate high-depth 3D concrete images. MST uses cosmic rays as
probes. Cosmic rays are high-energy charged particles which come to the
Earth’s atmosphere from outer space. In the atmosphere, cascades of new
particles are produced. The main type of particles that reach sea level are
muons. Muons are identical to electrons, but 200 times heavier. Muons can go
through large amounts of material as they do not scatter very much due to
their high mass. When traversing matter, Coulomb interactions take place
between the muons and the nuclei of the material. As a result, muons undergo a
series of scattering events and exit the material under an angle. The angular
distribution of scattered muons can be described by a Gaussian distribution
with a mean of zero and a standard deviation $\sigma_{\theta}$ described by
[24]:

$\sigma_{\theta}\approx\frac{13.6\text{MeV}}{pc\beta}\sqrt{\frac{T}{X_{0}}}\
[1+0.038\ln(\frac{T}{X_{0}})]$ (1)
$X_{0}\approx\frac{716.4A}{Z(Z+1)\ln(\frac{287}{\sqrt{Z}})}\ \
[\text{g}\cdot\text{cm}^{-2}]$ (2)

where p is muon’s momentum; $\beta$ is muon’s speed divided by the speed of
light c; T is the thickness of the material and $X_{0}$ its radiation length;
A is the atomic weight of the medium in g$\cdot$mol-1. $\sigma_{\theta}$
depends on the atomic number $Z$ of the traversed material. Under the
assumption that scattering occurs in a single location, and by reconstructing
the incoming and outgoing trajectories of the muons, the scattering angle
distribution can be reconstructed and thus information about the traversed
material can be inferred.

##  3 MST simulation

In this work, we use Monte Carlo simulations of a MST system. The simulated
MST system consists of detectors and a reinforced concrete block. The muons
were generated using the CRY library [26], specifically developed for cosmic
rays. GEANT4 [25] was used to simulate the propagation of the muons through
detectors and scanned objects.

The simulated detector system consists of 3 sets of 2 $\times$ 2 m2 resistive
plate chambers (RPCs) on one side of the volume under study and 3 sets of 2
$\times$ 2 m2 of RPCs on the other one. Three layers of detector sets on each
side of the scanned object were chosen to provide a precise reconstruction of
the muon direction and thus of the scattering angle.

RPCs consist of a gas volume enclosed between two glass panels over which a
high voltage is applied. Muons, which are electrically charged, ionize the gas
when traversing the detector. Due to the high voltage, a small avalanche is
created locally. The measurable signal induced by the avalanche can be used to
reconstruct the location where the muon traversed the RPC. The simulated RPCs
performance was modeled using the performance of RPCs that were built for a
container scanner prototype [27, 28]. The RPCs had a pitch of 1.5 mm, which
resulted in a position resolution of approximately 450 $\mu$m. One pair of
RPCs consists of both X and Y planes, orthogonal to each other, so that both x
and y coordinates of the muon paths can be detected, and the muon paths
calculated accurately. The thickness of one RPC is 6 mm. The X and Y planes
are 19 mm apart and the distance between the RPC pairs is between 56 and 59
mm. The space between top and bottom detector layers, where an object can be
scanned is 548 mm. The concrete block was placed in this volume. Reinforced
concrete was modeled as a rectangular, concrete-filled object with dimensions
of 200 cm $\times$ 200 cm $\times$ 34 cm. Inside the block, reinforcement bars
were arranged in two orthogonal layers to form a single, reinforcement grid.
Concrete was modeled as a material with a density of 2.3 g$\cdot$cm3, and the
steel reinforcement bars were simulated as iron bars with density of 7.87
g$\cdot$cm3. The density of concrete ranges from 2 to 2.5 g$\cdot$cm3. A
schematic drawing of the simulated setup is shown in figure 1.

Figure 1: Schematic drawing of the simulated setup. RPCs are above and below
the concrete block. Inside the concrete block a reinforcement grid is placed.

Multiple scenarios were simulated to perform a detailed analysis. For all
simulations the RPCs remained the same, whereas the contents of the concrete
were changed. The diameter of the bars varied from 6 mm to 20 mm. Spacings of
7.5, 10, 15 or 20 cm were used. An example with a single layer of
reinforcement grid is illustrated in figure 2. Figure 2a shows the ZX (front)
and ZY (side) projection of the concrete block. The top (YX) projection is
shown in figure 2b.

(a) Side (ZY) and front (ZX) views

(b) Top (YX) view

Figure 2: A schematic drawing of the concrete block used in the simulations.
Inside, two layers of reinforcement bars, forming a single grid were placed.

###  3.1 Bristol discriminator algorithm

A number of MST imaging approaches have been developed. All of them use
different ways to retrieve information from the incoming and outgoing muon
tracks [29, 30, 31]. We developed an algorithm which is described in detail in
[15]. It was successfully used to detect objects with a high atomic number
(high-Z objects) in cargo containers [15, 32], imaging of nuclear waste [12],
as well as for discrimination of high-Z materials in concrete-filled
containers [13] and detection of voids in concrete filled drums [11]. It was
also demonstrated that bars with a diameter of 33.7 $\pm$ 7.3 mm at 50 cm
depth can be located using that approach [21].

In our method incoming tracks are reconstructed using the three detector
planes above and outgoing tracks using the three detector planes below the
block. Subsequently, the hits are refitted under the assumption that the
incoming and outgoing track meet in a vertex, where the scattering is assumed
to have taken place in a single point. In reality this is not strictly true as
the muon actually performs a random walk through the concrete block. However,
the vertex assumption turns out to be a very useful one in practice. Our
method relies on the ”clusteredness” of high angle scatters in higher-Z
materials: in high-Z materials the scattering angles tend to be larger, and
larger scattering angles result in a well defined vertex. In addition, a
higher number of large-angle scatters occur in higher-Z materials. This makes
the method very sensitive to the detection of materials with a higher-Z inside
an object of a lower-Z, or the other way around.

An example of incoming, outgoing tracks and a vertex is shown in figure 3. The
scanned object is subdivided in voxels. A voxel size of 10 mm$\times$10
mm$\times$10 mm was used in this study. Each track-vertex-track combination is
assigned to the voxel where the vertex is reconstructed. Since the method
exploits the clusteredness of high angle scatters, only the $N$ most scattered
tracks assigned to each voxel are considered in further analysis. $N$ of 30
was used for this analysis. For each pair of remaining vertices in that voxel,
the weighted metric, $\widetilde{m_{ij}}$, is calculated:

$\widetilde{m_{ij}}=\frac{\|\boldsymbol{V_{i}-V_{j}}\|}{\theta_{i}\cdot\theta_{j}}$
(3)

where $\boldsymbol{V_{i}}$ is the position of the vertex of muon i;
$\theta_{i}$ is the corresponding scattering angle. The median of the weighted
metric distribution is calculated for every voxel. The median of that
distribution is then used as a discriminator [15]. An example of the median
discriminator distribution is shown in figure 4. In low atomic number (low-Z)
materials, vertices are located at larger distances as high-angle scattering
occurs less often than in denser materials. Hence, lower-Z materials
correspond to higher discriminator values. In figure 4 the discriminator
distributions for a case with concrete only and a concrete block with
reinforcement placed inside are shown. The reinforced block results in more
low discriminator values.

Figure 3: Incoming and outgoing muon tracks are reconstructed with RPCs. On
their basis, the vertex is determined.  Figure 4: Discriminator distributions
for a concrete block and a concrete block with single grid, where the bars’
diameter was 1.5 cm.

##  4 Detection of reinforcement bars

In this section a new detection algorithm will be presented which removes more
background. An example of its performance will be shown using a concrete with
a single grid made of bars with diameter of 1.5 cm, and a spacing of 15 cm. A
background scenario was defined as pure concrete block. This method is an
improved version of the algorithm published in [21]. In the previous
algorithm, for both background and reinforcement scenarios, for each voxel the
discriminator is calculated. Next, a number of sub-volumes is created for each
scenario and for each sub-volume, a discriminator distribution prepared. Then,
for each bin of a discriminator distribution, the absolute difference between
the discriminator values of the block containing concrete and the scenario
containing bars are taken. The resulting values are summed along the x, the y
and the z-axis. The same study is repeated for every sub volume, resulting in
three projections. A detailed description of the approach is in [21]. An
example of the front projection image (ZX) is shown in figure 5a.

For the new method, all the above mentioned steps are done but the background
subtraction was improved by generating two more concrete samples (the same
size as bar-scenario ones) and performing the same analysis using the two
concrete samples and thus generating final projection images for background
only sample. Then, the background projections were subtracted, bin-by-bin,
from the bar-concrete scenario projections. This method of background
subtraction was chosen as the background is not linear and thus resulted in
clear differences between bar and concrete. An example of the projection image
before and after additional background elimination is shown in figure 5. Areas
with higher signal clearly indicate the presence of iron, while areas with
lower signal show where the concrete is. Bars are clearly observable.

(a) single grid: front view

(b) single grid: front view

Figure 5: Result of the reinforcement imaging before (left) and after
background subtraction (right) for a single grid using 1.5 cm diameter bars.

###  4.1 Detection of the reinforcement grid

From a structural engineering perspective, the reinforcement is always fixed
in an orthogonal grid pattern. Due to its regular shape, the reinforcement
provides a periodic signal. This periodicity can be exploited to lower the
detection limit of the minimum diameter of the bars. This is done by
calculating the auto-correlation of the 2D signal distributions (like figure
5b), and looking for peaks in the Fourier spectrum of the auto-correlation.
The occurrence of peaks indicates the presence of a periodic structure and
thus of bar. The peak locations yield the spacing, while the peak amplitude
depend on the diameter of the bar.
In this method, a variation of the standard auto-correlation $R_{s}$ is used.
Since there is periodicity in both the x and y direction, the auto-correlation
is only evaluated in one direction, i.e. the x direction, as:

$R_{s}(\tau)=\int\limits_{y_{min}}^{y_{max}}\int\limits_{x_{min}}^{x_{max}}f(x^{\prime},y^{\prime})f(x^{\prime}+\tau,y^{\prime})dx^{\prime}dy^{\prime}$
(4)

The calculation was limited to the area within the acceptance of the detector,
and the presence of the reinforcement i.e., X$\in<$20;175$>$, Y$\in<$10;29$>$
in figure 5b. The result of the auto-correlation for the example case is shown
in figure 6. The triangular shape is due to the variation of the overlapping
area. It is observed as a triangular background and the triangular dependence
of the amplitude of the periodic structure. The periodic structure is due to
the reinforcement spacing. Before the Fourier transformation, that triangular
background needs to be subtracted. To estimate it, the complete series of
auto-correlation, $R_{b,k}$ is calculated, where:

$R_{b,k}=\int\limits_{y_{min}}^{y_{max}}\int\limits_{x_{min}}^{x_{max}}f(x^{\prime},y^{\prime})f(x^{\prime}+\Delta_{k}+\tau,y^{\prime})dx^{\prime}dy^{\prime}$
(5)

Here the function is shifted by an additional $\Delta_{k}$, where $k$
indicates the number of pixels the image has been shifted. The shift occurs in
a rolling mode, i.e. when a column is shifted further than $x_{max}$, it is
placed in column 0. This is illustrated in figure 7. This procedure leads to a
shift in the peaks, but the underlying triangular background shape remains the
same, as can be seen in figure 8. For each bin in the signal auto-correlation,
the minimum value of $R_{b,k}$ is subtracted. The result is shown in figure 9.
The graph still displays the triangular pattern in the amplitude, but the
triangular background under the function is removed.

Figure 6: Result of the auto-correlation for the example case.  Figure 7:
Illustration of the effect of the additional shift $\Delta_{k}$. Figure 8: The
auto-correlation for the signal sample and the complete series of auto-
correlations for all values of $\Delta_{k}$. Figure 9: The auto-correlation
for the signal sample after background subtraction.

Next, the background-subtracted signal is Fourier transformed using an
interface class for Fast Fourier Transforms, TVirtualFFT, provided by ROOT, a
data analysis package developed at CERN [33]. Figure 10 shows the result of
the FFT. The spectrum shows a series of clearly visible distinct peaks. Their
position is determined by the bar spacing and the peak amplitudes are related
to the bar diameter. A background scenario, consisting of a concrete block
without reinforcement, does not display any peaks. These results show that the
method is capable of detecting bar grids.

Figure 10: Fourier spectrum after background subtraction for the example case
with 1.5 cm diameter bars and a 15 cm spacing.

###  4.2 Variation of the bar diameter

The peak locations of the Fourier transform depend on the spacing, while the
peak amplitude is related to the bar diameter. Figure 11 shows the Fourier
transforms for reinforcement grid made of 20, 15, 10 and 8 mm diameter bars
with a 15 cm spacing, see figure 12 for a zoomed version of the plot. As
expected, having the same spacing, the peaks occur always at the same
normalized frequency values. With decreasing bar diameter, the amplitude of
the peaks also decreases. This is presented more clearly in figure 13, which
shows amplitude of the peak at 0.07 of the normalized frequency plot as a
function of bar diameter. The amplitude strongly increases with increasing the
diameter. Hence, the bar diameter can be measured based on normalized
frequency spectrum.

Figure 11: Normalized frequency spectrum for a reinforcement grid made of 8,
10, 15 or 20 mm diameter bars and spacing of 15 cm. Figure 12: Zoomed,
normalized frequency spectrum for a reinforcement grid made of 8, 10, 15 or 20
mm diameter bars and spacing of 15 cm. Figure 13: Amplitude of the peak at
0.07 of the normalized frequency distribution for different size of the bar
diameter and spacing fixed at 15 cm. There is a clear dependence between the
bar diameter and the amplitude; the bigger the diameter the higher the
amplitude.

###  4.3 Variation of the spacing

In the previous results, a spacing of 15 cm was used. Figure 14 shows the
Fourier spectrum for the reinforcement using 8 mm diameter bars with spacing
of 10, 15 and 20 cm, see figure 15 for a zoomed version of that figure. All of
the cases are clearly distinguishable from the background scenario. Moreover,
the peaks are located at different frequencies, which makes this method
suitable for the estimation of the spacing.

Figure 14: Normalized frequency spectrum for a reinforcement made of 8 mm
diameter bars, and spacing of 10 cm, 15 cm or 20 cm, respectively. Figure 15:
Zoomed, normalized frequency spectrum for a reinforcement made of 8 mm
diameter bars, and spacing of 10 cm, 15 cm or 20 cm, respectively.

###  4.4 Limits of the method

The smallest bars in normal use have a 6 mm diameter. In order to estimate
whether the method is capable of finding such a small bar, the Fourier
transform of grids with 8, 7 and 6 mm diameter bars were calculated using a 10
cm spacing and a one week worth of data taking. The results are shown in
figure 16 and a zoomed version in figure 17. Bars with 7 and 8 mm diameter at
normalized frequency values of 0.1 and 0.9 are clearly distinguishable from
the background. However, the signal for the 6 mm diameter case is less clear.
Please note that the peak locations are determined by the spacing and thus
only peaks at the right location need to be considered. To strengthen the 6 mm
diameter signal, two weeks worth data taking were used. The peaks become
clearer after two weeks of data taking, see figure 18 and a zoomed version in
figure 19. Figure 20 shows amplitude of the peak at 0.1 of the normalized
frequency spectrum. Clearly, the method can detect the smallest size bar in
use in practice for a 10 cm spacing. Amplitude of the bar with 6 mm diameter
is almost 6 times higher than a background sample.

Figure 16: Normalized frequency spectrum for a reinforcement made of 6, 7 or 8
mm diameter bars and spacing of 10 cm. Figure 17: Zoomed, normalized frequency
spectrum for a reinforcement made of 6, 7 or 8 mm diameter bars and spacing of
10 cm. Figure 18: Normalized frequency spectrum for a reinforcement made of 6,
7 or 8 mm diameter bars and spacing of 10 cm. Time of data taking was
increased to two weeks. Figure 19: Zoomed, normalized frequency spectrum for a
reinforcement made of 6, 7 or 8 mm diameter bars and spacing of 10 cm. Time of
data taking was increased to two weeks. Figure 20: Amplitude of the peak at
0.1 of the normalized frequency distribution for different size of the bar
diameter and spacing fixed at 10 cm. Amplitude of the bar with smallest
commonly used diameters is almost 6 times higher than a background sample.

To evaluate the limits of the method with respect to the spacing, a spacing of
20 cm was also considered. However, bigger spacing between bars results in a
smaller amount of steel in the scanning area. This results in a smaller signal
amplitude, as can be seen in figure 21, zoomed version in figure 22. The
signal for 6 mm diameter bars is now almost below background level. However,
when increasing the data taking time to two weeks (figure 23, zoomed version
in figure 24), even the smallest bar is clearly visible again.

Figure 21: Normalized frequency spectrum for a reinforcement made of 6, 7 or 8
mm diameter bars and spacing of 20 cm. Figure 22: Zoomed, normalized frequency
spectrum for a reinforcement made of 6, 7 or 8 mm diameter bars and spacing of
20 cm. Figure 23: Normalized frequency spectrum for a reinforcement made of 6,
7 or 8 mm diameter bars and spacing of 20 cm. Time of data taking was
increased to two weeks. Figure 24: Zoomed, normalized frequency spectrum for a
reinforcement made of 6, 7 or 8 mm diameter bars and spacing of 20 cm. Time of
data taking was increased to two weeks.

All of the scenarios presented in this publication used a single reinforcement
grid. One possible scenario not shown here consists of multiple reinforcement
grids. Adding more layers of grid will not reduce the ability of the detection
in contrary to existing scanning methods. These geometries include more iron
in the scanning area and thus the signal from steel is stronger and the time
of data taking can be limited.

##  5 Summary

Inspection of ageing, reinforced concrete structures is a world-wide challenge
and needs novel non-destructive evaluation techniques with large penetration
depths to precisely ascertain the configuration of reinforcement and the
internal condition of the structure and steelwork, which can possibly contain
some impurities like voids. Muon scattering tomography offers a technique that
suits those needs. A method was presented to locate reinforcement placed in a
large-scale concrete object. The reinforcement was simulated as two layers of
2 m long bars, forming a grid, placed at a fixed distance from each other
inside a large concrete block. The technique exploits the periodicity of the
bars in a reinforcement grid by considering the Fourier-transformed signal.
The presence of a grid leads to peaks in the normalized Fourier frequency
spectrum. Peaks locations are determined by the grid spacing and their
amplitude by the bar diameters. It is therefore possible to estimate both bar
diameter and spacing with this method. Using only one week worth of data
taking, bars with a diameter of 7 mm and larger, could easily be detected for
a 10 cm spacing. The signal for 6 mm diameter bar exceeds the background and
but becomes very clear after two weeks of data taking. Increasing the spacing
to 20 cm results in a smaller amount of iron in the scanning area, thus longer
data taking is required. It has been shown that this method enables the
detection of the smallest bars in practical use within one or two weeks of
data taking time and standard spacing. This is a very important result for
non-destructive evaluation of civil structures.

## References

  * [1] J. H. Bungey, “Sub-surface radar testing of concrete: A review,” Construction and Building Materials, vol. 18, pp. 1–8, Feb. 2004.
  * [2] A. Tarussov, M. Vandry, and A. De La Haza, “Condition assessment of concrete structures using a new analysis method: Ground-penetrating radar computer-assisted visual interpretation,” Construction and Building Materials, vol. 38, pp. 1246–1254, Jan. 2013.
  * [3] H. Rathod, S. Debeck, R. Gupta, and B. Chow, “Applicability of GPR and a rebar detector to obtain rebar information of existing concrete structures,” Case Studies in Construction Materials, vol. 11, p. e00240, Dec. 2019.
  * [4] C. W. Chang, C. H. Lin, and H. S. Lien, “Measurement radius of reinforcing steel bar in concrete using digital image GPR,” Construction and Building Materials, vol. 23, pp. 1057–1063, Feb. 2009.
  * [5] B. Milovanović and I. Banjad Pečur, “Review of Active IR Thermography for Detection and Characterization of Defects in Reinforced Concrete,” Journal of Imaging, vol. 2, p. 11, June 2016\.
  * [6] C. Maierhofer, R. Arndt, and M. Röllig, “Influence of concrete properties on the detection of voids with impulse-thermography,” Infrared Physics & Technology, vol. 49, pp. 213–217, Jan. 2007.
  * [7] P. Burrascano, S. Callegari, A. Montisci, M. Ricci, and M. Versaci, Ultrasonic Nondestructive Evaluation Systems: Industrial Application Issues.  Springer, Dec. 2014.
  * [8] S. Laureti, M. Ricci, M. N. I. B. Mohamed, L. Senni, L. A. J. Davis, and D. A. Hutchins, “Detection of rebars in concrete using advanced ultrasonic pulse compression techniques,” Ultrasonics, vol. 85, pp. 31–38, Apr. 2018.
  * [9] A. du Plessis and W. P. Boshoff, “A review of X-ray computed tomography of concrete and asphalt construction materials,” Construction and Building Materials, vol. 199, pp. 637–651, Feb. 2019.
  * [10] P. Zhang, F. H. Wittmann, P. Lura, H. S. Müller, S. Han, and T. Zhao, “Application of neutron imaging to investigate fundamental aspects of durability of cement-based materials: A review,” Cement and Concrete Research, vol. 108, pp. 152–166, June 2018.
  * [11] M. Dobrowolska, J. Velthuis, L. Frazão, and D. Kikoła, “A novel technique for finding gas bubbles in the nuclear waste containers using muon scattering tomography,” Journal of Instrumentation, vol. 13, no. 05, p. P05015, 2018.
  * [12] C. Thomay, J. Velthuis, T. Poffley, P. Baesso, D. Cussans, and L. Frazão, “Passive 3d imaging of nuclear waste containers with muon scattering tomography,” Journal of Instrumentation, vol. 11, no. 03, p. P03008, 2016\.
  * [13] L. Frazão, J. Velthuis, C. Thomay, and C. Steer, “Discrimination of high-z materials in concrete-filled containers using muon scattering tomography,” Journal of Instrumentation, vol. 11, no. 07, p. P07020, 2016.
  * [14] M. J. Weekes, A. F. Alrheli, D. Barker, D. Kiko?a, A. K. Kopp, M. Mhaidra, J. P. Stowell, L. F. Thompson, and J. J. Velthuis, “Material identification in nuclear waste drums using muon scattering tomography and multivariate analysis,” 2021.
  * [15] C. Thomay, J. Velthuis, P. Baesso, D. Cussans, P. Morris, C. Steer, J. Burns, S. Quillin, and M. Stapleton, “A binned clustering algorithm to detect high-z material using cosmic muons,” Journal of Instrumentation, vol. 8, no. 10, p. P10013, 2013.
  * [16] L. J. Schultz, G. S. Blanpied, K. N. Borozdin, A. M. Fraser, N. W. Hengartner, A. V. Klimenko, C. L. Morris, C. Orum, and M. J. Sossong, “Statistical reconstruction for cosmic ray muon tomography,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 1985–1993, 2007\.
  * [17] G. Anbarjafari, A. Anier, E. Avots, A. Georgadze, A. Hektor, M. Kiisk, M. Kutateladze, T. Lepp, M. Magi, V. Pastsuk, H. Plinte, and S. Suurpere, “Atmospheric ray tomography for low-z materials: implementing new methods on a proof-of-concept tomograph,” 2021.
  * [18] E. Niederleithinger, S. Gardner, T. Kind, R. Kaiser, M. Grunwald, G. Yang, B. Redmer, A. Waske, F. Mielentz, U. Effner, C. Kopp, A. Clarkson, F. Thomson, and M. Ryan, “Muon tomography of a reinforced concrete block – first experimental proof of concept,” 2020.
  * [19] P. Checchia, M. Benettoni, G. Bettella, E. Conti, L. Cossutta, M. Furlan, F. Gonella, J. Klinger, F. Montecassiano, G. Nebbia, et al., “Infn muon tomography demonstrator: past and recent results with an eye to near-future activities,” Philosophical Transactions of the Royal Society A, vol. 377, no. 2137, p. 20180065, 2019.
  * [20] J. Durham, E. Guardincerri, C. Morris, J. Bacon, J. Fabritius, S. Fellows, D. Poulson, K. Plaud-Ramos, and J. Renshaw, “Tests of cosmic ray radiography for power industry applications,” AIP Advances, vol. 5, p. 067111, 2015\.
  * [21] M. Dobrowolska, J. Velthuis, A. Kopp, M. Perry, and P. Pearson, “Towards an application of muon scattering tomography as a technique for detecting rebars in concrete,” Smart Materials and Structures, vol. 29, no. 5, p. 055015, 2020.
  * [22] “Rebar sizes: what you need to know, https://www.cannonsteelsltd.co.uk/rebar-sizes-what-you-need-to-know/, accessed: 10.02.2021.”
  * [23] “B785 reinforcement mesh, https://www.reinforcementproductsonline.co.uk/b785-standard-mesh-2.html, accessed: 23.02.2021.”
  * [24] S. Eidelman, K. Hayes, K. e. Olive, M. Aguilar-Benitez, C. Amsler, D. Asner, K. Babu, R. Barnett, J. Beringer, P. Burchat, et al., “Review of particle physics,” Physics Letters B, vol. 592, no. 1, 2004.
  * [25] S. Agostinelli, J. Allison, K. a. Amako, J. Apostolakis, H. Araujo, P. Arce, M. Asai, D. Axen, S. Banerjee, G. Barrand, et al., “Geant4 a simulation toolkit,” Nuclear instruments and methods in physics research section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 506, no. 3, pp. 250–303, 2003.
  * [26] C. Hagmann, D. Lange, and D. Wright, “Cosmic ray shower generator (cry) for monte carlo transport codes,” in Nuclear Science Symposium Conference Record, 2007. NSS’07. IEEE, vol. 2, pp. 1143–1146, IEEE, 2007.
  * [27] P. Baesso, D. Cussans, C. Thomay, and J. Velthuis, “Toward a rpc-based muon tomography system for cargo containers.,” Journal of Instrumentation, vol. 9, no. 10, p. C10041, 2014.
  * [28] P. Baesso, D. Cussans, C. Thomay, J. Velthuis, J. Burns, C. Steer, and S. Quillin, “A high resolution resistive plate chamber tracking system developed for cosmic ray muon tomography,” Journal of Instrumentation, vol. 8, no. 08, p. P08006, 2013.
  * [29] L. J. Schultz, Cosmic ray muon radiography.  PhD thesis, Portland State University Portland, OR, USA, 2003.
  * [30] L. J. Schultz, G. S. Blanpied, K. N. Borozdin, A. M. Fraser, N. W. Hengartner, A. V. Klimenko, C. L. Morris, C. Orum, and M. J. Sossong, “Statistical reconstruction for cosmic ray muon tomography,” IEEE transactions on Image Processing, vol. 16, no. 8, pp. 1985–1993, 2007.
  * [31] M. Stapleton, J. Burns, S. Quillin, and C. Steer, “Angle statistics reconstruction: a robust reconstruction algorithm for muon scattering tomography,” Journal of Instrumentation, vol. 9, no. 11, p. P11019, 2014\.
  * [32] C. Thomay, J. Velthuis, P. Baesso, D. Cussans, C. Steer, J. Burns, S. Quillin, and M. Stapleton, “A novel markov random field-based clustering algorithm to detect high-z objects with cosmic rays,” IEEE Transactions on Nuclear Science, vol. 62, no. 4, pp. 1837–1848, 2015.
  * [33] “Tvirtualfft class reference, https://root.cern/doc/master/classtvirtualfft.html, accessed: 10.12.2020.”