# Experiments of ASR-based mispronunciation detection for children and adult
English learners

###### Abstract

Pronunciation is one of the fundamentals of language learning, and it is
considered a primary factor of spoken language when it comes to an
understanding and being understood by others. The persistent presence of high
error rates in speech recognition domains resulting from mispronunciations
motivates us to find alternative techniques for handling mispronunciations. In
this study, we develop a mispronunciation assessment system that checks the
pronunciation of non-native English speakers, identifies the commonly
mispronounced phonemes of Italian learners of English, and presents an
evaluation of the non-native pronunciation observed in phonetically annotated
speech corpora. In this work, to detect mispronunciations, we used a phone-
based ASR implemented using Kaldi. We used two non-native English labeled
corpora;  (i) a corpus of Italian adults contains 5,867 utterances from 46
speakers  (ii) a corpus of Italian children consists of 5,268 utterances from
78 children.  Our results show that the selected error model can discriminate
correct sounds from incorrect sounds in both native and nonnative speech, and
therefore can be used to detect pronunciation errors in non-native speech. The
phone error rates show improvement in using the error language model. The ASR
system shows better accuracy after applying the error model on our selected
corpora.

Index Terms: ASR, l2 learners, detection of pronunciation errors, computer-
assisted pronunciation training (CAPT)

##  1 Introduction

The number of people who are learning a second language (L2) worldwide is
increasing. Consequently, the need to evaluate and grade their pronunciation
is becoming an important topic. The most challenging part of learning a new
language is the pronunciation part due to the fact that it is challenging to
imitate sounds that are different to those of the native language's phoneme
inventory [1, 2, 3]. [4] defines mispronunciation as surface pronunciation
forms differing from canonical pronunciation forms. Phoneme level
mispronunciation refers to the interference of a second language learner's
native language during speech production, where foreign sounds are produced
similar to a phoneme in their native language. Most of the classroom's
pronunciation activities rely on the teacher to monitor, evaluate, and provide
feedback on student pronunciation. This traditional technique does not seem
adequate to correct student pronunciation, and it tends to be costly and time-
consuming.

Given these constraints, the growing tendency to assess non-native languages
leads to increased interest in automatic proficiency assessment of speech and
boosts Computer Assisted Language Learning (CALL) tools in the field of
language teaching for L2 learners [5, 6, 7]. CALL tools are designed to
recognize words or sentences uttered by L2 learners by using an ASR system.
The CAPT system is one of CALL's essential tools designed for automatically
evaluating and detecting the learner's pronunciation errors. In the CAPT
system, pronunciation evaluation can happen at two levels [3]:  (i) detecting
specific pronunciation errors  (ii) an overall assessment of a speaker's
proficiency (i.e., goodness of pronunciation (GoP) [8, 9, 10]).

This study seeks to examine the use of ASR to empower learners to practice and
improve their pronunciation on their own. Using a well-designed ASR system
allows students to work autonomously while also offering flexibility to
improve their pronunciation. The persistent presence of high error rates in
speech recognition domains resulting from mispronunciations motivates us to
find alternative techniques for handling mispronunciations. We use two corpora
(for more information, see section 4) of both children and adult Italian
speakers. These corpora contain both spontaneous and read-aloud tasks.

The innovation of our system is to use error rules in our language model
(error language model) based on the most common errors that were seen in L2
learners' speech, in order to improve our detection system (see section 3).
Error detection of mispronounced input is similar to the process in which an
annotator manually indicates pronunciation errors. Mainly, the topic of
interest in this study is the mispronunciation assessment at the phone level.
Ideally, our system should be able to detect errors the same as human
annotators do.

##  2 Related Work

Segmental features have been a subject of phonetics from the 1950s up to now
[11]. The most common segmental features for investigating the pronunciation
of L2 learners on automatic speech assessment are consonant features: stop
closure duration, aspiration, and vowel features: vowel duration. In one of
the first studies on the assessment of pronunciation at a phone level, [12]
found that the scoring accuracy for the assessment of errors was 80-92$\%$ for
non-native English speakers (73 speakers).

Early research by [13] used two kinds of acoustic models to conduct automatic
mispronunciation detection:  (i) a model trained on native-speaker
pronunciation  (ii) a model trained on non-native speech.  They used an
acoustic model to calculate the log probability for each predicted phone in
both models. They then calculated the difference between these two probability
scores and used it as a metric for rating the pronunciation's quality. The
result showed that the log-likelihood ratioLLR had a better performance than
the log-posteriors method. According to [14], posterior probabilities and log-
likelihood scores were the methods that were most correlated with word and
phone level human assessments of pronunciation.

As an example of the earliest work in this field, we can refer to the FLUENCY
system [15] to detect pronunciation problems at both the phonetic and the
prosodic levels (i.e., segmental and suprasegmental levels). They used the
SPHINX-II speech recognizer to evaluate and detect phone errors and prosodic
information (namely prosodic problems) of non-native speakers of French,
German, Hebrew, Hindi, Italian, Mandarin, Portuguese, Russian, and Spanish,
who learned English as their second language. It was reported that using ASR
technology while learning a foreign language can reduce embarrassment and
enhance learning for learners.

[16] investigated three ASR systems for nativeness evaluation in their study:
a GMM-HMM system, a DNN-HMM system, and a GMM-HMM system using DNN for feature
extraction. The feature sets were categorized into fluency, rhythm,
pronunciation, grammar, and vocabulary for segmental (at the consonant and the
vowel level) and suprasegmental measurements of non-native speech in automatic
non-native speech assessment. We will base our work on these features in our
ASR system.

The above studies are a small set of examples that are related to segmental
features. We focus on segmental features, in which we observe the specific
phoneme-level errors made by second language learners. For learners, the most
challenging part of learning a new language is the pronunciation part, due to
the fact that it is challenging to imitate sounds that are different to those
of the native language's phoneme inventory [3]. Using a well-designed ASR
system allows students to work autonomously while also offering flexibility
that can lead to the improvement of their pronunciation.

##  3 Methods

In the following sections, we describe the methods (language model and
acoustic model) used to identify pronunciation errors made by Italian learners
of English, to develop an ASR system to improve their L2 pronunciation.

###  3.1 Speech file transcriptions: Ideal, Manual, & ASR

For this study, the three transcriptions at the phone level were considered:
Ideal, Manual, and ASR output. All the transcriptions are automatically time-
aligned to the related speech signals:

  * •

Ideal (IDE) (reference) refers to the canonical word's pronunciation.

  * •

Manual (MAN) (reference) is the hand-corrected version of the ideal data at
the phoneme level. Trained annotators corrected files manually using Praat
tool [17], noting substitution, insertion, and deletion at the phoneme level.

  * •

ASR output is the results of ASR processing given a set of phones as input
(see section 3.2.1).

Given these three representations (i.e., IDE vs. MAN, IDE & ASR, and MAN &
ASR), the comparison between specific pairs of phone sequences can provide us
the following information:

IDE vs. MAN output: this gives us a true map of the errors made by Italian
speakers when trying to speak English. The comparison of IDE & MAN could be
used to build some models of the errors made by L2 speakers.; IDE vs. ASR
output: this gives us a feeling of what an automatic system can detect for a
new utterance, where no manual annotation is available; MAN vs. ASR output:
this tells us how well an automatic system detects errors. The
mispronunciation at the segmental level (i.e., phonetic) was categorized into
three kinds at the phone level: substitution (i.e., a phoneme is replaced with
another), insertion (i.e., an extra phoneme is inserted), and deletion (a
certain phoneme is deleted) [3].

###  3.2 Proposed approach

####  3.2.1 Language model

The basic idea of language models is to provide a probability of a sentence or
sequence of words; these probabilities are combined with the acoustic
likelihood of the sequence and generate the resulting hypothesis [18]. In this
work phone n-grams are used as a language model to allow the generation of
phone sequences not constrained by only the words appearing in the lexicon;
$n$-grams are trained on the phonetic transcription of the audio data, with
$n$ ranging from 1 to 5.

Apart from the n-gram model, another novel approach is the application of an
error language model based on the most frequent errors (hereafter error
model). For the error model, we used lexical information by providing the ASR
with the canonical pronunciations of the (known) words to be recognized and
other pronunciations obtained by applying phonetic rules. These rules were
defined manually by looking at the most common errors resulting from the
5-gram phone model (Table 1).

####  3.2.2 Acoustic Model

The acoustic model is learned from a set of audio recordings and their
corresponding transcripts. We trained hybrid GMM-DNN models on English and
Italian speech data from the child and adult Italian speakers. Based on the
common Kaldi recipe 111Kaldi: http://kaldi-asr.org/doc/, the selected acoustic
models' features are:  (i) initial GMM models trained on MFCC acoustic
features with LDA transform and speaker adaptive training [19] (ii) use of
i-vectors [20] of size 100, stacked to the MFCCs  (iii) TDNNs trained using
LF-MMI [21] .

##  4 Data

The description of each corpus will be given below;  (i) a corpus of Italian
children (ChildEn) 222This corpus was designed and collected by ITC-irst:
http://www.itc.it [22] (ii) corpus of Italian adults (ISLE) [23] , who are
learning English as their second language. Detecting mispronunciations in ASR
requires corpora with labeling at the phonetic level: we selected these two
corpora because they were manually labeled in terms of pronunciation quality
by humans, and both were manually transcribed at phone-level.

ChildEn consists of 5,268 utterances, with an overall duration of 3h:28m:26s
from 78 children at about ten years of age. The selected students had been
studying English at school for 3 or 4 years.

ISLE is a non-native speech dataset that contains utterances recorded by
intermediate-level adult German and Italian learners of English. The audio
files are 17h:54m:44s in total. The Italian section contains 23 Italian
speakers.

###  4.1 Phone set

We created a phone list of each phoneme that includes canonical pronunciation
and every possible mispronunciation of English phones by L2 learners; in other
words, we have created a phoneme dictionary that contains both English and
Italian phones in order to be able to capture all possible mispronunciations.
According to [24], Italian and English share only 40% of their phones.
Therefore, we expect to see more phonological interference from Italian when
L2 learners need to pronounce the phones in English. Moreover, in Italian, the
relationship between spelling and pronunciation is straightforward compared to
English. For example, in Italian, the letter 'a' is pronounced /a/, but in
English, pronunciation and spelling are not strictly related to each other.
For example the letter 'u' is pronounced /u/, /V/ or // in
English333http://archive.is/zsxA.

###  4.2 Evaluation

According to Figure 1, the most common mispronunciations for Italian speakers
occur when the English target phone is not in the phoneme set of Italian
(e.g., English phoneme /ax/). These phones are not contrastive for Italian
speakers, leading to the speaker attempting to substitute the phonetically-
closest phone from the Italian phoneme set or delete that phone. By ``close",
we mean that the acoustic signal has similar formants in both languages or the
orthographic representation (spelling) is similar to spelling in Italian.

Figure 1: Phoneme errors distributions of ISLE (left graph) & ChildEn (Right
graph).

[5] provided a list of pronunciation errors that can be considered in ASR
studies: Phonemic deletion, phonemic insertion, and phonemic substitution. For
assessing the performance of a phoneme, the phone error rate (PER) will be
used. The phone-error rate calculated the log-likelihood of a predicted
phoneme given the acoustic signal [14, 25]. The PER takes into account the
errors related to phoneme substitutions (S), phoneme deletions (D), phoneme
insertions (I), and P stands for the number of phones. If there are P phones
in the reference transcript, and the ASR output has S, D, and I, then multiply
by 100:

$\mathrm{PER}=\frac{\mathrm{S}+\mathrm{D}+\mathrm{I}}{\mathrm{P}}*100$ (1)

##  5 Results

###  5.1 Detection output

In this study, we are concerned with identifying mispronounced phonemes by L2
learners and improving our ASR system; we are interested in the PER and
accuracy of our system.

We used Kaldi to perform speech recognition for phoneme error detection using
the dictionary of words and phonemes. The acoustic model was trained with
mixed speech data (i.e., children and adults), and the utterances were force-
aligned based on the newly adapted word transcription. In our ASR system, we
considered the following procedures in the acoustic model;  (i) speech
recognition based on the phone-level n-gram model  (ii) forced-alignment based
on the existing word transcriptions, including the transcriptions modified
using the mispronunciation data  (iii) GMM classifier on using the native and
the non-native acoustic model.  The following sections compare the output of
phoneme sequences to the gold standard, which leads us to identify
pronunciation errors.

####  5.1.1 N-gram output

The ASR was run on 1-gram, 2-gram, 3-gram, 4-gram, and 5-gram of phones. We
first trained the algorithm introduced on our data to obtain the baseline
values for the top frequent errors to determine the rules for the error model.

According to [18], the common n-gram models, when there is sufficient training
data, are trigram, 4-gram, or even 5-gram. We trained all the n-gram models,
but we saw that performance was much better with the 5-gram language model.
For the rest of the n-grams, we only report the PERs for ASR & MAN, ASR & IDE,
and IDE & MAN. The reason is that the 5-gram model captures more context than
other models, and thus we chose to report in full detail only the 5-gram
results based on the number of substitutions, insertions, and deletions. For
this reason, the 5-gram model was used for finding modified transcriptions.
The PERs of 5-gram in ISLE and ChildEn are 42% and 38% for ASR & MAN
comparison, respectively. As we have enough data for training, we considered
the output of 5-gram models to develop our error model rules. The ASR systems'
performance that used n-gram models was low; the PER for each n-gram model was
between 33-52%.

####  5.1.2 Error model output

Thus far, we have tried to improve the PER to estimate the ASR system's
performance for mispronunciation recognition of L2 learners of English. The
second model used the predefined rules (Table 1) from the n-gram model to
train native and non-native speech data to check if the output of our ASR
system will be improved compared to the n-gram model; we applied the forced-
alignment by using the adapted lexicon.

The comparison of our models' performance (n-gram model and error model) was
made by calculating PER. We choose the best model based on the PER metric.
Interestingly, the new ASR system performs better in terms of PER for ChildEn,
and the error rules showed better improvement for the ASR system. The speech
recognition phone error rates are typically greater for adults than children.
The low improvement in ISLE might be due to more complex and long sentences
used in this corpus that lead to error propagation differently. The other
possible reason might be due to the recordings' quality (i.e., noise in the
background).

Overall, the highest Average Accuracy (see Eq. 1) to date was obtained using
the error model. The accuracy results range from 72%-76% for correct error
detection of phones in ChildEn (Figure 2).

1\. ISLE

2\. ChildEn Figure 2: Overall detection accuracy for ISLE (top graph). &
ChildIt (bottom graph)

##  6 Discussion

Focusing on the smaller units would allow students to focus more on specific
aspects of their pronunciation. However, the evaluation of smaller units for
pronunciation assessment has a higher uncertainty compared to the evaluation
of longer units [10]. For pronunciation training, especially for L2 learners,
automatic pronunciation evaluation can play an important role, for example, by
using ASR systems to evaluate the pronunciation of non-native input and
quantify how close the speech is to a native-like pronunciation.

One of the challenges in using non-native speech for automatic recognition is
the diversity of allophones, accents, invented words by L2 learners, and
longer hesitations. We identified high-priority phones (i.e., the phones used
for creating the error rules (see Table 1) and evaluated our system based on
them. Since achieving a perfect native-like pronunciation is an unrealistic
goal for adult learners, we focused on the specific mispronounced phones
summarized in Table 1. As an example:

  * •

dh $\rightarrow$ dh/d: ``dh'' can be replaced by itself or ``d''.

  * •

n d $\rightarrow$ n d/: ``d'' can be deleted if it is in the sequence of ``n
d".

  * •

er $\rightarrow$ er r/R/: means that our adapted lexicon1 will have three
transcriptions for the word “HER” (i.e., ``hh er" (ideal phonetic
transcription), ``hh er" R, & ``hh er" r.

Table 1: Error rules: Substitutions, Deletions, & Insertions

Substitution rules | Deletion rules | Insertion rules
---|---|---
Original | Sub | Original | Del | Original | Insert
ih – | ih/i/iy | n d – | n d/ | er – | er r/R/
dh – | dh/d | l d – | l d/ | aa – | aa r/R/
ax – | ax/a/o/oh | th d – | th d/ | ao – | ao r/R/
z – | z/s | s t – | s t/ | p l – | p ax/o l
r – | r/R | n t – | n t/ | b l – | b ax/o l
uh – | uh/u | ay k – | ay k/ | k l – | k ax l
|  | ae k – | ae k | g l – | g ax l

This model was trained on the phone errors observed in our corpora. We created
a set of rules to consider these errors in our corpora and implemented our
system based on the adapted lexicon. From the output of our ASR system, it is
clear that the selected architecture is more capable of detecting
mispronounced phones. In fact, our ASR systems' outputs showed that by
applying n-gram and error models, the model proposed in this work obtained the
lowest PER and better accuracy for both ISLE and ChildEn. In other words, in
our system, the model that produced the best PER is the error language model
with a PER of 23%, which is better than the rate in previous studies.

Previous works in speech recognition report that word error rate (WER) for
human annotators on native data is around 5% in comparison to WER on non-
native speech data, which is as high as 30% for automatic annotations [8]. Our
ASR system may not perform better than this human-level performance from human
annotators in detecting mispronounced phonemes in non-speech data. The best
recent performance report on WER comes from an ETS project, in which the
authors trained an ASR system on 800 hours of speech data, and the reported
WER was 28.5% [26]. By considering the error language model, we can see that
the ASR system performs better at recognizing the phones. We can see that our
ASR system selected the transcription defined in the new adapted lexicon based
on our error model.

Improving ASR will provide more speaking possibilities for learners to learn a
new language. Using ASR can improve the traditional class to a more learner-
centered environment with less anxiety. Apart from the segmental aspect, the
suprasegmental aspect also plays a role in pronunciation training, but the ASR
for the suprasegmental part is less reliable and still needs more work to
improve it.

##  7 Conclusions

In this study, we worked on the language model - at the phone level - to
extract the mispronounced phones and trained an acoustic model of native and
non-native training examples to detect mispronunciations. We compared the PER
of our ASR system using native and non-native speech data to determine our
error model hypothesis's validity. Our system's innovation uses error rules in
our language model (error model) based on the most common errors seen in L2
learners' speech to improve our detection system. Our error rules were defined
based on the common errors of Italian speakers who learn English as their
second language. The error detection system records the learners’ utterances,
and the ASR-based detector will provide the phone-level transcription. As the
final step, a list of possible feedbacks can be given to L2 learners based on
their pronunciation errors. Our ASR system needs to be adapted to the mother
tongue of the L2 learners since different L1s can cause different
pronunciation errors.

## References

  * [1] A. Huensch, ``The pronunciation teaching practices of university-level graduate teaching assistants of French and Spanish introductory language courses,'' _Foreign Language Annals_ , vol. 52, no. 1, pp. 13–31, 2019.
  * [2] S. M. McCrocklin, ``Pronunciation learner autonomy: The potential of Automatic Speech Recognition,'' _System_ , vol. 57, no. February, pp. 25–42, 2016\.
  * [3] M. A. Peabody, ``Methods for pronunciation assessment in computer aided language learning,'' Ph.D. dissertation, Massachusetts Institute of Technology, 2011.
  * [4] G. Fant, _Speech sounds and features._ The MIT Press, 1973.
  * [5] S. M. Witt, ``Automatic error detection in pronunciation training: Where we are and where we need to go,'' _Proceedings of the International Symposium on Automatic Detection of Errors in Pronunciation Training (IS ADEPT)_ , vol. 6, pp. 1–8, 2012.
  * [6] A. Neri, C. Cucchiarini, and W. Strik, ``Automatic speech recognition for second language learning: how and why it actually works.'' in _In Proc. ICPhS_ , 2003, pp. 1157–1160.
  * [7] G. Kawai and K. Hirose, ``A call system using speech recognition to teach the pronunciation of japanese tokushuhaku,'' in _STiLL-Speech Technology in Language Learning_ , 1998.
  * [8] K. Zechner, ``What did they actually say? Agreement and disagreement among transcribers of non-native spontaneous speech responses in an English proficiency test,'' in _International Workshop on Speech and Language Technology in Education_ , 2009.
  * [9] R. Downey, H. Farhady, R. Present-Thomas, M. Suzuki, and A. Van Moere, ``Evaluation of the usefulness of the versant for english test: A response,'' _Language Assessment Quarterly_ , vol. 5, no. 2, pp. 160–167, 2008.
  * [10] S. Witt and S. Young, ``Phone-level pronunciation scoring and assessment for interactive language learning,'' _Speech Communication_ , vol. 30, no. 2-3, pp. 95–108, feb 2000.
  * [11] H. Ryu, H. Hong, S. Kim, and M. Chung, ``Automatic pronunciation assessment of Korean spoken by L2 learners using best feature set selection,'' _2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2016_ , 2017.
  * [12] S. M. Witt, ``Use of speech recognition in computer-assisted language learning,'' Ph.D. dissertation, University of Cambridge Cambridge, United Kingdom, 1999.
  * [13] O. Franco, Horacio and Neumeyer, Leonardo and Kim, Yoon and Ronen, ``Automatic pronunciation scoring for language instruction organization,'' in _1997 IEEE international conference on acoustics, speech, and signal processing_. IEEE, 1997, pp. 1471–1474.
  * [14] L. Kim, Yoon and Franco, Horacio and Neumeyer, ``Automatic pronunciation scoring of specific phone segments for language instruction,'' in _Fifth European Conference on Speech Communication and Technology_ , 1997\.
  * [15] M. Eskenazi, ``Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype,'' _Language Learning and Technology_ , vol. 2, no. 2, pp. 62–76, 1999.
  * [16] J. Tao, S. Ghaffarzadegan, L. Chen, and K. Zechner, ``Exploring deep learning architectures for automatically grading non-native spontaneous speech,'' in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 6140–6144.
  * [17] P. Boersma and D. Weenink, ``Praat: Doing phonetics by computer (Version 6.0. 14),'' _Retrieved from (last access: 29.04. 2018)_ , 2016.
  * [18] D. Jurafsky and J. H. Martin, ``N-gram language models,'' _Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition_ , 2018.
  * [19] F. Miao, Yajie and Zhang, Hao and Metze, ``Speaker adaptive training of deep neural network acoustic models using i-vectors,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 11, pp. 1938–1949, 2015.
  * [20] M. Madikeri, Srikanth and Dey, Subhadeep and Motlicek, Petr and Ferras, ``Implementation of the standard i-vector system for the kaldi speech recognition toolkit,'' Idiap, Tech. Rep., 2016.
  * [21] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, ``Purely sequence-trained neural networks for asr based on lattice-free mmi.'' in _Interspeech_ , 2016, pp. 2751–2755.
  * [22] A. Batliner, M. Blomberg, S. D'Arcy, D. Elenius, D. Giuliani, M. Gerosa, C. Hacker, M. Russell, S. Steidl, and M. Wong, ``The PF_STAR children's speech corpus,'' in _Ninth European Conference on Speech Communication and Technology_ , 2005.
  * [23] W. Menzel, E. Atwell, P. Bonaventura, D. Herron, P. Howarth, R. Morton, and C. Souter, ``The ISLE corpus of non-native spoken English,'' in _2nd International Conference on Language Resources and Evaluation, LREC 2000_ , vol. 2. European Language Resources Association, 2000, pp. 957–964.
  * [24] S. Browning, ``Analysis of Italian children's English pronunciation,'' _Report contributed to the EU FP5 PF STAR Project_ , 2004.
  * [25] K. Kawai, Goh and Hirose, ``A CALL system using speech recognition to teach the pronunciation of Japanese tokushuhaku,'' in _STiLL-Speech Technology in Language Learning_ , 1998.
  * [26] L. Chen, K. Zechner, S. Y. Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, ``Automated Scoring of Nonnative Speech Using the SpeechRaterSM v. 5.0 Engine,'' _ETS Research Report Series_ , vol. 2018, no. 1, pp. 1–31, dec 2018.


††thanks:

# Super-resolution reconstruction of turbulent flow at various Reynolds
numbers based on generative adversarial networks

Mustafa Z. Yousif  Linqi Yu  Hee-Chang Lim  Corresponding author,
<EMAIL_ADDRESS>School of Mechanical Engineering, Pusan National University,
2, Busandaehak-ro 63beon-gil, Geumjeong-gu, Busan, 46241, Rep. of KOREA

###### Abstract

This study presents a deep learning-based framework to reconstruct high-
resolution turbulent velocity fields from extremely low-resolution data at
various Reynolds numbers using the concept of generative adversarial networks
(GANs). A multiscale enhanced super-resolution generative adversarial network
(MS-ESRGAN) is applied as a model to reconstruct the high-resolution velocity
fields, and direct numerical simulation (DNS) data of turbulent channel flow
with large longitudinal ribs at various Reynolds numbers are used to evaluate
the performance of the model. The model is found to have the capacity to
accurately reproduce high-resolution velocity fields from data at two
different low-resolution levels in terms of the quantities of velocity fields
and turbulent statistics. The results further reveal that the model is able to
reconstruct velocity fields at Reynolds numbers that are not used in the
training process.

##  I Introduction

Turbulence has a chaotic nature with multiple spatio-temporal flow scales,
making experimental measurements and numerical simulations unpredictable and
sophisticated and requiring a considerable cost to describe the flow structure
effectively. At the same time, an abundance of high-fidelity data can be
generated using state-of-the-art particle image velocimetry (PIV) and direct
numerical simulation (DNS). This opens the door to the discovery of new data-
driven methods able to make use of the data and the construction of models
that can tackle different problems in the research area of turbulent flows.
Machine learning has demonstrated a strong ability to tackle highly non-linear
problems in various fields. Further, in recent years, many deep learning
algorithms have been applied practically in the field of fluid dynamics [1,
2]. Deep learning is a subset of machine learning in which deep neural
networks are used for classification, prediction and feature extraction [3].
Various deep learning-based approaches have been recently applied to deal with
various turbulence problems, including turbulence modelling [4, 5, 6, 7],
turbulent flow prediction [8, 9] and turbulent flow control [10, 11]. In terms
of the super-resolution reconstruction of turbulent flow fields, deep
learning-based high-resolution image reconstruction has exhibited great
potential for practical application in the reconstruction of high-fidelity
turbulent flows. This can be attributed to the recent development in
supervised and unsupervised deep learning algorithms that significantly
outperform traditional handcrafted super-resolution techniques, such as
bicubic interpolation [12, 13]. Motivated by the remarkable ability of
convolutional neural networks (CNNs) to tackle the spatial mapping of data,
Fukami et al. [14, 15] proposed models based on a CNN for the spatial and
spatio-temporal super-resolution reconstruction of turbulent flows. Their
results revealed satisfactory outcomes in the reconstruction of velocity and
vorticity fields from extremely low-resolution data and a good prediction of
the temporal evolution for the intervals the model was trained for.

Liu et al. [16] applied a CNN-based model to reconstruct high-resolution
turbulent flow data from low-resolution data with multiple temporal paths and
reported that their model had better reconstruction performance compared with
static CNN-based models. Kim et al. [17] applied a cycle-consistent generative
adversarial network (CycleGAN) [18] to reconstruct flow fields from low-
resolution DNS and LES data as an unsupervised deep learning technique to
reconstruct turbulence with high resolution using unpaired training data.
Their results revealed that the method had better reconstruction accuracy
compared with a bicubic interpolation and CNN-based model.

Cai et al. [19] presented a model based on CNN for estimating high-resolution
velocity fields from PIV measurements and reported that their model had better
performance than conventional cross-correlation algorithms. Further, Morimoto
et al. [20] applied a CNN-based model to estimate velocity fields using PIV
measurements of flow around a square cylinder with missing regions. Their
results revealed a relatively good estimation of the missing regions in the
velocity fields. Deng et al. [21] applied a super-resolution generative
adversarial network (SRGAN) [22] and enhanced SRGAN (ESRGAN) [7] to
reconstruct high-resolution velocity fields using PIV measurements of flow
around a cylinder. Their results demonstrated good agreement with the ground
truth data, and ESRGAN was found to have better reconstruction capability
compared with SRGAN. Recently, Yousif et al. [23] proposed multiscale ESRGAN
(MS-ESRGAN) with a physics-based loss function to reconstruct high-fidelity
turbulent flow fields from spatially limited data. They reported that the
model has a strong ability to reconstruct high-resolution turbulence even from
extremely coarse data.

In all the aforementioned studies, the models were trained to reconstruct
high-resolution flow fields at a specific Reynolds number. However, most
turbulent flow problems, in which the flow parameters can be changed spatially
and temporally, result in various Reynolds numbers.

In this study, we consider the reconstruction of high-resolution turbulent
velocity fields at certain salient Reynolds numbers. In particular, we apply
MS-ESRGAN to reconstruct high-resolution velocity fields at different Reynolds
numbers from extremely low-resolution data. To evaluate the performance of MS-
ESGAN, DNS calculations of turbulent channel flow with large longitudinal ribs
are performed at different Reynolds numbers.

The remainder of this paper is organised as follows. Section 2 explains the
methodology for reconstructing high-resolution flow fields using the proposed
deep learning model. Section 3 describes the generation of the training data
using DNS. Section 4 discusses the testing results obtained using the proposed
model. Finally, Section 5 presents the conclusions of the study.

##  II Methodology

Goodfellow et al. [24] first introduced the concept of the generative
adversarial networks (GANs). Different types of GANs have since been proposed
to tackle various image transformation and super-resolution problems [22, 25,
7, 18]. In GAN, two adversarial neural networks, called the generator ($G$)
and the discriminator ($D$), compete with each other. In particular, $G$
generates fake images similar to real images, and $D$ tries to distinguish the
fake ones from the real ones. The goal of the training process is to train $G$
to generate fake images that are difficult for $D$ to distinguish. This
process can be expressed as a min-max two-player game with a value function
$V(D,G)$ such that:

$\begin{split}\begin{subarray}{c}min\\\
G\end{subarray}~{}\begin{subarray}{c}max\\\
D\end{subarray}~{}V(D,G)=\mathbb{E}_{x_{r}\sim P_{data}(x_{r})}[{\rm
log}D(x_{r})]+\mathbb{E}_{z\sim P_{z}(z)}[{\rm log}(1-D(G(z)))],\end{split}$
(1)

where $x_{r}$ is the image from the ground truth data and $P_{data}(x_{r})$ is
the distribution of the real image. $\mathbb{E}$ represents the operation used
to calculate the average of all the data in the training mini-batch. In the
term second to the right in Eq. 1, $z$ is a random vector used as an input to
$G$, and $D(x_{r})$ represents the probability that the image is real and not
generated by the generator. $G(z)$ is the output from $G$ and is expected to
generate an image that is similar to the real image, such that the value of
$D(G(z))$ becomes as large as possible. At the same time, $D$ attempts to
increase the value of $D(x_{r})$ and decrease the value of $D(G(z))$. Thus,
during the training process, $G$ is trained to minimise $V(D,G)$, and $D$ is
trained to maximise $V(D,G)$. After successful training, $G$ is expected to
produce an image with a distribution similar to the real image that $D$ cannot
distinguish from the real image.

Figure 1: MS-ESRGAN architecture: (a) the generator ($\beta=0.2$ is the
residual scaling parameter), and (b) the discriminator

This study uses a deep learning framework based on ESRGAN [7] to reconstruct
high-resolution flow fields from extremely low-resolution data. We applied the
newly developed MS-ESRGAN model [23] to reconstruct the high-resolution
velocity fields at different Reynolds numbers. Figure 1(a) and (b) shows the
basic architectures of $G$ and $D$ in the MS-ESRGAN, respectively. $G$
consists of a deep convolution neural network represented by residuals in
residual dense blocks (RRDBs) and multiscale part (MSP) . The low-resolution
input image is first fed to $G$ and passed through a convolution layer and
then through a series of RRDBs. MSP, consisting of three parallel
convolutional sub-models with different kernel sizes, is applied to the data
features that the RRDBs extract, as explained in Table 1. Finally, the outputs
of the three branches are summed and passed through a final convolutional
layer to generate a high-resolution fake image $(x_{f})$. The fake and real
images are fed to $D$ and passed through a series of convolutional, batch
normalisation (BN) and leaky rectified linear unit (ReLU) layers. The data are
then passed through a final convolutional layer. The non-transformed
discriminator outputs using the real and fake images $C(x_{r})$ and $C(x_{f})$
are used to calculate the relativistic average discriminator value ($D_{Ra}$)
[26] as follows:

$D_{Ra}(x_{r},x_{f})=\sigma(C\left(x_{r})\right)-\mathbb{E}_{x_{f}}\left[C(x_{f})\right],$
(2)

$D_{Ra}(x_{f},x_{r})=\sigma(C\left(x_{f})\right)-\mathbb{E}_{x_{r}}\left[C(x_{r})\right],$
(3)

where $\sigma$ is the sigmoid function. In Eqs. 2 and 3, $D_{Ra}$ represents
the probability that the output from $D$ using the real image is more
realistic than the output using the fake image.

The discriminator loss is then defined as:

$L_{D}^{Ra}=-\mathbb{E}_{x_{r}}\left[{\rm
log}(D_{Ra}(x_{r},x_{f}))\right]-\mathbb{E}_{x_{f}}\left[{\rm
log}(1-D_{Ra}(x_{f},x_{r}))\right].$ (4)

The adversarial loss of the generator can be expressed in a symmetrical form
as:

$L_{G}^{Ra}=-\mathbb{E}_{x_{r}}\left[{\rm
log}(1-D_{Ra}(x_{r},x_{f}))\right]-\mathbb{E}_{x_{f}}\left[{\rm
log}(D_{Ra}(x_{f},x_{r}))\right].$ (5)

The combined loss function of the generator is defined as:

$\mathcal{L}_{G}=\lambda_{1}L_{G}^{Ra}+\lambda_{2}L_{pixel}+\lambda_{3}L_{perceptual},$
(6)

where $L_{pixel}$ is the error calculated based on the pixel difference
between the generated data and the data obtained from the ground truth data.
$L_{perceptual}$ represents the difference in the features extracted from the
real and fake data. The pre-trained CNN VGG-19 [27] is employed to extract the
features by using the output of three different layers[23]. Here,
$\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the coefficients used to
balance the loss terms whose values are empirically set to be 2, 2,000 and 4,
respectively. In this study, the mean squared error (MSE) is used to calculate
$L_{pixel}$ and $L_{perceptual}$. The adaptive moment estimation (ADAM)
optimisation algorithm [28] is applied to update the model weights. The
training data are divided into mini-batches, and the size of each mini-batch
is set to 16.

Table 1: MSP architecture

$1^{st}$ branch |  $2^{nd}$ branch |  $3^{rd}$ branch
---|---|---
Conv.(3, 3) |  Conv.(5, 5) |  Conv.(7, 7)
UpSampling(2, 2) |  UpSampling(2, 2) |  UpSampling(2, 2)
Conv.(3, 3) |  Conv.(5, 5) |  Conv.(7, 7)
LeakyReLU |  LeakyReLU |  LeakyReLU
UpSampling(2, 2) |  UpSampling(2, 2) |  UpSampling(2, 2)
Conv.(3, 3) |  Conv.(5, 5) |  Conv.(7, 7)
LeakyReLU |  LeakyReLU |  LeakyReLU
UpSampling(2, 2) |  UpSampling(2, 2) |  UpSampling(2, 2)
Conv.(3, 3) |  Conv.(5, 5) |  Conv.(7, 7)
LeakyReLU |  LeakyReLU |  LeakyReLU
Add ($1^{st}$ branch, $2^{nd}$ branch, $3^{rd}$ branch)

##  III Training data generation

In this study, a fully developed turbulent channel flow with large
longitudinal ribs is used as a test case. This case has been widely studied
experimentally and numerically for a wide range of Reynolds numbers [29, 30,
31, 32, 33], making it suitable for assessing our model. To generate the data,
the DNS calculation is performed. Figure 2 shows the schematic of the
computational domain used in this study. The domain contained four large
longitudinal ribs along the streamwise direction ($x$) and equally distributed
in the spanwise direction ($y$). The domain dimensions are $24h$, $32h$ and
$10h$ in the $x,y$ and the ground-normal ($z$) directions, respectively. Here,
$h$ is the height of each rib, which is set as 1. The size of each rib is
$24h\times 2h\times h$. A section normal to the streamwise direction that
contains one of the ribs is used to reconstruct the high-resolution velocity
fields.

Figure 2: Schematic of the computational domain

The momentum equation for an incompressible viscous fluid can be expressed as:

$\frac{\partial{\bf u}}{\partial t}+{\bf u}\cdot\nabla{\bf
u}=-\frac{1}{\rho}\nabla p+{\it\nu}\nabla^{2}{\bf u},$ (7)

where ${\bf u}$ is the velocity of the fluid, $\rho$ is the density, $p$ is
the pressure and $\nu$ is the kinematic viscosity. The continuity equation of
the fluid is defined as:

$\nabla\cdot{\bf u}=0.$ (8)

A version of the in-house finite-difference code CGLES, originally written by
Thomas and Williams [34], is used to perform the DNS calculation. The friction
Reynolds numbers used for training in this study ($Re_{\tau}=u_{\tau}H/\nu$)
are set to 292, 450 and 850, respectively, where $u_{\tau}$ is the friction
velocity and $H$ is the channel height. Cartesian uniform meshes are used in
all directions, and the second-order central differencing scheme is applied to
all spatial derivatives. The Adams-Bashforth scheme is employed for time
advancement using the pressure projection method. The continuity is forced
implicitly by solving the Poisson equation for pressure using a parallel
multi-grid method. A no-slip condition is applied at the ground and the walls
of the ribs, and the periodic boundary condition is applied in the streamwise
and spanwise directions, while a free-slip condition is applied to the top of
the domain. The flow is driven by applying a pressure gradient.

To avoid any interpolation of the training data at the three Reynolds numbers
used in this study, the mesh size is fixed to $768\times 1024\times 320$, with
element dimension $\Delta x^{+}=\Delta y^{+}=\Delta z^{+}\approx 2.7$. The
time step $\Delta t^{+}$ is set as 0.21, where the superscript $+$ indicates
that the quantity is made dimensionless using the wall units $\nu$ and
$u_{\tau}$. The grid size is carefully chosen to ensure it is sufficient to
solve all the turbulent flow scales of the case with the highest $Re_{\tau}$,
i.e. 850. This could be achieved by estimating the smallest Kolmogorov length
scale $\eta$ in the flow. By assuming that the energy production roughly
balances the dissipation, we can write the following for the latter:
$\varepsilon=-\overline{u^{\prime}w^{\prime}}dU/dz$, where
$\overline{u^{\prime}w^{\prime}}$ is Reynolds shear stress; $u^{\prime}$ and
$w^{\prime}$ are the fluctuations of the streamwise and ground-normal velocity
components, respectively and $U$ is the mean streamwise velocity. It is found
that at $z\approx 2h$, the shear stress has a maximum value of approximately
$0.7u_{\tau}^{2}$, so that $\eta\approx$ $1.3u_{\tau}^{3}\textfractionsolidus
h$. Hence, $\eta\approx 0.0056h=0.18\Delta$, leading to $\Delta\approx
0.6\eta$. Moin & Mahesh [35] explain that the smallest length scale that must
be resolved is typically much larger than the Kolmogorov scale ($\sim 10\eta$
). Based on this fact, we can ensure that the present simulation setup
captures the entire dissipation spectrum for all the Reynolds numbers used in
this study. At the same time, the maximum Courant-Friedrichs-Lewy number (CFL)
is maintained at $<0.3$ to ensure the stability of the simulations.

The training data are obtained from each simulation with 10,000 snapshots from
a single ($y-z$) plane. The size of the plane is fixed as $y/h\times
z/h=8\times 10$, which is equivalent to a grid size of $256\times 320$. The
interval between the collected snapshots of the flow fields is ten times the
time step used in the simulations. The low-resolution training data are
obtained by downsampling the data from the DNS using max pooling. Two
downsampling levels are used in this study, $\times 8$ and $\times 16$, which
are equivalent to sections of sizes of $32\times 40$ and $16\times 20$,
respectively. To prepare the data for training, they are normalised using the
min–max normalisation function to produce values between 0 and 1.

##  IV Results and discussion

###  IV.1 Velocity fields reconstruction (Reynolds numbers are used in the
training process)

In this section, the capacity of the model to reconstruct high-resolution
velocity fields at the three Reynolds numbers used in the training process
($Re_{\tau}$ = 292, 550 and 850) from the downsampled data at two low-
resolution levels ($\times 8$ and $\times 16$) is thoroughly examined and
discussed. It is worth noting that all of the reconstructed velocity fields
are based on the test data that are not used in the training process.

Figures 3-5 show the reconstructed instantaneous fields of the streamwise
velocity ($\tilde{u}$), spanwise velocity ($\tilde{v}$) and ground-normal
velocity ($\tilde{w}$) at the three Reynolds numbers and two resolution levels
used in this study. Here, $\sim$ indicates that the quantity is normalised
using the min–max normalisation function. The reconstructed velocity fields
closely match the DNS data, as indicated in the figures. When the $\times 8$
resolution level is employed, there is no discernible change in the
reconstructed velocity fields. Nonetheless, there is a slight under-prediction
in several velocity fields regions when the $\times 16$ resolution level is
used. In addition, increasing the Reynolds number is found to have no
influence on the correctness of the reconstruction, with the instantaneous
velocity fields being successfully reconstructed at all three Reynolds
numbers.

Figure 6 shows the probability density function (PDF) of the reconstructed
velocity components. The figure reveals a reasonable level of agreement
between the PDF of the reconstructed velocity fields and that obtained from
the DNS data, even for the highest Reynolds number and the $\times 16$
resolution level, demonstrating the ability of the model to reconstruct high-
resolution velocity fields for all three Reynolds numbers and the two low-
resolution levels.

Figure 7 shows the root mean square (RMS) profiles of the reconstructed
velocity components ($\tilde{u}_{rms}$, $\tilde{v}_{rms}$ and
$\tilde{w}_{rms}$) averaged over 100,000 snapshots and along the upper wall of
the rib. It can be observed from the figure that the RMS values of all the
velocity components are in fair agreement with the DNS results when the
$\times 8$ data are used and for all three Reynolds numbers. However, a slight
deviation is observed for the case in which the $\times 16$ data are used.
This can be attributed to the limited information on the high-frequency
fluctuations of the velocity components when extremely low-resolution data are
used.

To estimate how exact it is possible to make the spatial correlations of the
reconstructed velocity components, the spanwise two-point correlation
($R_{ii}(\Delta y)$) of each of the reconstructed velocity components is
calculated along the upper wall of the rib at $z=2h$. Here, $i$ represents the
velocity component. As shown in Figure 8, the correlations are generally in
good agreement with the results from the DNS, with no reduction in the
accuracy caused by increasing the Reynolds number. Nevertheless, an offset is
observed when extremely low-resolution data are used, which can be attributed
to the insufficient spatial information on the velocity components. All the
previous results reveal that the model used in this study is able to
successfully reproduce high-resolution velocity fields with a commendable
accuracy not affected by increasing the Reynolds number. Nevertheless, they
also demonstrate that the reconstruction accuracy is slightly affected when
extremely low-resolution data are used as input to the model.

Figure 3: Reconstructed instantaneous velocity fields at $Re_{\tau}=292$
Figure 4: Reconstructed instantaneous velocity fields at $Re_{\tau}=550$
Figure 5: Reconstructed instantaneous velocity fields at $Re_{\tau}=850$
Figure 6: Probability density functions of the reconstructed velocity
components at (a) $Re_{\tau}=292$, (b) $Re_{\tau}=550$ and (c) $Re_{\tau}=850$
Figure 7: RMS profiles of the reconstructed velocity components at (a)
$Re_{\tau}=292$, (b) $Re_{\tau}=550$ and (c) $Re_{\tau}=850$ Figure 8: Two-
point correlations of the reconstructed velocity components at (a)
$Re_{\tau}=292$, (b) $Re_{\tau}=550$ and (c) $Re_{\tau}=850$

###  IV.2 Velocity fields reconstruction (Reynolds numbers are not used in the
training process)

The capability of the model is further examined using data of velocity fileds
at Reynolds numbers that are not used in the training process. Here, two
Reynolds numbers are used, $Re_{\tau}$ = 400 and 600 at two different low-
resolution levels ( $\times 8$ and $\times 16$). The test is used to
investigate the ability of the model to interpolate the reconstructed velocity
fields, since the used Reynolds numbers are in the range of those the model is
trained with.

As shown in Figures 9 and 10, the reconstructed instantaneous velocity fields
exhibit acceptable agreement with the DNS data for both tested Reynolds
numbers. The reconstructed velocity fields also exhibit a similar level of
precision when the training Reynolds numbers are used. In addition, increasing
the Reynolds number is found to have no influence on the accuracy of the
reconstruction.

As shown in Figure 11, the PDF of the reconstructed velocity components shows
no significant difference from that obtained from the DNS data. However, the
difference is more noticeable when the $\times 16$ low-resolution level is
used than when the training Reynolds numbers are used. As mentioned in the
previous section, this can be attributed to the effect of having limited
spatial information, which influenced the mapping ability of the model,
resulting in less-accurate interpolation in the regions of the small-scale
eddies.

The RMS profiles of the reconstructed velocity fields as plotted in Figure 12
show no noticeable difference with the DNS results when the $\times 8$ data
are used. However, similar to the case in which the training Reynolds numbers
are used, a slight under- and over-prediction can be seen in the RMS values in
the case when $\times 16$ data are used.

Figure 13 shows $R_{ii}(\Delta y)$ of the reconstructed velocity components.
The correlations reveal a good match with the DNS results, with no noticeable
reduction in accuracy for increasing Reynolds numbers. As expected, an offset
is observed when an extremely low-resolution is used for two of the Reynolds
numbers.

The results in this section demonstrate that the model is able to reconstruct
high-resolution velocity fields at Reynolds numbers falling within the range
of the training process with acceptable precision. Similar to the case in
which the training Reynolds numbers are used, the reconstruction precision
exhibit less sensitivity to increasing Reynolds number than it does to
decreasing resolution level.

Figure 9: Reconstructed instantaneous velocity fields at $Re_{\tau}=400$
Figure 10: Reconstructed instantaneous velocity fields at $Re_{\tau}=600$
Figure 11: Probability density functions of the reconstructed velocity
components at (a) $Re_{\tau}=400$ and (b) $Re_{\tau}=600$ Figure 12: RMS
profiles of the reconstructed velocity components at (a) $Re_{\tau}=400$ and
(b) $Re_{\tau}=600$ Figure 13: Two-point correlations of the reconstructed
velocity components at (a) $Re_{\tau}=400$ and (b) $Re_{\tau}=600$

###  IV.3 Reconstruction error

To further explore the performance of the model in terms of its reconstruction
accuracy, the reconstruction error is statistically analysed using root mean
square error (RMSE), as reported in Figure 14. As observed in Figure 14(a) and
(b), the RMSE values for the case in which the training Reynolds numbers are
used are proportional to the increase in the Reynolds number. As expected, the
RMSE values tend to increase with the decrease of the resolution level due to
the lack of information about the velocity fields when extremely low-
resolution data are used. It can also be observed that when the reconstructed
velocity fields are represented by Reynolds numbers not used in the training
process, the error values rapidly increase and further increase with
decreasing resolution level.

Figure 14: Root mean square error of the reconstructed velocity components
using two low-resolution levels: (a) $\times 8$ and (b) $\times 16$

Hence, the RMSE can be represented as:

${\rm RMSE}={\rm RMSE}_{\rm Reynolds~{}number}+{\rm RMSE}_{\rm
interpolation}+{\rm RMSE}_{\rm resolution},$ (9)

where ${\rm RMSE_{Reynolds~{}number}}$ is the error due to the increase of
$Re_{\tau}$, ${\rm RMSE_{interpolation}}$ is the error due to the
interpolation between the Reynolds numbers used in the training process to
obtain the velocity fields at the Reynolds numbers falling within the training
range, i.e. $Re_{\tau}$ = 292 –850 and ${\rm RMSE_{resolution}}$ is the error
representing the effect of the resolution level. Based on this decomposition
of the RMSE, we can see that the combined effect of ${\rm
RMSE_{interpolation}}$ and ${\rm RMSE_{resolution}}$ can increase the error
values by a factor of three. Nevertheless, since the maximum RMSE in this
study is on the order of $10^{-2}$, the MS-ESRGAN performance is entirely
acceptable when compared with the results obtained by Deng et al. [21]. These
results further indicate that MS-ESRGAN has remarkable potential for
reconstructing high-resolution turbulent velocity fields from extremely low-
resolution data at a wide range of Reynolds numbers not necessarily used in
the training process.

##  V Conclusions

This study presented a deep learning-based method to reconstruct high-
resolution turbulent velocity fields from extremely low-resolution data at
various Reynolds numbers. In particular, MS-ESRGAN was implemented to map the
low-resolution velocity fields to the high-resolution ones. Velocity data of
turbulent channel flow with large longitudinal ribs obtained from DNS were
then used to validate the model. Three different Reynolds numbers were used in
the training process.

First, the capability of the model to reconstruct high-resolution velocity
fields at the Reynolds numbers used in training based on two different low-
resolution levels was examined. The results revealed that the model could
successfully reproduce the high-resolution instantaneous velocity fields with
commendable accuracy, even for the lowest-resolution level used in this study.
Although the PDF, $R_{ii}(\Delta y)$ and RMS profiles did exhibit a slight
deviation when the lowest-resolution level was used, the results still had
acceptable accuracy.

The ability of the model to reconstruct velocity fields at Reynolds numbers
that were not used in the training process was then examined using two
different Reynolds numbers that fell within the training Reynolds numbers
range. Here, both the instantaneous velocity fields and statistical analysis
results matched the DNS data well, indicating that the model has great
potential to interpolate between velocity fields that lie within the training
Reynolds number range.

The maximum reconstruction error also exhibited acceptable values, indicating
MS-ESRGAN has remarkable ability to map extremely low-resolution velocity
fields at a wide range of Reynolds numbers to high-resolution ones with
commendable accuracy.

The results from this study indicate that GAN-based model, such as MS-SRGAN,
has the ability to map limited spatial turbulence data to high-fidelity data
using flow fields information represented by a few specified Reynolds numbers
that determine the minimum, intermediate and maximum values of the
interpolation.

###### Acknowledgements.

This work was supported by ’Human Resources Program in Energy Technology’ of
the Korea Institute of Energy Technology Evaluation and Planning (KETEP),
granted financial resource from the Ministry of Trade, Industry & Energy,
Republic of Korea (no. 20214000000140). In addition, this work was supported
by the National Research Foundation of Korea (NRF) grant funded by the Korea
government (MSIP) (no. 2019R1I1A3A01058576).

## Data Availability

The data that supports the findings of this study are available within this
article.

## References

  * Brunton, Noack, and Koumoutsakos [2020] S. L. Brunton, B. R. Noack, and P. Koumoutsakos, “Machine learning for fluid mechanics,” Annual Review of Fluid Mechanics 52(1), 477–508 (2020).
  * Kutz [2017] J. N. Kutz, “Deep learning in fluid dynamics,” Journal of Fluid Mechanics 814, 1–4 (2017).
  * LeCun, Bengio, and Hinton [2015] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521(7553), 436–444 (2015).
  * Duraisamy and Xiao [2019] K. Duraisamy and G. I. H. Xiao, “Turbulence modeling in the age of data,” Annual Review of Fluid Mechanics 51(1), 357–377 (2019).
  * Gamahara and Hattori [2017] M. Gamahara and Y. Hattori, “Searching for turbulence models by artificial neural network,” Physical Review Fluids 2(5), 054604 (2017).
  * Ling, Kurzawski, and Templeton [2016] J. Ling, A. Kurzawski, and J. Templeton, “Reynolds averaged turbulence modelling using deep neural networks with embedded invariance,” Journal of Fluid Mechanics 807, 155–166 (2016).
  * [7] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, C. C. Loy, Y. Qiao, and X. Tang, “Esrgan: Enhanced super-resolution generative adversarial networks,” ArXiv:1809.00219 [Cs]. http://arxiv.org/abs/1809.00219 .
  * Lee and You [2019] S. Lee and D. You, “Data-driven prediction of unsteady flow over a circular cylinder using deep learning,” Journal of Fluid Mechanics 879, 217–254 (2019).
  * Srinivasan  _et al._ [2019] P. A. Srinivasan, L. Guastoni, H. Azizpour, P. Schlatter, and R. Vinuesa, “Predictions of turbulent shear flows using deep neural networks,” Physical Review Fluids 4(5), 054603 (2019).
  * Fan  _et al._ [2020] D. Fan, L. Yang, Z. Wang, M. S. Triantafyllou, and G. E. Karniadakis, “Reinforcement learning for bluff body active flow control in experiments and simulations,” Proceedings of the National Academy of Sciences 117(42), 26091–26098. (2020).
  * Rabault  _et al._ [2019] J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi, “Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control,” Journal of Fluid Mechanics 865, 281–302 (2019).
  * Bashir  _et al._ [2021] S. M. A. Bashir, Y. Wang, M. Khan, and Y. Niu, “A comprehensive review of deep learning-based single image super-resolution,” PeerJ Computer Science 7, e621 (2021).
  * Keys [1981] R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Transactions on Acoustics, Speech, and Signal Processing 29(6), 1153–1160 (1981).
  * Fukami, Fukagata, and Taira [2019] K. Fukami, K. Fukagata, and K. Taira, “Super-resolution reconstruction of turbulent flows with machine learning,” Journal of Fluid Mechanics 870, 106–120 (2019).
  * Fukami, Fukagata, and Taira [2021] K. Fukami, K. Fukagata, and K. Taira, “Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows,” Journal of Fluid Mechanics 909 (2021).
  * Liu  _et al._ [2020] B. Liu, J. Tang, H. Huang, and X. Y. Lu, “Deep learning methods for super-resolution reconstruction of turbulent flows,” Physics of Fluids 32(2), 025105 (2020).
  * Kim  _et al._ [2021] H. Kim, J. Kim, S. Won, and C. Lee, “Unsupervised deep learning for super-resolution reconstruction of turbulence,” Journal of Fluid Mechanics 910 (2021).
  * Zhu  _et al._ [2017] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” IEEE International Conference on Computer Vision (ICCV) 2242–2251 (2017).
  * Cai  _et al._ [2019] S. Cai, S. Zhou, C. Xu, and Q. Gao, “Dense motion estimation of particale images via a convolutional neural network,” Experiments in Fluids 60, 73 (2019).
  * Morimoto, Fukami, and Fukagata [2021] M. Morimoto, K. Fukami, and K. Fukagata, “Experimental velocity data estimation for imperfect particle images using machine learning,” Physics of Fluids 33(8), 087121 (2021).
  * Deng  _et al._ [2019] Z. Deng, C. He, Y. Liu, and K. C. Kim, “Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework,” Physics of Fluids 31(12), 125111 (2019).
  * Ledig  _et al._ [2017] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejan, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” ArXiv:1609.04802 [Cs, Stat] (2017).
  * [23] M. Z. Yousif, L. Yu, and H. Lim, “High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network,” ArXiv:2109.04250 [Physics]. http://arxiv.org/abs/2109.04250 .
  * Goodfellow  _et al._ [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y.Bengio, “Generative adversarial nets,” Advances in Neural Information Processing Systems 27 (2014).
  * Mirza and Osindero [2014] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” https://arxiv.org/abs/1411.1784v1 (2014).
  * relativistic discriminator: A key element missing from standard GAN [2018] T. relativistic discriminator: A key element missing from standard GAN, “A synthetic-eddy-method for generating inflow conditions for large-eddy simulations.” ArXiv:1807.00734 [Cs, Stat] (2018).
  * K.Simonyan and Zisserman [2015] K.Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” ArXiv:1409.1556 [Cs] (2015).
  * Kingma and Ba [2017] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” ArXiv:1412.6980 [Cs] (2017).
  * Castro  _et al._ [2021] I. P. Castro, J. W. Kim, A. Stroh, and H. C. Lim, “Channel flow with large longitudinal ribs,” Journal of Fluid Mechanics 915 (2021).
  * Lee and Hwang [2018] J. H. Lee and H. G. Hwang, “Secondary flows in turbulent boundary layers over longitudinal surface roughness,” Physical Review Fluids 3(1), 014608 (2018).
  * Vanderwel  _et al._ [2019] C. Vanderwel, A. Stroh, J. Kriegseis, B. Frohnapfel, and B. Ganapathisubramani, “The instantaneous structure of secondary flows in turbulent boundary layers,” Journal of Fluid Mechanics 862, 545–870 (2019).
  * Vanderwel and Ganapathisubramani [2015] C. Vanderwel and B. Ganapathisubramani, “Effects of spanwise spacing on large-scale secondary flows in rough-wall turbulent boundary layers,” Journal of Fluid Mechanics 774 (2015).
  * Zampiron, Cameron, and Nikora [2020] A. Zampiron, S. Cameron, and V. Nikora, “Secondary currents and very-large-scale motions in open-channel flow over streamwise ridges,” Journal of Fluid Mechanics 887 (2020).
  * Thomas and Williams [1997] T. G. Thomas and J. J. R. Williams, “Development of a parallel code to simulate skewed flow over a bluff body,” Journal of Wind Engineering and Industrial Aerodynamics 67-78, 155–167 (1997).
  * Moin and Mahesh [1998] P. Moin and K. Mahesh, “Direct numerical simulation: A tool in turbulence research,” Annual Review of Fluid Mechanics 30(1), 539–578 (1998).
  * Wang, Wu, and Xiao [2017] J.-X. Wang, J.-L. Wu, and H. Xiao, “A physics informed machine learning approach for reconstructing reynolds stress modeling discrepancies based on dns data,” Physical Review Fluids 2(3), 034603 (2017).