# Testing gravitational redshift based on microwave frequency links onboard
China Space Station

Wenbin Shen<EMAIL_ADDRESS>School of Geodesy and Geomatics, Wuhan
University, Wuhan 430079, China.  Time and Frequency Geodesy Center, Wuhan
University, Wuhan 430079, China.  State Key Laboratory of Information
Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan
430079, China.  Pengfei Zhang  School of Geodesy and Geomatics, Wuhan
University, Wuhan 430079, China.  Time and Frequency Geodesy Center, Wuhan
University, Wuhan 430079, China.  Ziyu Shen<EMAIL_ADDRESS>School of
Resource, Environmental Science and Engineering, Hubei University of Science
and Technology, Xianning, Hubei, China.  Rui Xu  School of Geodesy and
Geomatics, Wuhan University, Wuhan 430079, China.  Time and Frequency Geodesy
Center, Wuhan University, Wuhan 430079, China.  Xiao Sun  State Key Laboratory
of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan
University, Wuhan 430079, China.  Mostafa Ashry  School of Geodesy and
Geomatics, Wuhan University, Wuhan 430079, China.  Time and Frequency Geodesy
Center, Wuhan University, Wuhan 430079, China.  State Key Laboratory of
Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan
University, Wuhan 430079, China.  Abdelrahim Ruby  School of Geodesy and
Geomatics, Wuhan University, Wuhan 430079, China.  Time and Frequency Geodesy
Center, Wuhan University, Wuhan 430079, China.  State Key Laboratory of
Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan
University, Wuhan 430079, China.  Wei Xu  Kuangchao Wu  Yifan Wu  An Ning  Lei
Wang  Lihong Li  Chenghui Cai  School of Geodesy and Geomatics, Wuhan
University, Wuhan 430079, China.  Time and Frequency Geodesy Center, Wuhan
University, Wuhan 430079, China.

###### Abstract

In 2022 China Space Station (CSS) will be equipped with atomic clocks and
optical clocks with stabilities of $2\times 10^{-16}$ and $8\times 10^{-18}$,
respectively, which provides an excellent opportunity to test gravitational
redshift (GRS) with higher accuracy than previous results. Based on high-
precise frequency links between CSS and a ground station, we formulated a
model and provided simulation experiments to test GRS. Simulation results
suggest that this method could test the GR at the accuracy level of $(0.27\pm
2.15)\times 10^{-7}$, more than two orders in magnitude higher than the result
of the experiment of a hydrogen clock on board a flying rocket more than 40
years ago.

††preprint: APS/123-QED

##  I Introduction

Scientists paid great attention to testing the gravitational redshift (GRS)
effect, one of three classic predictions of general relativity theory (GRT).
In 1976 scientists conducted the Gravity Probe-A (GP-A) experiment mission to
test GRS [1]. A hydrogen atomic clock was on board a rocket in this mission,
which flew about two hours in space. Based on the microwave links between the
rocket and several ground stations, the GP-A experiment has tested the GRS at
the accuracy level of $7\times 10^{-5}$. To improve the accuracy level, one
should consider two aspects. In one aspect, we try to use better clocks with
higher stability and accuracy. In another aspect, we try to create a condition
that the gravitational potential difference between two points is as
significant as possible. For instance, using optic clocks with stabilities
around $10^{-18}$, Katori’s group [2] measured a height of 450 m with an
accuracy of 4.1 cm, by which the accuracy of testing GRS achieves $9.1\times
10^{-5}$, a little worse than the previous rocket-experiment result of
$7\times 10^{-5}$ [3]. Another example, based on observations from two
eccentric-orbit Galileo satellites E14 and E18 (the stabilities of the onboard
clocks are around $10^{-15}$/day), several groups tested GRS with an accuracy
level around $2.5\sim 4.5\times 10^{-5}$ [4, 5, 6].

Recently, a few space atomic clock projects have been put forward. For
example, the Atomic clock Ensemble in Space (ACES) mission onboard the
international space station (ISS) and the China Space Station (CSS) mission.
The ACES mission will be equipped with an atomic clock with long-term
stability $2\times 10^{-16}$ [7, 8], and use its two independent time and
frequency transfer links, including three microwave links (MWL) and two
European Laser Timing (ELT) optical links to distribute time and frequency
scale to the ground stations. By using one uplink and two downlinks MWL, a
tri-frequency combination method for time-frequency transfer is constructed
according to the accuracy of the ACES atomic clock [9, 10]. Verified by
simulation experiments the tri-frequency combination method may test GRS at a
level of $10^{-6}$ [11, 12]. The CSS and ACES are different in design. The CSS
will be equipped with an optical atomic clock whose long-term stability is
$8\times 10^{-18}$ [13, 14, 11, 12], and two up-microwave links from a ground
station to CSS and two down-microwave links from CSS to the ground station
will be extablished. One uplink signal and one downlink signal have the same
frequency of 30.4 GHz but different circular polarization directions.
Considering the MWL characters, the errors caused by the propagation will be
eliminated. This study created a new frequency transfer model used in the
space-ground frequency transfer to test GRS, which is at least an order of
magnitude higher than the results given by previous studies.

##  II China Space Station

The CSS designed three modules, one core module (CM) and two experimejnts
modules (EM). The CSS has recently been launched to an orbit with a height
around 400 km above the ground since April 29, 2021. From June 17 to September
16, 2021, three Chinese astronauts stayed in the TianHe core module of CSS to
prepare for future experiments. Since October 16, another three Chinese
astronauts have been successively launched in the core module for further
preparatory work. The CSS will stay in orbits for at least ten years. For the
purpose of time and frequency applications, three or more ground stations have
been or will be established, including for instance Beijing, Xi’An, Shanghai,
and Wuhan stations. In addition, it will be extablished the space-ground
signals links between CSS and ground stations, including microwave links and
laser links.

Experiments module II will be equipped with a set of clocks attached to the
core module moving in orbit in September 2022. The set of clocks consists of a
hydrogen clock with daily stability of $2\times 10^{-15}$, a cold-atom
microwave fountain clock with daily stability of $2\times 10^{-16}$, and an
optical atomic clock with long-term stability of $8\times 10^{-18}$ [13, 14,
11, 12]. There will be four microwave links between CSS and a ground station
of interest for time and frequency transfer. The MWL consist of two uplinks
and two downlinks as listed in Table 1. Compared with the ACES mission, there
are two changes. One is that the CSS mission has an additional uplink with its
frequency equal to the frequency of one of the downlink signal. Another is
that the uplink and downlink signals are left-hand and right-hand circularly
polarized, respectively. In ACES, there are three microwave signals without
polarization, hence it is designed that the three signals have different
frequencies to distinguish the up and downlinks. While, for the CSS mission
when a microwave signal is polarized, incident upon an antenna from a given
direction, it will be received by the antenna with same polarization character
(say both right-hand circularly polarized). This technology is referred to as
polarization match. The antenna with corresponding polarization character is
selected according to different polarization natures of the microwaves to
distinguish the uplink and downlink signals [15].

]  Parameter Value Altitude $\sim$400 km Uplink frequency 30.4 GHz, 26.8 GHz
(both left-hand circularly polarized) Downlink frequency 30.4 GHz, 20.8 GHz
(both right-hand circularly polarized) Orbit inclination $41.5^{\circ}$
Minimum visible elevation angle $5^{\circ}$ Observation cutoff elevation angle
$15^{\circ}$

Table 1:  Parameters of China Space Station (CSS) and frequency links.

Here we use only the uplink and downlink signals both with frequency 30.4 GHz
(see Fig. 1), and other two links with different frequencies (see Table 1)
will not be used in this study. Sicne the frequencies of the uplink and
downlink are the same, the uninterested frequency shifts caused by the Dopper
effects, troposphere and ionosphere effects are the same. Making a difference
between the up and downlinks, we may cancel the uninterested frequency shifts
and double the GRS, which provides an opportunity to test GR with higher
accuracy.

Figure 1:  One up-link and one down-link of microwave signals between the
China Space Station (CSS) noted as S and a ground station (E) at Wuhan, China.
The up-link ($f_{1}$) and down-link ($f_{2}$) are respectively left- and
right-hand circularly polarized signals both with the frequency 30.4 GHz. The
antenna and the atomic clock at ground are linked by a transceiver. The
received signals from CSS are compared with the local signals generated by the
ground clock.

##  III Relativistic model for frequency transfer of the CSS

The up-link and down-link signals between CSS (denoted as $S$) and ground
station (denoted as $E$) satisfy the following expression:

$\displaystyle f_{IJ}^{r}=$ $\displaystyle
f_{IJ}^{e}-\Delta{U_{IJ}}+\Delta{f_{{\rm{Dop}}1-IJ}}+\Delta{f_{{\rm{Dop}}2-IJ}}$
(1) $\displaystyle+\Delta{f_{{\rm{tro-
IJ}}}}+\left({\Delta{f_{ion1-IJ}}+\Delta{f_{{\rm{ion}}2-IJ}}}\right)$
$\displaystyle-\Delta{f_{ti-IJ}}-\Delta{f_{pl-IJ}}+{\varepsilon_{IJ}}$

where $I,J=E,S,I\neq J$, $E$ and $S$ denote Earth and space station,
respectively; $f_{IJ}^{r}$ is the recieved frequency of the signal at $J$
coming from $I$, $f_{IJ}^{e}$ is the emitting frequency of the signal at $I$
toward $J$; $\Delta{U_{IJ}}={U_{J}}-{U_{I}}$, ${U_{I}}$ is the gravitational
potential at $I$, $\Delta{f_{{\rm{Dop}}1-IJ}}$ and
$\Delta{f_{{\rm{Dop}}2-IJ}}$ are the first and second order Doppler frequency
shifts of the signals transmitting from $I$ to $J$, $\Delta{f_{{\rm{tro-
IJ}}}}$ is the frequency shift of the signal transmitting from $I$ to $J$
caused by the troposphere, $\Delta{f_{ion1-IJ}}$ and
$\Delta{f_{{\rm{ion}}2-IJ}}$ are the first- and second-order frequency shifts
of the signal transmitting from $I$ to $J$ caused by the ionosphere,
$\Delta{f_{ti-IJ}}$ and $\Delta{f_{pl-IJ}}$ are frequency shifts caused by
tides and external celestial bodies (including the sun, moon, Mercury, Venus,
Mars, Jupiter, etc.), respectively, ${\varepsilon_{IJ}}$ are the un-modeled
errors of the signals transmitting from $I$ to $J$, including for instance the
clock errors, random errors and other noises. As presently designed in CSS,
the uplink signal with frequency ${f_{1}}=30.4$ GHz is left-hand circularly
polarized, and the downlink signal with frequency ${f_{2}}=30.4$ GHz is right-
hand circularly polarized [13, 14]. Hence, we have
$\Delta{f_{{\rm{ion}}2-ES}}=\Delta{f_{{\rm{ion}}2-SE}}$ [16]. In addition,
taking into account the following relationships,
$\Delta{U_{IJ}}=-\Delta{U_{JI}}$,
$\Delta{f_{{\rm{Dop}}1-IJ}}=\Delta{f_{{\rm{Dop}}1-JI}}$,
$\Delta{f_{Dop2-IJ}}=\Delta{f_{{\rm{Dop}}2-JI}}$, $\Delta{f_{{\rm{tro-
IJ}}}}=\Delta{f_{{\rm{tro-JI}}}}$,
$\Delta{f_{{\rm{ion}}1-IJ}}=\Delta{f_{{\rm{ion}}1-JI}}$, $\Delta{f_{ti-
IJ}}=-\Delta{f_{ti-JI}}$, and $\Delta{f_{pl-IJ}}=-\Delta{f_{pl-JI}}$, from Eq.
(1), subtracting $f_{ES}^{r}$ from $f_{SE}^{r}$, we obtain

$\displaystyle\Delta{U_{SE}}=\frac{{f_{SE}^{r}-f_{ES}^{r}}}{2}-\Delta{f_{ti-
SE}}-\Delta{f_{pl-SE}}+\varepsilon$ (2)

where $\varepsilon=-\left({{\varepsilon_{SE}}-{\varepsilon_{ES}}}\right)/2$.
On the right-hand side of Eq. (2), the first term is directly observed. The
second and third terms, $\Delta{f_{ti-SE}}$ and $\Delta{f_{pl-SE}}$, can be
calculated by relevant models. For instance, the tides can be corrected by the
software ETERNA [17, 18, 19], and general formulas can directly correct the
influences by external celestial bodies [20, 21]. In Eq. (2), both the first-
and second-order ionospheric frequency shifts vanish because one left-hand
(up-link) and another right-hand (down-link) circularly polarized wave signals
are designed in CSS. In practice, since Eq.(2) is not rigorous, and the uplink
signal’s path does not coincide with the downlink signal’s path, after the
combination of the up and down frequency signals, there are still following
residual errors: first- and second-order Doppler frequency shifts, troposphere
frequency shift, ionosphere frequency shift, as shown in Table 2.

At a given time $t$, we can obtain an observation of $\Delta{U_{ES}}$.
Comparing the observation with the corresponding true (model) value
$\Delta{V_{ES}}$, we can get the offset between the observed value and the
model value, expressed in relative difference as

$\displaystyle\alpha=\frac{\Delta U{(t)_{ES}}-\Delta
V{(t)_{ES}}}{V{(t)_{ES}}}$ (3)

which discribes the deviation of the observed result based on GRT from the
true (model) value. If GRT holds, $\alpha=0$.

##  IV simulation experiments

In our simulation experiments, we chose Luojia time and frequency station
(LTFS) at Wuhan as the ground station, the coordinate of which is
($30^{\circ}31^{\prime}51.90274^{\prime\prime}$ N,
$114^{\circ}21^{\prime}25.83516^{\prime\prime}$ E, 25.728 m). The CSS
parameters and signals’ original emitted frequency values are shown in Table
1. The observations in our simulation experiments are the received frequency
signals between the ground station LTFS and CSS, denoted as $f_{IJ}^{r}$,
which includes various frequency shifts caused by different factors, as
expressed by Eq. (1). Here we simulate the first-order Doppler frequency
shifts, relativistic frequency shifts (including gravitational redshift and
second-order Doppler effect), atmospheric frequency shift, tidal frequency
shift, and clock noises to obtain the received frequency values.

In the simulation experiments, we set the stability of the onboard optical
clock of CSS as $2\times 10^{-15}$/s, and its long-term stability is about
$8\times 10^{-18}$. The other effects that appeared in Eq. (1) are simulated
by the orbit parameters data, which can be downloaded from
http://www.celestrak.com/NORAD/elements/stations.txt, including the position,
velocity, and acceleration information about the CSS. To obtain a better
result, we set the observation cutoff elevation angle of the CSS larger than
$15^{\circ}$. In the simulation, the accuracies of radial orbit position and
velocity are set as 10 cm and 1 mm/s, respectively [11, 12].

In the simulation of the atmospheric effects on the frequency, we select the
Saastamoinen model [22], VMF1(Vienna Mapping Function) [23], and
meteorological parameters in the space region near and around the LTFS to
simulate the tropospheric frequency shifts. We select IRI (International
Reference Ionosphere) model [24] and IGRF-13 (international geomagnetic
reference field) model [25] to simulate the first- and second-order
ionospheric frequency shifts. We use the IERS (International Earth Rotation
and Reference Systems) Convention 2010 [26] to calculate the tidal harmonic
parameters and consequently simulate the frequency shifts caused by the tidal
effect.

According to the simulation setup, we may obtain the observed values
$f_{IJ}^{r}$ based on emitting frequency values. By Eq. (2), the value of
$\Delta{U_{SE}}$ could be calculated. We use EGM2008 model [27] to calculate
the gravitational potentials at CSS and ground station to obtain
$\Delta{V_{ES}}$ as the true value.

We have conducted simulation experiments to estimate the feasibility of a GRS
test based on CSS. The experiment lasts for one day, from May 22 00:00 to May
23 00:00, 2021. The orbit data were generated by a Two-line Element Set (TLE)
[28], which is used to describe a spacecraft orbit and to predict its position
and velocity. The trace of CSS during one day was depicted in Fig. 2. The
gravitational potential at the space station was calculated by EGM2008 [29].
The precision of EGM2008 is about several centimeters at the CSS’ orbit. The
ground station LTFS is located at Wuhan, China, and its gravitational
potential can be determined by leveling at the precision of centimeter-level
or better. Suppose $\alpha=0$, and then we can calculate the true GRS value
(model value) based on the general relativity theory (GRT).

Figure 2:  The orbit of CSS in Earth Centered Earth Fix (ECEF) coordinates
during the observations in one day. The ground station is located at Wuhan,
China ($30^{\circ}31^{\prime}51.90274^{\prime\prime}$ N,
$114^{\circ}21^{\prime}25.83516^{\prime\prime}$ E, 25.728 m). The observation
period is from June 1, 00:00 to June 2, 00:00, 2021. The light-colored dash
lines denote when the CSS and CSS and Wuhan station are not intervisible. The
solid black lines denote when CSS and Wuhan station are intervisible and
observation values are available.

The next step is considering various error sources and simulating the
observation values for a real case. For example, one of the primary error
sources in the GRS test is clock instability. In our experiment the stability
of on-board clock is set as $2\times 10^{-15}/\sqrt{\tau}$ ($\tau$ in second),
same stability as the designed by CSS mission. The stability of ground clock
is supposed to be better, reaching $1\times 10^{-15}/\sqrt{\tau}$.

According to Allan [30], the clock noises are classified as five different
types of stochastic noises, including Random Walk Frequency Modulation (RWFM),
Flicker Frequency Modulation (FFM), White Frequency Modulation(WFM), Flicker
Phase Modulation (FPM) and White Phase Modulation (WPM). We analyzed the
ratios of the five kinds of noises to the clock signals and restore the clock
signals according to the different ratios. We set the stabilities of ground
clock and on-board clock as $1.0\times 10^{-15}/\sqrt{\tau}$ and $2.0\times
10^{-15}/\sqrt{\tau}$, respectivily (see Table 2). Then we use a python
library Allantools (https://github.com/aewallin/allantools) to generate the
series of clock frequency errors based on the characteristics of atomic
clocks. Their total Allan deviations [30] are depicted in Fig. 3.

Figure 3:  The total Allan deviation of the clocks’ frequency stability. The
stability of on-board clock is about $2.0\times 10^{-15}/\sqrt{\tau}$ ($\tau$
in second), and the stability of ground clock is about $1.0\times
10^{-15}/\sqrt{\tau}$.

Another important influence factor is the frequency shift caused by the
atmosphere. The local weather station can record the meteorological elements
of the ground station. Then, the signal’s tropospheric frequency shift can be
estimated based on the GPT2-mapping function. The ionospheric frequency shift
can be calculated according to International Reference Ionosphere (IRI) model
[31]. The precision of the atmosphere model is regarded as residual errors and
added to the frequency shift observation. Other error sources such as the
orbit error of CSS, the EGM2008 error, tidal correction residual are also
considered. Each of them is simulated as Gaussian errors plus a randomly set
systematic offset. The magnitudes of error sources are listed in Table 2.

Table 2:  Error sources and their magnitudes in the simulation experiments.

Error sources | Magnitudes
---|---
On-board clock stability | $2.0\times 10^{-15}/\sqrt{\tau}$
Ground clock stability | $1.0\times 10^{-15}/\sqrt{\tau}$
Troposphere influence residual | 5% of the tropospheric frequency shift after combination∗
Ionosphere influence residual | 10% of the ionospheric frequency shift after combination∗
Position accuracy of CSS |  $\pm$0.1 m
Velocity accuracy of CSS |  $\pm 1.0\times 10^{-3}$m/s
Gravitational potential model accuracy |  $\pm$0.3 $m^{2}/s^{2}$ (CSS), $\pm$0.1 $m^{2}/s^{2}$ (ground)
Tide influence residual | 0.1 $m^{2}/s^{2}$

$*$: Combinantion of up and down frequency signals, exprssed as eq.(2)

Once the simulated observation frequency values are obtained, they are
compared with the true frequency shift values and part of the offset between
them (a short period lasts for about 5 minutes) are depicted in Fig. 4. We can
see that the most prominent offset component comes from clock error, which is
at least two magnitudes larger than other error influences. The influence of
ionosphere frequency shift shows apparent fluctuation because the total
electron content (TEC) distribution varies at different latitudes, and the
extent of path discrepancy of uplink and downlink signal also varies at
different elevating angles.

Figure 4:  Relative frequency offset between theoretical predictions and
simulated observations of frequency shift during one slot period (around 5
min, from June 01, 09:34:18 to June 01, 09:38:52). The main error influence
factors are denoted in different colors, marked in the left-bottom corner. The
most prominent error source is the clock error, at least two magnitudes larger
than other error influences.

There are four sections (periods) in which observation data are available (see
the solid black line in Fig. 2) during the one day’s experiment. The result of
one-day experiment is $0.17\pm 3.05\times 10^{-16}$, where $0.17\times
10^{-16}$ is the mean offset between the true value and the experiment result,
corresponding to $1.68\times 10^{-7}$ for the offset of $\alpha$, and $\pm
3.05\times 10^{-16}$ is the uncertainty (STD). The offset of $\alpha$ reflects
the precision of the GRS test experiment. However, the result of only one
simulation experiment might be influenced by accidental factors. Therefore, we
repeated the above experiment 40 times for the same day under different random
seeds of errors generation to estimate the precision of the GRS test, and the
results of $\alpha$ offset of 40 experiments are depicted in Fig. 5. We can
see that the mean offset of $\alpha$ is $2.7\times 10^{-8}$, with the standard
deviation of $2.15\times 10^{-7}$.

Figure 5:  The results of 40 different simulation experiments. In each
experiment, we use the same parameters as listed in Table 1 with different
random seeds of errors generation.

##  V Conclusion

Based on the uplink and downlink frequency signals in free space between the
CSS and a ground station, we established a formulation for determining the
gravitational potential difference between the CSS and the ground station.
Based on this formulation, simulation results have shown that the GRS could be
tested at around $2\times 10^{-7}$, at least two orders in magnitude higher
than the presently highest accuracy level. In addition, the proposed
formulation in this study could be a new approach applied to determining the
geopotential difference between arbitrary two ground stations. In contrast,
the conventional approach is gravimetry plus leveling [32], which is laborious
and low-efficient [33]. Suppose two ground stations equipped with optical
clocks with the stability of $1\times 10^{-18}$ can simultaneously observe the
frequency signals from CSS, the accuracy of determining the geopotential
difference between the two ground stations could achieve a level of $0.1$
m2/s2, because subtraction of the observation between CSS and one station and
that between CSS and another station will cancel the common error sources
stemming from CSS. With the rapid development of time-frequency science, the
optical-atomic clocks with unprecedented accuracy and stability have been
generated [34, 35, 36, 37, 38, 39]. It dramatically expands the clocks’
application scopes in various branches. Besides its application in
geopotential determination, this formulation can also be applied in height
propagation between two datum stations separated by oceans, unifying the
vertical height systems around the world [33].

###### Acknowledgements.

This study is supported by the National Natural Science Foundation of China
(NSFC) (Grant Nos. 42030105, 41721003, 41631072, 41874023, 41804012,
41974034), Space Station Project (Grant No. 2020-228), and Natural Science
Foundation of Hubei Province (Grant No. 2019CFB611).

## References

  * Vessot and Levine [1979] R. F. C. Vessot and M. W. Levine, General relativity and gravitation 10, 181 (1979).
  * Takamoto  _et al._ [2020] M. Takamoto, I. Ushijima, N. Ohmae, T. Yahagi, K. Kokado, H. Shinkai, and H. Katori, Nature Photonics 14, 411 (2020).
  * Vessot  _et al._ [1980] R. F. C. Vessot, M. W. Levine, E. M. Mattison, E. L. Blomberg, T. E. Hoffman, G. U. Nystrom, B. F. Farrell, R. Decher, P. B. Eby, C. R. Baugher, J. W. Watts, D. L. Teuber, and F. D. Wills, Physical Review Letters 45, 2081 (1980).
  * Kouba [2021] J. Kouba, GPS Solutions 25, 1 (2021).
  * Delva  _et al._ [2018] P. Delva, N. Puchades, E. Schönemann, F. Dilssner, C. Courde, S. Bertone, F. Gonzalez, A. Hees, C. Le Poncin-Lafitte, F. Meynadier, R. Prieto-Cerdeira, B. Sohet, J. Ventura-Traveset, and P. Wolf, Phys. Rev. Lett. 121, 231101 (2018).
  * Herrmann  _et al._ [2018] S. Herrmann, F. Finke, M. Lülf, O. Kichakova, D. Puetzfeld, D. Knickmann, M. List, B. Rievers, G. Giorgi, C. Günther, H. Dittus, R. Prieto-Cerdeira, F. Dilssner, F. Gonzalez, E. Schonemann, J. Ventura-Traveset, and C. Lammerzahl, Physical review letters 121, 231102 (2018).
  * Cacciapuoti and Salomon [2011] L. Cacciapuoti and C. Salomon, in  _Journal of Physics: Conference Series_ , Vol. 327 (IOP Publishing, 2011) p. 012049.
  * Meynadier  _et al._ [2018] F. Meynadier, P. Delva, C. le Poncin-Lafitte, C. Guerlin, and P. Wolf, Classical and Quantum Gravity 35, 035018 (2018).
  * Blanchet  _et al._ [2001] L. Blanchet, C. Salomon, P. Teyssandier, and P. Wolf, Astronomy & Astrophysics 370, 320 (2001).
  * Linet and Teyssandier [2002] B. Linet and P. Teyssandier, Phys. Rev. D 66, 024045 (2002).
  * Sun  _et al._ [2021] X. Sun, W.-B. Shen, Z. Shen, C. Cai, W. Xu, and P. Zhang, The European Physical Journal C 81, 1 (2021).
  * Shen  _et al._ [2021] Z. Shen, W.-B. Shen, T. Zhang, L. He, Z. Cai, X. Tian, and P. Zhang, Advances in Space Research 68, 2776 (2021).
  * Guo [2021] Y. M. Guo, in  _The 12th China Satellite Navigation Conference, CSNC 2021_ (2021).
  * Wang [2021] W. B. Wang, in  _The 12th China Satellite Navigation Conference, CSNC 2021_ (2021).
  * Elliot [1983] R. S. Elliot, IEEE Transactions on Antennas and Propagation 31, 1 (1983).
  * Hoque and Jakowski [2012] M. M. Hoque and N. Jakowski, Global navigation satellite systems: signal, theory and applications 10, 30090 (2012).
  * Schüller [2020] K. Schüller,  _Program System ETERNA-x et34-x-v80-* for Earth and Ocean Tides Analysis and Prediction_ (2020).
  * Hartmann and Wenzel [1995] T. Hartmann and H.-G. Wenzel, Geophysical research letters 22, 3553 (1995).
  * Wenzel [1997] H.-G. Wenzel, Tidal phenomena , 9 (1997).
  * Hoffmann [1961] B. Hoffmann, Physical Review 121, 337 (1961).
  * Kleppner  _et al._ [1970] D. Kleppner, R. F. C. Vessot, and N. F. Ramsey, Astrophysics and Space Science 6, 13 (1970).
  * Saastamoinen [1972] J. Saastamoinen, The use of artificial satellites for geodesy 15, 247 (1972).
  * Böhm and Schuh [2003] J. Böhm and H. Schuh,  _Vienna mapping functions_ (na, 2003).
  * Bilitza  _et al._ [1993] D. Bilitza, K. Rawer, L. Bossy, and T. Gulyaeva, Advances in Space Research 13, 15 (1993).
  * Baerenzung  _et al._ [2020] J. Baerenzung, M. Holschneider, J. Wicht, V. Lesur, and S. Sanchez, Earth, Planets and Space 72, 1 (2020).
  * Petit and Luzum [2010] G. Petit and B. Luzum,  _IERS conventions (2010)_ , Report (Bureau International des Poids et mesures sevres (france), 2010).
  * Pavlis  _et al._ [2008] N. K. Pavlis, S. A. Holmes, S. C. Kenyon, and J. K. Factor, in  _AGU Fall Meeting Abstracts_ (2008).
  * Weiping  _et al._ [2012] L. Weiping, Q. Hongxing, and D. Qi, in  _2012 IEEE International Conference on Information and Automation_ (IEEE, 2012) pp. 811–813.
  * Pavlis  _et al._ [2012] N. K. Pavlis, S. A. Holmes, S. C. Kenyon, and J. K. Factor, Journal of geophysical research: solid earth 117, 1 (2012).
  * Allan  _et al._ [1991] D. W. Allan, M. A. Weiss, and J. L. Jespersen, in  _Proceedings of the 45th Annual Symposium on Frequency Control 1991_ (IEEE, 1991) pp. 667–678.
  * Bilitza  _et al._ [2017] D. Bilitza, D. Altadill, V. Truhlik, V. Shubin, I. Galkin, B. Reinisch, and X. Huang, Space weather 15, 418 (2017).
  * Heiskanen and Moritz [1967] W. A. Heiskanen and H. Moritz,  _Physical geodesy(Book on physical geodesy covering potential theory, gravity fields, gravimetric and astrogeodetic methods, statistical analysis, etc)_ (W. H. Freeman and Company San Francisco, 1967).
  * Shen  _et al._ [2020] Z. Shen, W.-B. Shen, and S. Zhang, arXiv preprint arXiv:2008.05868 , 1 (2020).
  * Nicholson  _et al._ [2015] T. L. Nicholson, S. L. Campbell, R. B. Hutson, G. E. Marti, B. J. Bloom, R. L. McNally, W. Zhang, M. D. Barrett, M. S. Safronova, G. F. Strouse, _et al._ , Nature communications 6, 1 (2015).
  * Huntemann  _et al._ [2016] N. Huntemann, C. Sanner, B. Lipphardt, C. Tamm, and E. Peik, Physical review letters 116, 063001 (2016).
  * McGrew  _et al._ [2018] W. F. McGrew, X. Zhang, R. J. Fasano, S. A. Schäffer, K. Beloy, D. Nicolodi, R. C. Brown, N. Hinkley, G. Milani, M. Schioppo, T. H. Yoon, and A. D. Ludlow, Nature 564, 87 (2018).
  * Oelker  _et al._ [2019] E. Oelker, R. B. Hutson, C. J. Kennedy, L. Sonderhouse, T. Bothwell, A. Goban, D. Kedar, C. Sanner, J. M. Robinson, G. E. Marti, _et al._ , Nature Photonics 13, 714 (2019).
  * Nakamura  _et al._ [2020] T. Nakamura, J. Davila-Rodriguez, H. Leopardi, J. A. Sherman, T. M. Fortier, X. Xie, J. C. Campbell, W. F. McGrew, X. Zhang, Y. S. Hassan, _et al._ , Science 368, 889 (2020).
  * Brewer  _et al._ [2019] S. M. Brewer, J.-S. Chen, A. M. Hankin, E. R. Clements, C.-W. Chou, D. J. Wineland, D. B. Hume, and D. R. Leibrandt, Physical review letters 123, 033201 (2019).


11institutetext: Indian Institute of Information Technology Allahabad, India
11email<EMAIL_ADDRESS>
# MAG-Net: Multi-task attention guided network for brain tumor segmentation
and classification ††thanks: All authors have contributed equally.

Sachin Gupta  Narinder Singh Punn  Sanjay Kumar Sonbhadra  Sonali Agarwal

###### Abstract

Brain tumor is the most common and deadliest disease that can be found in all
age groups. Generally, MRI modality is adopted for identifying and diagnosing
tumors by the radiologists. The correct identification of tumor regions and
its type can aid to diagnose tumors with the followup treatment plans.
However, for any radiologist analysing such scans is a complex and time-
consuming task. Motivated by the deep learning based computer-aided-diagnosis
systems, this paper proposes multi-task attention guided encoder-decoder
network (MAG-Net) to classify and segment the brain tumor regions using MRI
images. The MAG-Net is trained and evaluated on the Figshare dataset that
includes coronal, axial, and sagittal views with 3 types of tumors meningioma,
glioma, and pituitary tumor. With exhaustive experimental trials the model
achieved promising results as compared to existing state-of-the-art models,
while having least number of training parameters among other state-of-the-art
models.

###### Keywords:

Attention Brain tumor Deep learning Segmentation.

##  1 Introduction

Brain tumor is considered as the deadliest and most common form of cancer in
both children and adults. Determining the correct type of brain tumor in its
early stage is the key aspect for further diagnosis and treatment process.
However, for any radiologist, identification and segmentation of brain tumor
via multi-sequence MRI scans for diagnosis, monitoring, and treatment, are
complex and time-consuming tasks.

Brain tumor segmentation is a challenging task because of its varied behavior
both in terms of structure and function. Furthermore, the tumor intensity of a
person differs significantly from each other. MRI is preferred over other
imaging modalities [4] for the diagnosis of brain tumor because of its non-
invasive property that follows from without the exposure to ionizing
radiations and superior image contrast in soft tissues.

Deep learning has shown advancement in various fields with promising
performance especially in the area of biomedical image analysis [24]. The
convolutional neural networks (CNN) [2] are the most widely used models in
image processing. The CNNs involve combination of convolution, pooling and
activation layers accompanied with the normalization and regularization
operations to extract and learn the target specific features for desired task
(classification, localization, segmentation, etc.). In recent years various
techniques have been proposed for identification (classification and
segmentation) of the brain tumor using MRI images that achieved promising
results [13, 30]. However, most of the approaches use millions of trainable
parameters that result in slower training and analysis time, while also having
high variance in results in case of limited data samples.

In order to overcome the aforementioned drawbacks, Ronneberger et al. [26]
proposed U shaped network (U-Net) for biomedical image segmentation. The model
follows encoder-decoder design with feature extraction (contraction path) and
reconstruction phases (expansion path) respectively. In addition, skip
connections are introduced to propagate the extracted feature maps to the
corresponding reconstruction phase to aid upsample the feature maps. Finally,
model produces segmentation mask in same dimensions as the input highlighting
the target structure (tumor in our case). Following the state-of-the-art
potential of the U-Net model, many U-Net variants are proposed to further
improve the segmentation performance. Attention based U-Net model [19] is one
such variant that tend to draw the focus of the model towards target features
to achieve better segmentation results. The attention filters are introduced
in the skip connections where each feature is assigned weight coefficient to
highlight its importance towards the target features. Despite achieving the
promising results, these models have millions of trainable parameter which can
be reduced by optimizing the convolution operation. This can be achieved by
incorporating the depthwise convolution operations [8] that is performed in
two stages: depthwise and pointwise convlutions. The reduction in the number
of the parameters and multiplications as compared to standard convolution
operation can represented as $1/r+1/f^{2}$, where $r$ is the depth of the
output feature map and $f$ is the kernel height or width [21]. The achieved
reduction in number of parameters and multiplications is $\sim 80\%$.
Following this context, attention guided network is proposed that uses
depthwise separable convolution for real time segmentation and classification
of the brain tumor using MRI imaging. The major contribution of the present
research work is as follows:

  * •

A novel model, Multi-task (segmentation and classification) attention guided
network (MAG-Net) is proposed for brain tumor diagnosis.

  * •

Optimization of training parameters using depthwise separable convolution. The
training parameters of the MAG-Net reduced from 26.0M to 5.4M.

  * •

MAG-Net achieved significant improvement in classification and segmentation as
compared to the state-of-the-art models while having limited data samples.

The rest paper is organized as follows: Section 2 describes the crux of
related work on brain tumor segmentation and classification. Section 3, talks
about the proposed architecture, whereas Section 4 discuses the training and
testing environment with experimental and comparative analysis. Finally,
concluding remarks are presented in Section 5.

##  2 Literature review

Identifying the brain tumor is a challenging task for the radiologists.
Recently, several deep learning based approaches are proposed to aid in faster
diagnosis of the diseases. Segmentation of the infected region is most common
and critical practice involved in the diagnosis. In addition, the segmented
region can be provided with label (classification) to indicate what type of
anomaly or infection is present in the image.

In contrast to the traditional approaches, Cheng et al. [7] proposed a brain
tumor classification approach using augmented tumor region instead of original
tumor region as RoI (region of interest). Authors utilized the bag of word
(BOW) technique to segment and extract local features from RoI. Dictionary is
used for encoding the extracted local features maps that are passed through
SVM (support vector machine) classifier. The approach outperformed the
traditional classification techniques with the accuracy of 91.28% but the
performance is limited by the data availability. In similar work, Ismael et
al. [14] proposed an approach of combining statistical features along with
neural networks by using filter combination: discrete wavelet transform
(DWT)(represented by wavelet coefficient) and Gabor filter (for texture
representation). For classification of the tumor, three layered neural network
classifier is developed using multilayer perceptron network that is trained
with statistical features. In contrast to Cheng et al. [7], authors also
achieved promising results on the limited data samples with an overall
accuracy of. 91.9%.

Recently, capsule network [12] has shown great performance in many fields
especially in biomedical image processing. Afshar et al. [1] proposed basic
capsnet with three capsules in last layer representing three tumor classes.
However, due to varied behavior (background, intensity, structure, etc.) of
MRI image, the proposed model failed to extract optimal features representing
the tumor structure. The author achieved the tumor classification accuracy of
78% and 86.5% using raw MRI images and tumor segmented MRI images
respectively. In another approach, Pashaei et al. [20] utilized CNN and kernel
extreme learning machine that comprises one hidden layer with 100 neurons to
increase the robustness of the model. With several experimental trials, the
authors achieved an accuracy of 93.68% but detects only 1% of the positive
pituitary tumor cases out of the total pituitary tumor case. Deepak et al. [9]
proposed a transfer learning approach that uses pre-trained GoogleNet model to
extract features (referred as deep CNN features) with softmax classifier in
the output layer to classify three tumor classes. Furthermore, the authors
combine the deep CNN features and SVM model to analyse the classification
performance. The authors achieved 97.1% accuracy but resulted in poor
performance by standalone GoogleNet model due to overfitting with limited
training image dataset, and misclassifications in meningioma tumor. In another
approach, Pernas et al. [11] proposed to process images in three different
spatial scales along with multi pathways feature scales for classification and
segmentation of brain tumor. The images are pre-processed with elastic
transform for preventing overfitting. The model analyses entire image and
classifies pixel by pixel in one of four possible output labels (i.e.
0-healthy, 1-meningioma, 2-glioma, and 3-pituitary tumor). The proposed
approach outperformed existing approaches with 97.3% classification accuracy,
but with poor segmentation performance. Following this context, in this
article multi-task attention guided network (MAG-Net) is proposed based on the
U-Net architectural design [26] that uses parallel depthwise separable
convolution layers for multi-level feature extraction along with an attention
mechanism to better extract tumor features for brain tumor classification and
generate the corresponding tumor mask.

Figure 1: Schematic representation of the architecture of MAG-Net model.

##  3 Proposed work

The proposed multi-task attention guided network (MAG-Net) model, as shown in
Fig. 1, focuses on reducing overall computation, better feature extraction and
optimizing the training parameters by reduction. The overall architectural
design consists of an encoder, decoder, and classification module with 5.4M
trainable parameters. The overall architectural design of the model is
inspired by the U-Net encoder-decoder style [23]. Due to its state-of-the-art
potential, this model is the most prominent choice among the researchers to
perform biomedical image segmentation [17].

In MAG-Net to reduce the number of training parameters without the cost of
performance, standard convolution operations are replaced with depthwise
separable convolution. In addition, the skip connections are equipped with
attention filters [19] to better extract the feature maps concerning the tumor
regions. The attention approach filters the irrelevant feature maps in the
skip connection by assigning weights to highlight its importance towards the
tumor regions. Besides, the encoder block is equipped with parallel separable
convolution filters of different sizes, where the extracted feature maps are
concatenated for better feature learning. These features are then passed to
the corresponding decoder blocks via attention enabled skip connections to aid
in feature reconstruction with the help of upsampling operation. The
bottleneck layer connects the feature extraction path to the feature
reconstruction path. In this layer filters of different sizes are used along
with the layer normalization. Furthermore, the classification is performed
using the extracted feature maps obtained from the final encoder block.

###  3.1 Encoder

To detect the shape and size of varying image like brain tumor it is required
to use separable convolution of different sizes. Inspired from the concept of
inception neural network [22] the encoder segment is consist of separable
convolutions of 1 x 1, 3 x 3, and 5 x 5 kernels. Each of separable
convolutions are followed by layer normalization. The extracted feature maps
are fused with add operation that are downsampled by max pooling operation.
Fig. 2, shows the proposed encoder architecture of MAG-Net model for some
input feature map, $\mathcal{F}_{i}\in\mathcal{R}^{w\times h\times d}$, where
$w$, $h$ and $d$ are the width, height and depth of the feature map.

Figure 2: Proposed MAG-Net 2D encoder module.

###  3.2 Decoder

The decoder component follows from the encoder block and that tend to
reconstruct the spatial dimension to generate the output mask in same
dimension as input. It consists of upsampling of the feature maps along with
the concatenation with the attention maps followed by a separable convolution
operation. Long skip connections [10] are used to propagate the attention
feature maps from encoder to decoder to recover spatial information that was
lost during downsampling in encoder. By using attention in the skip connection
it helps the model to suppress the irrelevant features.

###  3.3 Classification

This module classifies the brain tumor MRI images into respective classes i.e
meningioma, glioma, and pituitary tumor by utilizing the features extracted
from the encoder block. This encoder block act as backbone model for both
classification and segmentation, thereby reducing the overall complexity of
the model. In this classification block the feature maps of the last encoder
block act as input that are later transformed into 1D tensor by using global
average pooling. The pooled feature maps are then processed with multiple
fully connected layers. The classification output is generated from the
softmax activated layer that generates the probability distribution of the
tumor classes for an image.

##  4 Experiment and Results

###  4.1 Dataset Setup

The present research work utilizes Figshare [6] dataset that comprises of 2D
MRI scan with T1-weighted contrast-enhanced modality acquired from 233
patients to form a total of 3064 MRI scans. The T1 modality highlight distinct
features of the brain tumor with three classes representing the type of brain
tumor i.e. meningioma (708 slices), glioma (1426 slices), and pituitary (930
slices) forming 23%, 46.5%, and 30% class distribution in the dataset
respectively. The sample MRI slices of different tumor classes are presented
in Fig. 3. Dataset is randomly split into 80% training and 20% of the
validation set. The training and testing composition kept the same throughout
the experiment trails for comparative analysis.

Figure 3: A slice of MRI scan with T1 modality showing different tumor
classes: meningioma, glioma, and pituitary

###  4.2 Training and Testing

The MAG-Net model is trained and evaluated on the Figshare dataset. The
training phase is accompanied with early-stopping [3] to tackle the
overfitting problem, and Adam as a learning rate optimiser [27]. Cross entropy
based loss functions are most popularly used for model training and validating
segmentation and classification tasks. Following this, binary cross entropy
and categorical cross entropy functions are employed for training the model
for binary tumor mask generation and classification respectively. Binary cross
entropy (BCE, shown in Eq. 1) is a sigmoid activation [16] followed by cross
entropy loss [29] that compares each of the predicted probabilities to actual
output. Categorical cross entropy (CE, shown in Eq. 2) is a softmax activation
function followed by cross-entropy-loss that compares the output probability
over each tumor class for each MRI image.

$\mathcal{L}_{BCE}=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}.log(p(y_{i}))+(1-y_{i}).log(1-P(y_{i})))$
(1)

where $y$ represents actual tumor mask, $p(y)$ represents predicted tumor mask
and $N$ is the total number of images.

$\mathcal{L}_{CE}=\sum_{i}^{C}t_{i}log(f(s_{i}))$ (2)

where $C$ is the no. of class, $f(s_{i})$ is the probability of occurrence of
each class $t_{i}$ represents 1 for true label and 0 for others.

For segmentation the most popular evaluation matrics are dice coefficient
(shown in Eq. 3) and intersection-over-union (IoU / Jaccard index) (shown in
Eq. 4), and hence are utilized to evaluate the trained MAG-Net model. TP
defines correctly classified predictions FP defines wrongly classified, and FN
defines missed objects of each voxel.

$DiceCoefficient=\frac{2*TP}{2*TP+FP+FN}$ (3)

$IoU=\frac{TP}{TP+FP+FN}$ (4)

To evaluate classification module of the MAG-Net, accuracy, precision, recall,
f1-score and micro average metrics are considered for better quantification
and visualization of the performance of the model. Precision of the class, as
shown in Eq. 5, quantifies about the positive prediction accuracy of the
model. Recall is the fraction of true positive which are classified correctly
(shown in Eq. 6). F1-score quantifies the amount of correct predictions out of
all the positive predictions (shown in Eq. 7). Support quantifies the true
occurrence in the specified dataset of the respective class. Micro average
($\mu_{avg}$) (shown in Eq. 8, Eq. 9 and Eq. 10) is calculated for precision,
recall, and F1-score. To compute micro average ($\mu_{avg}$), the test dataset
is divided into two sub dataset, on each of which the true positive, false
positive and false negative predictions are identified.

$Precision=\frac{TP}{(TP+FP)}$ (5)

$Recall=\frac{TP}{(FN+FP)}$ (6)

$F1-score=\frac{2*Recall*Precision}{(Recall+Precision)}$ (7)

$\mu_{avg}(Precision)=\frac{TP_{1}+TP_{2}}{(TP_{1}+TP_{2}+FP_{1}+FP_{2})}$ (8)

$\mu_{avg}(Recall)=\frac{TP_{1}+TP_{2}}{(TP_{1}+TP_{2}+FN_{1}+FN_{2})}$ (9)

$\mu_{avg}(F1-score)=HM(\mu_{avg}(Precision),mu_{avg}(Recall))$ (10)

where $TP_{1}$, $FP_{1}$, and $FN_{1}$ belong to the first set and $TP_{2}$,
$FP_{2}$, and $FN_{2}$ belongs to the different sets. $HM$ is the harmonic
mean.

Figure 4: Qualitative results of brain tumor segmentation and classification
on MRI images (a, b, and c of different tumor classes) using MAG-Net model.

###  4.3 Results

The MAG-Net outputs the segmented mask of a given MRI image consisting of
tumor region corresponding to meningioma, glioma, and pituitary as classified
by the model. For randomly chosen MRI slices, Fig. 4 presents the segmentation
and classification results of model. The visual representation confirms that
the results are close to the ground truth of respective tumor classes.

Table 1:  Comparative analysis of the MAG-Net with the existing segmentation models on test dataset. Model | Accuracy | Loss | Dice coefficient | Jaccard index | Parameters
---|---|---|---|---|---
U-Net | 99.5 | 0.024 | 0.70 | 0.55 | 31M
wU-Net | 99.4 | 0.034 | 0.66 | 0.49 | 31M
Unet++ | 99.5 | 0.028 | 0.65 | 0.49 | 35M
MAG-Net | 99.52 | 0.021 | 0.74 | 0.60 | 5.4M
*bold quantities indicate the best results.

Table 1 represents the result of the proposed work for segmentation in the
form of accuracy, loss, dice coefficient, Jaccard index, and trainable
parameters along with comparative analysis with other popular approaches. The
proposed framework outperforms the other approaches in segmenting tumor with
the dice and IoU score of 0.74 and 0.60 respectively. In contrast to other
models, MAG-Net achieved best results with minimal trainable parameters. The
other popular approaches taken in comparative analysis for segmentation are
U-Net [15], U-Net++ [18, 31], and wU-Net. [5].

Table 2:  Comparative analysis of the MAG-Net with the existing classification models on test dataset using confusion matrix. Model |  Acc. |  Loss |  |  Meningioma |  Glioma |  Pituitary
---|---|---|---|---|---|---
|  |  |  Meningioma |  114 |  25 |  3
VGG16 |  93.15 |  0.26 |  Glioma |  13 |  271 |  1
|  |  |  Pituitary |  0 |  1 |  185
|  |  |  Meningioma |  114 |  21 |  7
VGG19 |  93.8 |  0.25 |  Glioma |  11 |  274 |  0
|  |  |  Pituitary |  0 |  1 |  185
|  |  |  Meningioma |  123 |  12 |  7
ResNet50 |  94.2 |  0.31 |  Glioma |  16 |  266 |  3
|  |  |  Pituitary |  1 |  0 |  185
|  |  |  Meningioma |  134 |  7 |  1
MAG-Net |  98.04 |  0.11 |  Glioma |  1 |  282 |  2
|  |  |  Pituitary |  1 |  0 |  185
*bold quantities indicate the best results.

Table 2 and Table 3 represent the results of the proposed work for
classification in the form of accuracy, loss, confusion matrix, and
classification report for meningioma, glioma, and pituitary tumor along with
comparative analysis with other state-of-the-art approaches: VGG-16 [28],
VGG-19 [28], and ResNet50 [25]. With exhaustive experimental trials it is
observed that MAG-Net outperformed the existing approaches with significant
margin in all the metrics.

Table 3:  Comparative analysis of the MAG-Net with the existing classification models on test dataset considering classification report as evaluation parameter. Model |  Classes |  Precision |  Recall |  f1-Score |  Support
---|---|---|---|---|---
VGG16 |  Meningioma |  0.90 |  0.80 |  0.85 |  142
Glioma |  0.91 |  0.85 |  0.93 |  285
Pituitary |  0.98 |  0.99 |  0.99 |  186
Micro avg. |  0.93 |  0.93 |  0.93 |  613
VGG19 |  Meningioma |  0.91 |  0.80 |  0.85 |  142
Glioma |  0.93 |  0.96 |  0.94 |  285
Pituitary |  0.96 |  0.99 |  0.98 |  186
Micro avg. |  0.93 |  0.93 |  0.93 |  613
ResNet-50 |  Meningioma |  0.88 |  0.87 |  0.87 |  142
Glioma |  0.93 |  0.99 |  0.94 |  285
Pituitary |  0.95 |  0.99 |  0.97 |  186
Micro avg. |  0.94 |  0.94 |  0.94 |  613
MAG-Net |  Meningioma |  0.99 |  0.94 |  0.96 |  142
Glioma |  0.98 |  0.99 |  0.98 |  285
Pituitary |  0.98 |  0.99 |  0.99 |  186
Micro avg. |  0.98 |  0.98 |  0.98 |  613
*bold quantities indicate the best results.

It is observed that unlike other state-of-the-art models, the MAG-Net model
achieved promising results due to the reduction in the overall computation,
better feature extraction and training parameters optimization. As shown in
Table 1 raw U-Net displayed similar performance but at the cost of large
number of trainable parameters. In the MAG-Net model, the encoder block is
developed by replacing convolution layers with parallel depthwise separable
convolution of various sizes connected in parallel which resulted in better
multi-scale feature learning for varying shapes and sizes of the tumor. For
reducing spatial loss during feature reconstruction, attention mechanism is
used in skip connections for better feature reconstruction. To reduce the
overall complexity of the model the feature extracted by encoder blocks are
reused to classify the type of brain tumor.

##  5 Conclusion

In this paper, the complex task of brain tumor segmentation and classification
is addressed using multi-task attention guided network (MAG-Net). This a U-Net
based model that features reduction in the overall computation, better feature
extraction and training parameters optimization. The proposed architecture
achieved significant performance on the Figshare brain tumor dataset by
exploiting the state-of-the-art advantages of U-Net, depthwise separable
convolution and attention mechanism. The MAG-Net model recorded the best
classification and segmentation results compared to the existing
classification and segmentation approaches. It is believed that this work can
also be extended to other domains involving classification and segmentation
tasks.

## Acknowledgment

We thank our institute, Indian Institute of Information Technology Allahabad
(IIITA), India and Big Data Analytics (BDA) lab for allocating the centralised
computing facility and other necessary resources to perform this research. We
extend our thanks to our colleagues for their valuable guidance and
suggestions.

## References

  * [1] Afshar, P., Mohammadi, A., Plataniotis, K.N.: Brain tumor type classification via capsule networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP). pp. 3129–3133. IEEE (2018)
  * [2] Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET). pp. 1–6. Ieee (2017)
  * [3] Brownlee, J.: Use early stopping to halt the training of neural networks at the right time. https://machinelearningmastery.com/how-to-stop-training-deep-neural-networks-at-the-right-time-using-early-stopping/ (2018), [Online; accessed April 17, 2021]
  * [4] Cancer.Net: Brain tumor: Diagnosis. https://www.cancer.net/cancer-types/brain-tumor/diagnosis (2020), [Online; accessed March 20, 2021]
  * [5] CarryHJR: Nested unet. https://github.com/CarryHJR/Nested-UNet/blob/master/model.py. (2020), [Online; accessed March 11, 2021]
  * [6] Cheng, J.: brain tumor dataset (4 2017), https://figshare.com/articles/dataset/brai\\\n\\_tumor\\_dataset/1512427
  * [7] Cheng, J., Huang, W., Cao, S., Yang, R., Yang, W., Yun, Z., Wang, Z., Feng, Q.: Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS one 10(10), e0140381 (2015)
  * [8] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1251–1258 (2017)
  * [9] Deepak, S., Ameer, P.: Brain tumor classification using deep cnn features via transfer learning. Computers in biology and medicine 111, 103345 (2019)
  * [10] Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. In: Deep learning and data labeling for medical applications, pp. 179–187. Springer (2016)
  * [11] Díaz-Pernas, F.J., Martínez-Zarzuela, M., Antón-Rodríguez, M., González-Ortega, D.: A deep learning approach for brain tumor classification and segmentation using a multiscale convolutional neural network. Healthcare 9(2), 153 (2021). https://doi.org/10.3390/healthcare9020153, https://app.dimensions.ai/details/publication/pub.1135094000andhttps://www.mdpi.com/2227-9032/9/2/153/pdf
  * [12] Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with em routing. In: International conference on learning representations (2018)
  * [13] Işın, A., Direkoğlu, C., Şah, M.: Review of mri-based brain tumor image segmentation using deep learning methods. Procedia Computer Science 102, 317–324 (2016)
  * [14] Ismael, M.R., Abdel-Qader, I.: Brain tumor classification via statistical features and back-propagation neural network. In: 2018 IEEE international conference on electro/information technology (EIT). pp. 0252–0257. IEEE (2018)
  * [15] Jain, A.: brain tumor segmentation u-net. https://github.com/adityajn105/brain-tumor-segmentation-unet (2020), [Online; accessed January 08, 2021]
  * [16] Jamel, T.M., Khammas, B.M.: Implementation of a sigmoid activation function for neural network using fpga. In: 13th Scientific Conference of Al-Ma’moon University College. vol. 13 (2012)
  * [17] Minaee, S., Boykov, Y.Y., Porikli, F., Plaza, A.J., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [18] MrGiovanni: U-net++ keras. https://github.com/MrGiovanni/UNetPlusPlus (2020), [Online; accessed March 12, 2021]
  * [19] Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., Glocker, B., Rueckert, D.: Attention u-net: Learning where to look for the pancreas (2018)
  * [20] Pashaei, A., Sajedi, H., Jazayeri, N.: Brain tumor classification via convolutional neural network and extreme learning machines. In: 2018 8th International conference on computer and knowledge engineering (ICCKE). pp. 314–319. IEEE (2018)
  * [21] Punn, N.S., Agarwal, S.: Chs-net: A deep learning approach for hierarchical segmentation of covid-19 infected ct images. arXiv preprint arXiv:2012.07079 (2020)
  * [22] Punn, N.S., Agarwal, S.: Inception u-net architecture for semantic segmentation to identify nuclei in microscopy cell images. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16(1), 1–15 (2020)
  * [23] Punn, N.S., Agarwal, S.: Multi-modality encoded fusion with 3d inception u-net and decoder model for brain tumor segmentation. Multimedia Tools and Applications pp. 1–16 (2020)
  * [24] Punn, N.S., Agarwal, S.: Modality specific u-net variants for biomedical image segmentation: A survey. arXiv preprint arXiv:2107.04537 (2021)
  * [25] raghakot: keras-resnet. https://github.com/raghakot/keras-resnet (2017), [Online; accessed March 18, 2021]
  * [26] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [27] Ruder, S.: An overview of gradient descent optimization algorithms (2017)
  * [28] Thakur, R.: step by step vgg16 implementation in keras for beginners. https://towardsdatascience.com/step-by-step-vgg16-implementation-in-keras-for-beginners-a833c686ae6c (2019), [Online; accessed March 20, 2021]
  * [29] Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07836 (2018)
  * [30] Zhou, T., Ruan, S., Canu, S.: A review: Deep learning for medical image segmentation using multi-modality fusion. Array 3, 100004 (2019)
  * [31] Zhou, Z., Siddiquee, M., Tajbakhsh, N., Liang, J.U.: A nested u-net architecture for medical image segmentation. arXiv preprint arXiv:1807.10165 (2018)