# Perturbation amplification near the stagnation point of blunt bodies

Eduardo Martini  Département Fluides, Thermique, Combustion, Institut Pprime,
CNRS-Universitè de Poitiers-ENSMA, 1 Bd Marie et Pierre Curie, Futuroscope
Chasseneuil, F86962, la Vienne, France  Commissariat à l’Énergie Atomique,
CESTA, 15 Avenue des Sablières,Le Barp, 331163, la Vienne, France  Clement
Caillaud  Département Fluides, Thermique, Combustion, Institut Pprime, CNRS-
Universitè de Poitiers-ENSMA, 1 Bd Marie et Pierre Curie, Futuroscope
Chasseneuil, F86962, la Vienne, France  Commissariat à l’Énergie Atomique,
CESTA, 15 Avenue des Sablières,Le Barp, 331163, la Vienne, France  Guillaume
Lehnasch  Département Fluides, Thermique, Combustion, Institut Pprime, CNRS-
Universitè de Poitiers-ENSMA, 1 Bd Marie et Pierre Curie, Futuroscope
Chasseneuil, F86962, la Vienne, France  Peter Jordan  Département Fluides,
Thermique, Combustion, Institut Pprime, CNRS-Universitè de Poitiers-ENSMA, 1
Bd Marie et Pierre Curie, Futuroscope Chasseneuil, F86962, la Vienne, France
Oliver Schmidt  Departamane of Mechanical and Aerospace Engineering,
University of California San Diego, 9500 Gilman Drive, La Jolla, 10587,
California, USA

###### Abstract

Different transition to turbulence routes for the flow around blunt bodies are
possible. Non-modal amplification of perturbations via the lift-up effect has
recently been explored to explain transition near the stagnation point in
axisymmetric bodies. However, only perturbations already present in the
boundary layer can be amplified, and the mechanisms by which free-stream
perturbations enter the boundary layer have not yet been fully explored. In
this study, we present an investigation of how disturbances enter the boundary
layer via the stagnation point. This linear mechanism is expected to dominate
over non-linear mechanisms previously identified on the formation of boundary
layer perturbations at low turbulence intensity levels. A parametric
investigation is presented, revealing trends with Reynolds and Mach numbers.

##  1 Introduction

Turbulent boundary layers enhance viscous drag and higher heat flux to the
bodies about which they develop. Predicting where a boundary layer will
transition from a laminar to a turbulent state is necessary to predict
aerodynamic performance. In supersonic flight, the transition point is also a
key design parameter as turbulent boundary layers exchange more heat with the
surface and thus require more robust thermal shielding.

Earlier work issued from local stability theory describes several paths for
transition in wall-bounded flows (Morkovin, 1969). In scenarios where
perturbations present in the free stream or due to surface roughness are
small, the mechanism that leads to transition is typically modal, i.e.,
unstable modes in the boundary layer grow exponentially and eventually trigger
transition. Another possibility is by-pass transition, which occurs for
moderate perturbations. In this case, non-modal mechanisms amplify small-but-
finite, disturbances. These may then reach significant amplitudes and lead to
flow transition.

As only perturbations within the boundary layer can lead to its transition, it
is necessary to understand their origin, i.e., how they penetrate to, or are
generated in, the boundary layer. For low Mach numbers, flow over a flat plate
is typically insensitive to acoustic perturbations. The mismatch of wave
number and frequency between the boundary layer structures and acoustic
perturbations leads to a small coupling between them. However, Goldstein
(1985) has shown that points of curvature discontinuity act as a length-scale
conversion mechanism, allowing acoustic waves to excite Tollmien-Schlichting
(TS) waves.

Brandt et al. (2004) investigated mechanisms by which free-stream vortical
perturbations can enter an incompressible boundary layer. Brandt, Schlatter,
and Henningson observed that for moderate free-stream turbulence intensity,
perturbations penetrate the flat-plate boundary layer mainly via non-linear
interactions, becoming more relevant at larger free-stream turbulence
intensities. Amplification of pre-existing boundary layer structures was
dominant for lower turbulence intensities and larger integral wavelengths. The
study suggests that such vortical structures may penetrate the boundary layer
at the leading edge, although this mechanism was not investigated. Thus, such
a linear mechanism is expected to dominate receptivity at low turbulent
intensities.

For higher Mach numbers, different receptivity mechanisms may be observed. For
example, in supersonic flows there are regions for which Mack waves (Mack,
1965) have acoustic wavelengths and can thus be directly excited by acoustic
waves (Tam, 1978).

Several studies on this receptivity mechanism rely on numerical integration of
the flow dynamics. In these, smooth surfaces, i.e., with continuous
curvatures, are typically used, preventing the results from being contaminated
by the length-conversion mechanism identified by Goldstein. Leading edges
shaped as super-ellipses (Buter and Reed, 1994; Wanderley and Corke, 2001; Lin
et al., 1992; Shahriari et al., 2016) and parabolic bodies (Haddad and Corke,
1998) are among the examples studied. Haddad and Corke have identified that
the receptivity of TS waves decreases with Reynolds number on a paraboloid
body, i.e., with increasing bluntness, a result which contrasts with a more
recent review on thick flat plates with curved leading edges (Shahriari et
al., 2016). The reason for this discrepancy is unclear, but they may be
related to different flow regimes. The study of Haddad and Corke may only be
valid around the stagnation point, and, while the super-ellipse studied by
Shahriari et al. is formally smooth, the surrounding flow can exhibit strong
gradients, whose contribution to the receptivity mechanism is unclear. As a
favorable pressure gradient stabilizes TS waves near the stagnation point, TS
waves amplitudes are typically small around it. However, they can be amplified
downstream, where the pressure gradient vanishes. Mack waves exhibit a similar
behavior in supersonic flows due to the low velocities near the stagnation
point.

Nevertheless, there are circumstances where transition is observed close to
the leading edge, e.g., in hypersonic blunt bodies (Paredes et al., 2017). For
these configurations, different mechanisms have been proposed, including, on
one hand, perturbation amplification in the entropy layer (Fedorov and Tumin,
2004; Paredes et al., 2020), which then enters the boundary layer when it
swallows the entropy layer and, on the other, leading-edge modal instabilities
(Lin and Malik, 1996, 1997). However, the former can only lead to transition
after the ingestion point, which occurs further downstream than the observed
transition point, and the latter are only active in the presence of a
crossflow. When no crossflow is present, such modal mechanisms cannot explain
these transitions, motivating the study of non-modal mechanisms in the nose
region (Paredes et al., 2017).

The lift-up is a non-modal mechanism that is ubiquitous in shear flow. It
consists of streamwise vortices that move fast flow from regions of higher
velocities to regions of lower velocities, and vice versa, creating regions of
fast and slow streamwise velocities known as streaks. Under a parallel flow
assumption, linear stability theory predicts amplification factors to scale
with $Re^{2}$ for spanwise wavelengths of the order of two boundary layer
thickness in both compressible and incompressible flows (Andersson et al.,
1999; Luchini, 2000; Schmid and Henningson, 2012; Hanifi et al., 1996). Large
perturbation amplification is also observed in spatially evolving boundary
layers (Monokrousos et al., 2010; Paredes et al., 2017; Towne et al., 2022;
Kamal et al., 2023).

The dynamics of vorticity entering the boundary layer via the leading edge
have received less attention. Obrist and Schmid (2003) studied the problem
using Hiemenz’s self-similar flow model. However, there are virtually no
studies investigating the problem for curvatures typical of a leading edge.

In this study, we present a systematic investigation of non-modal mechanisms
at the leading edge using a paraboloid body as a flow model. We systematically
investigate the role of bluntness and compressibility to provide a cartography
of the phenomena.

The paper is structured as follows. Section 2 presents the flow around the
paraboloid body. Section 3 presents the numerical methods used in the linear
stability analysis. Section 4 presents aç study of the perturbation
amplification mechanisms near the stagnation point using the resolvent
framework. Section 5 expands the results to investigate compressible flows,
with Mach number up to $1.2$. Section 6 presents the final conclusions.

##  2 Paraboloid body

The body surface studied is given by

$\displaystyle x=(z^{2}+y^{2})-1/4,$ (1)

with the free-stream flow moving towards $+z$. The flow parameters are made
non-dimensional using the freestream density, viscosity, and velocity. As the
reference length, we use the curvature diameter of the leading edge.

Three coordinate systems are considered. Cylindrical coordinates are used in
all computations, which will be detailed later. Paraboloid coordinates
$(\sigma,\tau)$ are used and defined in section 2.1 to obtain an inviscid
potential flow analytically. Finally, body-fitted coordinates $(s,d)$ are
defined as the distance from the leading edge along the body surface and the
distance from the body surface. They are used for visualization only. Figure 1
illustrates the problem and coordinate systems used.

Figure 1: Illustration of the geometry and the coordinate systems used. Red
and blue lines illustrate parabolic and body-fitted coordinates, respectively.
The thick black line indicates the body surface. The dotted circle is tangent
to the leading edge. The arrow indicates the freestream velocity. Points with
a distance of multiples of $0.5$ length units from the stagnation point are
shown for reference.

###  2.1 Inviscid and incompressible flow

The potential flow around the body is obtained analytically using parabolic
coordinates, $(\sigma,\tau,\phi)$, defined as,

$\displaystyle z+\mathrm{i}r=$ $\displaystyle(\sigma+\mathrm{i}\tau)^{2},$
$\displaystyle r=$ $\displaystyle\sqrt{(x^{2}+y^{2})},$ $\displaystyle\phi=$
$\displaystyle\tan^{-1}(y/x).$ (2)

The body surface is located at $\sigma=\sigma_{0}=-1/2$.

The flow potential function, $\psi$, satisfies the Laplace equation,

$\displaystyle\nabla^{2}_{\sigma,\tau,\phi}\psi=\frac{1}{\sigma}\frac{d}{d\sigma}\left(\sigma\frac{d\psi}{d\sigma}\right)+\frac{1}{\tau}\frac{d}{d\tau}\left(\tau\frac{d\psi}{d\tau}\right)+\frac{d^{2}\psi}{d\phi^{2}},$
(3)

with, as boundary conditions, no penetration at the body surface and the
convergence to the free-stream conditions far from the body. These conditions
read

$\displaystyle\frac{d\psi}{d\sigma}$ $\displaystyle=0$
$\displaystyle,\text{for }$ $\displaystyle\sigma=\sigma_{0}\quad\text{and}$
(4) $\displaystyle\psi$ $\displaystyle=z=(\sigma^{2}-\tau^{2})$
$\displaystyle,\text{for }$ $\displaystyle\sigma\to\infty.$ (5)

The solution to (3) under (4) and (5) is given by

$\displaystyle\psi=\sigma^{2}-\tau^{2}+2\sigma_{0}^{2}\log(\sigma).$ (6)

The velocity field is then recovered as

$\displaystyle U_{z}=\frac{\partial\psi}{\partial
x}=\frac{\partial\psi}{\partial\sigma}\frac{\partial\sigma}{\partial
x}+\frac{\partial\psi}{\partial\sigma}\frac{\partial\tau}{\partial x},$ (7)
$\displaystyle U_{r}=\frac{\partial\psi}{\partial
y}=\frac{\partial\psi}{\partial\sigma}\frac{\partial\sigma}{\partial
y}+\frac{\partial\psi}{\partial\sigma}\frac{\partial\tau}{\partial y}.$ (8)

The potential flow is illustrated in figure 2, where the stagnation point and
subsequent flow reacceleration are observed.

Figure 2: Potential flow around the paraboloid body.

###  2.2 Viscous incompressible flow

Reynolds numbers
---
$100$ | $500$ | $1\,000$ | $5\,000$ | $30\,000$ | $60\,000$ | $120\,000$ | $180\,000$ | $240\,000$ | $300\,000$
Table 1: List of Reynolds numbers investigated for the incompressible case.

A 2D axisymmetric model was used to obtain base flows for the viscous problem.
No-slip boundary conditions were applied at the body surface; outflow
conditions were used on the right-most edge of the domain, and the potential
flow was imposed on the left-most edge as an inflow condition, reducing the
domain size required to converge the solutions. The problem was time-marched
using the open-source code Nek5000 (Fischer and Patera, 1989) until the norm
of the velocity time derivative became smaller than $10^{-8}$.

This process was repeated for several Reynolds numbers between $100$ and
$300\,000$, listed in table 1. The domain size was adapted for each Reynolds
number: larger domains were used at lower Reynolds numbers to minimize
blockage effects, and smaller domains at larger Reynolds numbers to avoid
unnecessary computational cost. The Reynolds-number range was chosen to reach
values sufficiently high for an asymptotic trend to be observed, and lower
values where the behavior deviates from the trend.

Figure 3 illustrates the base flows obtained. For $Re\gtrapprox 5\,000$, the
flow is qualitatively similar to the potential flow shown in figure 2 but with
the formation of a boundary layer around the body surface. The base flows are
similar but with thinner boundary layers for higher Reynolds numbers.

Figure 3: Velocity magnitude of the baseflow for different Reynolds numbers.
For clearness, only selected cases are shown.

Figure LABEL:fig:bl_profiles shows the wall-tangential velocity profile for
$s=1$, i.e., at unit distance from the stagnation point. A thicker boundary
layer is formed for lower Reynolds numbers, and the flow at its edge has
larger velocities than the potential flow, compensating for the loss of mass
flow closer to the wall. This effect reduces with $Re$ and is negligible for
$Re>5\,000$. As expected, the flow converges to the potential solution away
from the wall for higher Reynolds numbers.

(a)

(b)

Figure 4:  Boundary layer profile and thickness for the flow around the
paraboloid body.

The decreasing velocity magnitude with the potential flow’s wall distance and
the wall’s no-slip boundary condition creates a velocity maximum at a distance
$d_{max}$ from the wall. We thus define the boundary layer thickness for the
problem as,

$\displaystyle\delta^{*}(s)=\int_{0}^{d_{max}}\left(\frac{U_{\parallel,\text{pot}}(s,d)-U_{\parallel}(s,d)}{U_{\parallel,\text{pot}}(s,0)}\right)\differential
d.$ (9)

where $U_{\parallel}$ indicates velocity component parallel to the wall, and
$U_{\parallel,\text{pot}}$ corresponds to the potential solution. Note that
$U_{\parallel,pot}(s,0)=1$ for a flat plate, and (9) recovers the classical
definition of the displacement thickness.

Figure LABEL:fig:bl_deltastar shows the evolution of the boundary layer
profile. Near the stagnation point, the boundary layer is formed with a
thickness that scales with $Re^{-1/2}$, as predicted when the region is
modeled by a Homann flow (Homann, 1936; Weidman, 2012). Downstream, the
boundary layer evolution reaches the asymptotic limit where it scales with
$Re^{-1/2}$ for $Re\gtrapprox 5\,000$.

##  3 Numerical method

We will perform resolvent analysis on a 2D axisymmetric flow to study the
perturbation amplification. The numerical method is derived from that used by
Schmidt et al. (2018) and Schmidt and Rist (2011, 2014), with the code adapted
to use curvilinear coordinates. We refer readers to the original document for
details of the code, describing next only its main features.

###  3.1 Resolvent analysis

The discretized linearized Navier-Stokes equations written in input-output
form in the frequency domain

$\displaystyle\begin{aligned}
(-\mathrm{i}\omega-\mathbf{A})\hat{\boldsymbol{u}}(\omega)&=\mathbf{B}\hat{\boldsymbol{f}}(\omega),\\\
\hat{\boldsymbol{y}}(\omega)&=\mathbf{C}\hat{\boldsymbol{u}}(\omega),\end{aligned}$
(10)

where $\mathbf{A}$ is the discretized representation of linearized Navier-
Stokes operator around a fixed point, $\mathbf{B}$ and $\mathbf{C}$ are
matrices introduced to limit input (forcing) and output (responses) spaces.
E.g., these can be used to limit the forcing and response to regions near the
boundary layer, avoiding the contamination of the results by free-stream modes
(Nogueira et al., 2020). The forcing and response vectors are, respectivelly,
$\hat{\boldsymbol{f}}=[\hat{f}_{\rho},\hat{f}_{u_{z}},\hat{f}_{u_{r}},\hat{f}_{u_{\theta}},\hat{f}_{T}]^{T}$,
and
$\hat{\boldsymbol{u}}=[\hat{\rho},\hat{u}_{z},\hat{u}_{r},\hat{u}_{\theta},\hat{T}]^{T}$
.

A direct relation between input and the output is obtained as

$\hat{\boldsymbol{y}}(\omega)=\mathbf{R}(\omega)\hat{\boldsymbol{f}}(\omega),$
(11)

where

$\mathbf{R}(\omega)=\mathbf{C}(-\mathrm{i}\omega-\mathbf{A})^{-1}\mathbf{B},$
(12)

is the resolvent operator (Schmid and Henningson, 2012).

A forcing term is said to be optimal if it maximizes the Rayleight ratio,

$\sigma^{2}(\omega)=\dfrac{\left\lVert\hat{\boldsymbol{y}}(\omega)\right\lVert_{\mathbf{W}_{y}}}{\left\lVert\hat{\boldsymbol{f}}(\omega)\right\lVert_{\mathbf{W}_{f}}}=\dfrac{\left\lVert\mathbf{R}(\omega)\hat{\boldsymbol{f}}(\omega)\right\lVert_{\mathbf{W}_{y}}}{\left\lVert\hat{\boldsymbol{f}}(\omega)\right\lVert_{\mathbf{W}_{f}}},$
(13)

and can be obtained from a singular value decomposition (SVD) of the weighted
resolvent operator
$\mathbf{R}^{\prime}(\omega)=\mathbf{W}_{y}^{-1/2}\mathbf{R}(\omega)\mathbf{W}_{f}^{1/2}$,
where $\mathbf{W}_{y/f}$ are the weight matrices for the response and forcing,
respectively. In this study, the weight matrices include integration
quadrature weight and an energy norm for compressible flows. Two different
energy norms are used, a kinetic,

$\displaystyle\lVert\hat{\boldsymbol{y}}\rVert^{2}=\int\overline{\rho}\left(|\hat{u}_{z}|^{2}+|\hat{u}_{\theta}|^{2}+|\hat{u}_{r}|^{2}\right)\differential
V$ (14)

and a compressible (Chu, 1965; Hanifi et al., 1996),

$\displaystyle\lVert\hat{\boldsymbol{y}}\rVert^{2}=\int\left(\frac{\overline{T}}{\overline{\rho}\gamma
M^{2}}|\hat{\rho}|^{2}+\overline{\rho}|\hat{u}_{z}|^{2}+\overline{\rho}|\hat{u}_{\theta}|^{2}+\overline{\rho}|\hat{u}_{r}|^{2}\frac{\overline{\rho}}{\overline{T}\gamma(\gamma-1)M^{2}}|\hat{T}|^{2}\right)dV.$
(15)

norms, where the overline variables refer to the baseflow fields, $\gamma$ is
the adiabatic coefficient, and $M$ is the Mach number.

Different strategies to compute the SVD can be used. For problems with only
one inhomogeneous direction, $\mathbf{R}(\omega)$ is a small matrix, and
standard tools are effective in obtaining the decomposition. On the other
hand, for three-dimensional problems, methods based on time-marching schemes
are typically used, which can be based on matrix-free approaches (Monokrousos
et al., 2010; Gómez et al., 2016; Martini et al., 2021; Farghadan et al.,
2021). For two-dimensional problems where one direction is homogeneous, one of
the most effective approaches is the use of sparse matrices to represent
$\mathbf{A}$. In this approach, the effect of the resolvent operator on a
vector is obtained by solving the linear system (10), which can be effectively
obtained using a lower-upper (LU) decomposition of its left-hand side. The
latter approach is employed in this work.

###  3.2 Curvilinear Coordinates

In order to construct the linearized operator, derivative matrices for the
domain are required. These matrices were constructed by mapping a square
domain (described with coordinates $\xi_{i}$), for which derivatives are
computed using standard finite differences, into the desired domain (described
with coordinates $x_{i}$), as illustrated in figure 5. Centered and uncentered
fourth-order finite-difference schemes were used on the bulk of the domain and
at the domain boundaries, respectively.

In general, the mapping between $\xi_{i}$ and $x_{i}$ is given implicitly: for
a given node in the original mesh, $x_{i}(\xi_{j})$ corresponds to the
coordinate of the same node on the mapped domain. The inverse function,
$\xi_{i}(x_{j})$, is likewise available. Henceforth, we will refer to
$x_{i}(\xi_{j})$ simply as mapping, and to $\xi_{i}(x_{j})$ as inverse
mapping.

Figure 5: Illustration of mapping between the computational ($\xi_{i}$) and
physical ($x_{i}$) meshes.

Using the chain rule, the derivative of a function $g$ is written as

$\partial_{x_{i}}g={J}_{ij}\partial_{\xi_{j}}g,$ (16)

where $J_{ij}=\frac{\partial\xi_{i}}{\partial x_{j}}$ are $(i,j)$ entries of
$\mathbf{J}$, the Jacobian of the mapping function. As only differentiation
matrices w.r.t. $\xi_{i}$ are a priori available, we construct first
$J_{ij}^{-1}=\frac{\partial x_{j}}{\partial\xi_{i}}$. We then obtain
$\mathbf{J}$ by inverting $\mathbf{J}^{-1}$.

Eq. (16) can be iterated to obtain second-order derivatives. However, this
increases the effective stencil of the method, which increases the cost of
solving the resulting linear systems. To avoid larger stencils, we describe
the second derivative as

$\displaystyle\begin{aligned}
\partial_{x_{i}}\partial_{x_{j}}g&=\partial_{x_{i}}\left({{J}}_{jq}\partial_{\xi_{p}}g\right)\\\
&=\left(\partial_{x_{i}}{{J}}_{jp}\right)\partial_{\xi_{p}}g+{{J}}^{-1}_{jp}\left(\partial_{x_{i}}\partial_{\xi_{p}}g\right)\\\
&=\left(\partial_{x_{i}}{{J}}_{jp}\right)\partial_{\xi_{p}}g+{{J}}^{-1}_{jp}{{J}}_{iq}\partial_{\xi_{q}}\partial_{\xi_{j}}g,\end{aligned}$
(17)

computing $\left(\partial_{x_{i}}{{J}}_{ij}\right)$ via the matrix identity

$\partial_{x_{i}}{{\mathbf{J}}}={\mathbf{J}}\left(\partial_{x_{i}}{\mathbf{J}}^{-1}\right){\mathbf{J}}.$
(18)

Finally, the term $\partial_{x_{i}}{\mathbf{J}}^{-1}$, reads

$\displaystyle\partial_{x_{i}}J_{lm}^{-1}$
$\displaystyle=J_{ji}^{-1}\partial_{\xi_{i}}J_{lm}^{-1}=J^{-1}_{ij}\dfrac{\partial^{2}x_{l}}{\partial\xi_{i}\partial\xi_{m}}.$
(19)

With this approach, the stencil size in the physical domain ($x_{i}$) has the
same size as that of the computational domain ($\xi_{i}$).

##  4 Receptivity

We start by investigating the receptivity of the $Re=30\,000$ baseflow. A
Cartesian grid with $100\times 100$ points was mapped around the body, with
nodes clustered near the surface to resolve the boundary layer. The
convergence of all the results presented was checked, and variations less than
$1\%$ when using a finer mesh were observed.

As will be shown next, the spatial support of the forcing/response modes can
extend far up/downstream. Fully capturing these structures requires a large
domain, making the analyses too costly. Since the study aims to investigate
how free-stream disturbances penetrate the boundary layer, the downstream
domain was limited to $s\lessapprox 2$. The domain upstream was limited to
$z\gtrapprox-6$. Down/upstream of these limits, a sponge zone is used to
dampen perturbations before the boundary of the computation domain.

Figure 6 shows the frequency dependency of the leading gains for different
azimuthal wavenumbers. The largest gain is found for $\omega=0$, first
evidence that the lift-up mechanism is dominant around the leading edge. We
henceforth concentrate the study on gains associated with zero-frequency
disturbances.

Figure 6: Gain variation with frequency for the $Re=30\,000$ flow.
Leading/suboptimal gains in solid/dashed lines for different azimuthal numbers
$m$.

(a)

(b)

Figure 7: Leading gains (a) and suboptimal gains for the dominant azimuthal
number (b). The unlabelled lines in (a) correspond to $Re=100,500,1\,000,$ and
$5\,000$, from the bottom up.

We next perform a systematic parametric investigation, computing gains at
$\omega=0$ for different Reynolds and azimuthal numbers. The numerical grid
was adapted for each scenario as the boundary layer thickness varied
considerably between the different cases. The Cartesian grid size between
$75\times 75$ and $100\times 175$ gridpoints were used, with the last number
corresponding to the wall-normal direction near the body surface, to
discretize the boundary layer. The results are presented in Figure
LABEL:fig:gains_opt, which shows that both the maximum gain and the
corresponding azimuthal wavenumber increase with $Re$. Figure
LABEL:fig:gains_subopt shows that the difference between the leading and the
suboptimal gains tends to also increase with $Re$, which points towards an
increasing selectivity of disturbances.

(a)

(b)

Figure 8: Leading forcing and response modes (a) and the curl of the forcing
mode (b) for $Re=5\,000$ and $m=9$. The modes are normalized by their largest
absolute value. Color scales indicate the real part of the modes. Figure 9:
Leading and suboptimal forcing and response modes. Results for $Re=5\,000$ and
$m=9$. The modes are normalized by their largest absolute value. Color scales
indicate the real part of the modes.  Figure 10: Spatial support of the
leading forcing and response modes. From top to bottom, results for
$Re=5\,000$, $30\,000$, and $60\,000$, with azimuthal wave number
corresponding to the highest gain for each $Re$. The modes are normalized by
their largest absolute value. Color scales indicate the real part of the
modes.

To illustrate the forcing and response modes, Figure 8 shows their support for
the highest gain observed for $Re=5\,000$ and $m=9$. Although this case has a
moderate gain, the relatively thick boundary layer facilitates visualization
of the flow features. The modes for higher Reynolds numbers are qualitatively
similar. Both forcing and response modes are concentrated in the $z$ and
$\theta$ components. The forcing mode excites vorticity perturbations upstream
of the leading edge. Over the surface, the response mode is mostly dominated
by the streamwise velocity component. These observations suggest that the
dominant mechanism at play consists of free-stream streamwise vortices
penetrating the boundary layer at the stagnation point and creating streamwise
streaks via the lift-up mechanism. The $\rho$ component of the vorticity being
active in creating the streaks near the stagnation point; and the $z$
component being active further downstream, where the flow becomes more aligned
with the $z$ axis. The spatial support of the forcing modes extends upstream
with the Reynolds numbers, and thus, resolving it requires large domains. We
limit the upstream domain, as it captures the relevant physical mechanism.

Figure 9 compares the support for the optimal and suboptimal force and
response modes. The suboptimal modes exhibit nodes in the radial direction for
both the forcing and response modes, similar to the streak force and response
modes in jets (Wang et al., 2021). The sub-optimal modes for all cases studied
here presented a similar structure.

Evolution of the modes with Reynolds number is shown in figure 10. The forcing
modes move close to the axis of symmetry for higher $Re$, which can be
interpreted as the region for which the streamlines converge to, or close to,
the boundary layer. The response modes are increasingly concentrated in the
boundary layer and move upstream and closer to the body surface with $Re$.
Figure 11 shows the contour of the response modes scaled by the boundary layer
thickness using the wall-fitted coordinates $(s,d)$, previously shown in
figure 1. The modes are distributed around the boundary layer displacement
thickness. A slight wall-normal displacement of the modes is observed for the
highest $Re$ numbers. Whether this trend persists for higher Reynolds numbers
or its implication is unclear.

Figure 12 shows the trends associated with the optimal gains, $\sigma_{opt}$,
and the corresponding azimuthal wavenumbers, $m_{opt}$, with $Re$. For $Re\geq
30\,000$, $\sigma_{opt}$ follows a $Re^{1/2}$ scaling, while $m_{opt}$ scales
with $Re^{0.34}$. These scalings contrast with those of a flat plate in two
respect. The gains scale with the boundary layer thickness
($Re_{\delta^{*}}=Re^{1/2}$), instead of its square, as found for flat plates.
This difference is possibly due to the finite domain used here: note that the
flat-plate $Re$ scaling is obtained for locally parallel approximations, which
assumes infinite domains. Another trend is the increase in the spanwise
wavelength in terms of the boundary layer thickness, with $m\propto
Re^{0.34}=Re_{\delta^{*}}^{0.68}$. We speculate that this difference may be
due to the axisymmetric geometry: a given value of $m$ implies a different
“spanwise” wavelength ($2\pi r/m$) for different positions at the body.
Different azimuthal numbers are optimally amplified locally at each position
as the boundary layer thickness increases with the distance to the stagnation
point. A trade-off between different amplification rates at different
positions may be responsible for the observed trend.

Figure 11: Isocontours of half of the leading response mode peak value for
different Reynolds numbers. The displacement thickness for $Re=300\,000$ is
shown for reference.  Figure 12: Reynolds number scaling of the maximum gain
and the corresponding azimuthal number. Dotted lines show a linear fit for
$Re\geq 3\times 10^{4}$.

##  5 Compressibility effects

Having explored receptivity of the incompressible paraboloid flow, we now
assess the effect of compressibility for increasing Mach numbers. We
investigate Mach numbers $0.3$, $0.6$, $0.9$, and $1.2$, and fix the Reynolds
number to the value of $30\,000$, for which the asymptotic trend observed in
Figure 12 is reached. Baseflows are obtained using the code SU2 (Economon et
al., 2016). Freestream values and adiabatic walls are used as boundary
conditions. Baseflows were converged until the residuals were smaller than
$10^{-8}$. The resulting baseflows are illustrated in Figure 13. Figure 14
shows the temperature and viscosity profile along the body surface. Higher
temperature and viscosity are observed near the stagnation point.

We smoothed the shock discontinuities for the stability analysis of the
supersonic flow cases, as shown in figure 13, and we used a finer mesh with
$200\times 200$ grid points. We considered two cases, one using a kinetic
energy norm (14), i.e., restricting the force- and response-mode norm to the
velocity components, and another considering temperature and density
variations, using a compressible energy norm (15). While it can be argued to
be more generic, it has been argued that velocity perturbations are more
likely to trigger flow transition. We thus choose to study both.

Figure 15 shows the gains associated with flows with increasing Mach numbers.
Results are similar to those of the incompressible flow, with a monotonic
increase in value with Mach number, which is consistent with the results
presented by Tumin and Reshotko (2001). The reduction of the optimal spanwise
wavenumber shown in their study is also observed in figure 15. The trend of
the peak gain for each condition is summarized in figure 16.

The support of the optimal forcing and response modes are shown in figure 17.
The support of the forcing terms of the momentum equations remains
qualitatively similar, and the shock, located around $z\approx 2$, does not
seem to affect these terms. Extra forcing terms for the continuity and energy
equations appear, showing a transition for the regions before and after the
shock. Again, the flow responses are qualitatively similar, but with density
and temperature fluctuations becoming more pronounced for higher Mach numbers.

Figure 13: Illustration of the velocity profiles for different Mach numbers
for $Re=3\times 10^{4}$. Figure 14: Viscosity (dashed lines) and temperature
(solid lines) along the paraboloid wall. Figure 15: Gains for the leading zero
frequency mode for different Mach numbers. Solid lines indicate results where
forcing and responses are restricted to the momentum equations and velocity
components. Dashed lines consider temperature and density fluctuations. Figure
16: Gains trend with Mach number.

Figure 17: Forcing and response modes for $m=21$ modes for the compressible
flow cases. Color scales indicate the real part of the modes.

##  6 Conclusions

We have presented a parametric investigation of non-modal mechanisms at the
leading edge of blunt bodies. Reynolds- and Mach-numbers trends associated
with the perturbations optimally amplified by the flow were identified.

Optimal perturbations were obtained using resolvent analysis. To study a
paraboloid body, a curvilinear grid formulation was implemented in a linear
stability code. The baseflow used for analysis was obtained by time marching
the non-linear flow equations.

Close to the stagnation point, maximum perturbation amplification occurs for
zero-frequency disturbances. The optimal modes indicate a receptivity
mechanism associated with free-stream vorticity that enters the boundary layer
in the vicinity of the stagnation point and then uses the lift-up effect to
create streaky structures at the body surface.

In the incompressible regime, a parametric investigation reveals that for
increasing Reynolds number, i.e., increasing bluntness, the maximum
amplification factor increases with $Re^{1/2}$. The result contrasts with the
reduction of receptivity of Tollmien-Schlichting waves with bluntness
previously reported for the same geometry. Increases in Mach number also lead
to larger perturbation amplification. These trends reinforce the importance of
non-modal mechanisms for blunt bodies at high Reynolds numbers in the
incompressible and compressible limits.

The results complement previous receptivity studies. The trends presented are
consistent with the switch from modal to non-modal transition mechanisms in
hypersonic flows around blunt bodies (Paredes et al., 2017). In the low Mach
regime, the decreasing receptivity of Tollmien-Schlichting waves with
bluntness (Haddad and Corke, 1998) points towards a similar phenomenon in
incompressible flow: as bluntness scales, the role of non-modal mechanisms
tends to overcome that of modal mechanisms, potentially leading to a
transition to turbulent close to the stagnation point.

Finally, the linear mechanism by which free-stream disturbances penetrate the
boundary layer complements previously studied transition routes for flow over
a flat plate. While for moderate free-stream disturbances in the
incompressible limit, non-linear receptivity is the dominant mechanism by
which perturbations enter the boundary layer Brandt et al. (2004), at lower
turbulence intensities, a linear receptivity mechanism tends to dominate. We
have thus studied one such mechanism: the ingestion of vortical disturbances
at the stagnation point.

This work also complements previous studies of the blunt body paradox by
exploring perturbation amplification trends near the stagnation point. The
paradox comes from observations that, in hypersonic flows, the transition
points move downstream with increasing bluntness up to a critical bluntness,
after which it quickly moves upstream. The trends of the perturbation
amplification with Reynolds number presented in this work is consistent with
investigations indicating that the transition reversal is due to a non-modal
mechanism, and can be used as a departure point for further studies of the
phenomena.

## Acknowledgments

This work is part of the project TRANSITION supported by Région Nouvelle-
Aquitaine under grant 2018-1R10220. This work has been granted access to the
HPC resources of IDRIS under the allocations A0092A10868 and A0112A10868 made
by GENCI (Grand Equipement National de Calcul Scientifique).

## References

  * Andersson et al. [1999] P. Andersson, M. Berggren, and D. S. Henningson.  Optimal disturbances and bypass transition in boundary layers.  _Physics of Fluids_ , 11(1):134–150, Jan. 1999\.  ISSN 1070-6631, 1089-7666.  10.1063/1.869908.
  * Brandt et al. [2004] L. Brandt, P. Schlatter, and D. S. Henningson.  Transition in boundary layers subject to free-stream turbulence.  _Journal of Fluid Mechanics_ , 517:167–198, Oct. 2004.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112004000941.
  * Buter and Reed [1994] TA. Buter and HL. Reed.  Boundary layer receptivity to free-stream vorticity.  _Physics of Fluids_ , 6(10):3368–3379, 1994.
  * Chu [1965] B.-T. Chu.  On the energy transfer to small disturbances in fluid flow (Part I).  _Acta Mechanica_ , 1(3):215–234, 1965.
  * Economon et al. [2016] T. D. Economon, F. Palacios, S. R. Copeland, T. W. Lukaczyk, and J. J. Alonso.  SU2: An Open-Source Suite for Multiphysics Simulation and Design.  _AIAA Journal_ , 54(3):828–846, Mar. 2016.  ISSN 0001-1452, 1533-385X.  10.2514/1.J053813.
  * Farghadan et al. [2021] A. Farghadan, A. Towne, E. Martini, and A. Cavalieri.  A randomized time-domain algorithm for efficiently computing resolvent modes.  In _AIAA Paper_ , pages 2021–2896. American Institute of Aeronautics and Astronautics, Aug. 2021.  ISBN 978-1-62410-610-1.  10.2514/6.2021-2896.
  * Fedorov and Tumin [2004] A. Fedorov and A. Tumin.  Evolution of Disturbances in Entropy Layer on Blunted Plate in Supersonic Flow.  _AIAA Journal_ , 42(1):89–94, Jan. 2004.  ISSN 0001-1452, 1533-385X.  10.2514/1.9033.
  * Fischer and Patera [1989] P. F. Fischer and A. T. Patera.  Parallel spectral element methods for the incompressible Navier-Stokes equations.  In _Solution of Superlarge Problems in Computational Mechanics_ , pages 49–65. Springer, 1989.
  * Goldstein [1985] M. E. Goldstein.  Scattering of acoustic waves into Tollmien-Schlichting waves by small streamwise variations in surface geometry.  _Journal of Fluid Mechanics_ , 154:509–529, May 1985.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112085001641.
  * Gómez et al. [2016] F. Gómez, H. M. Blackburn, M. Rudman, A. S. Sharma, and B. J. McKeon.  A reduced-order model of three-dimensional unsteady flow in a cavity based on the resolvent operator.  _Journal of Fluid Mechanics_ , 798:R2, 2016.  10.1017/jfm.2016.339.
  * Haddad and Corke [1998] O. M. Haddad and T. C. Corke.  Boundary layer receptivity to free-stream sound on parabolic bodies.  _Journal of Fluid Mechanics_ , 368:1–26, Aug. 1998.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112098001311.
  * Hanifi et al. [1996] A. Hanifi, P. J. Schmid, and D. S. Henningson.  Transient growth in compressible boundary layer flow.  _Physics of Fluids_ , 8(3):826–837, Mar. 1996\.  ISSN 1070-6631, 1089-7666.  10.1063/1.868864.
  * Homann [1936] F. Homann.  Einfluß großer zähigkeit bei strömung um zylinder.  _Forschung auf dem Gebiet des Ingenieurwesens A_ , 7(1):1–10, 1936.
  * Kamal et al. [2023] O. Kamal, M. T. Lakebrink, and T. Colonius.  Global receptivity analysis: Physically realizable input–output analysis.  _Journal of Fluid Mechanics_ , 956:R5–undefined, 2023.  ISSN 1469-7645.  10.1017/jfm.2023.48.
  * Lin et al. [1992] N. Lin, H. L. Reed, and W. S. Saric.  Effect of Leading-Edge Geometry on Boundary-Layer Receptivity to Freestream Sound.  In M. Y. Hussaini, A. Kumar, and C. L. Streett, editors, _Instability, Transition, and Turbulence_ , pages 421–440, New York, NY, 1992. Springer New York.  ISBN 978-1-4612-2956-8.
  * Lin and Malik [1996] R.-S. Lin and M. R. Malik.  On the stability of attachment-line boundary layers. Part 1. The incompressible swept Hiemenz flow.  _Journal of Fluid Mechanics_ , 311:239–255, 1996.
  * Lin and Malik [1997] R.-S. Lin and M. R. Malik.  On the stability of attachment-line boundary layers. Part 2. The effect of leading-edge curvature.  _Journal of Fluid Mechanics_ , 333:125–137, 1997.
  * Luchini [2000] P. Luchini.  Reynolds-number-independent instability of the boundary layer over a flat surface: Optimal perturbations.  _Journal of Fluid Mechanics_ , 404:289–309, Feb. 2000.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112099007259.
  * Mack [1965] LM. Mack.  The stability of the compressible laminar boundary layer according to a direct numerical solution(Stability of compressible laminar boundary layer according to direct numerical solution).  _1965._ , pages 329–362, 1965.
  * Martini et al. [2021] E. Martini, D. Rodríguez, A. Towne, and A. V. Cavalieri.  Efficient computation of global resolvent modes.  _Journal of Fluid Mechanics_ , 919:A3, 2021.  ISSN 0022-1120.  10.1017/jfm.2021.364.
  * Monokrousos et al. [2010] A. Monokrousos, E. Åkervik, L. Brandt, and D. S. Henningson.  Global three-dimensional optimal disturbances in the Blasius boundary-layer flow using time-steppers.  _Journal of Fluid Mechanics_ , 650:181–214, May 2010.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112009993703.
  * Morkovin [1969] M. V. Morkovin.  On the Many Faces of Transition.  In C. S. Wells, editor, _Viscous Drag Reduction_ , pages 1–31. Springer US, Boston, MA, 1969.  ISBN 978-1-4899-5581-4 978-1-4899-5579-1.  10.1007/978-1-4899-5579-1_1.
  * Nogueira et al. [2020] P. A. S. Nogueira, A. V. Cavalieri, A. Hanifi, and D. S. Henningson.  Resolvent analysis in unbounded flows: Role of free-stream modes.  _Theoretical and Computational Fluid Dynamics_ , pages 1–14, 2020\.  10.1007/s00162-020-00519-x.
  * Obrist and Schmid [2003] D. Obrist and P. J. Schmid.  On the linear stability of swept attachment-line boundary layer flow. Part 2. Non-modal effects and receptivity.  _Journal of Fluid Mechanics_ , 493:31–58, Oct. 2003.  ISSN 00221120, 14697645.  10.1017/S0022112003005780.
  * Paredes et al. [2017] P. Paredes, M. M. Choudhari, and F. Li.  Blunt-body paradox and transient growth on a hypersonic spherical forebody.  _Physical Review Fluids_ , 2(5):053903, May 2017\.  ISSN 2469-990X.  10.1103/PhysRevFluids.2.053903.
  * Paredes et al. [2020] P. Paredes, M. M. Choudhari, and F. Li.  Mechanism for frustum transition over blunt cones at hypersonic speeds.  _Journal of Fluid Mechanics_ , 894:A22, July 2020.  ISSN 0022-1120, 1469-7645.  10.1017/jfm.2020.261.
  * Schmid and Henningson [2012] P. J. Schmid and D. S. Henningson.  _Stability and Transition in Shear Flows_ , volume 142.  Springer Science & Business Media, 2012.
  * Schmidt and Rist [2011] O. T. Schmidt and U. Rist.  Linear stability of compressible flow in a streamwise corner.  _Journal of Fluid Mechanics_ , 688:569–590, 2011.  10.1017/jfm.2011.405.
  * Schmidt and Rist [2014] O. T. Schmidt and U. Rist.  Viscid–inviscid pseudo-resonance in streamwise corner flow.  _Journal of Fluid Mechanics_ , 743:327–357, 2014.  10.1017/jfm.2014.31.
  * Schmidt et al. [2018] O. T. Schmidt, A. Towne, G. Rigas, T. Colonius, and G. A. Brès.  Spectral analysis of jet turbulence.  _Journal of Fluid Mechanics_ , 855:953–982, 2018.  10.1017/jfm.2018.675.
  * Shahriari et al. [2016] N. Shahriari, D. J. Bodony, A. Hanifi, and D. S. Henningson.  Acoustic receptivity simulations of flow past a flat plate with elliptic leading edge.  _Journal of Fluid Mechanics_ , 800:R2, Aug. 2016.  ISSN 0022-1120, 1469-7645.  10.1017/jfm.2016.433.
  * Tam [1978] C. K. W. Tam.  Excitation of instability waves in a two-dimensional shear layer by sound.  _Journal of Fluid Mechanics_ , 89(2):357–371, Nov. 1978.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112078002645.
  * Towne et al. [2022] A. Towne, G. Rigas, O. Kamal, E. Pickering, and T. Colonius.  Efficient global resolvent analysis via the one-way Navier–Stokes equations.  _Journal of Fluid Mechanics_ , 948:A9, Oct. 2022.  ISSN 0022-1120, 1469-7645.  10.1017/jfm.2022.647.
  * Tumin and Reshotko [2001] A. Tumin and E. Reshotko.  Spatial theory of optimal disturbances in boundary layers.  _Physics of Fluids_ , 13(7):2097–2104, July 2001\.  ISSN 1070-6631, 1089-7666.  10.1063/1.1378070.
  * Wanderley and Corke [2001] J. B. V. Wanderley and T. C. Corke.  Boundary layer receptivity to free-stream sound on elliptic leading edges of flat plates.  _Journal of Fluid Mechanics_ , 429:1–21, Feb. 2001.  ISSN 0022-1120, 1469-7645.  10.1017/S0022112000002548.
  * Wang et al. [2021] C. Wang, L. Lesshafft, A. V. G. Cavalieri, and P. Jordan.  The effect of streaks on the instability of jets.  _Journal of Fluid Mechanics_ , 910:A14, Mar. 2021.  ISSN 0022-1120, 1469-7645.  10.1017/jfm.2020.963.
  * Weidman [2012] P. D. Weidman.  Non-axisymmetric Homann stagnation-point flows.  _Journal of Fluid Mechanics_ , 702:460–469, 2012.  ISSN 00221120.  10.1017/jfm.2012.197.


# Advancing Generative AI for Portuguese with Open Decoder
Gervásio PT*

###### Abstract

To advance the neural decoding of Portuguese, in this paper we present a fully
open Transformer-based, instruction-tuned decoder model that sets a new state
of the art in this respect. To develop this decoder, which we named Gervásio
PT*, a strong LLaMA 2 7B model was used as a starting point, and its further
improvement through additional training was done over language resources that
include new instruction data sets of Portuguese prepared for this purpose,
which are also contributed in this paper. All versions of Gervásio are open
source and distributed for free under an open license, including for either
research or commercial usage, and can be run on consumer-grade hardware, thus
seeking to contribute to the advancement of research and innovation in
language technology for Portuguese.

Keywords: Portuguese, large language model, decoder, open source, open
license, open distribution

Advancing Generative AI for Portuguese with Open Decoder
Gervásio PT*

Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, António Branco
---
University of Lisbon
NLX - Natural Language and Speech Group, Department of Informatics
Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal
{rsdsantos, jrsilva, luis.gomes, jarodrigues<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

This paper presents a model that is the first competitive, 7 billion
parameter, fully open and fully documented large language model of the decoder
family of Transformers that is prepared specifically for the Portuguese
language, by means of instruction tuning, for both the European variant,
spoken in Portugal (PTPT) and the American variant, spoken in Brazil (PTBR).
By being fully open, it is open source and openly distributed for free under a
free license, including for research and commercial purposes. By being fully
documented, the new datasets that were specifically developed for its
construction can be reused, its development can be reproduced, and reported
performance scores can be independently assessed. By being fully open and
documented, its further development and improvement is openly available to the
community.

In the last half decade, the neural approach to natural language processing
became pervasive, with virtually any language processing task attaining top
performance under the Transformer architecture (Vaswani et al., 2017).
Initially proposed and explored in an encoder-decoder setup (Raffel et al.,
2020), subsequent research has shown the particular strengths of separate
encoder-only and decoder-only solutions (Devlin et al., 2019; He et al., 2021;
Brown et al., 2020), with decoders becoming specially notable with the
availability of ChatGPT to the general public (Ouyang et al., 2022; OpenAI,
2023).

Among the thousands of natural languages spoken in the world, English is the
one whose research is, by a huge margin, better funded and thus the one for
which more language resources exist, including the gigantic collections of
text that are necessary to train top performing large language models.
Consequently, the largest and best performing monolingual models have been
developed for this particular language (Touvron et al., 2023b; He et al.,
2021).

Seeking to build on the strength of such monolingual models, multilingual
models have also been developed. Typically, they are trained over datasets
where relatively small portions of data from a few other languages are added
to the data from English (Devlin et al., 2019; Scao et al., 2022).
Interestingly, these models have shown competitive performance in handling
tasks in languages other than English, leveraged by the massive volume of data
thus made available and outdoing the meager results that would otherwise be
obtained if a monolingual model had been trained only in the data available
for those languages alone (Pires et al., 2019).

In order to further mitigate the relative data scarceness impacting the non-
English languages, further approaches have been undertaken that include the
continuation of the self-supervised training with monolingual data from a
specific language. This continuation of causal language modelling (CLM) has
been experimented with over multilingual models or even monolingual English
models. Research has shown that when such training is appropriately continued,
the performance of the resulting model for that specific language exceeds the
performance of the baseline model on that language, whose training has not
been thus continued (Kaplan et al., 2020; Rodrigues et al., 2023).

By exploiting this approach of continuing the training of a previous strong
foundation model, we contribute a new model with instruction tuning to foster
the technological preparation of the Portuguese language. To the best of our
knowledge, this is the first decoder under the Transformer architecture that
is both (i) specifically improved for Portuguese, covering two variants of
this language, namely PTBR and PTPT, and (ii) fully open, that is it
cumulatively complies with all the features of being open source and openly
distributed for free under a most permissive license (including for research
and for commercial purposes). The model is available at
[https://huggingface.co/PORTULAN].

To the best of our knowledge and at the time of writing, Gervásio represents
the state of the art reported in the literature for open, 7 billion parameter
decoders for Portuguese, surpassing the model it is based on as well as
otherdecoders for Portuguese of similar size. The release of Gervásio,
alongside the instruction datasets used to train it and which are also a novel
contribution of this paper, seeks to contribute to foster research and
innovation for the language technology for Portuguese.

The remainder of this paper is organized as follows: Related work is covered
in the next Section 2; the data used to train and test the model is presented
in Section 3; Section 4 describes the decoder for Portuguese created in this
study and Section 5 presents and discusses the results of its evaluation. The
last Section 6 offers concluding remarks.

## 2\. Related work

In this section we discuss previous results and resources in the literature
that are related to the aim of the present paper. We first address decoders
for Portuguese that are publicly reported or publicly distributed, and then we
address the available options concerning the base model that can be used to be
continued to be trained with Portuguese data.

### 2.1. Decoders for Portuguese

Looking for decoders specifically developed or improved for Portuguese that
are publicly distributed and for which it is possible to find a publicly
available report, to the best of our knowledge there can be found only two
that, with 7 billion parameters or more, match or surpass the size of Gervásio
PT* contributed in the present paper, namely the Sabiá models with 7 and 65
billion parameters (Pires et al., 2023) It is worth noting that: these two
models were developed for only one of the variants of Portuguese, PTBR, but
not for PTPT; the 65 billion parameter model is reported in that publication
but it is not distributed; and the 7 billion parameter model is distributed in
a non open license, being its reuse restricted to research purposes only.

Other decoders that at the time of writing the present paper, can be found of
comparable size are not documented, besides being also only for only one of
the variants of Portuguese, namely PTBR: Boana, Cabra, Cabrita, Canarim.111All
on HuggingFace, at lrds-code/boana-7b-instruct, nicolasdec/CabraMistral7b-0.2,
22h/open-cabrita3b, and dominguesm/canarim-7b, respectively.

The other decoders, numbering about a dozen, that can be found for Portuguese
have a smaller size, and are also only for PTBR. The largest of these, with 3
billion parameters and named Cabrita 3B, is distributed in Hugging Face (HF)
and documented in a non peer-reviewed publication (Larcher et al., 2023). The
second largest is
Aira222https://huggingface.co/nicholasKluge/Aira-2-portuguese-1B7 and it is
presented in HF as having 1.7 billion parameters and being based on Bloom. No
evaluation results in benchmarks or downstream tasks for it are reported, it
has a residual number of downloads from HF and, being based on Bloom, it
inherits the restrictions from Bloom’s license and it is thus not fully open
as Gervásio.

Common to these encoders other than Sabiá, which are of similar or smaller
size, is that while they are publicly distributed, no public detailed
presentation of them seems to be provided, be it an implementation report or a
paper, either in pre-print or in peer-reviewed versions. This hampers knowing,
among other aspects, which datasets were used for their training and thus
hampers sensible comparison with other related work and models, which may risk
being evaluated in datasets where they were trained.

Turning to Sabiá, while there is a paper with its reporting (Pires et al.,
2023), this model was developed by a commercial company and the variant with 7
billion parameters is not openly distributed, with its license restricting its
use only for research, a restriction inherited from the license of LLaMA 1
(Touvron et al., 2023a), which was taken as its base model. The variant with
65 billion parameter, in turn, does not appear to be publicly distributed.
Sabiá is reported to have been obtained by continuing the training of LLaMA 1
both in its 7 billion and 65 billion parameter versions. A third version of
Sabiá was trained over GPT-J (Wang, 2021), with 6 billion parameters. All of
these were trained for the PTBR variant of Portuguese only.

Looking into the collection of tasks reported to have been used to evaluate
Sabiá, one finds a few that are common with the evaluation of Gervásio, namely
BoolQ and WSC, from SuperGLUE, and SST2, from GLUE, which were also machine
translated into PTBR to evaluate Sabiá.

Additionally, Sabiá’s authors present its performance scores in a few other
downstream tasks whose datasets did not result from machine translation from
English ones, but were developed originally in PTBR.

The performance scores from Sabiá’s publication are repeated in Section 5,
side by side with related scores of the Gervásio PTBR, for American
Portuguese. Against this background, and as it will be discussed at length in
that Section, at the time of this writing and to the best of our knowledge,
Gervásio offers the state of the art in terms of fully open encoders
specifically improved for Portuguese in both PTPT and PTBR variants, and it is
the first 7 billion parameter decoder specifically developed and distributed
for the PTPT variant.

### 2.2. Base models

In this connection, it is worth noting also that not only Gervásio happens to
be the top performing 7 billion open encoder for Portuguese, but also that at
the time of its development, it adopted the best possible setup and base code
available given the goals and requirements assumed for its construction.

There is a number of multilingual encoders reported in the literature, such as
mBART (Liu et al., 2020), XLM-R (Conneau et al., 2020), mT5 (Xue et al.,
2021), ByT5 (Xue et al., 2022), XGLM (Lin et al., 2022), mGPT (Shliazhko et
al., 2023), Bloom (Scao et al., 2022), and LLaMA (Touvron et al., 2023b), to
which the promising English open models Mistral (Jiang et al., 2023) and
Pythia (Biderman et al., 2023) were added in our considerations of the options
available. From these possibilities, many had to be excluded given their non-
open license, leaving only those from the Mistral, Bloom, Pythia, and LLaMA
families as viable bases on which to build Gervásio.

From these, we decided to leave out Mistral given that, unlike the others, it
is indicated to have been developed with no guardrails or other possible
state-of-the-art preventive measures available that could help mitigate
possible ethical issues.

From the remainder three models left, Bloom is distributed under a RAIL
license,333https://huggingface.co/spaces/bigscience/license which hampers its
use in some important application domains, such as law and healthcare, and
thus it was left aside.

Finally, as LLaMA models appear to generally deliver better performance than
similarly sized Pythia models in the Hugging Face’s Open LLM
Leaderboard,444https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
we adopted LLaMA for our base model. In this leaderboard, LLaMA appears as
superior to all the other models mentioned above, except possibly to Mistral,
for which it is a matching or close alternative option, with the important
advantage over Mistral though of safeguarding ethical aspects to the extent
possible given the current status of knowledge concerning foundation models.

## 3\. Data

In this section we present the datasets we developed or reused to train and
evaluate Gervásio.

### 3.1. Developed datasets

To benefit from the advantages of instruction tuning over standard supervised
fine-tuning (Wei et al., 2022), and to keep some alignment with mainstream
benchmarks for English, we resorted to tasks and respective datasets in the
GLUE (Wang et al., 2018) and the SuperGLUE (Wang et al., 2019) collections.

#### Task selection

We selected those datasets where the outcome of their machine translation into
Portuguese could preserve, in the target language, the linguistic properties
at stake and thus be acceptable for the purposes of this paper.

For instance, the COLA dataset from the GLUE benchmark contains examples of
grammatical and non-grammatical expressions from English. This dataset had to
be put aside given that an automatic machine translator typically delivers
grammatical expressions in the target language, even if the source expression
is not grammatical, defeating the purpose of the benchmark.

From GLUE, we resorted to the following four tasks: (i) MRPC (paraphrase
Detection), (ii) RTE (recognizing Textual Entailment), (iii) STS-B (semantic
textual similarity), and (iv) WNLI (coreference and natural language
inference). And from SuperGLUE, we included these other four tasks: (i) BoolQ
(yes/no question answering), (ii) CB (inference with 3 labels), (iii) COPA
(reasoning), and (iv) MultiRC (question answering).

#### Task translation

To machine translate into, respectively European and American Portuguese, we
resorted to DeepL,555https://www.deepl.com which was set for each one of these
variants, respectively.

#### Task templates

Instruction templates have been manually crafted for each task. These take the
various fields in the dataset and arrange them into a prompt by, for instance,
appending “Frase 1:” (Eng. “Sentence 1:”) before the first sentence of an
example in the RTE dataset. These templates are listed in full detail in a
dedicated webpage,666https://huggingface.co/datasets/PORTULAN/extraglue of
which one example is provided below in the Annex A.

task | #exs.tra | #exs.aug | total
---|---|---|---
STS-B | 5749 | 5749 | 11498
WNLI | 635 | 1270 | 1905
BoolQ | 9427 | 28281 | 37708
CB | 250 | 500 | 750
MultiRC | 27243 | 81729 | 108972
Total #exs | 43304 | 117529 | 160833
Total #tok pt | 17.9M | 50.1M | 68.0M
Total #tok br | 17.8M | 50.6M | 68.4M
Table 1: Size of translated (tra) and augmented (aug) training datasets, in
number of examples (#exs). The number of examples is identical for both
variants. Token counts (#tok) concern examples only and do not include the
instruction or the context examples in few-shot mode.

#### Training data

For continuing causal language modelling (CLM) with Portuguese data, we used
the datasets STS-B and WNLI, from GLUE, and BoolQ, CB and MultiRC, from
SuperGLUE, machine translated into Portuguese twice, once for PTPT, and
another time for PTBR.

For CLM, each training instance includes the task instruction followed by one
or more examples taken from the training partition of that task (including the
respective gold answers).

Every instance from the training partitions is seen twice during CLM: once
where it is the only example in the respective training instance (that is, it
is not preceded by other examples — zero-shot mode); and once where it is
preceded by other, 1 to n randomly selected examples (few-shot mode), where n
is the largest number possible given the sequence length in CLM.777Exceptions
were BoolQ and MultiRC, which given the size of their examples and the maximum
sequence length of the model, allowed zero-shot mode only. Instances,
examples, modes and values for n are shuffled.

Statistics on the training datasets are in Table 1. Taking into account the
instructions, the examples in few-shot mode and the two subsets, one for zero-
shot mode and the other for few-shot mode, altogether, the CLM resorted to a
83 million token dataset (83.1M for PTPT and 83.6 for PTBR) when we trained
our model.

#### Testing data

For testing, we reserved the translated datasets MRPC (similarity) and RTE
(inference), from GLUE, and COPA (reasoning/qa), from SuperGLUE, which were
taking as representatives of three major types of tasks, and were not seen
during training in CLM.

Each testing prompt includes the task instruction followed by an instance from
the validation partition (without the gold label). This instance may be
preceded by zero (in zero-shot prompting) or by a few examples (in few-shot
prompting) taken from the training partition (these examples include the
respective gold labels).

#### Augmented datasets

Following (Iyer et al., 2023), we employ data augmentation techniques to
enhance the size and diversity of our dataset. This involves repurposing the
tasks in various ways, such as generation of answers from MultiRC, question
generation from BoolQ, and other relevant modifications. These are presented
in the Annex B. Table 1 summarizes the number of examples in the augmented
datasets we arrived at. We did not perform data augmentation for any dataset
reserved for testing.

### 3.2. Reused datasets

For further testing our decoder, in addition to the testing data described
above, we also reused some of the datasets that had been resorted to by (Pires
et al., 2023) for American Portuguese to test the Sabiá model and that were
originally developed with materials from Portuguese: ASSIN 2 RTE (entailment)
and ASSIN 2 STS (similarity) (Real et al., 2020), BLUEX (question answering)
(Almeida et al., 2023), ENEM 2022 (question answering) (Nunes et al., 2023)
and FaQuAD (extractive question-answering) (Sayama et al., 2019). To secure
comparability with that model, we filtered out these datasets and prepared
their test instances as indicated in the Annex of the Sabiá paper.888We did
not reuse TweetSentBR because its distribution is discontinued; ENEM Challenge
because it is very similar to ENEM 2022, which was already on board; and
FaQuAD because its domain is very narrow (viz. higher education institutions).

Statistics on the testing datasets are show in Table 2.

translated tasks | #exs
---|---
MRPC | 408
RTE | 277
COPA | 100
subtotal | 785
reused tasks | #exs
ASSIN2 RTE | 2448
ASSIN2 STS | 2448
BLUEX | 178
ENEM 2022 | 118
FaQuAD | 63
subtotal | 5256
Table 2: Size of translated and reused testing datasets, in number of examples
(#exs). The number of examples is identical for both variants. Reused tasks
are pt-br only.

## 4\. Models

The Gervásio models are based on the LLaMA 2 (Touvron et al., 2023b) model
with 7 billion parameters. LLaMA 2 is a open-sourced decoder-based Transformer
that has achieved state-of-the-art results in various natural language
processing tasks in the English language. In comparison with previous decoder-
based models, such as LLaMA 1 (Touvron et al., 2023a), the main reasons for
its superiority are the use of a larger context length of 4096 tokens and the
extensive volume of data it was trained on, a volume that is currently lacking
for the Portuguese language. More specifically, the LLaMA 2 model is
pretrained using 2  _trillion_ tokens from publicly available sources. The
Gervásio models aim to advance generative AI capacity to handle the Portuguese
language by further pretraining it on the data we have curated for Portuguese
language variants.

Regarding the details of the decoder architecture, the model has a hidden size
of 4096 units, an intermediate size of 11,008 units, 32 attention heads, 32
hidden layers, and a tokenizer obtained using the Byte-Pair Encoding (BPE)
algorithm implemented with SentencePiece (Kudo and Richardson, 2018),
featuring a vocabulary size of 32,000.

We adopted the LLaMA 2 implementation provided by Hugging Face (Wolf and et
al., 2020) as our codebase. For this purpose, we employed the Transformers
library in conjunction with Accelerate (Gugger et al., 2022), Flash Attention
(Dao et al., 2022) and DeepSpeed (Rasley et al., 2020).

#### Fine-tuning

In accordance with the previously described architecture and pre-trained
model, we applied supervised fine-tuning for each variant of Portuguese, PTPT
and PTBR. The training objective was causal language modeling (CLM) using the
training data specified in Section 3.

It is noteworthy that we implemented the zero-out technique during the fine-
tuning process, as outlined in (Touvron et al., 2023b). Specifically, while
the entire prompt received attention during fine-tuning, only the response
tokens were subjected to back-propagation.

In terms of hyper-parameters, we aimed to closely match those utilized in
(Touvron et al., 2023b). Consequently, both models were trained with a
learning rate of $2\times 10^{-5}$, a weight decay of 0.1, a two-epoch
training regime without warm-up, and to ensure the same number of tokens back-
propagated per step, we employed an input sequence of 512 tokens with a batch
size of 16 and 16 accumulation steps.

Due to hardware limitations that imposed a shorter sequence length (512)
compared to the base model (4096), instead of the typical practice of
concatenating all training examples and then dividing them into batches with
the same input sequence length, we separate each example individually. In
other words, each example occupies the full input sequence length.

To achieve this, we adapted the tokenizer of the base model to accept padding
to allow grouping examples with different size into batches while preserving
the original input sequence length.

Considering the substantial discrepancy in dataset sizes between the training
set and the pre-training corpus used for the base model, with the latter being
orders of magnitude larger, and given the language shift from English to
Portuguese, we were uncertain about the expected loss behavior. We observed
that both models exhibited convergence, featuring in the training steps an
initial acceleration in terms of loss decay followed by a deceleration. This
behavior suggests the inherent ability of the base model to adapt its focus to
a new language, especially considering that the tokenizer was not retrained
for Portuguese.

For the model training process, we resorted to an a2-megagpu-16gb Google Cloud
A2 VM, equipped with 16 GPUs, 96 vCPUs, and 1.360 GB of RAM. The training of
each model took approximately two hours.

## 5\. Evaluation and discussion

To assess Gervásio models, we resorted to the test sets introduced above in
Section 3. For every task under evaluation, we use the respective evaluation
metrics commonly found in the literature, typically the F1 score or the
Pearson correlation coefficient, as indicated below.

In this connection, it is worth noting that in a text generation task where
the generated text is evaluated against a gold label, various responses may
arise in which the generated text does not match any of the predefined
classes. In such cases, the response was considered different from the correct
label and thus incorrect. To maintain the integrity of the generated text,
which corresponds to the final label, in tasks where the answer is a word,
like “sim” ou “não” (Eng. “yes” or “no”), we only considered the first word
provided as the response, after trimming any leading whitespace. In tasks
where the outcome involve classes consisting of single digit numeric value,
only the first digit is accepted as the response.

Regarding the hyper-parameters relevant in inference time for the decoder to
generate responses to the test tasks, we employed a temperature setting of
1.0, greedy decoding, a beam search value of 1, and applied top-k filtering
with a threshold of 50.

Each performance score reported below is the average of the outcome of three
independent runs using different seeds.

Model | MRPC | RTE | COPA
---|---|---|---
Gervásio ptbr | 0.7822 | 0.8321 | 0.2134
LLaMA 2 | 0.0369 | 0.0516 | 0.4867
LLaMA 2 Chat | 0.5432 | 0.3807 | 0.5493
Gervásio ptpt | 0.7273 | 0.8291 | 0.5459
LLaMA 2 | 0.0328 | 0.0482 | 0.3844
LLaMA 2 Chat | 0.5703 | 0.4697 | 0.4737
Table 3: F1 scores for ptbr and ptpt tasks translated from GLUE and SuperGLUE,
not seen during training. Best scores for each task are in bold.

#### Tasks from GLUE and SuperGLUE

Each language variant of Gervásio was evaluated with the respective translated
version of the test tasks selected from GLUE and SuperGLUE. The evaluation
scores are displayed in Table 3.

The LLaMA 2 and LLaMA 2 Chat models were evaluated by us over the Portuguese
data for both variants by following also the same approach used for Gervásio,
described above.

Model | ENEM 2022 | BLUEX | RTE | STS
---|---|---|---|---
Gervásio ptbr | 0.1977 | 0.2640 | 0.7469 | 0.2136
LLaMA 2 | 0.2458 | 0.2903 | 0.0913 | 0.1034
LLaMA 2 Chat | 0.2232 | 0.2959 | 0.5546 | 0.1750
Sabiá-7B | 0.6017 | 0.7743 | 0.6487 | 0.1363
Table 4: Evaluation (F1 for RTE, Accuracy for ENEM 2022 and BLUEX, Pearson for
STS) in data sets originally developed for American Portuguese, not seen
during training. Best scores in bold.

#### Other downstream tasks

Gervásio PTBR was also evaluated in the downstream tasks whose data sets were
not translated from English but originally developed for Portuguese. The
evaluation scores are displayed in Table 4. For Sabiá, the results presented
therte are those reported in the respective publication (Pires et al., 2023).

#### Discussion

The first important result worth underling is that Gervásio largely
outperforms its baseline LLaMA 2 in all tasks by both models, as reported in
Table 3, except for the PTR model on the COPA task.

This demonstrates that it was rewarding to continue the causal language
modeling of LLaMA 2 with the Portuguese data, even though LLaMA 2 had been
pre-trained over a overwhelming majority of English data, and also despite the
Portuguese dataset used to continue its pre-training being tiny (1.8 billion
tokens) when compared to the one used for LLaMA 2 (2 trillion tokens).999 To
further examine the outlier score of COPA in ptbr, we proceeded with cross
evaluation. The PTPT model shown quite similar scores for both the PTPT and
PTBR datasets, which seems to indicate that the possible cause for the outlier
value did not occur with the construction of the PTBR dataset. The PTBR model,
in turn, run over the PTPT testset, shown again an outlier score, similar to
the outlier score obtained for PTBR, which may indicate the root of the
difference occurs with the training sets. In fact, the base model LLaMA was
trained on 2.8 Billion tokens of Portuguese (0.09% of the total 2 Trillion
tokens used for its training in English), where PTBR texts were most probably
in much superior number than PTPT ones, given the respective demographics.
This indicates in which measure the two training conditions for PTBR and PTPT
may differ. Nevertheless, if this larger exposure to PTBR data, by the
starting model LLaMA, was the cause for the outlier value with COPA, then it
will remain to expalin why the score for MRPC and RTE are in line for both
PTBR and PTPT. We leave this for future research.

Another result from the values in Table 3 is aligned with similar results that
had been found in (Rodrigues et al., 2023). The different performance scores
of Gervásio for each of the language variants reinforce that it is relevant to
have a specific version of the model for each language variant.

Turning to Table 4, one finds the results obtained with datasets originally
developed for PTBR, thus not having been obtained by machine translation. For
two of the tasks, namely RTE and STS, the performance scores obtained here
repeat the same contrast obtained with the other test datasets translated into
Portuguese whereby Gervásio PTBR greatly outperforms its baseline LLaMA 2.

For the two other tasks, ENEM 2022 and BLUEX, in turn, Gervásio does not show
clear advantage over its starting model. This difference in performance seems
to be justified by the different type of tasks in each group. Gervásio seems
to cope better with tasks concerned with comparing sentences (RTE, with binary
decision, and STS, with 6-way decision), rather than with tasks concerned with
question answering (ENEM2022, with 5-way, and BLUEX, with 4-way), likely less
exercised in the training set.

The scores of Sabiá in Table 4 invite to contrast them with Gervásio’s but
such comparison needs to be taken with some caution.

First, these are a repetition of the scores presented in the respective paper
Pires et al. (2023), which only provide results for a single run of each task,
while scores of Gervásio are the average of three runs, with different seeds.

Second, the evaluation methods adopted by Sabiá are sui generis, and different
from the one’s adopted for Gervásio. Following Gervásio’s decoder nature as a
generative model, our scores are obtained by matching the output generated by
Gervásio against the ground labels. Sabiá, in turn, followed a convoluted
approach away from its intrinsic generative nature, by “calculating the
likelihood of each candidate answer string based on the input text and
subsequently selecting the class with the highest probability” (Pires et al.,
2023, p.231), which forces the answer to be one of the possible classes and
likely facilitates higher performance scores than Gervásio’s, whose answers
are generated without constraints.

Third, to evaluate Sabiá, the examples included in the few-shot prompt are
hand picked, and identical for every test instance in each task (Pires et al.,
2023, p.4). To evaluate Gervásio, the examples were randomly selected to be
included in the prompts.

Even taking these considerations into account, it is noticeable that the
results in Table 4 indicate performance scores for Gervásio that are clearly
better than for Sabiá, over the same two test tasks where it also excels over
its starting model.

Given that Gervásio, in addition, is distributed as an fully open model, and
Sabiá is publicly available for research only, all these circumstances seems
to speak for Gervásio’s advantage in terms of its usage for research and
commercial purposes.

## 6\. Conclusion

This paper contributes new, instruction-tuned large language models of the
encoder family of Transformers specifically developed for the Portuguese
language, as well as the instruction datasets used to train and evaluate them.

The models are openly available for free and with no registration required
under an MIT license at https://huggingface.co/PORTULAN, where the respective
datasets are also openly available for free and with no registration required.

With a 7 billion parameter, these models have an unique set of features for
their size. They are fully open: they are open source; and they are openly
distributed, under an open license, thus including for either research or
commercial purposes. They are the most encompassing models for the Portuguese
language: they cover both the European variant, spoken in Portugal, and the
American variant, spoken in Brazil; and the model for the European variant it
is the first of its class, known in the literature. They show a competitive
performance: they outperform other models of similar size publicly reported,
thus representing the state of the art. They are fully documented: the new
datasets that were specifically developed for its construction can be reused
and its development can be reproduced; and reported performance scores can be
independently assessed.

By being fully open and fully documented, its further development and
improvement is openly available to the community.

Also, given their size, these models can still be run on consumer-grade
hardware with technological solutions currently available, thus being a
contribution to the advancement of research and innovation in language
technology for Portuguese.

Future work will include taking these models as the inaugural members of a
future family of fully open decoders for Portuguese with a range of other
sizes, and characteristics and for other variants of Portuguese.

## Acknowledgements

This research was partially supported by: PORTULAN CLARIN—Research
Infrastructure for the Science and Technology of Language, funded by Lisboa
2020, Alentejo 2020 and FCT (PINFRA/22117/2016); ACCELERAT.AI—Multilingual
Intelligent Contact Centers, funded by IAPMEI (C625734525-00462629); and
GPTPT—Transformer-based Decoder for the Portuguese Language, funded by FCT
(CPCA-IAC/AV/478395/2022).

## 7\. Annex A: Template example

As an example, here we describe the template used for the RTE task in PTPT. In
this task, two sentences are given and the task consists in determining
whether the first sentence entails the second. Each instance in the dataset
contains the fields premise, hypothesis and labels. The template describes how
to handle these fields, usually by prepending some string to their contents,
as well as defining the initial instruction.

instruction

“Nesta tarefa vais receber duas frases. Indica se a primeira frase implica
claramente a segunda frase. Ou seja, indica se se conclui que a segunda frase
é verdadeira desde que a primeira frase seja verdadeira. Deves responder ‘sim’
se a primeira frase implica a segunda frase ou deves responder ‘não’ no caso
contrário.” (Eng. “In this task you’ll receive two sentences. Indicate whether
the first sentence clearly entails the second sentence. That is, indicate
whether one can conclude that the second sentence is true as long as the first
sentence is true. You should answer ‘yes’ if the first sentence entails the
second sentence or ‘no’ otherwise.”)
This is the instruction that is given at the beginning of the input.

premise

“Frase 1:” (Eng. “Sentence 1:”)
This is placed before the contents of the ‘premise’ field of the RTE instance.

hypothesis

“Frase 2:” (Eng. “Sentence 2:”)
This is placed before the contents of the ‘hypothesis’ field of the RTE
instance.

pre-label

“Resposta:” (Eng. “Answer:”)
This is placed before the answer.

labels

“0” $\to$ “sim”, “1” $\to$ “não”
This is a mapping from the 0/1 labels used in the RTE dataset to the yes/no
labels that are asked for in the instructions for the task.

Applying the template above to an instance gives something like what is shown
below.

Nesta tarefa vais receber duas frases. Indica se a primeira frase implica
claramente a segunda frase. Ou seja, indica se se conclui que a segunda frase
é verdadeira desde que a primeira frase seja verdadeira. Deves responder ‘sim’
se a primeira frase implica a segunda frase ou deves responder ‘não’ no caso
contrário
---
Frase 1: Em 1969, redigiu o relatório que propunha a expulsão do partido do
grupo Manifesto. Em 1984, após a morte de Berlinguer, Natta foi eleito
secretário do partido.
Frase 2: A Natta apoiou o grupo do Manifesto.
Resposta: não

In addition, a separator string formed by 3 to 5 consecutive ‘=’ (equals)
symbols is inserted between each instance in the training data. And, during
few-shot inference, each instance is headed by “Exemplo $n$” (Eng. “Example
$n$”), with increasing $n$, and within each instance its few-shot examples are
delimited by a separator string formed by 3 or 4 consecutive ‘-’ (hyphen) or
‘*’ (asterisk) symbols.

## 8\. Annex B: Instruct training tasks

The base tasks and their augmented counterparts that together form the
training data are:

STS-B

for semantic textual similarity, with augmented STS-B Aug1 for generation of a
sentence with a STS score of 0/1/2/3/4/5

WNLI

for coreference and natural language inference, with augmented WNLI Aug1 for
generating an hypothesis with Positive/Negative inference, and WNLI Aug2 for
generating a premise with Positive/Negative inference

BoolQ

for Yes/No question answering, with augmented BoolQ Aug1 for question
generation with Yes/No answer based on an excerpt, and BoolQ Aug2 for excerpt
generation with Yes/No answer to a question

CB

for inference with labels Entailment (E), Contradiction (C) and Neutral (N),
with augmented CB Aug1 for generating an hypothesis with label E/C/N, and CB
Aug2 for generating a premise with label E/C/N

MultiRC

for question answering, with augmented MultiRC Aug1 for question generation,
MultiRC Aug2 for excerpt generation, and MultiRC Aug3 for answer generation

## 9\. Bibliographical References

##

  * Almeida et al. (2023) Thales Sales Almeida, Thiago Laitz, Giovana K. Bonás, and Rodrigo Nogueira. 2023.  BLUEX: A benchmark based on Brazilian leading universities entrance exams.  _arXiv preprint arXiv:2307.05410_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _arXiv preprint arXiv:2304.01373_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Gugger et al. (2022) Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. 2022.  Accelerate: Training and inference at scale made simple, efficient and adaptable.  https://github.com/huggingface/accelerate.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  DeBERTa: Decoding-enhanced BERT with disentangled attention.  In _International Conference on Learning Representations_.
  * Iyer et al. (2023) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2023.  OPT-IML: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71.
  * Larcher et al. (2023) Celio Larcher, Marcos Piau, Paulo Finardi, Pedro Gengo, Piero Esposito, and Vinicius Caridá. 2023.  Cabrita: closing the gap for foreign languages.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, et al. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Nunes et al. (2023) Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, and Rodrigo Nogueira. 2023.  Evaluating GPT-3.5 and GPT-4 models on Brazilian university admission exams.  _arXiv preprint arXiv:2303.17003_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pires et al. (2023) Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023.  Sabiá: Portuguese large language models.  In _Intelligent Systems_ , pages 226–240, Cham. Springer Nature Switzerland.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? _arXiv preprint arXiv:1906.01502_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21:5485–5551.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Real et al. (2020) Livy Real, Erick Fonseca, and Hugo Gonçalo Oliveira. 2020.  The ASSIN 2 shared task: A quick overview.  In _Computational Processing of the Portuguese Language_ , pages 406–412, Cham. Springer International Publishing.
  * Rodrigues et al. (2023) João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023.  Advancing neural encoding of Portuguese with Transformer Albertina PT-*.  In _Progress in Artificial Intelligence_.
  * Sayama et al. (2019) Hélio Fonseca Sayama, Anderson Viçoso Araujo, and Eraldo Rezende Fernandes. 2019.  FaQuAD: Reading comprehension dataset in the domain of Brazilian higher education.  In _2019 8th Brazilian Conference on Intelligent Systems (BRACIS)_ , pages 443–448.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Shliazhko et al. (2023) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2023.  mGPT: Few-shot learners go multilingual.  _arXiv preprint arXiv:2204.07580_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wang (2021) Ben Wang. 2021.  Mesh-Transformer-JAX: Model-parallel implementation of transformer language model with JAX.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wolf and et al. (2020) Thomas Wolf and et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Xue et al. (2022) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022.  ByT5: Towards a token-free future with pre-trained byte-to-byte models.  _Transactions of the Association for Computational Linguistics_ , 10:291–306.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_.

## 10\. Language Resource References

*

##

  * Almeida et al. (2023) Thales Sales Almeida and Thiago Laitz and Giovana K. Bonás and Rodrigo Nogueira. 2023.  _BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams_.  HuggingFace.
  * Nunes et al. (2023) Desnes Nunes and Ricardo Primi and Ramon Pires and Roberto Lotufo and Rodrigo Nogueira. 2023.  _ENEM 2022 (Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams)_.  GitHub.
  * Real et al. (2020) Real, Livy and Fonseca, Erick and Gonçalo Oliveira, Hugo. 2020.  _ASSIN 2 (The ASSIN 2 Shared Task: A Quick Overview)_.  HuggingFace.
  * Sayama et al. (2019) Sayama, Hélio Fonseca and Araujo, Anderson Viçoso and Fernandes, Eraldo Rezende. 2019.  _FaQuAD: Reading Comprehension Dataset in the Domain of Brazilian Higher Education_.  HuggingFace.
  * Touvron et al. (2023) Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others. 2023.  _Llama 2 7B (Llama 2: Open foundation and fine-tuned chat models)_.  HuggingFace.
  * Wang et al. (2019) Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel. 2019.  _Superglue: A stickier benchmark for general-purpose language understanding systems_.  HuggingFace.
  * Wang et al. (2018) Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel. 2018.  _GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding_.  HuggingFace.