# A JWST NIRSpec phase curve for WASP-121b: dayside emission strongest
eastward of the substellar point and nightside conditions conducive to cloud
formation

Thomas Mikal-Evans Max Planck Institute for Astronomy, Königstuhl 17, D-69117
Heidelberg, Germany  David K. Sing Department of Earth & Planetary Sciences,
Johns Hopkins University, Baltimore, MD, USA  Department of Physics &
Astronomy, Johns Hopkins University, Baltimore, MD, USA  Jiayin Dong Flatiron
Research Fellow  Center for Computational Astrophysics, Flatiron Institute,
New York, NY, USA  Daniel Foreman-Mackey Center for Computational
Astrophysics, Flatiron Institute, New York, NY, USA  Tiffany Kataria Jet
Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove
Drive, Pasadena, CA 91001  Joanna K. Barstow School of Physical Sciences, The
Open University, Walton Hall, Milton Keynes, MK7 6AA, UK  Jayesh M. Goyal
School of Earth and Planetary Sciences (SEPS), National Institute of Science
Education and Research (NISER), HBNI, Jatni, Odisha 752050, India  Nikole K.
Lewis Department of Astronomy and Carl Sagan Institute, Cornell University,
122 Sciences Drive, Ithaca, NY 14853, USA  Joshua D. Lothringer Department of
Physics, Utah Valley University, Orem, UT, USA  Nathan J. Mayne Department of
Physics and Astronomy, Faculty of Environment Science and Economy, University
of Exeter, EX4 4QL, UK  Hannah R. Wakeford University of Bristol, HH Wills
Physics Laboratory, Tyndall Avenue, Bristol, BS8 1TL, UK  Duncan A. Christie
Max Planck Institute for Astronomy, Königstuhl 17, D-69117 Heidelberg, Germany
Zafar Rustamkulov Department of Earth & Planetary Sciences, Johns Hopkins
University, Baltimore, MD, USA

(Received December 13, 2022; Revised December 27, 2022; Accepted December 29,
2022)

###### Abstract

We present the first exoplanet phase curve measurement made with the JWST
NIRSpec instrument, highlighting the exceptional stability of this newly-
commissioned observatory for exoplanet climate studies. The target, WASP-121b,
is an ultrahot Jupiter with an orbital period of 30.6 hr. We analyze two
broadband light curves generated for the NRS1 and NRS2 detectors, covering
wavelength ranges of 2.70-3.72$\,\mu\textnormal{m}$ and
3.82-5.15$\,\mu\textnormal{m}$, respectively. Both light curves exhibit
minimal systematics, with approximately linear drifts in the baseline flux
level of 30 ppm/hr (NRS1) and 10 ppm/hr (NRS2). Assuming a simple brightness
map for the planet described by a low-order spherical harmonic dipole, our
light curve fits suggest that the phase curve peaks coincide with orbital
phases $3.36\pm 0.11^{\circ}$ (NRS1) and $2.66\pm 0.12^{\circ}$ (NRS2) prior
to mid-eclipse. This is consistent with the strongest dayside emission
emanating from eastward of the substellar point. We measure planet-to-star
emission ratios of $3,924\pm 7\,$ppm (NRS1) and $4,924\pm 9\,$ppm (NRS2) for
the dayside hemisphere, and $136\pm 8\,$ppm (NRS1) and $630\pm 10\,$ppm (NRS2)
for the nightside hemisphere. The latter nightside emission ratios translate
to planetary brightness temperatures of $926\pm 12\,$K (NRS1) and $1,122\pm
10\,$K (NRS2), which are low enough for a wide range of refractory condensates
to form, including enstatite and forsterite. A nightside cloud deck may be
blocking emission from deeper, hotter layers of the atmosphere, potentially
helping to explain why cloud-free 3D general circulation model simulations
systematically over-predict the nightside emission for WASP-121b.

Exoplanet astronomy (486), Exoplanet atmospheres (487)

††journal: Astrophysical Journal Letters††software:  NumPy (van der Walt et
al., 2011), SciPy (Virtanen et al., 2020), Matplotlib (Hunter, 2007), JWST
Python pipeline (Bushouse et al., 2022), FIREFly (Rustamkulov et al., 2022a),
starry (Luger et al., 2019), PyMC3 (Salvatier et al., 2016), pysynphot (STScI
Development Team, 2013) ††facilities:  JWST(NIRSpec)

##  1 Introduction

It is crucial to explore the coupling between the dayside and nightside
hemispheres of tidally-locked planets if we are to develop a global
understanding of their atmospheres. One of the most effective means of
constraining the chemical, thermal, and dynamical properties of both
hemispheres is to measure planetary emission spectra over the course of a full
planetary orbit (e.g. Knutson et al., 2007; Cowan et al., 2012; Stevenson et
al., 2014; Kreidberg et al., 2018; Arcangeli et al., 2019; Feng et al., 2020;
Irwin et al., 2020; Mikal-Evans et al., 2022). To this end, our team observed
a spectroscopic phase curve for the ultrahot Jupiter WASP-121b using JWST
NIRSpec with the G395H grating. In this letter, we present an analysis of two
broadband light curves generated from this dataset, one for each of the NRS1
and NRS2 detectors. As well as providing some preliminary scientific results,
our purpose is to provide a brief report on the stability of JWST and NIRSpec
when performing long-stare time series measurements lasting tens of hours.

The target, WASP-121b, is an inflated gas giant with a mass of
$1.183_{-0.062}^{+0.064}\,M_{J}$, a radius of $1.753\pm 0.036\,R_{J}$ measured
at red-optical wavelengths, and an orbital period of $30.59820\pm 0.00001\,$hr
(Delrez et al., 2016; Bourrier et al., 2020). Secondary eclipse measurements
made with HST have shown that the dayside atmosphere has a thermal inversion,
with a near-infrared brightness temperature close to 2,700 K (Evans et al.,
2017; Mikal-Evans et al., 2020). Evidence has been uncovered for numerous UV-
optical absorbers in the atmosphere that are likely responsible for the
dayside thermal inversion, such as VO, SiO, V, Fe, Mg, and Ca (e.g. Evans et
al., 2018; Hoeijmakers et al., 2020; Lothringer et al., 2021).

Phase curve measurements with HST have shown that temperatures on the
nightside of WASP-121b are likely low enough for refractory condensates
containing V, Fe, Mg, and Ca to form (Mikal-Evans et al., 2022). Since these
latter elements are observed in the gas-phase at the day-night terminator, if
such clouds do form they are presumably recirculated back to the dayside
hemisphere where they are vaporised before settling to the deeper layers of
the atmosphere. Meanwhile, non-detections of Ti and Al at the day-night
terminator reported by Hoeijmakers et al. (2020, 2022) are consistent with
these elements being cold-trapped in deeper layers of the nightside hemisphere
as perovskite (CaTiO3) and corundum (Al2O3).

From these previous studies, we are already observing the important connection
between the dayside and nightside hemispheres of WASP-121b. One of the
ultimate goals of our JWST program will be to provide significantly tighter
constraints on the dayside and nightside properties. This should be possible
thanks to the JWST data having a much higher signal-to-noise and spectral
resolution than the existing HST data, as well as the more favorable planet-
to-star emission ratio at the longer wavelengths covered by NIRSpec.

##  2 Observations and data reduction

Figure 1: Pointing jitter along the $x$ and $y$ axes of the NRS1 (top panel)
and NRS2 (bottom panel) detectors, as measured by the FIREFly code.

Our team observed the WASP-121 system over a full planetary orbit on 2022
October 14-15 using the NIRSpec instrument as part of program GO-1729 (P.I.
Mikal-Evans, co-P.I. Kataria). Observations were made without interruption for
37.8 hr, commencing 145 min prior to secondary eclipse ingress and continuing
until 105 min after egress of the following secondary eclipse. We used the
G395H grating, providing wavelength coverage between
2.70-5.15$\,\mu\textnormal{m}$ at $R\sim 3000$. For reading the detector, we
employed the SUB2048 subarray option with the NRSRAPID readout pattern. Each
integration consisted of 42 groups, translating to integration times of 38.8
seconds and an overall duty cycle of 99%.

To extract the target spectra from the data frames, we used the FIREFLy code
(Fast InfraRed Exoplanet Fitting for Lightcurves; Rustamkulov et al., 2022a,
b). The data reduction begins with a customized reduction of stage 1 from the
JWST Python pipeline (Bushouse et al., 2022). Detector “$1/f$” noise (Birkmann
et al., 2022; Rustamkulov et al., 2022a; Schlawin et al., 2020) is removed at
the group level, masking out the spectral trace pixels and removing the median
value from each column of the remaining background pixels. Bad pixels and
cosmic rays are then flagged as outliers, both in the time series and
spatially on the 2D images. Movement of the spectra along the $x$ and $y$ axes
of the detector are measured using cross-correlation, exhibiting standard
deviations at the level of 0.002 pixels in both directions for NRS1 and in the
$y$ direction for NRS2, while a higher standard deviation of 0.005 pixels is
measured in the $x$ direction for NRS2 (Figure 1). This level of jitter is
similar to previously reported on-sky performance (Rigby et al., 2022;
Espinoza et al., 2022), though demonstrated here on a much longer timescale.
The 2D data frames are then adjusted using interpolation to account for the
small pointing shifts. We find that this alignment procedure can dampen $x$
and $y$ correlations in the subsequent spectrophotometry, as it effectively
takes into account the redistribution of flux across adjacent pixels. However,
residual $x$ and $y$ trends may still remain due to intrapixel sentivities
(Birkmann et al., 2022). Next, a fourth order polynomial was used to trace the
center of the point-spread-function (PSF) along the detector. Counts were then
summed within an aperture centered on the trace with a width of 4.9 pixels for
NRS1 and 5.5 pixels for NRS2, linearly interpolating the counts in those
pixels where only a fraction was contained within the aperture. The aperture
widths were selected after trialing a number of values and selecting those
that minimized the scatter of the in-eclipse photometric time series. To
obtain the wavelength calibration, we extrapolated JWST pipeline data products
across the detector edge pixels which did not have an assigned wavelength.

##  3 Light curve analysis

Figure 2: Panels (a) and (b): Raw broadband light curves for the NRS1 and NRS2
detectors, respectively. Orange lines show the best-fit light curve models.
Blue lines show the corresponding instrument baseline trends. Grey horizontal
lines are calibrated to the bottoms of the first eclipse, to highlight the
instrumental drift over the course of the observation. Panels (c) and (d): The
same light curves focusing on the planetary emission signal and after
correcting for the instrument baseline trends shown by the blue lines in
panels (a) and (b). Panels (e) and (f): Black points show the residuals
between the data and best-fit models for the NRS1 and NRS2 detectors,
respectively. Orange lines show the residuals after applying a Gaussian filter
to smooth the random noise. Green and purple shading show the times of transit
and eclipse, respectively, with darker ranges corresponding to ingress and
egress times.

We generated two light curves by summing each extracted 1D spectrum between
dispersion columns 300-2042 for NRS1 and 5-2010 for NRS2, conservatively
encompassing the full passbands of both detectors. The resulting raw light
curves are shown in the top two panels of Figure 2. We modeled these light
curves using the starry package (Luger et al., 2019) for the star-planet
signal, multiplied by a linear trend in time to account for instrumental
drift. For the planet brightness distribution, a simple dipole map was
adopted, comprised of the $Y_{0,0}$ and $Y_{1,0}$ spherical harmonics. For the
stellar brightness distribution, a quadratic limb darkening profile was
assumed. In defining the data log-likelihood, we assumed that the data were
normally distributed with standard deviations given by the pipeline Poisson
uncertainties and an additional high-frequency systematics noise term combined
in quadrature.

We fitted the NRS1 and NRS2 light curves jointly, with the following
parameters shared across both light curves: the stellar mass ($M_{\star}$) and
logarithmic radius ($\log_{10}R_{\star}$); the differences in the orbital
period ($\Delta P$) and transit mid-time ($\Delta T_{c}$) from the values
reported by Bourrier et al. (2020); the orbital inclination ($i$); and the
planetary mass ($M_{p}$). Separate sets of the remaining model parameters were
fitted for each light curve in the joint fit, namely: the stellar limb
darkening coefficients ($u_{1}$, $u_{2}$); the logarithmic planetary radius
($\log_{10}R_{p}$); the overall amplitude of the planetary brightness map
($A$) and the coefficient of the $Y_{1,0}$ spherical harmonic ($y_{1,0}$); the
rotational offset of the planetary brightness map relative to a map with a
brightness peak centered at the substellar point ($\Delta\phi$); the
coefficients of the linear instrumental drift trend ($f_{0}$, $f_{1}$); and
the high-frequency systematics noise term ($\sigma_{\rm{syst}}$). The priors
we adopted for each parameter are shown by the dashed lines in Figure 3. For
$M_{\star}$, $\log_{10}R_{\star}$, $i$, and $M_{p}$, these were normal priors
based on the posteriors reported in Bourrier et al. (2020). Broad normal
priors were also adopted for $\log_{10}R_{p}$, $\Delta T_{c}$, $u_{1}$,
$u_{2}$, $A$, $y_{1,0}$, $\Delta\phi$, $f_{0}$, and $f_{1}$. Half-normal
priors only allowing positive values were adopted for $\sigma_{\rm{syst}}$. We
marginalized the resulting posterior distribution using the No-U-Turn-Sampling
(NUTS; Hoffman & Gelman, 2014) method implemented by PyMC3 (Salvatier et al.,
2016).

##  4 Results

Figure 3: Dark blue histograms show posterior distributions for model
parameters that were varied in the light curve fitting. Light blue histograms
show corresponding distributions for other parameters that were subsequently
derived. Posterior medians and $\pm 34$% credible intervals are listed in
black font. Red dashed lines show the priors. For the normal priors, the mean
$\mu$ and standard deviation $\sigma$ are listed in red font. Half-normal
priors were adopted for the $\sigma_{\rm{syst}}$ parameters, allowing only
positive values and with scale $\sigma$ listed in red font. Vertical axes are
not labeled, as arbitrary normalizations have been applied to all plotted
distributions.

The maximum a posteriori (MAP) light curve models are shown in Figure 2.
Posterior distributions for each of the parameters that were varied in the
fitting are shown as dark blue histograms in Figure 3, while the light blue
histograms show the corresponding distributions for other parameters that were
calculated from the latter, namely: the stellar radius $R_{\star}$; the
planet-to-star radius ratio $R_{p}/R_{\star}$; the planet-to-star flux ratio
$F_{p}/F_{\star}$ for the dayside and nightside hemispheres of the planet; and
the corresponding planetary brightness temperatures $T_{b}$. The brightness
temperatures were calculated by assuming
$F_{p}/F_{\star}=(R_{p}/R_{\star})^{2}\,B_{p}(T_{b})/F_{\star}$, where $B_{p}$
denotes a black body function for the planetary flux, integrated over the
instrument passband. In deriving distributions for $T_{b}$, uncertainties in
the host star properties were accounted for by first randomly drawing values
for the stellar effective temperature ($T_{\star}$), surface gravity ($\log
g_{\star}$), and metallicity ([M/H]) from the posterior distributions reported
in Delrez et al. (2016), which were assumed to be normal. For each of these
sets of stellar properties, we used pysynphot (STScI Development Team, 2013)
to obtain a corresponding ATLAS9 stellar model (Castelli & Kurucz, 2003) that
was then integrated over the instrument passband to obtain an estimate for
$F_{\star}$. The resulting set of $F_{\star}$ values, along with the
$R_{p}/R_{\star}$ and $F_{p}/F_{\star}$ posterior samples obtained from the
light curve fits, were used to derive the final distributions for $T_{b}$.

We measure a significantly larger $R_{p}/R_{\star}$ value of
$0.123668_{-0.000069}^{+0.000068}$ for NRS2, compared to
$0.122551_{-0.000063}^{+0.000062}$ for NRS1. This difference can potentially
be explained by the wavelength-dependent opacity of the planetary atmosphere.
We find that the planetary brightness map is shifted eastward by $3.36\pm
0.11\,^{\circ}$ ($30\sigma$) for NRS1 and by $2.66\pm 0.12\,^{\circ}$
($22\sigma$) for NRS2. We also make significant detections of the planetary
nightside emission in both passbands: $136\pm 8$ ppm ($17\sigma$) for NRS1 and
$630\pm 10$ ppm ($63\sigma$) for NRS2. These relative emission values
translate to nightside brightness temperatures of $926_{-12}^{+12}\,$K and
$1,122_{-10}^{+10}\,$K, respectively. For the dayside hemispheres, we obtain
corresponding brightness temperatures of $2,762_{-32}^{+30}\,$K for the NRS1
passband and $2,768_{-39}^{+39}\,$K for the NRS2 passband. We note that the
brightness temperature uncertainties are dominated by the stellar model
uncertainties. When we only account for the uncertainties in $F_{p}/F_{\star}$
and $R_{p}/R_{\star}$ derived from the light curve fits while holding the
stellar properties fixed to the best-fit values reported by Delrez et al.
(2016), the uncertainties on the dayside brightness temperatures reduce by an
order-of-magnitude from around 30-40 K (Figure 3) to 3 K. By contrast, the
uncertainties associated with dayside brightness temperatures measured
previously with the Spitzer Space Telescope in the 3-5$\,\mu\textnormal{m}$
wavelength range have been dominated by the $F_{p}/F_{\star}$ uncertainty. For
example, Garhart et al. (2020) reported dayside brightness temperatures for
WASP-121b of $2,490\pm 77\,$K and $2,562\pm 66\,$K in the
$3.6\,\mu\textnormal{m}$ and $4.5\,\mu\textnormal{m}$ Infrared Array Camera
(IRAC) passbands, which cover similar wavelength ranges to NRS1 and NRS2.

The long-term instrumental drift is found to be mild for both detectors. For
NRS1, we obtain a negative drift of 1,172 ppm over the course of the
observation, corresponding to a rate of approximately 30 ppm/hr. NRS2 is even
better-behaved, exhibiting a positive drift of only 416 ppm from start to
finish, at a rate of approximately 10 ppm/hr. The standard deviation of the
light curve residuals are 127 ppm and 161 ppm for the NRS1 and NRS2 detectors,
respectively. This is 39% and 17% higher than the Poisson uncertainties
derived from the instrument pipeline, respectively. As such, we derive high-
frequency systematics noise values ($\sigma_{\rm{syst}}$) of $90\pm 2$ ppm for
NRS1 and $80\pm 4$ ppm for NRS2.

##  5 Discussion

The small positive values that we measure for $\Delta\phi$ in both the NRS1
($3.36\pm 0.11\,^{\circ}$) and NRS2 ($2.66\pm 0.12\,^{\circ}$) passbands
translate to phase curves with maxima shifted prior to mid-eclipse. This in
turn suggests that the eastern half of WASP-121b’s dayside hemisphere is
hotter on average than the western hemisphere, which is qualitatively
consistent with the predictions of general circulation model (GCM) simulations
of hot Jupiters (e.g. Showman & Guillot, 2002; Showman et al., 2009; Komacek
et al., 2022). However, GCMs typically predict eastward phase curve shifts
that are significantly higher than what we have measured here for WAP-121b;
closer to $10^{\circ}$ or more (e.g. Parmentier & Crossfield, 2018). Small
phase curve shifts relative to GCM predictions have also been measured for a
number of other hot Jupiters with properties similar to WASP-121b (e.g. Zhang
et al., 2018; Kreidberg et al., 2018). For example, Kreidberg et al. (2018)
measured an eastward phase shift of $2.0\pm 0.7\,^{\circ}$ for WASP-103b in
the Spitzer IRAC $3.6\,\mu\textnormal{m}$ channel, compared to a shift of
$9.2^{\circ}$ predicted by a GCM presented in the same study. For ultrahot
planets like WASP-121b and WASP-103b – which both have dayside brightness
temperatures well above $2,500\,$K – Lorenz drag resulting from the motion of
thermally ionized gas through the planetary magnetic field may inhibit the
advection of heat throughout the atmosphere, reducing the observed phase curve
offsets (Perna et al., 2010). Alternatively, even if the hottest region of the
dayside hemisphere is located significantly eastward of the substellar point,
the presence of nightside clouds can reduce the observed phase shift in the
resulting light curve (Parmentier et al., 2021). We plan to investigate the
latter possibility further in future analyses by including higher-order
spherical harmonics terms for the planetary brightness map, which will allow
for a sharper delineation in the brightness of the dayside and nightside
hemispheres.

Our dayside and nightside emission measurements are plotted in Figure 4
against the 3D GCM predictions from Parmentier et al. (2018), which assume
$1\times$ and $5\times$ solar metallicity, chemical equilibrium, and do not
include clouds. The models exhibit spectral features of H2O and CO, which are
in emission on the dayside due to a thermal inversion and absorption on the
nightside due to a cooling temperature profile (Figure 4). The agreement
between the measured dayside emission (3,924$\pm 7$ ppm) and the $5\times$
solar GCM prediction (3,916 ppm) across the NRS1 passband is very close to
$1\sigma$. However, the dayside emission measured across the NRS2 passband
(4,924$\pm 9$ ppm) is $19\sigma$ higher than the $5\times$ solar GCM
prediction (3,916 ppm). For the nightside emission, the measurements across
the NRS1 ($136\pm 8\,$ppm) and NRS2 ($630\pm 10\,$ppm) passbands are
$>10\sigma$ lower than both the $1\times$ solar metallicity (588 ppm for NRS1
and 1,080 ppm for NRS2) and $5\times$ solar metallicity (413 ppm for NRS1 and
789 ppm for NRS2) GCM predictions. Such large discrepancies are unsurprising,
given the small measurement uncertainties and the fact that the GCMs were not
tuned to match the data. Still, it is notable that the measured nightside
brightness temperatures are both substantially lower than predicted by the GCM
simulations. Furthermore, the Parmentier et al. (2018) GCMs did not include
the dissociation and recombination of hydrogen, which if anything would result
in the predicted nightside temperatures being higher than those shown in
Figure 4 due to the release of latent heat (Bell & Cowan, 2018; Tan & Komacek,
2019; Komacek et al., 2022), exacerbating the discrepancy.

Clouds – which were also not included in the Parmentier et al. (2018) GCMs –
might help explain our lower-than-predicted nightside brightness temperature
measurements. Previous HST observations have shown that the nightside
temperature profile of WASP-121b does not have a thermal inversion at the
near-infrared photosphere, instead cooling with decreasing pressure (Mikal-
Evans et al., 2022). If an optically-thick cloud deck blankets the nightside
hemisphere, it could block the emission from deeper, hotter layers of the
atmosphere, lowering the observed brightness temperature (Mendonça et al.,
2018; Keating et al., 2019; Beatty et al., 2019; Parmentier et al., 2021).
Alternatively, Komacek et al. (2022) have shown that if nightside cloud is
patchy rather than uniform, the resulting brightness temperatures can be
higher than cloud-free predictions. However, even for patchy cloud, Komacek et
al. (2022) also identified scenarios in which nightside brightness
temperatures could be lower than predicted by cloud-free models, depending on
the precise assumptions made for the cloud properties, such as the size
distribution and radiative properties of the cloud particles.

The nightside brightness temperatures that we measure for WASP-121b in the
NRS1 ($925_{-11}^{+11}\,$K) and NRS2 ($1127_{-6}^{+6}\,$K) passbands fall
below the condensation temperatures of silicates, such as enstatite and
forsterite (Visscher et al., 2010; Wakeford et al., 2017), which are expected
to be abundant in the nightside atmospheres of hot Jupiters (e.g. Gao &
Powell, 2021). Therefore, our measurements reveal nightside conditions for
WASP-121b that do appear to be conducive to the formation of silicate clouds,
strengthening the possibility that clouds may prove key to understanding why
our measured nightside brightness temperatures are significantly lower than
predicted by the Parmentier et al. (2018) GCMs shown in Figure 4. Further work
will be required to investigate if the difference in the nightside brightness
temperatures obtained for the two passbands could be caused by the wavelength-
dependent opacity of atmospheric layers above a cloud deck. For these future
analyses, the dayside and nightside emission spectra will be extracted and
interpreted, rather than the broad passbands considered here.

Figure 4: Black diamonds show the measured dayside (top panel) and nightside
(bottom panel) planet-to-star emission levels. Note that the measurement
uncertainties are smaller than the diamond symbols. Predictions from the
cloud-free 3D GCM simulations of Parmentier et al. (2018) are also shown for
heavy element enrichments of $1\times$ solar (blue lines) and $5\times$ solar
(orange lines). Circle and square symbols show, respectively, the $1\times$
and $5\times$ solar GCM predictions binned to the light curve passbands, which
are shown as green lines in the top panel.

It should also be stressed that the analysis we have presented here is
preliminary. In particular, we have adopted a very simple light curve model
and it is evident by eye that low-amplitude correlations still remain in the
time series of residuals (Figure 2). We did experiment with including the $x$
and $y$ pointing coordinates shown in Figure 1 as additional linear
decorrelation variables in the light curve fits, but found that this did not
appreciably reduce the scatter in the residuals, nor did it affect the derived
parameter distributions shown in Figure 3. Tidal deformation of WASP-121b
could also subtly affect the observed phase curve (Cowan et al., 2012; Wahl et
al., 2021). We did use starry to perform some preliminary light curve fits
treating the planetary oblateness as a free parameter, but found that our
basic conclusions were unaffected. Further work is required to determine if an
oblate shape for the planet is supported by the data. Additional possibilities
yet to be considered include adding higher-order spherical harmonics terms for
the planetary brightness map and allowing for nonlinear instrumental baseline
trends. These investigations are ongoing and could conceivably affect the
constraints that we ultimately obtain for parameters of interest, such as the
planetary nightside emission and brightness phase offsets.

In the meantime, our phase curve measurement for WASP-121b demonstrates the
overall high level of stability that JWST NIRSpec is capable of maintaining
for a single-stare observation lasting 37.8 hr. The approximately-linear drift
observed in the baseline flux level is extremely mild in comparison to the
instrumental systematics that have affected past HST and Spitzer datasets
(e.g. Kreidberg et al., 2018; Mikal-Evans et al., 2022). We are hopeful that
the additional 80-90 ppm high-frequency systematics noise observed here in our
preliminary analysis can be further reduced as more on-sky calibrations become
available, along with continued refinement of the data analysis, such as
improved treatment of the $1/f$ detector noise. Based on other recent analyses
of NIRSpec data (Alderson et al., 2022; Rustamkulov et al., 2022b), we also
anticipate that simple models fitted to light curves generated over narrower
wavelength ranges than the broad passbands we have considered in the present
study will achieve residual scatters closer to Poisson predictions.

##  6 Conclusion

We have measured a full-orbit phase curve for the ultrahot Jupiter WASP-121b
using JWST NIRSpec. The resulting light curves generated across broad
passbands for the NRS1 and NRS2 detectors exhibit minimal systematics over the
37.8 hr observation. We find that the phase curve peaks are shifted prior to
mid-eclipse by $3.36\pm 0.11\,^{\circ}$ (NRS1) and $2.66\pm 0.12\,^{\circ}$
(NRS2), suggesting that the eastern region of the dayside hemisphere is hotter
on average than the western region. The measured dayside emission in the NRS1
passband is in good agreement with a cloud-free GCM assuming $5\times$ solar
metallicity; however, the same GCM underpredicts the dayside emission in the
NRS2 passband by $19\sigma$. For the nightside emission, cloud-free GCM
simulations assuming $1\times$ and $5\times$ solar metallicity significantly
over-predict the data. This observation could possibly be explained by
nightside clouds blocking the emission from deeper, hotter layers of the
atmosphere. The corresponding nightside brightness temperatures are $<1200$ K
in both passbands, which is cool enough for various condensates to form,
including silicates such as enstatite and forsterite.

The authors are grateful to the anonymous referee for constructive feedback
that improved the paper. Support for JWST program GO-1729 was provided by NASA
through a grant from the Space Telescope Science Institute, which is operated
by the Association of Universities for Research in Astronomy, Inc., under NASA
contract NAS 5-26555. JKB was supported by a Science and Technology Facilities
Council Ernest Rutherford Fellowship. NM was partly supported by a Science and
Technology Facilities Council Consolidated Grant [ST/R000395/1], the
Leverhulme Trust through a research project grant [RPG2020-82] and a UKRI
Future Leaders Fellowship [grant number MR/T040866/1].

All of the JWST data used in this paper were obtained from the Mikulski
Archive for Space Telescopes (MAST) at the Space Telescope Science Institute.
The specific observations analyzed can be accessed via
http://dx.doi.org/10.17909/23j6-ng29 (catalog 10.17909/23j6-ng29)

## References

  * Alderson et al. (2022) Alderson, L., Wakeford, H. R., Alam, M. K., et al. 2022, arXiv e-prints, arXiv:2211.10488
  * Arcangeli et al. (2019) Arcangeli, J., Désert, J.-M., Parmentier, V., et al. 2019, A&A, 625, A136
  * Beatty et al. (2019) Beatty, T. G., Marley, M. S., Gaudi, B. S., et al. 2019, AJ, 158, 166
  * Bell & Cowan (2018) Bell, T. J., & Cowan, N. B. 2018, ApJ, 857, L20
  * Birkmann et al. (2022) Birkmann, S. M., Ferruit, P., Giardino, G., et al. 2022, A&A, 661, A83
  * Bourrier et al. (2020) Bourrier, V., Ehrenreich, D., Lendl, M., et al. 2020, A&A, 635, A205
  * Bushouse et al. (2022) Bushouse, H., Eisenhamer, J., Dencheva, N., et al. 2022, JWST Calibration Pipeline, Zenodo, v1.7.0, Zenodo, doi:10.5281/zenodo.7038885
  * Castelli & Kurucz (2003) Castelli, F., & Kurucz, R. L. 2003, in Modelling of Stellar Atmospheres, ed. N. Piskunov, W. W. Weiss, & D. F. Gray, Vol. 210, A20
  * Cowan et al. (2012) Cowan, N. B., Machalek, P., Croll, B., et al. 2012, ApJ, 747, 82
  * Delrez et al. (2016) Delrez, L., Santerne, A., Almenara, J. M., et al. 2016, MNRAS, 458, 4025
  * Espinoza et al. (2022) Espinoza, N., Úbeda, L., Birkmann, S. M., et al. 2022, arXiv e-prints, arXiv:2211.01459
  * Evans et al. (2017) Evans, T. M., Sing, D. K., Kataria, T., et al. 2017, Nature, 548, 58
  * Evans et al. (2018) Evans, T. M., Sing, D. K., Goyal, J. M., et al. 2018, AJ, 156, 283
  * Feng et al. (2020) Feng, Y. K., Line, M. R., & Fortney, J. J. 2020, AJ, 160, 137
  * Gao & Powell (2021) Gao, P., & Powell, D. 2021, ApJ, 918, L7
  * Garhart et al. (2020) Garhart, E., Deming, D., Mandell, A., et al. 2020, AJ, 159, 137
  * Hoeijmakers et al. (2020) Hoeijmakers, H. J., Seidel, J. V., Pino, L., et al. 2020, A&A, 641, A123
  * Hoeijmakers et al. (2022) Hoeijmakers, H. J., Kitzmann, D., Morris, B. M., et al. 2022, arXiv e-prints, arXiv:2210.12847
  * Hoffman & Gelman (2014) Hoffman, M. D., & Gelman, A. 2014, Journal of Machine Learning Research, 15, 1593\.  https://jmlr.org/papers/v15/hoffman14a.html
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science and Engineering, 9, 90
  * Irwin et al. (2020) Irwin, P. G. J., Parmentier, V., Taylor, J., et al. 2020, MNRAS, 493, 106
  * Keating et al. (2019) Keating, D., Cowan, N. B., & Dang, L. 2019, Nature Astronomy, 3, 1092
  * Knutson et al. (2007) Knutson, H. A., Charbonneau, D., Allen, L. E., et al. 2007, Nature, 447, 183
  * Komacek et al. (2022) Komacek, T. D., Tan, X., Gao, P., & Lee, E. K. H. 2022, ApJ, 934, 79
  * Kreidberg et al. (2018) Kreidberg, L., Line, M. R., Parmentier, V., et al. 2018, AJ, 156, 17
  * Lothringer et al. (2021) Lothringer, J. D., Rustamkulov, Z., Sing, D. K., et al. 2021, ApJ, 914, 12
  * Luger et al. (2019) Luger, R., Agol, E., Foreman-Mackey, D., et al. 2019, AJ, 157, 64
  * Mendonça et al. (2018) Mendonça, J. M., Malik, M., Demory, B.-O., & Heng, K. 2018, AJ, 155, 150
  * Mikal-Evans et al. (2020) Mikal-Evans, T., Sing, D. K., Kataria, T., et al. 2020, MNRAS, 496, 1638
  * Mikal-Evans et al. (2022) Mikal-Evans, T., Sing, D. K., Barstow, J. K., et al. 2022, Nature Astronomy, 6, 471
  * Parmentier & Crossfield (2018) Parmentier, V., & Crossfield, I. J. M. 2018, in Handbook of Exoplanets, ed. H. J. Deeg & J. A. Belmonte, 116
  * Parmentier et al. (2021) Parmentier, V., Showman, A. P., & Fortney, J. J. 2021, MNRAS, 501, 78
  * Parmentier et al. (2018) Parmentier, V., Line, M. R., Bean, J. L., et al. 2018, A&A, 617, A110
  * Perna et al. (2010) Perna, R., Menou, K., & Rauscher, E. 2010, ApJ, 719, 1421
  * Rigby et al. (2022) Rigby, J., Perrin, M., McElwain, M., et al. 2022, arXiv e-prints, arXiv:2207.05632
  * Rustamkulov et al. (2022a) Rustamkulov, Z., Sing, D. K., Liu, R., & Wang, A. 2022a, ApJ, 928, L7
  * Rustamkulov et al. (2022b) Rustamkulov, Z., Sing, D. K., Mukherjee, S., et al. 2022b, arXiv e-prints, arXiv:2211.10487
  * Salvatier et al. (2016) Salvatier, J., Wiecki, T. V., & Fonnesbeck, C. 2016, PeerJ Computer Science, 2, e55
  * Schlawin et al. (2020) Schlawin, E., Leisenring, J., Misselt, K., et al. 2020, AJ, 160, 231
  * Showman et al. (2009) Showman, A. P., Fortney, J. J., Lian, Y., et al. 2009, ApJ, 699, 564
  * Showman & Guillot (2002) Showman, A. P., & Guillot, T. 2002, A&A, 385, 166
  * Stevenson et al. (2014) Stevenson, K. B., Désert, J.-M., Line, M. R., et al. 2014, Science, 346, 838
  * STScI Development Team (2013) STScI Development Team. 2013, pysynphot: Synthetic photometry software package, , , ascl:1303.023
  * Tan & Komacek (2019) Tan, X., & Komacek, T. D. 2019, ApJ, 886, 26
  * van der Walt et al. (2011) van der Walt, S., Colbert, S. C., & Varoquaux, G. 2011, Computing in Science Engineering, 13, 22
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261
  * Visscher et al. (2010) Visscher, C., Lodders, K., & Fegley, Bruce, J. 2010, ApJ, 716, 1060
  * Wahl et al. (2021) Wahl, S. M., Thorngren, D., Lu, T., & Militzer, B. 2021, ApJ, 921, 105
  * Wakeford et al. (2017) Wakeford, H. R., Visscher, C., Lewis, N. K., et al. 2017, MNRAS, 464, 4247
  * Zhang et al. (2018) Zhang, M., Knutson, H. A., Kataria, T., et al. 2018, AJ, 155, 83


# PixT3: Pixel-based Table To Text generation

Iñigo Alonso,
HiTZ Center - Ixa,
University of the
Basque Country UPV/EHU
<EMAIL_ADDRESS>
Eneko Agirre
HiTZ Center - Ixa,
University of the
Basque Country UPV/EHU
<EMAIL_ADDRESS>
Mirella Lapata
Institute for Language,
Cognition and Computation,
School of Informatics,
University of Edinburgh
<EMAIL_ADDRESS>

###### Abstract

Table-to-Text has been traditionally approached as a linear language to text
problem. However, visually represented tables are rich in visual information
and serve as a concise, effective form of representing data and its
relationships. When using text-based approaches, after the linearization
process, this information is either lost or represented in a space inefficient
manner. This inefficiency has remained a constant challenge for text-based
approaches making them struggle with large tables. In this paper, we
demonstrate that image representation of tables are more space-efficient than
the typical textual linearizations, and multi-modal approaches are competitive
in Table-to-Text tasks. We present PixT3, a multimodal table-to-text model
that outperforms the state-of-the-art (SotA) in the ToTTo benchmark in a pure
Table-to-Text setting while remaining competitive in controlled Table-to-Text
scenarios. It also generalizes better in unseen datasets, outperforming ToTTo
SotA in all generation settings. Additionally, we introduce a new intermediate
training curriculum to reinforce table structural awareness, leading to
improved generation and overall faithfulness of the models.

PixT3: Pixel-based Table To Text generation

Iñigo Alonso, HiTZ Center - Ixa, University of the Basque Country UPV/EHU
<EMAIL_ADDRESS>Eneko Agirre HiTZ Center - Ixa, University of the
Basque Country UPV/EHU<EMAIL_ADDRESS>Mirella Lapata Institute for Language,
Cognition and Computation, School of Informatics, University of Edinburgh
<EMAIL_ADDRESS>

##  1 Introduction

Data-to-text generation is the task of generating accurate textual
descriptions from a non-linguistic structured input such as tables, knowledge
bases, tuples, or graphs (Reiter and Dale, 1997; Covington, 2001; Gatt and
Krahmer, 2018). Practical uses of this task include generating biographical
information descriptions (Lebret et al., 2016), creating sport summaries based
on game statistics (Wiseman et al., 2017), and generating interesting facts
from Wikipedia tables Korn et al. (2019). The main objective in these
applications is to represent relevant information in the input through natural
language descriptions.

Figure 1: Overview of our multimodal Table-to-Text pipeline.

In a pure Data-to-Text setting, this task is underspecified, meaning that the
same table may be described by multiple textual descriptions, all of them
equally correct, as each one can focus on different subsets of the input data.
In a more precise setting, a selection of the contents used in the
description, also referred as Content Selection, are provided alongside the
input data. This setting is often referred as Controlled Data-to-Text Parikh
et al. (2020a).

Various Data-to-Text approaches have emerged to address this challenge in both
settings. Methods include leveraging the structural information of the input
data (Puduppully et al., 2019; Chen et al., 2020b; Wang et al., 2022), using
neural templates (Wiseman et al., 2018), or focusing on content planning (Su
et al., 2021; Puduppully et al., 2022). Other techniques (Chen et al., 2020a,
c; Aghajanyan et al., 2022; Kasner and Dusek, 2022) also report performance
gains in terms of fluency and generalization when leveraging large-scale pre-
trained language models (Devlin et al., 2019; Raffel et al., 2020).

A common feature across all these methods is their unimodal approach to this
task. Data-to-Text, particularly in the context of Table-to-Text, has been
traditionally approached as a linear language to text problem. This is
primarily due to tables being predominantly comprised of textual data, and
computer vision models historically having limited visual language
understanding capabilities. However, visually represented tables are rich in
visual information and serve as a concise, effective form of representing data
and its relationships. When using text-based approaches, after the
linearization process, this information is either lost or represented in a
space inefficient manner. This space inefficiency has remained a constant
challenge for text-based Table-to-Text approaches making them struggle when
tables require an elevated amount of tokens to be represented. In the case of
controlled table-to-text generation, current approaches have deviated from
utilizing tables altogether and have shifted form controlled Table-to-Text to
simply controlled text parsing setting. This shift has rendered the latest
models susceptible to generalization and practically ineffective in scenarios
where content selection is not explicitly provided.

Recent work Kim et al. (2022); Davis et al. (2022) has significantly advanced
the field of visual language understanding, with Vision Transformer (ViT)
Dosovitskiy et al. (2021) based approaches demonstrating proficiency in
various natural language processing tasks, including language modeling Rust et
al. (2023), visual document understanding Huang et al. (2022), and visual
question answering Masry et al. (2022). Building on top of this architecture,
models like Pix2Struct Lee et al. (2023) extend the capabilities of ViTs,
allowing transfer learning for visually-situated language tasks, and
addressing the limitations associated with fixed resolution.

We leverage the transfer learning capabilities of Pix2Struct to demonstrate
that approaching Table-to-Text generation as an Image-to-Text problem can
improve the input size limitations of current approaches while remaining
competitive against the latest methods.

Our mayor contributions are as follows:

  * •

We present PixT3, a multimodal table-to-text model that outperforms the state-
of-the-art (SotA) in the ToTTo benchmark Parikh et al. (2020b) in a pure
Table-to-Text setting while remaining competitive in controlled Table-to-Text
scenarios. PixT3 is also more robust to generalization and does not degrade
when the full table is part of the input in controlled Table-to-Text. Notably,
PixT3 achieves this with an equivalent parameter count as the current SotA,
and a simplified end-to-end architecture.

  * •

We introduce a new intermediate training curriculum to reinforce table
structural awareness, leading to improved generation and overall faithfulness
of the models.

  * •

We present a new Controlled Table-to-Text generation dataset based on
Logic2Text Chen et al. (2020c) including the content selection values of the
target descriptions in the form of highlighted cells.

##  2 Method

###  2.1 Background

Previous attempts at approaching Table-to-Text from a multimodal perspective
have relied on external Optical Character Recognition systems (OCR) to first
extract the text and structure from table images and feed the output to a
linearized text-based pipeline Dash et al. (2023). OCR systems are often
noisy, add engineering complexity, reduce adaptability, and may lead to an
overall increment of computational cost. However, the main difference between
our two approaches is that, in this approach, the table image is considered a
limitation rather than a rich source of information. In order to leverage this
information a pixel-only model needs to be utilized.

Recent progress with OCR-free pixel-only visual language undestanding models
like Donut Kim et al. (2022), Dessurt Davis et al. (2022), and Pix2Struct Lee
et al. (2023) have bridged the gap with OCR-based pipelines, being the latter
the top performant in multiple visual language understanding
benchmarks111ChartQA, AI2D, OCR VQA, Ref Exp, Widget Cap, Screen2Words. We
refer the reader to the original Pix2Struct publication for a more in-depth
comparison between these models..

###  2.2 Pix2Struct

Pix2Struct is a ViT based model pretrained on 80M screenshots of web pages
extracted from the URLs in the C4 corpus Raffel et al. (2020). In the same way
as ViTs, Pix2Struct splits the input image into patches of 16x16 pixels each,
linearly embed each of them, adds position embeddings, and feed the resulting
sequence of vectors to a standard Transformer Vaswani et al. (2017) encoder.

However, ViTs are typically pre-trained in low resolutions and fixed aspect
ratios, and tend to be very sensitive to resolution changes. Supporting
variable resolutions and multiple aspect ratios is crucial for Table-to-Text
tasks because, in the same way as with many other visually-situated language
tasks, table sizes and aspect ratios can be very diverse. Pix2Struct addresses
this by first re-scaling the input up or down to extract the maximal number of
fixed-size patches that fit within a given sequence length. Then, they replace
the typical 1-dimensional absolute positional embedding with a 2-dimensional.
This adds resolution flexibility and removes any aspect ratio distortion.

Similarly to Rust et al. (2023) and Davis et al. (2022), Pix2Struct is first
warmed up with a reading curriculum to improve training stability and
finetuning performance. Then the model was pre-trained with a screenshot
parsing objective. This objective consists on generating a simplified version
of the HTML subtree that represents a highlighted area of a web page
screenshot. On top of this they introduce a BART-like Lewis et al. (2020)
learning signal to the pre-training by masking 50% of the text in the input
and then requiring the model to produce the entire subtree. This results in a
pre-trained model that can be finetuned on tasks containing visually-situated
language. Lee et al. (2023) pretrained two model variants, a base model with
282M parameters, and a large model with 1.3B parameters.

Pix2Struct obtains SotA results in many visual language understanding
benchmarks across four domains: illustrations, user interfaces, natural
images, and documents. However, none of them included Data-to-Text or Table-
to-Text. The weights for both the base and large versions of these finetuned
Pix2Struct variants are publicly available222https://github.com/google-
research/pix2struct.

###  2.3 PixT3

PixT3 is an image-encoder-text-decoder model based on Pix2Struct Lee et al.
(2023). While the complete pipeline would be considered Table-to-Text as it
receives text-formatted tables, this is a purely Image-to-Text model. An
overview of the complete pipeline is depicted in figure 1. To create this
model we finetuned a "vanilla" version of Pix2Struct on our image rendered
version of ToTTo dataset (See 2.4 for a broader description of the rendering
process). We follow the original Pix2Struct finetuning process as closely as
possible.

###  2.4 Finetuning Dataset

We use ToTTo Parikh et al. (2020b) as our training dataset for PixT3. ToTTo is
an open-domain English controlled Table-to-Text dataset with 120,761 training
examples. Each example contains a Wikipedia table, its page and section title,
a set of highlighted table cells, and a one-sentence description. The set of
highlighted cells represent the Content Selection required to produce the
target description. The human revised sentences, alongside the inclusion of
the Content Selection, make this dataset a high-quality unique dataset to
train and test, for both controlled and plain Table-to-Text settings.

In order to train an Image-to-Text model like Pix2Struct with this dataset we
created a pipeline that first, parses the table to HTML, and then renders the
HTML into an image. As the page and section title of each table contain
crucial information to generate the target description, we follow Lee et al.
(2023) and we render this text and add it on top of the rendered table to
create the input image for our pixel-only model. We create three different
versions of this dataset depending on the Data-to-Text setting. Note that for
controlled generation setting we only render the page and section title
alongside a concatenation of the values of the highlighted cells. This last
version of the dataset was created to fairly compare our model against many of
the other controlled Table-to-Text models. We do this to much the input
information of these models as many of them exclude the table completely. We
use the same table style for the first two dataset versions.

####  2.4.1 Image compression and truncation

Although the architecture of Pix2Struct allows us to set a custom amount of
input patches, an increase in the input length still comes at a computational
cost. We follow Lee et al. (2023) with an input length of 2048 patches of 16 ×
16 pixels each to represent all input images. This results in a maximum image
input size of 524,288 pixels. Once the size of the rendered table plus title
exceeds this amount of pixels, the input image is downscaled. While many of
the tables in ToTTo fit into this limit, other datasets like ToTTo contain
tables that exceed this size. Specifically, 41.74% of tables in ToTTo exceed
this size, being 5% of the total table images larger than 8.3M pixels (32,768
patches). Downscaling these rendered tables results in a 16 times reduction,
and hence, a hit in performance as this extreme size reduction makes the text
within the table unreadable. A chart showing how this re-scaling affects the
performance of the models can be found in Appendix A.

We see that the average performance remains constant up to the point of down-
scaling. Thus, it is worth trying to minimize downscaling as much as possible.
In order to do that, we truncate the image of the rendered table to fit within
a maximum downscaling factor of 0.39, meaning that images that would require
further compression are first compressed to 39% of their original size and
then truncated from right to left until they fit into 2048 patches of 16 × 16
pixels. A full analysis to identify the optimal point between truncation and
compression is presented in Appendix B.

###  2.5 Intermediate training

The first qualitative analysis we conducted over the descriptions produced by
PixT3 showed that 45% of faithfulness errors in the Table-to-Text setting were
caused by a disregard of table rules. This is, either the produced text
exhibited a relation between two unrelated table cells, or the model miss
interpreted the amount of rows and columns in the table. One of the basic
principles of tables establish that each cell is related and only related to
the cells in the same column and the same row. With this in mind, we decided
to expose the model to an intermediate training before finetuning it on ToTTo.
This acts as a structural learning curriculum to reinforce the notion of this
rule on PixT3. This variant of PixT3 intermediately trained with this
structural learning curriculum is denominated PixT3Struct.

In a similar fashion to Chen et al. (2023), we find a self supervised
objective (SSL).

####  2.5.1 Intermediate training Dataset

The task of reinforcing the structural notion of tables to the model resembled
the widely known research area of Table Structure Recognition (TSR).
Therefore, we consider using two famous TSR datasets, ICDAR2021 Kayal et al.
(2021) and TableBank Li et al. (2019). However, both of these datasets focus
fully on structure recognition with no text assigned to the target cells,
requiring us to fill cells with synthetic text. To avoid generating synthetic
text, we also considered using ToTTo. However, many SSL objectives required
the model to produce long sequences that where unnecessary for our learning
curriculum, slowing down the training. Therefore we decided to make use of the
table rendering pipeline we developed to turn ToTTo into a image-to-text
dataset (See 2.4) and generate a synthetic dataset specifically "tailored" for
our use-case while being extra careful with the effects of synthetic text on
the downstream task (See section 3.2 for more details).

Our intermediate training dataset is comprised of 135,400 synthetic tables,
being 120,000 for training and two sets of 7700 for validation and test. The
structure of each table in terms of size, and column and row spans, was
randomly determined following ToTTo’s training set distribution. We caped the
generation at a maximum of 20 columns and 75 rows. As the text within each
cell acts as an identifier instead of a meaningful content, the text of each
cells is comprised of a random combination of one to five random English
alphabet characters and digits.

####  2.5.2 SSL objective

A common approach that can be found in literature is masking Devlin et al.
(2019). Masking is difficult to implement in Table-to-Text as table values are
not correlated to other values and thus a masked cell can not be predicted
from other cells in its context. Although table values can be arranged in a
way where each value cell is in fact correlated to its neighbours,
experimentation conducted in this study has show that such kind of objectives
do not contribute positively to improving performance in downstream tasks
(Appendix C shows the experimentation we conducted). Another common self
supervised objective found in literature is table linearization as shown in
Chen et al. (2023). However, this pre-training objective scales poorly with
the table size, as producing an entire linearization of a table would result
in long generations leading to slow pretraining. Therefore, we propose a new
self supervised objective that requires models to capture the relations
between cells within a table with a minimal amount of tokens to produce.

Figure 2: Example of intermediate training Dataset

The objective consists on highlighting one random cell in each example of our
intermediate training dataset. The model is then trained to produce a sorted
list of cells within the same column and row. We follow the same pseudo HTML
notation used in Pix2Struct to format our output sequence to ease the model’s
transition form its last screenshot parsing objective to this new one. Note
that we consider tables with a heterogeneous structure where where cells can
span across multiple columns and rows. For cases where the highlighted cell
spans across many rows or columns, the expected sequence will contain all
cells in the related rows and columns. An example of this objective can be
seen in figure 2.

##  3 Experimental Setup

###  3.1 Implementation

As mentioned in section 2.2, there are two pretrained Pix2Struct models: a
282M parameter _base_ , and a 1.3B parameter _large_ variants. Alongside
these, the checkpoints for all 8 versions of all finetuned models in different
visually-situated language understanding benchmarks are also publicly
available. All experiments were carried out using the _base_ version of each
model. We later apply all the acquired knowledge to train a _large_ variant.

####  3.1.1 PixT3 and PixT3Struct base

We train a variant of PixT3 and PixT3Struct for every Table-to-Text setting
using its corresponding rendered ToTTo dataset with a maximum down-scaling
factor of 0.39 (See section 2.4 for more details). Both models are based on
Pix2Struct without any fine-tuning. Fine-tuned versions, such as _docvqa_ ,
converged faster than the ’vanilla’ Pix2Struct probably thanks to the
similarity between the two generation objectives. However, after 8,500 steps,
the latter narrowed the performance gap, ultimately outperforming the fine-
tuned one, and resulting in a longer but more stable training. All _base_
models were fine-tuned with a batch size of 8 and a gradient accumulation of
32 for 14K steps on one A100 GPU. The checkpoints for each model were saved at
the epoch with the best performance over the valiadation set. All models use
an input sequence length of 2048 patches and are optimized using AdamW
Loshchilov and Hutter (2017). We use a learning rate scheduler with a linear
warmup of 1000 steps to 0.0001, followed by cosine decay to 0. The decoder
maximum sequence length is 50 tokens, as this would cover 97.49º% of the
training target sequences.

####  3.1.2 PixT3 and PixT3Struct large

Only one _large_ model was fine-tuned on the controlled Table-to-Text setting
with a batch size of 2 and gradient accumulation of 32 for 17K steps on one
A100 GPU. We use a use a learning rate scheduler with a linear warmup of 1000
steps to 0.00005. As shown in table 1, despite the parameter increase, the
model performed similarly to the its base sibling. These results go in line
with other studies An et al. (2022) with the ToTTo dataset where a T5 model
Raffel et al. (2020) with 5B parameters performed similarly to its 220M
counterpart.

####  3.1.3 Benchmark

We selected the published top performant model in the ToTTo leader-board333A
model named SKY appears to have a higher score in the ToTTo leader-board.
However, we where not able to verify its results and use it because, as of
this paper’s writing, it remains unpublished. to compare our image-based
approach against the text-based controlled Table-to-Text SotA. By the time of
this writing, this model is CoNT An et al. (2022). In the same way as with
PixT3 and PixT3Struct, we train a variation of CoNT with the ToTTo dataset,
for each setting, using the original publicly available training
script444https://github.com/Shark-NLP/CoNT.

###  3.2 Intermediate training

For the intermediate training of PixT3Struct we trained a Pix2Struct ’vanilla’
for 1.4k steps on the SSL objective described in section LABEL:app:ssl1. To
prevent the synthetic text in the intermediate training dataset from breaking
the language understanding capabilities acquired by the model during pre-
training, we conducted two experiments. One training the model freezing the
weights of the decoder, and another fully training it. We see that the frozen
one converged at a BLEU of 39.2 in 1.4k steps, at the same time the fully
trained reached a BLEU of 97.59. We then followed the same ToTTo fine-tuning
for both models and saw no degradation or difference between the two models,
with the fully trained model consistently exhibiting better performance.

###  3.3 Zero shot generalization

In order to evaluate the generalisation capabilities of our model, we evaluate
our model in a zero shot setting in other datasets and we compare its
performance against CoNT.

####  3.3.1 Controlled-Logic2Text Dataset

To the best of our knowledge, ToTTo is a unique dataset, containing not only a
set of human curated descriptions but also the content selection information
required to produce each of the target sentences. This allows ToTTo to serve
as a benchmark for: Table-to-Text, controlled Table-to-Text, and controlled
generation. Consequently, finding a comparable dataset to evaluate our models
across these three settings becomes a challenging task.

For this, we use the Logic2Text dataset Chen et al. (2020c) that contains a
similar set of human produced Table-to-Text examples paired with Logical Forms
representing, in graph like structures, the semantics of its paired target
texts. Similar to the highlighted cells in ToTTo, these Logical Forms control
the generation of the description. We follow Alonso and Agirre (2023) to trace
the values mentioned in these Logical Forms back to the cells in the input
table, obtaining the list of cells required to produce the given descriptions.
These cells serve the same purpose as ToTTo’s highlighted cells, adding the
content selection required for the controlled Table-to-Text and controlled
generation settings.

##  4 Results

###  4.1 Benchmarks

As mentioned in the section 2.4, we utilize the ToTTo benchmark to train and
evaluate PixT3, PixT3Struct against its top performant model, CoNT. For the
automatic metrics, we employ the same metrics used to rank participants in the
competition, namely BLEU Papineni et al. (2002), PARENT Dhingra et al. (2019)
(similar to BLEU but with a stronger emphasis on recall with the n-grams not
only from the reference but also from the table; this serves as a proxy of
faithfulness), and BLEURT Sellam et al. (2020). We evaluate these models in
three generation settings: Table-to-Text, controlled Table-to-Text, and
controlled generation. Then, we demonstrate the generalization capabilities of
our model compared to the SotA in ToTTo under the same settings. Finally, we
evaluate the faithfulness of these models through a human evaluation test.

| Dev Set (All) | Dev Set (Non) | Test Set (All)
---|---|---|---
Model | BLEU | PAR | BLEU | PAR | BLEU | PAR | BLEURT
Reference (upper bound) | 100.0 | 86.61 | 100.0 | 85.98 | - | - | -
T5-3B | 48.4* | 57.8* | 40.4* | 53.3* | 49.5* | 58.4* | 0.230*
T5-220M | 47.7* | 57.1* | 39.6* | 52.6* | - | - | -
CoNT | 49.2* | 59.4* | 41.5* | 55.0* | 49.1* | 58.9* | 0.238*
PixT3 (C. generation) | 45.7 | 55.66 | 37.9 | 50.68 | 45.4 | 55.52 | 0.1785
CoNT (C. Table-to-Text) | 23.8 | 29.28 | 19.4 | 26.14 | 23.9 | 29.24 | -0.2886
PixT3 (C.Table-to-Text) | 46.0 | 54.53 | 38.7 | 49.99 | 45.2 | 54.16 | 0.1628
PixT3-large (C. Table-to-Text) | 45.3 | 53.88 | 37.8 | 49.13 | - | - | -
PixT3Struct (C. Table-to-Text) | 46.2 | 55.05 | 38.7 | 50.58 | 45.4 | 54.67 | 0.1673
CoNT (Table-to-Text) | 21.7 | 25.84 | 17.7 | 23.51 | 21.6 | 25.77 | -0.3664
PixT3 (Table-to-Text) | 24.5 | 27.92 | 20.9 | 26.43 | 24.5 | 27.79 | -0.3375
PixT3Struct (Table-to-Text) | 24.8 | 28.25 | 20.9 | 26.39 | 24.7 | 28.29 | -0.3358
Table 1: Automatic evaluation of T5, ToTTo SotA, PixT3, and PixT3Struct in
each of the three generation settings. All models were trained or fine-tuned
with the ToTTo dataset in their corresponding setting. All PixT3 models are
the _base_ variant except for the explicitly specified _large_ variant. *
reported results.

###  4.2 ToTTo results

Table 1 compares PixT3 and PixT3Struct against CoNT and T5. We train each
model with its corresponding dataset setting (See section 2.4 for more
details).

Table-to-Text Both PixT3 and PixT3Struct outperform CoNT, with PixT3Struct
being the top-performing model in this setting. Although the difference
between PixT3 and PixT3Struct is small, PixT3Struct consistently achieves the
highest performance across all metrics on both the development and test sets.
This leads to the conclusion that, even though the intermediate training
curriculum was originally aimed at reducing table structure-related
faithfulness errors, it also had a positive impact on automatic evaluation
results.

The overall low results are primarily attributed to the lack of control in the
generation process rather than reduced fluency of the produced descriptions.
In this setting, models are required to perform content selection, and any
variation here can result in entirely different descriptions. N-gram
overlapping-based automatic metrics become particularly punitive in such
cases, underscoring the necessity for a different set of metrics for this
specific setting. This also highlights the underspecified nature of the pure
Table-to-Text setting.

Controlled Table-to-Text This setting showcases the largest gap between our
models and the SotA. We can see an almost two times improvement over the
version of CoNT trained to use the full table as well as the highlighted
cells, rather than just the highlighted cells in its original variant. We can
see how addind the fuull table to CoNT drastically degrades its performance.
PixT3 model variants not only it do not degrade when adding the full table,
but also improves slightly. In this setting we see once again how the
intermediately trained PixT3Struct achieves the highest performance across all
metrics.

We can also see how the addition of more parameters does not contribute on
improving the performance of the model in this setting. The _large_ version of
the model with 1.3B parameters performs similarly its 282M parameter sibling.
We can see a similar behaviour between T5-3B and T-220M.

Controlled generation In this setting, PixT3 is unable to surpass ToTTo’s SotA
CoNT, falling 3.5 BLEU points behind on the overall development set and 3.7 on
the test set. This highlights the suboptimal nature of approaching a text-only
controlled generation task as Image-to-Text. Since many features present in
images are absent in this setting, a text-oriented approach would be more
optimized for the task at hand.

For CoNT we use the reported results in the original publication An et al.
(2022). We validated these results by replicating CoNT in this setting with
the original code.

We do not present PixT3Struct in this comparison because pre-training a model
on a table structural setting would be useless in a table-free downstream
task.

###  4.3 Generalization

Table 2 shows the results of the evaluation on Controlled-Logic2Text in a
zero-shot setting (refer to Section 3.3.1 for more details on this dataset).
We observe that PixT3 outperforms CoNT not only in the usual settings (plain
and controlled Table-to-Text) but also in controlled generation.

These results also demonstrate that, in comparison to CoNT, the performance of
both of our models is not only unaffected by the inclusion of the full table
in the controlled Table-to-text setting but is also enhanced by it, resulting
in a performance increase of 1.5 points relative to the controlled generation
setting.

We also observe that the table structure intermediate training curriculum does
not contribute to the improvement of PixT3Struct on automatic metrics in the
zero-shot setting. We see two possible reasons to this: firstly, the automatic
metrics are so low that any enhancement of PixT3Struct over PixT3 is
overshadowed by more substantial differences in the structure of the
descriptions of both datasets. The second reason is connected to the goal of
the table structure learning curriculum. The benefits of this curriculum
become more evident in the faithfulness evaluation.

Model | BLEU | PARENT | BLEURT
---|---|---|---
Controlled Generation
CoNT | 20.9 | 63.30 | -1.0859
PixT3 | 22.5 | 61.81 | -1.0834
Controlled Table-to-Text
CoNT | 12.9 | 43.35 | -1.1504
PixT3 | 24.1 | 58.48 | -1.0532
PixT3Struct | 22.7 | 58.79 | -1.0584
Table-to-Text
CoNT | 11.5 | 37.27 | -1.2214
PixT3 | 13.3 | 38.65 | -1.1973
PixT3Struct | 12.7 | 38.14 | -1.2070
Table 2: Automatic evaluation in a zero shot setting for each of the three
generation settings. All models were trained or fine-tuned in the ToTTo
dataset and evaluated on the Controlled-Logic2Text dataset. Both datasets in
their corresponding setting. All PixT3 models are the _base_ variant.

###  4.4 Human evaluation

We conducted a human evaluation to evaluate the faithfulness of the generated
descriptions555We conducted this evaluation using Prolific
https://www.prolific.com. We evaluated 100 descriptions for every model in
each of the three generation settings. Each description corresponds to the
generation of each model for the same 100 examples of the development set of
ToTTo. We presented 90 evaluators with an uncompressed image of a rendered
table, its page and section title, and one model generated description. We
also added 100 reference descriptions to measure deviation between
evaluations. Each participant was asked to determine whether each description
was "True" or "False" based on the information provided in the table and/or
its title and subtitle.

The evaluation concluded with a moderate agreement among evaluators, with a
Fleiss’ Kappa coefficient of 0.55 (Fleiss, 1971). Table 3 shows the results of
this evaluation, revealing that both PixT3 and PixT3Struct achieve a higher
percentage of "True" descriptions compared to CoNT in both Table-to-Text and
controlled Table-to-Text settings. Furthermore, observe thatnour model’s
capacity to maintain performance when incorporating the table as input extends
to faithfulness. Additionally, the intermediate pretraining of PixT3Struct
contributed to its improvement over PixT3. This leads PixT3Struct to slightly
improve the faithfulness results of CoNT in its original, top-performing
setting.

Model | C. Gen. | C. T2T | T2T
---|---|---|---
Reference | 87% | 84% | 89%
CoNT | 76% | 16% | 35%
PixT3 | 69% | 69% | 77%
PixT3Struct | - | 72% | 78%
Table 3: Human faithfulness evaluation results. Percentage of descriptions
rated as faithful by 90 human annotators for each of the model variants of the
settings: controlled generation (C. Gen), controlled Table-to-Text (C. T2T),
and Table-to-Text (T2T)

##  5 Related Work

Data-to-text generation is a long-established research line. It is the task of
generating accurate textual descriptions from a non-linguistic structured
input such as tables, knowledge bases, tuples, or graphs (Reiter and Dale,
1997; Covington, 2001; Gatt and Krahmer, 2018). Over time, multiple techniques
have been developed to approach this task in different ways, including
leveraging the structural information of the input data (Puduppully et al.,
2019; Chen et al., 2020b; Wang et al., 2022), using neural templates (Wiseman
et al., 2018), or focusing on content planning (Su et al., 2021; Puduppully et
al., 2022). Other techniques (Chen et al., 2020a, c; Aghajanyan et al., 2022;
Kasner and Dusek, 2022) also report performance gains in terms of fluency and
generalization when leveraging large-scale pre-trained language models (Devlin
et al., 2019; Raffel et al., 2020). Practical applications of Data-to-Text
include generating biographical information descriptions (Lebret et al.,
2016), creating sport summaries based on game statistics (Wiseman et al.,
2017), and generating interesting facts from Wikipedia tables Korn et al.
(2019).

Visual Language Understanding, on the other hand, studies the interactions
between language and vision. This are has traditionally approached by
separating text and images into distinct channels thought techniques like
Optical Character Recognition OCR Srihari et al. (2003). Recent work Kim et
al. (2022); Davis et al. (2022) has significantly advanced the field of visual
language understanding, with Vision Transformer (ViT) Dosovitskiy et al.
(2021) based approaches demonstrating proficiency in various natural language
processing tasks, including language modeling Rust et al. (2023), visual
document understanding Huang et al. (2022), and visual question answering
Masry et al. (2022). Building on top of this architecture, models like
Pix2Struct Lee et al. (2023) extend the capabilities of ViTs, allowing
transfer learning for visually-situated language tasks, and addressing the
limitations associated with fixed resolution.

Recent research efforts have explored multimodal approaches to Table-to-Text
generation. However, to the best of our knowledge, no prior work has
approached the entire pipeline of table-to-text generation in an end-to-end
manner, treating table-to-text as as a visually-situated language
understanding problem. While works such like Dash et al. (2023) have proposed
similar rendering pipelines for tables, their focus only on producing a table-
rendered version of the WeatherGov Liang et al. (2009) dataset, a somewhat
dated single-domain dataset widely known for the lack of diversity on its
table descriptions. Furthermore, existing works like TableVLM, which shares
similarities with our intermediate training through synthetic table structure
generation, primarily concentrate on Table Structure Recognition (TSR).

##  6 Conclusion

In this paper, we described a multimodal approach to Data-to-Text generation,
specifically focusing on Table-to-Text scenarios. Traditionally treated as a
linear language-to-text problem, we introduced a novel approach by leveraging
the capabilities of Vision Transformer-based models, particularly Pix2Struct,
to treat Table-to-Text generation as an Image-to-Text problem.

Our contributions are substantial and multifaceted. First, we introduced
PixT3, a multimodal Table-to-Text model, which not only outperformed the SotA
in pure Table-to-Text settings but also remained competitive in controlled
scenarios. Additionally, PixT3 exhibited enhanced robustness to
generalization, even when the entire table was part of the input.

Second, in order to enhance its structural awareness, we introduced a novel
intermediate training curriculum that resulted in improved generation quality
and overall faithfulness of the models.

Lastly, we presented a new Controlled Table-to-Text generation dataset based
on Logic2Text, incorporating content selection values in the form of
highlighted cells. This dataset provides a valuable resource for evaluating
and advancing Controlled Table-to-Text approaches.

In conclusion, our work demonstrates the efficacy of treating Table-to-Text
generation as an Image-to-Text problem, offering improvements in space
efficiency, model performance, and generalization. PixT3, along with our novel
training curriculum and dataset, validates this approach, and contributes to
the field of Data-to-Text generation by opening a the way for future research
and applications in this domain.

## Acknowledgements

## References

  * Aghajanyan et al. (2022) Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2022.  HTLM: Hyper-text pre-training and prompting of language models.  In _International Conference on Learning Representations_.
  * Alonso and Agirre (2023) Iñigo Alonso and Eneko Agirre. 2023.  Automatic logical forms improve fidelity in table-to-text generation.  _Expert Systems with Applications_ , page 121869.
  * An et al. (2022) Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. 2022.  Cont: Contrastive neural text generation.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 2197–2210. Curran Associates, Inc.
  * Chen et al. (2023) Leiyuan Chen, Chengsong Huang, Xiaoqing Zheng, Jinshu Lin, and Xuanjing Huang. 2023.  TableVLM: Multi-modal pre-training for table structure recognition.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2437–2449, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a.  Logical natural language generation from open-domain tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7929–7942, Online. Association for Computational Linguistics.
  * Chen et al. (2020b) Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020b.  KGPT: Knowledge-grounded pre-training for data-to-text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8635–8648, Online. Association for Computational Linguistics.
  * Chen et al. (2020c) Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020c.  Logic2Text: High-fidelity natural language generation from logical forms.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2096–2111, Online. Association for Computational Linguistics.
  * Covington (2001) Michael A Covington. 2001.  Building natural language generation systems.  _Language_ , 77(3):611–612.
  * Dash et al. (2023) Amanda Dash, Melissa Cote, and Alexandra Branzan Albu. 2023.  Weathergov+: A table recognition and summarization dataset to bridge the gap between document image analysis and natural language generation.  In _Proceedings of the ACM Symposium on Document Engineering 2023_ , DocEng ’23, New York, NY, USA. Association for Computing Machinery.
  * Davis et al. (2022) Brian Davis, Bryan Morse, Brian Price, Chris Tensmeyer, Curtis Wigington, and Vlad Morariu. 2022.  End-to-end document recognition and understanding with dessurt.  In _European Conference on Computer Vision_ , pages 280–296. Springer.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019.  Handling divergent reference texts when evaluating table-to-text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4884–4895, Florence, Italy. Association for Computational Linguistics.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _Journal of Artificial Intelligence Research_ , 61:65–170.
  * Huang et al. (2022) Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , MM ’22, page 4083–4091, New York, NY, USA. Association for Computing Machinery.
  * Kasner and Dusek (2022) Zdeněk Kasner and Ondrej Dusek. 2022.  Neural pipeline for zero-shot data-to-text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3914–3932, Dublin, Ireland. Association for Computational Linguistics.
  * Kayal et al. (2021) Pratik Kayal, Mrinal Anand, Harsh Desai, and Mayank Singh. 2021.  Icdar 2021 competition on scientific table image recognition to latex.  In _Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV 16_ , pages 754–766. Springer.
  * Kim et al. (2022) Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022.  Ocr-free document understanding transformer.  In _European Conference on Computer Vision (ECCV)_.
  * Korn et al. (2019) Flip Korn, Xuezhi Wang, You Wu, and Cong Yu. 2019.  Automatically generating interesting facts from wikipedia tables.  In _Proceedings of the 2019 International Conference on Management of Data_ , SIGMOD ’19, page 349–361, New York, NY, USA. Association for Computing Machinery.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213, Austin, Texas. Association for Computational Linguistics.
  * Lee et al. (2023) Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2019) Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. 2019.  Tablebank: A benchmark dataset for table detection and recognition.
  * Liang et al. (2009) Percy Liang, Michael Jordan, and Dan Klein. 2009.  Learning semantic correspondences with less supervision.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 91–99, Suntec, Singapore. Association for Computational Linguistics.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Masry et al. (2022) Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022.  ChartQA: A benchmark for question answering about charts with visual and logical reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2263–2279, Dublin, Ireland. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Parikh et al. (2020a) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020a.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Parikh et al. (2020b) Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020b.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of EMNLP_.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with entity modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2023–2035, Florence, Italy. Association for Computational Linguistics.
  * Puduppully et al. (2022) Ratish Puduppully, Yao Fu, and Mirella Lapata. 2022.  Data-to-text generation with variational sequential planning.  _Transactions of the Association for Computational Linguistics (to appear)_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997.  Building applied natural language generation systems.  _Natural Language Engineering_ , 3(1):57–87.
  * Rust et al. (2023) Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023.  Language modelling with pixels.  In _The Eleventh International Conference on Learning Representations_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Srihari et al. (2003) Sargur N. Srihari, Ajay Shekhawat, and Stephen W. Lam. 2003.  _Optical Character Recognition (OCR)_ , page 1326–1333. John Wiley and Sons Ltd., GBR.
  * Su et al. (2021) Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021.  Plan-then-generate: Controlled data-to-text generation via planning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 895–909, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Wang et al. (2022) Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. 2022.  Robust (controlled) table-to-text generation with structure-aware equivariance learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5037–5048, Seattle, United States. Association for Computational Linguistics.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wiseman et al. (2018) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2018.  Learning neural templates for text generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3174–3187, Brussels, Belgium. Association for Computational Linguistics.

##  Appendix A Performance degradation over compression

We measure the table sizes in total amount of pixels. The perfect solution
would be to compute a BLEU and PARENT per prediction and see how the metric
behaves when we increase the table size. But that would consume time and the
plot would probably be very noisy. Thus we split the dataset into 10 segments
and we compute the mean BLEU and PARENT for each segment. We compute 20
segments in a logarithmic scale as seen in figure 3. We then compute BLEU and
PARENT for each segment, we can see the performance for each segment in figure
4.

Figure 3: Amount of examples in ToTTo train set at each segment on a
logarithmic scale. Figure 4: PixT3 performance over size. ToTTo automatic
evaluation metrics for each segment. Vertical lines show degree of compression
range at key segments.

##  Appendix B Optimal parameters for down-scaling and truncating.

###  B.1 Truncation

In order to perform the least destructive cut to the rendered table image
possible, we analyze the highlighted cell location distribution across 120000
training samples of ToTTo.

Figure 5: Distribution of highlighted cells across examples in the train set.

###  B.2 Down-scaling table images.

Although we knew the effects of down-scaling on performance, we didn’t measure
the effects of truncating. Consequently, we didn’t also know what affected
performance more, the truncation or down-scaling. Therefore we launched a set
of experiments where we trained Pix2Struct DocQA models on dataset versions of
max_scale 0.87, 0.58, 0.39, 0.26 and 0.00. Being the latter an equivalent of
the original all-downscale / no-truncate version of the dataset. According to
the results shown in figure 6, we see that the most consistent, optimal
maximum down-scaling factor is 0.39.

Figure 6: evaluation results over training for each compression factor in the
rendered tables.

##  Appendix C SSL objectives for intermediate training

We conducted a set of experiemnts to indentify the best SSL objective for our
structural learning curriculum. Apart from the already selected objective we
also considered adapting masking to Table-to-Text. This objective consisted
in, given a random structured table, fill each cell with text correlated with
its position on the table, mask random cells and make the model predict them.
This objective resulted in fast training as the generated sequence contained a
minimal amount of tokens. Also allowing us to mask many cells per example,
reinforcing the signal for each example. Figure 7 shows an example of this
objective.

Figure 7: Example of a contextual self-supervised training objective.

We trained a Pix2Struct for 5 epochs with the model converging to 98.16 BLEU.
Later experiments showed that models intermediately trained on this objective
before finetuning performed worse than the ones trained with the one described
in section 2.5.2.

A possible reason of the under-performance of this objective was the fact that
the model does not need to pay attention to all the cells in the column and
row to guess the cell’s vale, as the closest neighbours would already lead to
the answer. Thus we decided to combine both SSL objectives, the one we
described in section 2.5.2 and this one. The results showed that Pix2Struc
models intermidiately trained with this combination still lag behind the ones
trained only with the one described in section 2.5.2.