As indicated above in Sect. 5.2.1, TIPS simulates the readout process for each
intermediate frame of the MACC mode in spectroscopy with 15 groups of 16
frames separated by 11 drops. The signal slope and the quality factor of each
pixel is then derived following Kubik et al. (2016), as it is implemented in
the onboard payload data processing. Non-linearity is applied afterwards on
the slope by using the inverse function of the non-linearity correction model,
but its impact on the quality factor is ignored. The slope, in ADU, is encoded
on 16 bits and the quality factor on 8 bit.

We show in Fig. 16 an example image produced by TIPS for the NISP instrument
when emulating readout for the spectroscopic channel. Like the photometric
channel, we have the same $4\times 4$ detector mosaic on the left. For closer
inspection, a small region of a detector is shown on the right containing the
characteristic features of slitless spectroscopy.

###  5.3 EXT ground-based surveys simulator

Lastly, the ground-based simulator SIM-EXT is a generic simulation software
package to produce accurate pixel data for various ground telescope and
instrument systems. Euclid will rely on external optical photometry
complementing its own infrared bands in order to obtain accurate photometric
redshifts. The necessary photometry in the Northern hemisphere will be
provided by the Ultraviolet Near Infrared Optical Northern
Survey101010https://www.skysurvey.cc (UNIONS), a joint effort between the
Canada-France Imaging Survey (CFIS; Ibata et al. 2017) for the $u$ and $r$
bands, the Javalambre-Euclid Deep Imaging Survey (JEDIS) in the $g$ band, the
Panoramic Survey Telescope and Rapid Response System (Pan-STARRS; Chambers et
al. 2016) in the $i$ band, and the Wide Imaging with Subaru Hyper Suprime
Camera (HSC; Miyazaki et al. 2017) of the Euclid Sky survey (WISHES) for the
$z$ band. For the Southern sky, it is anticipated that the Dark Energy Survey
(DES; The Dark Energy Survey Collaboration 2005) and the $Rubin$ Observatory
Legacy Survey of Space and Time111111Hereafter, we use Rubin or LSST
interchangeably when referring to $Rubin$-specific software, hardware and data
products. will provide all $ugriz$ photometry (LSST Science Collaboration et
al. 2009; Ivezić et al. 2019).

Figure 17: The six ground-based EXT surveys simulated by SIM-EXT. From Top
Left to Bottom Right, the detector layout for DECam ($2$°.$2$ / 570 Mpix),
LSSTCam ($3$°.$5$/ 3.2 Gpix), MegaCam ($1$°/ 380 Mpix), JPCam ($3$°, 1.2
Gpix), Pan-STARRS PS 1 ($3$°/ 1.2 Gpix), and HSC ($1$°.$5$/ 870 Mpix). Figure
18: A simulated DES exposure in the $r$ band. The left image shows the entire
62 detector mosaic covering a $2\hbox to0.0pt{.\hss}^{\circ}2$ diameter field-
of-view. The right image shows a detailed 10$\arcmin$ wide section of a single
detector.

The function of SIM-EXT differs from that of the other Euclid instrument
simulators. Rather than producing extremely detailed simulations for each EXT
survey (a notably costly effort), its purpose is to provide realistic imaging
data that can flow inside the ground segment pipeline with the same TU inputs
as the other Euclid channels. The output data products of SIM-EXT vary per EXT
survey in accordance with the SGS strategies designated for each one. For
example, data processing of raw DES data will be performed within the Euclid
SGS whereas the detrended and/or calibrated data products produced by the
other respective experiments and synergies (e.g. UNIONS and LSST) will be
received by the Euclid SGS when available. Therefore, for DES we produce
outputs consisting of raw images and master calibration frames. For LSST, only
raw images are produced with a choice of units: e- or ADU. Moreover, the FITS
format for LSST images produced by SIM-EXT version 2.14.2 is similar to the
e-image format defined by the Dark Energy Science Collaboration (DESC) image
simulator imSim version
0.1.0121212https://github.com/LSSTDESC/imSim/tree/0.1.0 described in Sánchez
et al. (2020). This output format is required for SIM-EXT LSST frames to be
processed using the $Rubin$ Observatory LSST Science Pipelines release
16.0131313https://pipelines.lsst.io/releases/v16_0.html (LSST Dark Energy
Science Collaboration 2012; Jurić et al. 2017; Bosch et al. 2018; Ivezić et
al. 2019). This allows the SGS to emulate ingesting official $Rubin$
Observatory calibrated data products for joint pixel processing with the
Euclid data products. For the Northern Surveys, a software package called
EXT$\\_$NS$\\_$SIM$\\_$AFTERBURNER developed by the Organisation Unit for
External Surveys (OU-EXT) is used in conjunction with SIM-EXT to provide
level-2 processed data products consisting of detrended frames, source
catalogues, and PSF models.

##### Sources

The EXT ground-survey simulator performs the generation of all TU sources such
as real and model stars, Flagship, primeval universe galaxies, high-$z$, and
strong lensed sources. Using the same SimThumbnails as the Euclid simulators,
it renders the exact same light profiles including weak lensing distortions.
As the PSF is assumed to be monochromatic, it does not require the input
spectra and uses precomputed band fluxes, thus accelerating the computation of
these simulations.

##### Background and transmission

As most of the EXT surveys have already started their observations, we have
realistic background magnitudes and zeropoints from their respective reduced
image analysis. We use the measurements taken from actual observations
contained with the MDB Survey Tables as inputs to the simulation, which
provide a very realistic distribution of sky brightness and atmospheric
extinction. For the EXT surveys, such as Rubin, that have not started their
observations, we can model realistic distributions from the available models
of the sky brightness (Yoachim et al. 2016) and passbands (Kahn et al. 2010;
Claver et al. 2014; Connolly et al. 2014).

##### PSF

The PSF is significantly simpler in most EXT simulations, and it is modelled
as either a simple Gaussian or a Moffat profile. From a realistic distribution
of seeing FWHM, we can replicate the average optical quality for each survey.
In the case of DES, we obtained a large sample of PSF models extracted from
real data analysis using PSFEx (Bertin 2011). We randomly select a model,
enabling variable and complex PSF simulations across the focal plane for the
ground-based photometry.

##### Optics and detectors

The design of SIM-EXT takes advantage of the common characteristics of the
ground-based instruments to perform simulations of various surveys. These
surveys are configurable through the MDB instrument file providing a
description of the optics, detector layout and electronics. Figure 17 shows
the focal plane layouts for the DES Dark Energy Camera (DECam), Rubin’s LSST
Camera (LSSTCam), CFIS Mega Camera (MegaCam), JEDIS (JPCam), Pan-STARRS PS1,
and WISHES HSC, respectively. In detail, the camera detector layout for DECam
was derived from Flaugher et al. (2015). The detector focal plane layout for
the LSST camera (LSSTCam) was defined with the information provided in Ivezić
et al. (2019). For simplicity, our camera model is an approximation to
LSSTCam, as all 189 detectors in our focal plane are models of either
LSSTCam’s e2V sensor or LSSTCam’s ITL sensor, not a combination of both as
will be used during operations. The detector layout for MegaCam, JPCam, Pan-
STARRS PS1, and HSC were each inferred, respectively, from a set of real
exposures provided courtesy of the CFIS, JEDIS, Pan-STARRS, and WISHES teams.
For a majority of the surveys, we deliver a perfectly calibrated WCS by
writing the true WCS in the header of the output images. For DES, we
reproduced a certain degree of ‘un-calibration’ by writing a slightly
different WCS in the output headers. The input WCS are obtained with the
astrometry tool SCAMP (Bertin 2006) applied on actual data. If real data are
not available, we apply a simpler estimation of the WCS (without distortions).
Additionally, we use master bias and master flats from the actual observations
to include realistic pixel-to-pixel variations and faithful readout patterns.
With the same model, we can create individual flat field and bias calibration
images that have their own noise realisation.

A sample image of the SIM-EXT simulator configured to produce a DES exposure
can be seen in Fig. 18. On the left, we show the 62 detector mosaic and on the
right a close-up image of one of the CCDs. In addition to the features
discussed above, we have the additional detector effect of tree rings acquired
from sampling from real DES flat data; these are circular patterns due to
variations of the silicon dopant concentration in the CCD chip.

With all four instrument simulators presented in this section, we can produce
complex and detailed simulations for a large variety of channels and
configurations for the same universe, environment, and common models. In the
following section, we present our procedure for launching a large-scale
production over segments of the Euclid Wide Survey using our suite of
simulation codes.

##  6 SC8 simulation processing and results

In this section, we present the application of our SIM pipeline framework for
SC8 and the corresponding results. We first begin with an in-depth description
of our simulation production workflow as developed for SC8.

###  6.1 Simulation planning and orchestration

The previously defined elements (i.e. TU catalogues, simulators, etc.) enable
the generation of mock observation sets of data to be processed and analyzed.
However, for the production of large data sets in a distributed and controlled
manner these simulation elements need to be connected and coordinated. Here we
describe the additional components of our software framework which fulfill
this objective: the simulation request, the SimPlanner, and the SIM pipeline.

The simulation request is one of the inputs of the SIM pipeline together with
a mission configuration and the catalogues. One can specify some aspects of
the simulations such as (i) channel: VIS, NISP-P, NISP-S, EXT-LSST, EXT-DES,
(ii) image type: Science, Deep, Flat, Bias. (iii) pointings or area to
simulate, and (iv) instrumental effects to be enabled/disabled.

The simulation request is handled by the SimPlanner, a program that
constitutes the entry point of the SIM pipeline and allows us to orchestrate
large sets of simulations. It parses the information in the simulation request
and the survey files and configures the corresponding simulator with the
appropriate parameters. The SimPlanner contains additional features such as
the generation of particular dithering patterns, in-pointing sequence,
enabling complex parameter exploration, configuration of flat field sequences,
and other high-level tasks that can be done at this point.

The SIM Pipeline framework allows multilevel splitting and parallelisation of
the processing, which is critical to producing large sets of simulations in a
reasonable time frame. The processing of the SC8 Main Production, as well as
the other productions in the scope of SC8 (e.g. Deep field) was carried out on
the Euclid SGS infrastructure.

Figure 19: Simulated raw single visit images for each Euclid imager: VIS,
NISP-P ($Y_{\scriptscriptstyle\rm E}$ band), NISP-S (NIS-RGS0) with the raw
outputs from three of the ground-based survey instruments in the $z$ band:
WISHES (HSC-$z$), DES (DES-$z$), and Rubin (LSST-$z$) over the same area of
sky in Euclid tile 79171 (Observation ID 18539) of the Pilot Phase.

###  6.2 Pilot Phase

SC8 began with a small test production of simulations in preparation for the
Main Run, called the Pilot Phase. This production had two primary goals: (1)
ensure the validity of the simulations and subsequent data products produced
during image processing by each OU in the SGS pipeline; (2) measure the amount
of resources, such as number of cores, peak RAM, and maximum wall time, needed
for each of the OU-SIM tasks, (3) perform the necessary optimisations to
guarantee feasibility of the larger scale SC8 Main Run with the available
resources.

The Pilot Phase contained four Science Wide observations covering the so-
called Euclid tile (Kümmel et al. 2022) (79171) with an area measuring
$\sim$$0.5\,{\rm deg}^{2}$. Euclid tiles are non-overlapping ecliptic
latitude-longitude rectangles (approximately the size of the Euclid field of
view) and oriented along the meridians. With no gaps between adjacent tiles,
they are essentially the unit of processing for high level OUs to ensure
complete coverage of the survey area when constructing the object catalogues
for WL and GC analysis. Even for an area this small, the multiple feedback
iterations with the other OUs resulted in a total production of 1866 raw
images for all simulation channels. In addition, all the calibration frames
required to carry out image processing (such as detrending, astrometric, and
photometric calibrations) for the Euclid instruments were also produced during
this phase and later reused for the data reduction during the Main Run. The
set of VIS calibration products consisted of a sequence of 60 bias images and
60 short-exposure flat-fields using the VIS calibration unit (VIS-CU). Since
the shutter will remain open during flat acquisition, there will be objects
entering into the field of view of the telescope, and we have simulated the
most luminous objects yielding a flux above the photon noise.

For the NISP instrument, we produced a sequence of 600 dark frames and 750
flat-field images with the respective LED intensities and exposure times for
the photometric and spectroscopic channels. We also simulated observations of
spectrophotometric standard stars and planetary nebulae to enable the
performance of NISP photometric and spectroscopic wavelength calibrations.
Finally, we also delivered a self-calibration sequence over a dense stellar
field consisting of multiple dithered overlapping images that were to be used
to improve the flat-fielding and the astrometric calibration.

In Table 5 in Appendix C, we list the number of files and the total size for
each image data product produced during the Pilot Phase. We show in Fig. 19
raw science images produced by the simulators for the VIS, NISP-P
($Y_{\scriptscriptstyle\rm E}$ band), NIS (RGS0), and three EXT ground surveys
in the $z$ band: WISHES (HSC), DES, and LSST over the same area of sky in
Euclid tile 79171 for one of the four Euclid observations (Observation ID
18539). The simulated Euclid images show considerably more prominent features
(e.g. ghosts, diffraction spikes, and CTI) in comparison to the simplified
images produced by the simulator for the EXT ground-based surveys which are
known or expected to experience similar effects.

###  6.3 Main run

The original plan of the SC8 Main run was to conduct a full end-to-end test of
the SGS pipeline using a large and complex simulation of 1286 observations
from the Euclid Wide Survey, covering a contiguous area of ${\sim}\,600\,{\rm
deg}^{2}$ with different sky conditions. All known instrumental effects were
to be included in the pixel data to enable performance and quality assessments
of the level-2 calibration pipelines. Furthermore, the large area would enable
the first integration test of the final processing elements of the Euclid SGS
pipeline (i.e. the level-3 cosmological measurements) with enough area to
measure the baryon acoustic peak in the galaxy two-point correlation function.

However, as explained in the previous section, the results of the Pilot Phase
indicated that this would present a major challenge as producing and
processing large catalogues and realistic images carries a high computational
cost. The SC8 Main run could not have been achieved in a reasonable time frame
with the resources available for the challenge. Therefore we have reverted to
simulate a subset of 331 observations grouped in 11 non-contiguous patches
preserving the variety of conditions: mostly stellar density and zodiacal
light as shown in Fig. 26. The final simulated area measuring
${\sim}\,165\,{\rm deg}^{2}$ was still sufficient to perform the integration
test of the processing elements of both the WL and GC experiments, the topmost
priority.

Figure 20: Distribution of the processing for the SC8 Main Production among
the SDCs. The labels in the image indicate the patch identifier and the
corresponding SDC responsible of processing the entire patch containing all
Euclid pointings and the overlapping EXT surveys. The black dots denote each
of the 331 observations of the Euclid Wide Survey and the squares represent
the 1036 Euclid processing tiles.

Figure 20 shows the 11 patches and the position of each Euclid observation
contained within each of them. The colour of each patch represents the SDC
tasked with processing the set of SC8 observations. The assignment of a region
to a SDC was according to their expected contribution and the resources
available. Moreover, the strategy of having non-contiguous patches processed
at the different SDCs was implemented to avoid data transfers between
processing centres. Each patch is composed of a set of Euclid tiles. There are
1039 Euclid tiles overlapping with the SC8 observations.

Over the selected Euclid targets in the sky, we simulated all three Euclid
channels: VIS, NISP-P, and NISP-S, respectively. The Main run also included
simulations of the EXT ground-based surveys in the Northern and Southern
hemispheres. The target area was partitioned into three non-overlapping
patches. All Northern surveys: CFIS, JEDIS, Pan-STARRS PS1, and WISHES covered
the same region above $30^{$\mathrm{\SIUnitSymbolDegree}$}$ in declination. We
partitioned the lower rectangular region along the diagonal into two sections:
Rubin observations spanned the area above the diagonal and DES covered the
opposing field below it. We refer the reader to Appendix C.4, which describes
with more detail the selections of the pointings for each EXT survey and
provides figures illustrating the EXT survey footprints in each band.

Table 1: Summary of the data products simulated in the scope of the SC8 Main Production. The data sets for VIS, NISP-P, NISP-S, and DES are stored as exposures, whereas for LSST, CFIS, JEDIS, Pan-STARRS, and WISHES each individual data file corresponds to a detector frame. The processing time corresponds to the sum of all single-thread sub-jobs wall time, as indicated in table 6 SC8 Main Channel |  Number of products |  Storage (TB) |  Processing time (days)
---|---|---|---
VIS |  1,324 |  8.7 |  5,200
NISP-P |  3,948 |  1.5 |  1,500
NISP-S |  1,324 |  5.1 |  4,700
DES |  2,370 |  2.4 |  600
LSST |  154,980 |  9.2 |  3,700
CFIS |  55,440 |  0.7 |  150
JEDIS |  12,096 |  1.7 |  210
Pan-STARRS |  63,300 |  1.9 |  570
WISHES |  47,586 |  0.5 |  100
Total |  342,368 |  31.7 |  16,730
Figure 21: Footprint of the SC8 Deep field, including 30 observations (denoted by the red circles), i.e. 120 images, near the Ecliptic North Pole. Table 2: Summary of the data products produced for SC8 Deep field production SC8 Deep Channel | Number of products | Storage (GB)
---|---|---
VIS | 120 | 809
NISP-P | 360 | 140
NISP-S | 120 | 469
Total | 840 | 1418

In Table 1, we report the number of data products and the amount of storage
required per survey channel participating in the SC8 Main run. We produced a
total of $342\,368$ data products amounting to 31.7 TB for the
$165\,\rm{deg}^{2}$ Euclid Wide Survey.

Figure 22: VIS and NIR imaging of a region in the SC8 Deep field (Euclid ile
306182, Observation ID 1914) resulting from the different stages of the SGS
processing flow. The two panels on the left show the simulated raw science
frames for the VIS $I_{\scriptscriptstyle\rm E}$ band (top-left), and NISP-P
$Y_{\scriptscriptstyle\rm E}$ band (bottom-left). The right panels show the
background-subtracted mosaic images (see text for details) of the same field
for the VIS $I_{\scriptscriptstyle\rm E}$ band (top-right) and NISP-P
$Y_{\scriptscriptstyle\rm E}$ band (bottom-right), produced, respectively,
during level-2 data processing by the OU-MER pipeline. Figure 23: A simulated
strong lens in the SC8 Deep field (Euclid tile 306182). Strong Lens 200026 was
identified in the VIS $I_{\scriptscriptstyle\rm E}$ band, NISP-P
$Y_{\scriptscriptstyle\rm E}$, $J_{\scriptscriptstyle\rm E}$, and
$H_{\scriptscriptstyle\rm E}$ bands by the strong lens detection algorithm
developed by the Strong Lensing Science Working Group using the MER
background-subtracted mosaic data. Figure 24: Imaging of a simulated strong
lens in the VIS $I_{\scriptscriptstyle\rm E}$ band, $Rubin$ ($g$+$r$+$i$)
bands and combined VIS and NISP-P ($I_{\scriptscriptstyle\rm E}$ \+
$Y_{\scriptscriptstyle\rm E}$ \+ $H_{\scriptscriptstyle\rm E}$) bands of the
SC8 Main run. Strong Lens 200094 was identified in Euclid tile 74088 by the
detection algorithms of the Strong Lensing Science Working Group. Figure 25:
Stacked and processed spectra from simulated SC8 NISP-S images. The colours in
the plot are identified as the following: blue line corresponds to the
measured spectrum, the brown line corresponds to continuum, orange line
corresponds to model and green line corresponds to estimated noise. This
galaxy was successfully identified at $z=1.29$.

Notably, even with shallower data than the actual $Rubin$ experiment, the
unprecedented enormity of the three billion pixel $Rubin$ camera with its
$3\hbox to0.0pt{.\hss}^{\circ}5$ ($9.6\,\rm{deg}^{2}$) field of view, combined
with imaging capability in five optical bands,141414We do not simulate the
sixth $Rubin$ broadband filter: $Y$, ranging from $948\,{\rm nm}$ to
$1060\,{\rm nm}$. yielded the largest data set. These results provide insight
on the volume of actual EXT ground-based data that has to be processed for the
Euclid mission. EXT data may require as much storage and computing time (or
more) as the Euclid data themselves, especially if performing pixel-level
processing of purely raw data products. Finally, the total wall time to
produce the entire SC8 Main data set was $16\,730$ days. The VIS simulations,
which remained full-featured, dominated the processing time with $5\,200$ days
followed by NISP-S at $4\,700$ days.

###  6.4 Deep run

For the SC8 Euclid Deep survey, we simulated a small area measuring $\sim
1\,\rm{deg}^{2}$ of the Euclid Deep Field around the North Ecliptic Pole.
Figure 21 shows the SC8 Deep field footprint and the number of layers to reach
the desired depth. We produced sets of full-featured simulations for VIS,
NISP-P, and NISP-S with 10 times more exposures totalling 840 amounting to
$\sim$1.4 TB (as reported in Table 2) and reaching a flux limit 2 mag fainter
than the nominal wide field. For NISP-S, we also ran simulations of the blue
grism (BGS00) in addition to the three other spectroscopic channels. We did
not produce simulations of any Northern EXT ground-based surveys for which no
current strategies have been defined at these fainter magnitude depths.
Nevertheless, the SC8 Deep field run was designed to be used for testing by
the level-2 SGS processing chains, which perform image and spectra
calibration, photometry, and coaddition for VIS and NISP, as well as PSF
modeling and the generation of optical distortion maps.

###  6.5 Outcomes of SC8

Here, we provide a brief summary of the SGS end-to-end performance in SC8 as a
more in-depth discussion goes beyond the scope of this work. Our total
production of simulations for the SC8 Main and Deep surveys yielded 33 TB of
data. The entire SC8 data set was transferred to the EAS and queried by the
level-2 OUs to initiate the data reduction pipelines, signaling the start of
the end-to-end integration test.

Data processing by the full SGS pipeline proceeded gradually with data quality
inspection by the respective OUs and updates to various software packages.
Nearly all level-3 processing elements151515The codes to compute the
photometric visibility mask and the covariance matrices for the galaxy two-
point correlation and power spectrum (high priority processing functions) were
not fully integrated. were successfully integrated. The full integration test
also served as an aid to identify and resolve performance issues concerning
infrastructure, bugs, and bottlenecks in software. It also provided an impetus
to further improve communications between interfacing OUs in order to maximise
SGS capabilities as we approach operational readiness.

We conclude this section by highlighting the work enabled by the SC8
simulations such as the testing of detection algorithms for strong lenses,
galaxies high-$z$ quasars and galaxy emission-lines with Euclid. As detailed
in Appendix C, along with the standard galaxy and standard stars, the object
catalogues also contained a variety of additional sources such as QSOs,
high-$z$ galaxies, strong lenses, and LT stars (brown dwarfs). From the sample
of 801 strong lenses, 285 sources were inserted into 216 Euclid tiles covering
the SC8 Main field. The catalogues containing these sources were provided as
inputs to the various simulator codes for the VIS, NISP-P, and EXT instruments
(i.e. ELViS, Imagem, TiPS, and SIM-EXT), which introduced instrumental and
environmental effects in each band to produce raw images that were then
processed by the SGS pipelines. In detail, VIS and the NISP-P calibrated
frames were produced by the respective OU level-2 pipelines for VIS and NIR
image reduction.

The VIS and NIR data products along with their corresponding flag maps (e.g.
cosmics) then proceeded through the SGS data processing chain as inputs to the
OU-MER level-2 pipeline, where they were stacked. The MER pipeline then
performed the background removal and used the flags to produce fully detrended
and background subtracted MER mosaics.

Figure 22 presents an example of raw (level-1) and processed (level-2) data
products on a zoom-in of Euclid tile 306182 in the SC8 Deep field. On the
left, we show the raw simulated science exposures for VIS (top-left) and
NISP-P (bottom-left; for $Y_{\scriptscriptstyle\rm E}$ band only) with the
characteristic features of these two instruments (described in Sects. 5.1 and
5.2.2, respectively). The two rightmost panels show the corresponding
calibrated, stacked, and background subtracted mosaic images produced by OU-
MER after processing the data products generated by OU-VIS and OU-NIR,
respectively.

These MER mosaic images and the source catalogues generated from them were
used, respectively, by the Strong Lensing and Primeval Universe Science
Working Groups to perform tests of their corresponding detection algorithms in
development.

Figure 23 shows a zoom-in cut-out of the strong lens 200026 in the VIS and NIR
deep simulations. The Einstein ring can be seen with fine detail by the VIS
instrument, whereas it is indiscernible in the Euclid NIR bands. Another of
the Strong Lensing Science Working Group detections was a strong lens system
(200094) in the SC8 Main run (Euclid tile 74088) in both MER processed Euclid
and $Rubin$ mosaic data. Cutout images of the system are shown in Fig. 24;
from left-to-right, we show a set of images of the Einstein ring as seen in
the VIS $I_{\scriptscriptstyle\rm E}$ band, combined Rubin $g$+$r$+$i$ bands,
and combined VIS and NISP-P ($I_{\scriptscriptstyle\rm
E}$+$Y_{\scriptscriptstyle\rm E}$+$H_{\scriptscriptstyle\rm E}$) bands. These
images demonstrate the resolving power of both Euclid and LSST, two
forthcoming contemporary and pivotal science programs. They also, however,
exemplify the exceptional image quality anticipated with the VIS camera alone
with its superb capability to deblend compact systems.

In Fig. 25, we show an example of a successfully calibrated, decontaminated,
stacked, and modelled galaxy emission-line spectra produced from the raw pixel
data of the NISP-S grism simulations in SC8 processed by the Organisation Unit
for Near Infrared Slitless Spectroscopy (OU-SIR) and Organisation for Spectral
Measurement (OU-SPE), respectively. The results in the figure show a clearly
identifying the redshifted H$\alpha$ emission line by the OU-SPE spectroscopic
emission line detection algorithm. The identified emission-lines were then
used by OU-SPE to estimate spectroscopic redshifts.

##  7 Future perspective

Following SC8, new features and improvements have been added to our simulation
framework. For example, to produce more realistic profiles, we implement deep
generative models, a machine learning framework presented in Euclid
Collaboration: Bretonnière et al. (2022) that can generate complex galaxy
stamps learned from a training set of real $Hubble$ Space Telescope
observations. These galaxy thumbnails include spiral arms and other features
that cannot be represented with analytical profiles as the ones we currently
use. The advantage of this approach is that it can draw complex stamps just
from parameterised information such as the one available in the TU galaxy
catalogue without complex morphological models in the Flagship galaxy mock.
These realistic stamps will enable the development of other complex algorithms
to measure shear and to deblend larger objects. Color gradient and its impact
on shear measurments have already been studied (Welikala & Kneib 2012) and an
implementation in the SIM pipeline is currently in progress.

For the NISP simulators, we introduced an additional background component
caused by persistence: trapped charges from one exposure are released on later
image acquisitions. In early ground-based calibrations, a persistence signal
was detected more than two hours after the initial exposure for bright
sources. Among other dependencies, the charge reached during an exposure,
exposure time and history of successive illuminations all affect the release
of charges trapped in a pixel. A preliminary model only takes into account the
single preceding exposure. For the persistence signal $I_{\rm persistence}$
that is added to the current signal, we use the following model:

$I_{\rm persistence}=\Big{(}p_{0}+10^{-5}p_{1}I_{-1})(1-{\rm
e}^{-I_{-1}/p_{2}}\Big{)}\,,$ (12)

where $I_{-1}$ is the previous image and $p_{i}$ are a set of measured
persistence parameters. Distinct $p_{i}$ values are used for each pixel,
previous exposure time and the elapsed time between exposures in the survey.
Additionally, we have released within the Euclid Consortium newer versions of
SIM-EXT (2.17+), which feature spatially-varying PSFs for CFIS and WISHES.
These are produced using PSFEx models for CFIS $u$ and $r$ bands and WISHES
$z$-band data provided to us courtesy of the CFIS and WISHES teams. Our Rubin
simulations now include nominal rotation of the camera per pointing acquired
from the OpSim
v2.0161616https://epyc.astro.washington.edu/~lynnej/opsim_downloads/fbs_2.0/
survey simulations (Jones et al. 2020). Furthermore, these simulations
currently implement the FITS format (described in Lim 2019) as well as the
camera layout and WCS description of DESC imSim version
0.6.2171717https://github.com/LSSTDESC/imSim/tree/v0.6.2 (Abolfathi et al.
2021). The file formatting and WCS changes fulfill the requisites of the
latest generation LSST Science Pipelines software to perform processing of
LSST simulations. Finally, SIM-EXT 2.17+ now provides simulated data products
in the $g$ band for the Waterloo-Hawaii-IfA $G$-band Survey (WHIGS) using the
Subaru telescope.

The complete set of calibration exposures for the Euclid instrument will
encompass a large variety of image types. In addition to the standard
calibration exposures: bias, dark, flats, we also produce calibration unit
exposures, out-of-focus exposures for phase diversity, different readout modes
for NISP, injection line frames, trap pumping for VIS CTI calibration, NISP
persistence model validation, bleeding and blooming calibration exposures. For
the PLM/instrument calibration exposures, we produce straylight checks and
dense stellar fields. Finally, for precise astrometric calibration, we
generate white dwarf and planetary nebulae observations.

Upon completion of PVPR, the Euclid Ground Segment will commence its third
Science Performance Verification (SPV3) phase. The focus of this campaign will
be on assessing the scientific performance of the Euclid mission. In detail,
we plan to produce nominal science exposures for the Wide and Deep areas
spanning a few tens of degrees to yield mock weak lensing and galaxy
clustering measurements (e.g. Cℓ and power spectrum multipoles) to enable
testing of the Euclid Consortium Science Working Groups analysis pipelines.

##  8 Summary and outlook

OU-SIM is responsible for providing detailed simulations to facilitate the
development of the SGS data processing pipeline as well as end-to-end
performance tests. In this work, we introduced our simulation framework for
providing a full pixel simulation pipeline for the entire Euclid mission and
presented its application to SC8, the eighth testing campaign of the Euclid
Ground Segment and its first end-to-end test. The SC8 data set was generated
from galaxy mocks built from the Flagship v1.0 simulation, the largest
cosmological $N$-body simulation at the time it was developed. Additional
sources were added to complement this massive simulation, such as high-
redshift QSOs and strong-lensed galaxies. Our stellar catalogue was
constructed from a combination of a very large and deep Besançon model and
real all-sky stellar catalogues. From these catalogues, we produced and
delivered simulated images for a simplified Euclid Wide Survey of $165\,{\rm
deg}^{2}$ and a full-featured $1\,{\rm deg}^{2}$ Euclid Deep Survey.

The framework we adopted for SC8 follows a highly modular design from the
onset, consisting of individual components for the TU catalogues inputs, the
pixel-level simulators for space and ground-based instruments, a set of common
tools for generating spectra and galaxy shapes, and a software API for
orchestrating large sets of simulations at any of the international Euclid SGS
data centers. Furthermore, all instrument models and reference surveys used in
SC8 as well as all past/future challenges and production cycles are stored in
the MDB repository. We also devised a validation program, consisting of suites
of tests for our simulation pipeline to ensure accuracy of the released data
products as well as adherence to scientific and technical requirements.

We will continue upgrading our simulation framework to allow large-scale
testing and improvements to the ground segment pipeline. We plan to provide
data products to aid the validation of very complex algorithms (e.g. VIS PSF
estimation) and which can be used to conduct mission performance evaluations
throughout this active development phase. Our longer term goals during Euclid
operations are to produce simulations to calibrate all possible systematics in
the data (e.g. shear measurement biases and selection biases) and for
interpretation of discrepancies or surprises found in the inflight data.

###### Acknowledgements.

We thank the OU-VIS, -NIR, -SIR, -EXT, and -MER teams for their assistance
with validation and providing valuable feedback on the simulations. We are
grateful to the System Team for the technical help and continuous support. We
thank the Cosmological Simulations Working Group and the Flagship team for the
outstanding galaxy mocks and support on their integration into OU-SIM. We
thank both VIS and NISP instrument teams for providing instrument models and
detailed explanations. We thank OU-SHE for their support with generating the
VIS PSF data set. We would like to thank Benjamin Clément from the Strong
Lensing Science Working Group, who produced the images of the strong lenses
detected in the SC8 Main and Deep fields which were presented in Figures 23
and 24. We thank Vincent Le Brun from the Organisation Unit for Spectral
Measurement (OU-SPE) for producing Figure 25. We also express our gratitude to
Dominique Boutigny, Jim Chiang, and Phillipe Gris of the Dark Energy Science
Collaboration as well as Jim Bosch, Tim Jenness, Lynne Jones, Kian-Tat Lim,
and the other members from Rubin Observatory Data Management and Survey
Strategy teams for providing support, useful insight and sample data sets to
aid us with upgrading the SGS Rubin simulations and ensuring compliance with
the LSST Science Pipelines. We would also like to thank Jesus Varela and
JEDIS, WISHES, Pan-STARRS and WHIGS teams for assisting us with information,
data and the models needed to produce the respective image simulations for the
Northern Surveys. We acknowledge all Science Data Centers (SDC-CH: University
of Geneva, SDC-DE: Max Planck Computing and Data Facility, SDC-ES: Port
d’Informació Científica, SDC-FI: CSC – IT Center for Science, SDC-FR: Centre
de calcul de l’IN2P3, SDC-IT: INAF Astronomical Observatory of Trieste/ALTEC
S.p.A. in Turin, SDC-NL: University of Groningen, SDC-UK: Institute for
Astronomy, University of Edinburgh, and SDC-US: Caltech/IPAC) that contributed
to the production of simulations for SC8. A portion of this work also used the
resources available at CosmoHub. CosmoHub has been developed by the Port
d’Informació Científica, maintained through a collaboration of the Institut de
Física d’Altes Energies and the Centro de Investigaciones Energéticas,
Medioambientales y Tecnológicas and the Institute of Space Sciences, and was
partially funded by the “Plan Estatal de Investigación Científica y Técnica y
de Innovación” program of the Spanish government. The author acknowledges
support from the Spanish Research Project PID2021-123012NA-C44 [MICINNFEDER].
This work has been financed by MCINN/AEI and the EU NextGeneration/PRTR
project EQC2021-007479-P. This study was supported by MICIIN with funding from
European Union NextGenerationEU(PRTR-C17.I1) and by Generalitat de Catalunya.
The authors acknowledge the Euclid Collaboration, the European Space Agency,
and a number of agencies and institutes that have supported the development of
Euclid, in particular the Academy of Finland, the Agenzia Spaziale Italiana,
the Belgian Science Policy, the Canadian Euclid Consortium, the French Centre
National d’Etudes Spatiales, the Deutsches Zentrum für Luft- und Raumfahrt,
the Danish Space Research Institute, the Fundação para a Ciência e a
Tecnologia, the Ministerio de Ciencia e Innovación, the National Aeronautics
and Space Administration, the National Astronomical Observatory of Japan, the
Netherlandse Onderzoekschool Voor Astronomie, the Norwegian Space Agency, the
Romanian Space Agency, the State Secretariat for Education, Research and
Innovation (SERI) at the Swiss Space Office (SSO), and the United Kingdom
Space Agency. A complete and detailed list is available on the Euclid web site
(http://www.euclid-ec.org).

## References

  * Abbott et al. (2018) Abbott, T. M. C., Abdalla, F. B., Allam, S., et al. 2018, ApJS, 239, 18
  * Abolfathi et al. (2021) Abolfathi, B., Alonso, D., Armstrong, R., et al. 2021, Ap&SS, 253, 31
  * Barbier et al. (2018) Barbier, R., Buton, C., Clemens, J.-C., et al. 2018, in High Energy, Optical, and Infrared Detectors for Astronomy VIII, ed. A. D. Holland & J. Beletic, Vol. 10709, International Society for Optics and Photonics (SPIE), 107090S
  * Behroozi et al. (2013) Behroozi, P. S., Wechsler, R. H., & Wu, H.-Y. 2013, ApJ, 762, 109
  * Bertin (2006) Bertin, E. 2006, in Astronomical Society of the Pacific Conference Series, Vol. 351, Astronomical Data Analysis Software and Systems XV, ed. C. Gabriel, C. Arviset, D. Ponz, & S. Enrique, 112
  * Bertin (2011) Bertin, E. 2011, in Astronomical Society of the Pacific Conference Series, Vol. 442, Astronomical Data Analysis Software and Systems XX, ed. I. N. Evans, A. Accomazzi, D. J. Mink, & A. H. Rots, 435
  * BGM web-service, OSU THETA (https://model.obs-besancon.fr/) BGM web-service, OSU THETA. https://model.obs-besancon.fr/, Besançon model of stellar population synthesis of the Galaxy
  * Bizyaev (2007) Bizyaev, D. 2007, in American Astronomical Society Meeting Abstracts, Vol. 211, American Astronomical Society Meeting Abstracts, 13.23
  * Bosch et al. (2018) Bosch, J., Armstrong, R., Bickerton, S., et al. 2018, PASJ, 70, S5
  * Bruzual & Charlot (2003) Bruzual, G. & Charlot, S. 2003, MNRAS, 344, 1000
  * Calzetti et al. (2000) Calzetti, D., Armus, L., Bohlin, R. C., et al. 2000, ApJ, 533, 682
  * Carretero et al. (2015) Carretero, J., Castander, F. J., Gaztañaga, E., Crocce, M., & Fosalba, P. 2015, MNRAS, 447, 646
  * Carretero et al. (2017) Carretero, J., Tallada, P., Casals, J., et al. 2017, in Proceedings of the European Physical Society Conference on High Energy Physics. 5-12 July, 488
  * Chambers et al. (2016) Chambers, K. C., Magnier, E. A., Metcalfe, N., et al. 2016, arXiv e-prints, 1612.05560
  * Claver et al. (2014) Claver, C. F., Selvy, B. M., Angeli, G., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9150, Modeling, Systems Engineering, and Project Management for Astronomy VI, ed. G. Z. Angeli & P. Dierickx, 91500M
  * Collett (2015) Collett, T. E. 2015, ApJ, 811, 20
  * Connolly et al. (2014) Connolly, A. J., Angeli, G. Z., Chandrasekharan, S., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9150, Modeling, Systems Engineering, and Project Management for Astronomy VI, ed. G. Z. Angeli & P. Dierickx, 915014
  * Cropper et al. (2014) Cropper, M., Pottinger, S., Niemi, S. M., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9143, Space Telescopes and Instrumentation 2014: Optical, Infrared, and Millimeter Wave, ed. J. Oschmann, Jacobus M., M. Clampin, G. G. Fazio, & H. A. MacEwen, 91430J
  * Delgado & Reuter (2016) Delgado, F. & Reuter, M. A. 2016, in Observatory Operations: Strategies, Processes, and Systems VI, ed. A. B. Peck, R. L. Seaman, & C. R. Benn, Vol. 9910, International Society for Optics and Photonics (SPIE), 991013
  * Dimauro et al. (2018) Dimauro, P., Huertas-Company, M., Daddi, E., et al. 2018, MNRAS, 478, 5410
  * Drimmel et al. (2003) Drimmel, R., Cabrera-Lavers, A., & López-Corredoira, M. 2003, A&A, 409, 205
  * Euclid Collaboration: Barnett et al. (2019) Euclid Collaboration: Barnett, R., Warren, S. J., Mortlock, D. J., et al. 2019, A&A, 631, A85
  * Euclid Collaboration: Blanchard et al. (2020) Euclid Collaboration: Blanchard, A., Camera, S., Carbone, C., et al. 2020, A&A, 642, A191
  * Euclid Collaboration: Bretonnière et al. (2022) Euclid Collaboration: Bretonnière, H., Huertas-Company, M., Boucaud, A., et al. 2022, A&A, 657, A90
  * Euclid Collaboration: Martinet et al. (2019) Euclid Collaboration: Martinet, N., Schrabback, T., Hoekstra, H., et al. 2019, A&A, 627, A59
  * Euclid Collaboration: Scaramella et al. (2022) Euclid Collaboration: Scaramella, R., Amiaux, J., Mellier, Y., et al. 2022, A&A, 662, A112
  * Euclid Collaboration: Schirmer et al. (2022) Euclid Collaboration: Schirmer, M., Jahnke, K., Seidel, G., et al. 2022, A&A, 662, A92
  * Faber & Jackson (1976) Faber, S. M. & Jackson, R. E. 1976, ApJ, 204, 668
  * Flaugher et al. (2015) Flaugher, B., Diehl, H. T., Honscheid, K., et al. 2015, AJ, 150, 150
  * Fosalba et al. (2015) Fosalba, P., Gaztañaga, E., Castander, F. J., & Crocce, M. 2015, MNRAS, 447, 1319
  * Frailis et al. (2019) Frailis, M., Belikov, A., Benson, K., et al. 2019, in Astronomical Society of the Pacific Conference Series, Vol. 521, Astronomical Data Analysis Software and Systems XXVI, ed. M. Molinaro, K. Shortridge, & F. Pasian, 612
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2018, A&A, 616, A1
  * Gaia Collaboration et al. (2023) Gaia Collaboration, Montegriffo, P., Bellazzini, M., et al. 2023, A&A, 674, A33
  * Gaspar Venancio et al. (2014) Gaspar Venancio, L. M., Laureijs, R., Lorenzo, J., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9143, Space Telescopes and Instrumentation 2014: Optical, Infrared, and Millimeter Wave, ed. J. Oschmann, Jacobus M., M. Clampin, G. G. Fazio, & H. A. MacEwen, 91430I
  * Grogin et al. (2011) Grogin, N. A., Kocevski, D. D., Faber, S. M., et al. 2011, ApJS, 197, 35
  * Hoekstra et al. (2017) Hoekstra, H., Viola, M., & Herbonnet, R. 2017, MNRAS, 468, 3295
  * Høg et al. (2000) Høg, E., Fabricius, C., Makarov, V. V., et al. 2000, A&A, 355, L27
  * Ibata et al. (2017) Ibata, R. A., McConnachie, A., Cuillandre, J.-C., et al. 2017, ApJ, 848, 128
  * Ilbert et al. (2009) Ilbert, O., Capak, P., Salvato, M., et al. 2009, ApJ, 690, 1236
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
  * Jones et al. (2020) Jones, R. L., Yoachim, P., Ivezic, Z., Neilsen, E. H., & Ribeiro, T. 2020, Survey Strategy and Cadence Choices for the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST)
  * Jurić et al. (2017) Jurić, M., Kantor, J., Lim, K. T., et al. 2017, in Astronomical Society of the Pacific Conference Series, Vol. 512, Astronomical Data Analysis Software and Systems XXV, ed. N. P. F. Lorente, K. Shortridge, & R. Wayth, 279
  * Kahn et al. (2010) Kahn, S. M., Kurita, N., Gilmore, K., et al. 2010, in Ground-based and Airborne Instrumentation for Astronomy III, ed. I. S. McLean, S. K. Ramsay, & H. Takami, Vol. 7735, International Society for Optics and Photonics (SPIE), 77350J
  * Kennicutt (1998) Kennicutt, Robert C., J. 1998, ARA&A, 36, 189
  * Koekemoer et al. (2011) Koekemoer, A. M., Faber, S. M., Ferguson, H. C., et al. 2011, ApJS, 197, 36
  * Kubik et al. (2016) Kubik, B., Barbier, R., Chabanat, E., et al. 2016, PASP, 128, 104504
  * Kümmel et al. (2022) Kümmel, M., Vassallo, T., Dabin, C., & Gracia Carpio, J. 2022, in Astronomical Society of the Pacific Conference Series, Vol. 532, Astronomical Society of the Pacific Conference Series, ed. J. E. Ruiz, F. Pierfedereci, & P. Teuben, 329
  * Kümmel et al. (2009) Kümmel, M., Walsh, J. R., Pirzkal, N., Kuntschner, H., & Pasquali, A. 2009, PASP, 121, 59
  * Laigle et al. (2016) Laigle, C., McCracken, H. J., Ilbert, O., et al. 2016, ApJS, 224, 24
  * Lastennet et al. (2002) Lastennet, E., Lejeune, T., Oblak, E., Westera, P., & Buser, R. 2002, Ap&SS, 280, 83
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, arXiv e-prints, 1110.3193
  * Lim (2019) Lim, K.-T. 2019, Large Synoptic Survey Telescope (LSST) Header Service Interface between the OCS and EFD
  * LSST Dark Energy Science Collaboration (2012) LSST Dark Energy Science Collaboration. 2012, arXiv, 1211.0310
  * LSST Science Collaboration et al. (2009) LSST Science Collaboration, Abell, P. A., Allison, J., et al. 2009, arXiv, 0912.0201
  * Maciaszek et al. (2022) Maciaszek, T., Ealet, A., Gillard, W., et al. 2022, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 12180, Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave, ed. L. E. Coyle, S. Matsuura, & M. D. Perrin, 121801K
  * Maciaszek et al. (2016) Maciaszek, T., Ealet, A., Jahnke, K., et al. 2016, in Space Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave, ed. H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, & E. C. Tong, Vol. 9904, International Society for Optics and Photonics (SPIE), 99040T
  * Markovič et al. (2017) Markovič, K., Percival, W. J., Scodeggio, M., et al. 2017, MNRAS, 467, 3677
  * Massey et al. (2014) Massey, R., Schrabback, T., Cordes, O., et al. 2014, MNRAS, 439, 887
  * Metcalf et al. (2019) Metcalf, R. B., Meneghetti, M., Avestruz, C., et al. 2019, A&A, 625, A119
  * Metcalf & Petkova (2014) Metcalf, R. B. & Petkova, M. 2014, MNRAS, 445, 1942
  * Miyazaki et al. (2017) Miyazaki, S., Komiyama, Y., Kawanomoto, S., et al. 2017, PASJ, 70, s1
  * Naghib et al. (2019) Naghib, E., Yoachim, P., Vanderbei, R. J., Connolly, A. J., & Jones, R. L. 2019, AJ, 157, 151
  * O’Donnell (1994) O’Donnell, J. E. 1994, ApJ, 422, 158
  * Pickles & Depagne (2010) Pickles, A. & Depagne, É. 2010, PASP, 122, 1437
  * Planck Collaboration: Abergel et al. (2014) Planck Collaboration: Abergel, A., Ade, P. A. R., Aghanim, N., et al. 2014, A&A, 571, A11
  * Planck Collaboration: Ade et al. (2016) Planck Collaboration: Ade, P. A. R., Aghanim, N., Arnaud, M., et al. 2016, A&A, 594, A13
  * Planck Collaboration: Aghanim et al. (2020) Planck Collaboration: Aghanim, N., Akrami, Y., Ashdown, M., et al. 2020, A&A, 641, A6
  * Polletta et al. (2007) Polletta, M., Tajer, M., Maraschi, L., et al. 2007, ApJ, 663, 81
  * Potter et al. (2017) Potter, D., Stadel, J., & Teyssier, R. 2017, Computational Astrophysics and Cosmology, 4, 2
  * Pozzetti et al. (2016) Pozzetti, L., Hirata, C. M., Geach, J. E., et al. 2016, A&A, 590, A3
  * Prevot et al. (1984) Prevot, M. L., Lequeux, J., Maurice, E., Prevot, L., & Rocca-Volmerange, B. 1984, A&A, 132, 389
  * Racca et al. (2016) Racca, G. D., Laureijs, R., Stagnaro, L., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9904, Space Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave, ed. H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, & E. C. Tong, 99040O
  * Robin & Creze (1986) Robin, A. & Creze, M. 1986, A&A, 157, 71
  * Rolland et al. (2007) Rolland, G., Pinheiro da Silva, L., Inguimbert, C., et al. 2007, in 2007 9th European Conference on Radiation and Its Effects on Components and Systems, 1–9
  * Rowe et al. (2015) Rowe, B. T. P., Jarvis, M., Mandelbaum, R., et al. 2015, Astronomy and Computing, 10, 121
  * Schultheis et al. (2014) Schultheis, M., Chen, B. Q., Jiang, B. W., et al. 2014, A&A, 566, A120
  * Secroun et al. (2018) Secroun, A., Barbier, R., Buton, C., et al. 2018, in High Energy, Optical, and Infrared Detectors for Astronomy VIII, ed. A. D. Holland & J. Beletic, Vol. 10709, International Society for Optics and Photonics (SPIE), 1070921
  * Secroun et al. (2016) Secroun, A., Serra, B., Clémens, J. C., et al. 2016, in High Energy, Optical, and Infrared Detectors for Astronomy VII, ed. A. D. Holland & J. Beletic, Vol. 9915, International Society for Optics and Photonics (SPIE), 99151Y
  * Sánchez et al. (2020) Sánchez, J., Walter, C. W., Awan, H., et al. 2020, MNRAS, 497, 210
  * Tallada et al. (2020) Tallada, P., Carretero, J., Casals, J., et al. 2020, Astronomy and Computing, 32, 100391
  * The Dark Energy Survey Collaboration (2005) The Dark Energy Survey Collaboration. 2005, arXiv, arXiv:astro-ph/0510346
  * Tully et al. (1975) Tully, R. B., de Marseille, O., & Fisher, J. R. 1975, in BAAS, Vol. 7, 426
  * Tylka et al. (1997) Tylka, A., Adams, J., Boberg, P., et al. 1997, IEEE Transactions on Nuclear Science, 44, 2150
  * van der Kruit & Searle (1982) van der Kruit, P. C. & Searle, L. 1982, A&A, 110, 61
  * Waczynski et al. (2016) Waczynski, A., Barbier, R., Cagiano, S., et al. 2016, in High Energy, Optical, and Infrared Detectors for Astronomy VII, ed. A. D. Holland & J. Beletic, Vol. 9915, International Society for Optics and Photonics (SPIE), 991511
  * Welikala & Kneib (2012) Welikala, N. & Kneib, J.-P. 2012, arXiv, 1202.0494
  * Williams et al. (2019) Williams, O. R., Begeman, K., Boxhoorn, D., et al. 2019, in Astronomical Society of the Pacific Conference Series, Vol. 521, Astronomical Data Analysis Software and Systems XXVI, ed. M. Molinaro, K. Shortridge, & F. Pasian, 120
  * Yoachim et al. (2016) Yoachim, P., Coughlin, M., Angeli, G. Z., et al. 2016, in Proc. SPIE, Vol. 9910, Observatory Operations: Strategies, Processes, and Systems VI, 99101A
  * Zoubian et al. (2014) Zoubian, J., Kümmel, M., Kermiche, S., et al. 2014, in Astronomical Society of the Pacific Conference Series, Vol. 485, Astronomical Data Analysis Software and Systems XXIII, ed. N. Manset & P. Forshay, 509

##  Appendix A Throughputs

Spectra and band fluxes were computed using the total transmission curves
estimates for VIS ($I_{\scriptscriptstyle\rm E}$), NISP-P
($Y_{\scriptscriptstyle\rm E}$, $J_{\scriptscriptstyle\rm E}$,
$H_{\scriptscriptstyle\rm E}$), NISP-S (Red and Blue Grisms), and each EXT
survey (e.g. Gaia, DES, LSST, and UNIONs) provided by each of the respective
instrument teams. We describe below the generation and/or acquisition of the
passbands for each instrument and their status for SC8 where applicable:

  * •

VIS The VIS total transmission was computed using the product of the MDB
parameters: the VIS quantum efficiency and the PLM VIS transmission curves
provided by the VIS Instrument team and Airbus Space and Defense,
respectively.

  * •

NISP We used the current best estimates of the NISP photometric passbands
($Y_{\scriptscriptstyle\rm E}$, $J_{\scriptscriptstyle\rm E}$,
$H_{\scriptscriptstyle\rm E}$) and the transmission curves for the red and
blue grisms produced by the NISP Instrument Development Team and validated and
crossed-checked by the SGS. The data sets used for SC8 are stored in the MDB,
and the latest current best estimates are described in Euclid Collaboration:
Schirmer et al. (2022).

  * •

Gaia As stated in §3.2.4, Gaia band fluxes were derived from the DR2 release
(Gaia Collaboration et al. 2018). Specifically, we used the revised passbands
provided at
https://www.cosmos.esa.int/documents/29201/1645651/GaiaDR2_Revised_Passbands_ZeroPoints.zip/54db454f-69cb-
ea0c-15be-f1b1f597f191.

  * •

DES We used the DR1 griz passbands (Abbott et al. 2018) available at
https://noirlab.edu/science/programs/ctio/filters/Dark-Energy-Camera.

  * •

LSST The baseline total throughput curves (CCDs+Filter+Optics+Atmosphere) at
airmass=1.2 (Kahn et al. 2010; Claver et al. 2014; Connolly et al. 2014) were
computed by the LSST system engineering team. For SC8, we used the earlier
release-1.4 total throughputs for $ugriz$ from
https://github.com/lsst/throughputs/tree/1.4/baseline.

  * •

CFIS The bandpasses for $u$ and $r$ filters, respectively, were generated with
the information for the MegaCam filter set available at https://www.cadc-
ccda.hia-iha.nrc-cnrc.gc.ca/en/megapipe/docs/filt.html.

  * •

JEDIS The JEDIS team provided full efficiency curves
(CCD+Filter+Optics+Atmosphere) in the $g$ band for 14 CCDs measured in a grid
of $12\times 13$ points, which we used to compute an average throughput.

  * •

Pan-STARRS PS1 The bandpass model for the $i$ band was derived from the data
available at http://ipp.ifa.hawaii.edu/ps1.filters/.

  * •

WISHES The bandpass data for the $z$ band was acquired from https://hsc-
release.mtk.nao.ac.jp/doc/index.php/survey-2/.

##  Appendix B Spectra and fluxes

###  B.1 Galaxy spectra reconstruction

Here we describe the process for reconstructing the observed spectra of the
galaxies from the parametrised information in the catalogue.

We begin by homogenising all wavelength-dependent elements into a common
uniform sampling to enable direct operation between models that were sampled
at different wavelength ranges and resolutions. To ensure the spectra cover
the wavelength range of interest and with enough resolution, we set a range
between $600\,\AA$ and $24,000\,\AA$, in steps of $1\,\AA$. With a spectral
resolution of $R=480$ in NISP-S, we are able to sample properly the dispersed
spectra.

Our reconstruction algorithm commences with the initialisation of a spectrum.
The SimSpectra library allows for the linear combination of two galaxy
templates, enabling a wider variety of colours and shapes, which gives more
realism to the simulation. We compute the combined template as follows:

$f_{\rm combined}(\lambda)=(1-i_{\rm frac})f_{\rm template}(\lambda)[i]+i_{\rm
frac}f_{\rm template}(\lambda)[i-1]\,,$ (13)

where $i_{\rm frac}$ is the fractional contribution of template with index $i$
in the combined template. Templates are ordered from early to late type galaxy
in the SED library. High-$z$ galaxies and QSOs use only one template. Next, we
apply one out of 5 dust extinction laws taken from Prevot et al. (1984);
Calzetti et al. (2000) and presented in Ilbert et al. (2009). The dust
extinction curve $k(\lambda)$ is scaled by the galaxy colour excess $E(B-V)$
and applied to the combined template, such that

$f_{\rm ext}(\lambda)=f_{\rm combined}(\lambda)\,10^{-0.4E(B-V)k(\lambda)}\,,$
(14)

where the extinction curve $k(\lambda)$ is

$k(\lambda)=-2.5\log_{10}\left(\frac{E_{j}(\lambda)}{E_{0}(\lambda)}\right)E(B-V)_{0.2}^{-1}\,.$
(15)

where $E_{j}(\lambda)$ corresponds to the extinction law used (Prevot et al.
1984; Calzetti et al. 2000; Ilbert et al. 2009), with $j=0$ corresponding to
no extinction, $j=1$ to the Prevot extinction law, $j=2$ to the Calzetti
extinction law with no 2175Å bump and $j=3$ and $j=4$ to the Calzetti
extinction law with different strengths of the 2175Å bump (see Ilbert et al.
2009 for details). For computational ease, we compute all the extinctions in
the filter passbands for a value of $E(B-V)=0.2$ and then re-scale the value
to the actual $E(B-V)$ in Eq. 14.

The next step is to shift the spectrum to the observed redshift (including
peculiar velocities), $z_{\rm obs}$. This operation is performed by scaling
the wavelength array elements by $(1+z_{\rm obs})$ and dividing the fluxes by
the same factor $(1+z_{\rm obs})$. With the redshifted spectrum, we can
measure the flux in the reference filter, and scale it to the reference
apparent AB magnitude. For standard and strongly lensed galaxies, the
reference filter is the SDSS $r$ passband redshifted to $z=0.1$. For high-
redshift galaxies and QSOs, the template is scaled according to the apparent
magnitude at $1500\,(1+z_{\rm obs})\,\AA$ and $1450\,(1+z_{\rm obs})\,\AA$
respectively.

For standard and strongly-lensed galaxies, the reference magnitude to scale
the spectrum is determined from the absolute magnitude in the SDSS $r$ band,
$M_{r}$. The $M_{r}$ magnitude is a product of the semi-analytic recipe in the
simulation, and is given in $h$-normalised units. The conversion from absolute
to apparent magnitude requires computing the luminosity distance $D_{\rm L}$
and the $K_{\rm Corr}$ correction.

We compute the luminosity distance with a Flat LCDM cosmology, setting the
$Hubble$ constant to $H_{\rm 0}=100\,{\rm km}\,{\rm s}^{-1}\,{\rm Mpc}^{-1}$,
and the non-relativistic matter density parameter to the value used in the
Flagship simulation $\Omega_{\rm m}=0.319$.

As our reference flux is not bolometric but computed within a particular
passband, we need the $K_{\rm Corr}$ correction term to transform from
observed to restframe flux to account for the galaxy redshift. This correction
term, expressed in magnitudes, is defined as

$K_{\rm Corr}=2.5\log_{\rm 10}\frac{f_{\rm restframe}}{f_{\rm observed}}\,;$
(16)

where we obtain both the rest-frame, $f_{\rm restframe}$, and the redshifted
$f_{\rm observed}$ fluxes integrated in the reference passband using the
following formula

$f_{\rm\nu}=\frac{\int
f_{\rm{\lambda}}\,R(\lambda)\frac{\lambda}{c}\,\rm{d}\lambda}{\int
R(\lambda)\,\rm{d}\lambda/\lambda}\,,$ (17)

where $R(\lambda)$ is the system response of the reference magnitude passband
in photon counts, resampled at $1\,\AA$ resolution. With the $K_{\rm Corr}$
correction and the luminosity distance $d_{\rm L}$ in parsec, we compute the
apparent magnitude,

$m_{r}=M_{r}+5\left[\log_{10}(d_{\rm L})-1\right]+K_{\rm Corr}\;.$ (18)

Note that the absolute magnitude and the luminosity distance are computed for
the same value of the $Hubble$ constant and therefore its dependence can be
disregarded in the equation making the apparent magnitude independent of its
value.

The scaled observed flux in ${\rm erg}\,{\rm cm}^{-2}\,{\rm s}^{-1}\,{\rm
Hz}^{-1}$ at the reference passband from the AB definition is then

$f_{{\rm\nu},r}=10^{-0.4m_{r}+48.6}\,,$ (19)

and the scaled continuum spectrum is therefore

$f_{\rm continuum}(\lambda)=f_{\rm
ext}(\lambda)\frac{f_{{\rm\nu},r}}{f_{\rm\nu,observed}}\,.$ (20)

At this stage, we add the emission lines to the scaled template. As emission
lines are already scaled and modelled with extinction, we compute them after
applying the dust to the scaled continuum. The central frequency of each line
is taken from vacuum or air, depending on whether the channel we simulate is
ground or space-based. In Table 3, we list the set of emission lines and their
corresponding central wavelength used for both air and vacuum in the
simulation.181818https://physics.nist.gov/PhysRefData/ASD/lines_form.html

Table 3: List of emission lines used in the simulated spectra of galaxies. Secondary lines are indicated with * and the corresponding flux ratio to the primary is indicated, as no secondary fluxes are explicitly indicated in the True Universe catalogue. Line | Vacuum (Å) | Air (Å) | ratio to primary
---|---|---|---
H$\alpha$ | 6564.614 | 6562.801 |
H$\beta$ | 4862.721 | 4861.363 |
O ii | 3727.090 | 3726.030 |
O ii* | 3729.880 | 3728.820 | 1:1
O iii | 5008.239 | 5006.843 |
O iii* | 4960.295 | 4958.911 | 1:3
N ii | 6585.270 | 6583.450 |
N ii* | 6549.860 | 6548.050 | 1:3
S ii | 6718.290 | 6716.440 |
S ii* | 6732.680 | 6730.820 | 1:1

We estimate the width of the emission line, $\sigma_{\rm el}$, in ${\rm
km}\,{\rm s}^{-1}$ following a luminosity-velocity dispersion relation (e.g.
Faber-Jackson Faber & Jackson 1976 and Tully-Fisher relation Tully et al.
1975) as shown below:

$\log_{10}(\sigma_{\rm el})=(-0.10+0.01z_{\rm obs})(M_{r}-3.0)-0.05z_{\rm
obs}\,.$ (21)

High-redshift galaxies and QSO templates already include emission lines. We
limit the approximation to a minimum velocity of $50\,{\rm km}\,{\rm s}^{-1}$
to avoid too narrow lines. We generate all 10 emission lines as Gaussian
profiles with the integrated flux as defined in the catalogue at the
corresponding central wavelength with a width equal to $\sigma_{\rm el}$. As
was done for continuum spectra, the set of reconstructed emission lines are
shifted in wavelength by the factor $(1+z_{\rm obs})$.

The velocity dispersion is converted to wavelength and scaled to the observed
frame as

$\sigma_{\rm el,\lambda}=\sigma_{\rm el}\frac{\mu_{i}}{c}(1+z_{\rm obs})\,.$
(22)

Then, we compute the spectrum containing only the emission line contribution,
$f_{\rm el}(\lambda)$, as

$f_{\rm el}(\lambda)=\sum_{i}^{10}\frac{[f_{\rm el}]_{i}}{\sigma_{\rm
el,\lambda}\sqrt{2\pi}}\exp{\left(-\frac{(\lambda-\mu_{i})^{2}}{2\sigma_{\rm
el,\lambda}^{2}}\frac{1}{1+z_{\rm obs}}\right)}\,,$ (23)

where $\mu_{i}$ and $[f_{\rm el}]_{i}$ are the central wavelengths and TU line
fluxes, respectively, for the 10 emission lines in the corresponding medium.

The resulting line profiles can now be added to the calibrated continuum
component of the spectra with the following expression

$f_{\rm galaxy}(\lambda)=f_{\rm continuum}(\lambda)+f_{\rm el}(\lambda)\,.$
(24)

With all the elements from the source galaxy, we introduce the effect of
lensing magnification with

$f_{\rm magnified}(\lambda)=f_{\rm galaxy}(\lambda)\,\mu\,,$ (25)

where $\mu$ is the magnification factor as defined in Eq. 4.

The final remaining step is to apply the Milky Way extinction to the magnified
galaxy spectra at the specific location in the sky. From the Planck dust maps
(Planck Collaboration: Abergel et al. 2014), we obtain a precise colour excess
$E(B-V)$, and following an O’Donnell (1994) extinction law with the ratio of
absolute to differential extinction $R_{V}=3.1$, we apply the MW extinction to
obtain the final incident spectra:

$f_{\rm incident}(\lambda)=f_{\rm
magnified}(\lambda)\,10^{-0.4\,A(\lambda)}\,,$ (26)

where the extinction law $A(\lambda)$ is the product of the extinction terms
$A(\lambda)=E(B-V)R_{V}$.

###  B.2 Stellar spectra reconstruction

With the already available apparent magnitude in the $H$ band we can simply
compute the template flux $f_{\rm\nu,template}$ in the observed band as
defined in Eq. 17 and the 2MASS $H$-band flux $f_{{\rm\nu},H}$ as Eq. 19 and
scale the star spectrum:

$f_{\rm star}(\lambda)=f_{\rm
template}(\lambda)\frac{f_{{\rm\nu},H}}{f_{\rm\nu,template}}\,.$ (27)

The only remaining step is to correct the Galactic extinction. The Tycho2
stars are not extinction corrected and therefore we do not apply further
correction. For the additional stars with BaSeL and LT spectra, we apply a
procedure similarly to the galaxy spectra as

$f_{\rm star}(\lambda)_{\rm ext}=f_{\rm star}(\lambda)\,10^{-0.4\,A(\lambda)}\
,$ (28)

with the O’Donnell (1994) law, $R_{\mathrm{V}}=3.1$, and the absorption
coefficient $A_{\mathrm{V}}$ determined assuming the Drimmel et al. (2003) 3D
Galactic extinction model.

Even though the wavelength range of the stellar templates cover the whole
spectrum of interest, the reddest part from $12\,000\,\AA$ onwards has a
reduced sampling in the Basel 2.2 library with $50\,\AA$ intervals. The
wavelength calibration in NISP-S has to be accurate to better than half a
pixel, which translates to approximately $6\,\AA$. Therefore, we would need
stellar spectra sampled at $2\,\AA$ to $3\,\AA$ intervals. For this reason we
replaced the feature-less low sampled redder part of the spectrum with real
high resolution measurements following the continuum as defined in the
template.

With support from OU-SIR, we did a fit for the spectral continuum for each
BaSEL spectrum, using functions provided by our redshift and spectra
measurements package EZ,191919http://pandora.lambrate.inaf.it/EZ/ resampled
the smooth continuum to a higher resolution, and added the continuum-
subtracted spectrum of a star taken from the NASA Infrared Telescope Facility
IRTF Spectral Library
202020http://irtfweb.ifa.hawaii.edu/~spex/IRTF_Spectral_Library/ to the high-
resolution continuum.

From the full IRTF library we have taken ten stars at random for each spectral
class (F, G, K, M - earlier classes O, B, and A are not covered by the
library), primarily of luminosity class III (giants stars) and V (main
sequence stars), but with at least one star of class either I (supergiants) or
IV (subgiants).

The choice of which IRTF continumm-subtracted spectrum to add to the BaSEL
continuum-only spectrum was based on broad-band color: the IRTF template with
the $Y-H$ color closer to the one stored in the original catalog was chosen as
the match. There was no attempt to include also the luminosity or other
properties in the match. So a very red star was assigned an M type spectrum,
without considering if the star luminosity was compatible with an M-type
object or not.

In this way we do not alter the colours of the spectra but we allow accurate
wavelength calibrations in the simulated NISP-S channel.

###  B.3 Synthetic photometry

We need to generate synthetic photometry from the high resolution spectra to
reproduce the passband fluxes that each instrument will measure. This
information is provided in the catalogue for the EXT surveys simulations, and
for validation purpose only for the Euclid instruments simulators, because
these later make use of the true spectra to take into account all chromatic
instrumental models.

Starting from the high resolution spectra, we compute the integrated flux
density of the source $f_{\nu}$ in ${\rm erg}\,{\rm cm}^{-2}\,{\rm
s}^{-1}\,{\rm Hz}^{-1}$ for a specific passband $R(\lambda)$ using Eq. (17).
Then, the synthetic magnitudes in the AB system are obtained with the
following transformation:

$m_{\rm syn}=-2.5\log_{10}(f_{\nu})-48.6\,.$ (29)

##  Appendix C SC8 preparation

###  C.1 Input catalogues

Galaxy and stellar catalogues containing the variety of sources described in
Sect. 3 were produced for both the SC8 Main (wide) and Deep fields and
ingested into the EAS. In particular, for the SC8 Main run, the spatial extent
of the galaxy and stellar catalogues exceeds ${\sim}\,600\,{\rm deg}^{2}$, the
original wide area survey planned for SC8. We added an additional margin to
ensure complete coverage for the EXT ground-based survey simulations. Figure
27 shows the footprint of the standard galaxy catalogues and we refer the
reader to Figure 26 in Sect. 3.2 which shows the footprint of the star
catalogues used for the SC8 Main run.

Standard galaxies, high-redshift galaxies, and QSOs are directly stored in
CosmoHub212121https://cosmohub.pic.es/ (Tallada et al. 2020) just after being
produced by SciPIC. Taking advantage of the interactive data exploration and
fast distribution capabilities of SciPIC, a dedicated team has validated the
catalogues. These three catalogues, plus the one containing the strongly
lensed galaxies, are then stored as distinct FITS files in a format defined in
the Euclid data model. The catalogues are further divided into different files
covering non-overlapping sky areas for each type of object. Each catalogue
file covers a HEALPix pixel of $N_{\rm SIDE}=2^{5}$ (corresponding to an area
of about $3.35\,{\rm deg}^{2}$), except for strong lensing galaxies which are
all included in a single file. All the FITS files along with their associated
metadata are subsequently ingested in the EAS.

Stellar catalogues were stored on a set of non-overlapping FITS files for each
kind of object, covering the original SC8 footprint with an area
${\sim}\,600\,{\rm deg}^{2}$. The catalogues are arranged on a grid with
different cell sizes depending on the stellar density of the area. As shown in
Fig. 26, there is an inverse relation between the area covered by a catalogue
file and the density of objects.

Figure 26: Density of objects (number of stars per ${\rm deg}^{2}$) in each
standard star catalogues file covering SC8 Main Production area, superimposed
with the Euclid observations and simulated EXT survey area.

In Table 4, we report the number of objects per galaxy/star type for both wide
and deep areas. For the SC8 Main run, we produced a total of ¿ 1 billion
objects of which 922M were standard Flagship galaxies (described in Sect. 3.1)
. The catalogues are divided among 3078 files for a total 254 GB of disk
space.

Figure 27: Number of objects in each standard galaxy catalogues file covering SC8 Main production area, superimposed with the Euclid observations and EXT simulated area. Table 4: Summary details for the galaxy and stellar catalogues used in SC8. Area |  Object Type |  Release |  Num obj |  Size [GB] |  Num files
---|---|---|---|---|---
SC8 Main |  Std gals |  1.10.18 |  922M |  224 |  747
QSOs |  1.10.15 |  2M |  0.5 |  562
High-$z$ |  1.10.19 |  11M |  2.5 |  562
SL gals |  SL_SC8 |  801 |  0.1 |  1
Std stars |  v15 |  77M |  15 |  532
LT stars |  v4_MLT |  1.8M |  0.4 |  532
SC8 Deep |  Std gals |  1.10.22 |  43M |  11 |  50
QSOs |  1.10.20 |  91K |  0.02 |  36
High-$z$ |  1.10.21 |  477K |  0.1 |  36
SL gals |  SL_DEEP |  201 |  0 |  1
Std stars |  v16 |  1.7M |  0.5 |  10
|  LT stars |  v6_MLT |  150K |  0.03 |  10

###  C.2 Feasibility analysis and validation

During the Pilot Phase, we ran profiling tests of each simulator code and
instrument channel for the six EXT surveys. Scaling up the total processing
time of the Pilot Phase to the SC8 Main run, we estimated a total simulation
time in excess of four million CPU hours to generate full-featured simulations
over the SC8 Main run target area. We assessed the cost of each feature
identifying those effects for which the computational expense was high, and
yet, had minimal impact on image quality and mission performance. This would
enable us to introduce simplifications to carry out the Main run production
within the appropriated computing budget. Finding the balance between feature
complexity and resource usage required several iterations, which thus
prolonged the time and effort to validate each resulting product. The
validation of the Pilot Phase data products and specific features contained
therein for each simulation channel (i.e. VIS, NISP-P, NISP-S, and EXT) was
performed by different teams using independent approaches, which we describe
further below.

For VIS, short-loop validation of the simulated Pilot Phase data products were
performed by the level-2 organisation units and reported any issues
identified. We, in turn, worked to resolve the errors reported, delivering new
VIS images to be analyzed until no further problems were raised.

For the NISP-P channel, validation of the simulation products was performed by
the NISP-P validators in SDC Finland. We performed a separate evaluation of
the technical requirements during the Pilot Phase, which revealed the high
computing cost for the NISP-P simulator Imagem would exceed the CPU time
constraints for the SC8 Main run. Therefore, we decided to reduce the intra-
pixel sampling to speed up the processing by a factor of 4. We verified the
astrometric accuracy of all sources to be generated below one milli-arcsecond
RMS for both the relative (between images) and absolute astrometry (with
respect to the TU catalogue). This ensured that a lower resolution intra-pixel
response simulation would not affect the positioning accuracy. The photometry
error was less than 0.02 mag with respect to the TU reference magnitude in
both aperture and PSF photometry, which also suggested a minimal dependence to
lower intra-pixel sampling.

In our validation of the spectroscopic channel for NISP, we verified the
$\chi^{2}$ distribution quality map was consistent with the expected
performance of the H2RG detector. Our validation tests confirmed the extracted
flux for stellar sources reproduced sufficiently accurately the expected TU
flux, with occasional outliers overestimating the flux due to contamination
from other sources. This was expected as the validation analysis was not
optimised to perform accurate decontamination of nearby spectra. The
polychromatic PSF was simplified by implementing a stacked monochromatic
version; improving the computational time by a factor of 7.

We validated the SIM-EXT science data products for astrometry, photometry,
background and PSFs for each EXT survey. We used the SourceExtractor software
to identify sources and measure fluxes and compared the derived positions and
fluxes with the reference TU catalogue values. The results of our validation
tests showed that the generated EXT data products were accurate and consistent
with the reference input values.222222It was later discovered during the
analysis of the SC8 Main data, that the input sky background values received
for the Northern Surveys were incorrect leading to shallower depths than
expected. The following simulation production cycle, which came after SC8,
included the appropriate sky brightness values. Finally, to help reduce
computational costs, we decided to simplify the galaxy simulation across all
simulation channels; rendering very small galaxies below $None$ with a simpler
exponential profile, instead of the more complex inclined Sérsic profile.

###  C.3 Pilot Phase processing

The main objective of the Pilot Phase was to profile and validate the OU-SIM
pipeline in preparation for the SC8 Main run. Extrapolations of the Pilot
Phase computing costs revealed the resources available for SC8 would not be
sufficient to generate full-featured simulations for a Euclid Wide Survey of
$600\,\rm{deg}^{2}$ within the appropriated time frame. While strategies were
being implemented to lower computing costs such as the reduction to a smaller
area of 165$\,\rm{deg}^{2}$ and the simplification of certain features for the
NISP instruments (e.g. intrapixel sampling and monochromatic stacked PSFs), as
many as 19 new features and data products (e.g. a cosmic ray map, a ghost map,
stars-only image, etc.) were requested to be simulated for VIS to test and
validate the OU-VIS processing and analysis software. Thus, while the Pilot
Phase area only included four Euclid observations overlapping one Euclid tile
and a small sample of EXT pointings per band, multiple productions were
launched with each new release due to new and/or modified features. In Table
5, we list all products produced for a given instrument channel during the
Pilot Phase, noting the number of images per production, number of production
sequences and the final number of products and storage requirements. For the
Euclid instruments, as each observation has 4 dithers a total of 16 images
were to be simulated for VIS, 48 images for NISP-P, and 16 images for NISP-S
for the Pilot Phase production. However, we performed 12, 14, and 6
productions, respectively for VIS, NISP-P, and NISP-S channels yielding 192,
672, and 96 data products. As seen in Table 5, multiple productions were also
carried out for each EXT survey. A total of 3,614 exposures were produced
during the Pilot Phase. Furthermore, for $Rubin$ and the Northern Surveys,
each simulated detector frame comprising an image was saved as an individual
file. Thus, the total number of data products ingested into the EAS was 64,679
amounting to 7.1 TB.

Table 5: Summary of the data products produced for SC8 Pilot Phase field production. The calibration frames were reused from previous challenges. Channel | Image type |  Original number of images |  Number of productions |  Final number of images |  Number of products per image |  Number of products |  Storage (GB)
---|---|---|---|---|---|---|---
VIS | Science Wide | 16 | 12 | 192 | 1 | 192 | 1294
VIS | Bias | 60 | 1 | 60 | 1 | 60 | 404
VIS | Flat | 60 | 1 | 60 | 1 | 60 | 404
NISP-P | Science Wide | 48 | 14 | 672 | 1 | 672 | 263
NISP-P | Dark | 600 | 1 | 600 | 1 | 600 | 234
NISP-P | Flat | 750 | 1 | 750 | 1 | 750 | 293
NISP-P | Self Cal | 180 | 1 | 180 | 1 | 180 | 70
NISP-P | Std Stars | 24 | 1 | 24 | 1 | 24 | 9
NISP-S | Science Wide | 16 | 6 | 96 | 1 | 96 | 375
NISP-S | Std Stars | 8 | 1 | 8 | 1 | 8 | 32
NISP-S | PN | 16 | 1 | 16 | 1 | 16 | 62
DES | Science Wide | 76 | 5 | 380 | 1 | 380 | 408
DES | Flat | 40 | 1 | 40 | 1 | 40 | 43
DES | Bias | 10 | 1 | 10 | 1 | 10 | 11
LSST | Science Wide | 65 | 4 | 260 | 189 | 49 140 | 3004
CFIS | Science Wide | 14 | 4 | 56 | 40 | 2240 | 30
JEDIS | Science Wide | 23 | 3 | 69 | 14 | 966 | 138
Pan-STARRS | Science Wide | 40 | 3 | 120 | 60 | 7200 | 221
WISHES | Science Wide | 7 | 3 | 21 | 103 | 2163 | 23

For each production sequence, we characterised the resources needed for each
processing element (i.e. memory, CPU, processing time, and storage). This was
to assess whether the resources originally requested to the HTC systems at
each SDC for each payload job would suffice for the SC8 Main run. In Table 6,
we report the final profiling results of the main processing elements of the
SIM pipeline described in Sect. 5 for each instrument.

Table 6: Resources required to simulate a given detector frame using the corresponding simulator, observed during SC8 Pilot Phase: Wall time, Resident Set Size (RSS), and Virtual Memory (VMEM). job |  wall time (h:mm) |  RSS (GB) |  VMEM (GB)
---|---|---|---
avg | max | avg | max | avg | max
VIS detector | 2:37 | 4:49 | 2.8 | 5.9 | 11.2 | 14.6
NISP-P detector | 0:34 | 1:14 | 4.3 | 5.9 | 4.8 | 6.6
NISP-S detector | 5:21 | 8:40 | 1.1 | 7.2 | 1.3 | 13.5
SIM_EXT detector |  |  |  |  |  |
DES | 0:06 | 0:16 | 0.5 | 1.0 | 0.8 | 1.4
LSST | 0:34 | 1:50 | 0.3 | 3.8 | 0.5 | 6.5
CFIS | 0:03 | 0:15 | 0.6 | 1.6 | 1.1 | 2.2
JEDIS | 0:16 | 0:49 | 1.2 | 1.8 | 1.5 | 2.2
Pan-STARRS | 0:08 | 1:18 | 0.6 | 1.4 | 0.9 | 2.0
WISHES | 0:03 | 0:12 | 0.7 | 1.9 | 1.2 | 2.6
NS Afterburner |  |  |  |  |  |
CFIS | 0:01 | 0:01 | 0.6 | 0.9 | 1.9 | 3.8
JEDIS | 0:09 | 0:23 | 4.9 | 6.9 | 5.8 | 7.4
Pan-STARRS | 0:05 | 0:58 | 1.4 | 2.0 | 2.1 | 5.3
WISHES | 0:00 | 0:02 | 0.6 | 0.8 | 1.2 | 4.8

We used the upper-bounds reported to issue a request for resources to the HTC
system, which had to be sufficiently higher than the observed maximum but
somewhat conservative to optimise SDC performance and capacity. Table 7 lists
the resource characterisations for each software release and the configuration
of each processing element used to produce image simulations for the SC8 Main
run.

Table 7: Program and version used for each of the main jobs in the SIM pipeline and resources requested to run on an HTC system job | project | version | n cpu | RSS (GB) | wall time (h)
---|---|---|---|---|---
VIS detector | ELViS | 1.20.3 | 1 | 8 | 6
NISP-P detector | SIM_Imagem | 1.9.11 | 1 | 8 | 10
NISP-S detector | SIM_TIPS_Simulator | 5.4.4 | 1 | 4 | 50
SIM_EXT detector | SIM_EXT | 2.14.2 | 1 | 4 | 6
NS Afterburner | EXT_NS_AFTERBURNER | 2.0.0 | 1 | 8 | 2

###  C.4 SC8 Main survey files

The SC8 Main area was reduced from a continuous region measuring
$600\,\rm{deg}^{2}$ to 11 patches with a total area of approximately
$165\,\rm{deg}^{2}$. Figure 28 shows the distribution of Euclid observations
among 11 discontiguous regions. For the EXT surveys, we subdivided the
original SC8 Main area into three non-overlapping patches. The area above
$30^{$\mathrm{\SIUnitSymbolDegree}$}$ in Declination was covered by the
Northern Surveys in certain broadbands: CFIS ($u$ and $r$ bands), JEDIS ($g$
band), Pan-STARRS ($i$ band), and WISHES ($z$ band). The rectangular region
below $30^{$\mathrm{\SIUnitSymbolDegree}$}$ in Declination was further sub-
divided into two sections along a diagonal to allow a range of sky conditions
in both segments. The area above the diagonal contained the set of Rubin
pointings for each broadband filter (e.g., $u$, $g$, $r$, $i$, and $z$ bands),
whereas the region below the diagonal contained only DES pointings for the
bands ($g$, $r$, $i$, and $z$ bands). OU-EXT provided a set of files
containing the respective observing cadence strategies for each survey from
which we selected exposures that intersect the SC8 Main area for processing.
The preparation of each survey file is described below and their corresponding
file contents are summarised in Table 8:

  * •

CFIS The pointings from the available CFIS data for SC8 preparation did not
cover the whole SC8 area of the sky. Therefore, the pointing pattern from
another area of the sky with similar declination was translated to the SC8
area as shown in Figure 29.

  * •

JEDIS By the start of SC8, JEDIS observations had yet to be performed. The
pointing pattern was generated following procedures outlined by the JEDIS
team. For any given sky location, the pattern consists of 14 overlapping
pointings with a dithering step of the size of the JEDIS CCD, so that each
point in the sky is imaged with all the 14 CCDs of the camera. The JEDIS SC8
footprint and depth in layer count can be seen in Figure 30.

  * •

Pan-STARRS A pre-selection of all pointings overlapping the SC8 area was
performed by OU-EXT. They reduced the original number of pointings to 6.5
times of the total in order to minimise data storage. To reach the required
depths, the exposure times were increased from $240.0\,{\rm s}$ to
$292.5\,{\rm s}$. The saturation limit of the CCDs was increased accordingly
in order to retain the saturation limit of the stars. Figure 31 shows the Pan-
STARRS PS1 SC8 footprint and the depth indicated by the contours denoting the
layer count.

  * •

WISHES The survey file in the $z$ band was built from real WISHES pointings
overlaying the SC8 area (as seen in Figure 32 which includes the corresponding
number of layers).

  * •

Rubin The LSST scheduler (Naghib et al. 2019) and associated simulated
pointing histories (OpSim outputs) provide many examples of survey strategy
options (Delgado & Reuter 2016; Connolly et al. 2014). For this work, a survey
table was derived courtesy of the Dark Energy Science Collaboration (DESC)
using the wide-fast-deep output from the v1.5 simulation,
footprint_sky_dustv1.5_10yrs (Jones et al. 2020), which we downsampled in
order to reach the desired limiting magnitudes in $ugriz$ for the Euclid Wide
Survey. We show in Figures 33 and 34, respectively, the number of layers in
the $u$ and $g$ bands for the $Rubin$ SC8 main survey. Magnitude zero points
estimates for each bandpass were provided by DESC. Furthermore, the same
position angle was used for all visits; therefore, we did not not account for
rotation of the camera denoted by the OpSim parameter ‘rottelpos’.

  * •

DES Similar to CFIS, the SC8 region is also outside of the DES footprint,
observations were extracted from the same range of Declination in the southern
celestial hemisphere at a shifted Right Ascension and transformed to cover the
SC8 area. An additional margin of 1.05 deg, half of DECam FOV, (Flaugher et
al. 2015), was applied to the selection to ensure that all exposures
overlapping the target area were included. Fig. 35 shows the resulting DES SC8
footprint and the blue contours indicate the number of layers to reach the
desired depth for bands $griz$.

Table 8: Number of exposures selected for processing and average number of layers in SC8 Main area for each of the survey-band pairs Survey | Bands |  Num exposures |  Num layers
---|---|---|---
Euclid |  $I_{\scriptscriptstyle\rm E}$, $J_{\scriptscriptstyle\rm E}$, $H_{\scriptscriptstyle\rm E}$, $Y_{\scriptscriptstyle\rm E}$, and Spectro | 331 | 4
DES |  $g$, $r$, $i$, $z$ | 593 | 10–15
LSST | $u$ | 201 | 10–15
$g$ | 133 | 5–10
$r$ | 130 | 5–10
$i$ | 139 | 5–10
$z$ | 197 | 10–15
CFIS |  $u$, $r$ | 673 | ${\sim}\,4$
JEDIS | $g$ | 864 | 15–20
Pan-STARRS | $i$ | 1055 | 20–30
WISHES | $z$ | 462 | ${\sim}\,4$
Figure 28: Footprint of the Euclid observations selected for Main SC8
production. Figure 29: Positions of the exposures and the number of layers for
the CFIS $u$ and $r$ bands. Figure 30: Positions of the exposures and the
number of layers for the JEDIS $g$ band. Figure 31: Positions of the exposures
and the number of layers for the Pan-STARRS $i$ band. Figure 32: Positions of
the exposures and the number of layers for the WISHES $z$ band. Figure 33:
Positions of the exposures and the number of layers for the LSST $u$ band. The
number of exposures and layers for the $u$ and $z$ bands are roughly equal.
Figure 34: Positions of the exposures and the number of layers for the LSST
$g$ band. The number of exposures and layers in $g$, $r$, and $i$ bands are
also approximately the same. Figure 35: Observations and the number of layers
in DES SC8 Main survey. Exposures of different bands ($g$, $r$, $i$, and $z$)
are at the same positions.