figure table

# Detailed Design of the Science Operations for the XRISM mission

Yukikatsu Terada  Saitama University, Graduate School of Science and
Engineering, Saitama-shi, Saitama, Japan  Japan Aerospace Exploration Agency,
Institute of Space and Astronautical Science, Sagamihara, Kanagawa, Japan
Matt Holland  National Aeronautics and Space Administration, Goddard Space
Flight Center, Greenbelt, Maryland, United States  Michael Loewenstein
National Aeronautics and Space Administration, Goddard Space Flight Center,
Greenbelt, Maryland, United States  Makoto Tashiro  Saitama University,
Graduate School of Science and Engineering, Saitama-shi, Saitama, Japan  Japan
Aerospace Exploration Agency, Institute of Space and Astronautical Science,
Sagamihara, Kanagawa, Japan  Hiromitsu Takahashi  Hiroshima University, School
of Science, Higashi-Hiroshima, Hiroshima, Japan  Masayoshi Nobukawa  Nara
University of Education, Department of Teacher Training and School Education,
Nara-shi, Nara, Japan  Tsunefumi Mizuno  Hiroshima University, School of
Science, Higashi-Hiroshima, Hiroshima, Japan  Takayuki Tamura  Japan Aerospace
Exploration Agency, Institute of Space and Astronautical Science, Sagamihara,
Kanagawa, Japan  Shin’ichiro Uno  Nihon Fukushi University, Faculty of Health
Sciences, Handa, Aichi, Japan  Shin Watanabe  Japan Aerospace Exploration
Agency, Institute of Space and Astronautical Science, Sagamihara, Kanagawa,
Japan  Chris Baluta  National Aeronautics and Space Administration, Goddard
Space Flight Center, Greenbelt, Maryland, United States  Laura Burns  National
Aeronautics and Space Administration, Goddard Space Flight Center, Greenbelt,
Maryland, United States  Ken Ebisawa  Japan Aerospace Exploration Agency,
Institute of Space and Astronautical Science, Sagamihara, Kanagawa, Japan
Satoshi Eguchi  Fukuoka University, Faculty of Science Department of Applied
Physics, Fukuoka-shi, Fukuoka, Japan  Yasushi Fukazawa  Hiroshima University,
School of Science, Higashi-Hiroshima, Hiroshima, Japan  Katsuhiro Hayashi
Japan Aerospace Exploration Agency, Institute of Space and Astronautical
Science, Sagamihara, Kanagawa, Japan  Ryo Iizuka  Japan Aerospace Exploration
Agency, Institute of Space and Astronautical Science, Sagamihara, Kanagawa,
Japan  Satoru Katsuda  Saitama University, Graduate School of Science and
Engineering, Saitama-shi, Saitama, Japan  Takao Kitaguchi  RIKEN, Nishina
Center, Wako, Saitama, Japan  Aya Kubota  Shibaura Institute of Technology,
Department of Electronic Information Systems, Saitama-shi, Saitama, Japan
Eric Miller  Massachusetts Institute of Technology, Kavli Institute for
Astrophysics and Space Research, Cambridge, Massachusetts, United States  Koji
Mukai  National Aeronautics and Space Administration, Goddard Space Flight
Center, Greenbelt, Maryland, United States  Shinya Nakashima  Japan Aerospace
Exploration Agency, Institute of Space and Astronautical Science, Sagamihara,
Kanagawa, Japan  Kazuhiro Nakazawa  Nagoya University, Department of Physics,
Nagoya, Aichi, Japan  Hirokazu Odaka  The University of Tokyo, Department of
Physics, Bunkyo, Tokyo, Japan  Masanori Ohno  Hiroshima University, School of
Science, Higashi-Hiroshima, Hiroshima, Japan  Naomi Ota  Nara Women’s
University, Department of Physics, Nara-shi, Nara, Japan  Rie Sato  Japan
Aerospace Exploration Agency, Institute of Space and Astronautical Science,
Sagamihara, Kanagawa, Japan  Makoto Sawada  RIKEN, Nishina Center, Wako,
Saitama, Japan  Yasuharu Sugawara  Japan Aerospace Exploration Agency,
Institute of Space and Astronautical Science, Sagamihara, Kanagawa, Japan
Megumi Shidatsu  Ehime University, Department of Physics, Matsuyama-shi,
Ehime, Japan  Tsubasa Tamba  The University of Tokyo, Department of Physics,
Bunkyo, Tokyo, Japan  Atsushi Tanimoto  The University of Tokyo, Department of
Physics, Bunkyo, Tokyo, Japan  Yuichi Terashima  Ehime University, Department
of Physics, Matsuyama-shi, Ehime, Japan  Yohko Tsuboi  Chuo University,
Department of Physics, Bunkyo, Tokyo, Japan  Yuusuke Uchida  Hiroshima
University, School of Science, Higashi-Hiroshima, Hiroshima, Japan  Hideki
Uchiyama  Shizuoka University, Faculty of Education, Shizuoka-shi, Shizuoka,
Japan  Shigeo Yamauchi  Nara Women’s University, Department of Physics, Nara-
shi, Nara, Japan  Tahir Yaqoob  National Aeronautics and Space Administration,
Goddard Space Flight Center, Greenbelt, Maryland, United States

###### Abstract

XRISM is an X-ray astronomical mission led by the Japan Aerospace Exploration
Agency (JAXA) and National Aeronautics and Space Administration (NASA), with
collaboration from the European Space Agency (ESA) and other international
participants, that is planned for launch in 2022 (Japanese fiscal year), to
quickly restore high-resolution X-ray spectroscopy of astrophysical objects
using the micro-calorimeter array after the loss of Hitomi satellite. In order
to enhance the scientific outputs of the mission, the Science Operations Team
(SOT) is structured independently from the instrument teams and the Mission
Operations Team (MOT). The responsibilities of the SOT are divided into four
categories: 1) guest observer program and data distributions, 2) distribution
of analysis software and the calibration database, 3) guest observer support
activities, and 4) performance verification and optimization activities.
Before constructing the operations concept of the XRISM mission, lessons on
the science operations learned from past Japanese X-ray missions (ASCA,
Suzaku, and Hitomi) are reviewed, and 15 kinds of lessons are identified by
categories, such as lessons on the importance of avoiding nonpublic (“animal”)
tools, coding quality of public tools in terms of both the engineering
viewpoint and calibration accuracy, tight communications with instrument teams
and operations teams, well-defined task division between scientists and
engineers, etc. Among these lessons, a) the importance of early preparation of
the operations from the ground stage, b) construction of an independent team
for science operations separate from the instrument development, and c)
operations with well-defined duties by appointed members are recognized as key
lessons for XRISM. Based on this, i) the task division between the mission and
science operations and ii) the subgroup structure within the XRISM team are
defined in detail as the XRISM Operations Concept. Based on this operations
concept, the detailed plan of the science operations is designed as follows.
The science operations tasks are shared among Japan, the US, and Europe and
are performed by three centers: the Science Operations Center (SOC) at JAXA,
the Science Data Center (SDC) at NASA, and European Space Astronomy Centre
(ESAC) at the ESA. The SOT is defined as a combination of the SOC and SDC. The
SOC is designed to perform tasks close to the spacecraft operations, such as
spacecraft planning of science targets, quick-look health checks, pre-pipeline
data processing, etc., and the SDC covers tasks regarding data calibration
processing (pipeline processing), maintenance of analysis tools, etc. The
data-archive and user-support activities are planned to be covered both by the
SOC and SDC. Finally, the details of the science operations tasks and the
tools for science operations are defined and prepared before launch. This
information is expected to be helpful for the construction of science
operations of future X-ray missions.

###### keywords:

The XRISM mission, science operations, operations concept, operations plan

*Yukikatsu Terada<EMAIL_ADDRESS>
##  1 Introduction

The X-Ray Imaging and Spectroscopy Mission (XRISM)[1] is an X-ray astronomical
mission led by the Japan Aerospace Exploration Agency (JAXA) and National
Aeronautics and Space Administration (NASA), in collaboration with the
European Space Agency (ESA) and other international partners, that is planned
for launch in 2022 (Japanese fiscal year) to restore high-resolution X-ray
spectroscopy after the loss of the Hitomi satellite[2]. The XRISM mission has
four scientific objectives[3]: 1) understanding the formation of the structure
of the universe and evolution of clusters of galaxies by measuring turbulent
and Doppler velocities at the 300 km/s level in spatially resolved
spectroscopy of clusters of galaxies, 2) understanding the circulation history
of baryonic matter in the universe from high-resolution spectroscopy of
phenomena such as supernova remnants and supernovae, 3) understanding the
transport and circulation of energy in the universe by observing feedback from
active galactic nuclei or outflow from super-massive black holes via high-
resolution spectroscopy, and 4) new science based on unprecedented high-
resolution X-ray spectroscopy, such as detailed diagnostics of collisional
ionization and photo-ionized plasma. To meet these scientific objectives of
the XRISM mission, the spacecraft and ground systems are designed to use the
X-ray micro-calorimeter array Resolve and the X-ray CCD camera Xtend on the
focal planes of the X-ray mirrors, which provide X-ray spectroscopy with a
high-energy resolution of $\leq$ 7 eV FWHM within a field of view (FOV) of
$2.9\times 2.9$ arcmin2 and imaging capability with a wide FOV of $30\times
30$ arcmin2, respectively, in the 0.3 to 12 keV band. This paper focuses on
the ground systems of the XRISM mission.

In order to maximize the scientific outputs from the XRISM mission, the
science operations of the mission also need to be well designed and performed
properly, namely, by conducting a guest observation program operating as a
public observatory under a well-supported system of guest observers and
providing well-calibrated observational data in the standard format for
astronomical use (i.e., flexible image transport system [FITS] format [4])
with simple and accurate analysis environments and tools [5].

This paper aims to describe the details of the development of the XRISM
Science Operations from the concept study to the detailed plans, as well as
give detailed descriptions of the preparations for the operation (such as
science operations manuals, tools, websites, etc) based on the SPIE Proceeding
in 2020[6]. Note that such descriptions on the detail of design of the science
operations may have sensitive topics for the project but the paper aims to
describe those as much as possible avoiding confidential technical ideas and
political issues for the XRISM project and agencies, because the authors
believe that this knowledge may help the design of science operations in near-
future high-energy missions. The rest of this paper is organized as follows.
We summarize the lessons learned from past X-ray missions in Section 2 as the
first step of the concept study, and summarize the concept of the operations
in Section 3. In Sections 4 and 5, the team structure and the details of the
science operations plan are summarized, respectively. Finally, Section 6
describes the timeline of the science operations and details of preparation of
tools in the ground systems for the XRISM Science Operations, and finally we
summarize this paper in Section 7.

##  2 Lessons for Science Operations Learned from Previous X-ray Missions

###  2.1 Summary of Lessons and Their Relations

As described in Section 1, the goal of science operations is to enhance or
maximize the scientific outputs of the mission. In science missions, the
activities required of science operations can be divided into the following
four categories.

> SO1
>
>
> Guest Observer (GO) program and data distribution
>
> SO2
>
>
> Distribution of analysis software and calibration database
>
> SO3
>
>
> GO supporting activities
>
> SO4
>
>
> Performance verification and optimization (PVO) activities

Many lessons were learned from the science operations in the series of
Japanese X-ray satellites, and although some of them require no changes,
others need to be addressed before the next mission. Table 1 summarizes the
relations among lessons learned from the Advanced Satellite for Cosmology and
Astrophysics (ASCA[7]), Suzaku[8], and Hitomi[2] missions, the details of
which are described in Sections 2.2, 2.3, and 2.4 below. Positive and negative
lessons are marked by + and - identifiers, respectively. Historically,
attempts were made to address negative lessons in the next mission. However,
this sometimes created another negative situation, which then needed to be
solved in a subsequent mission. All of the lessons learned from past X-ray
missions were considered by XRISM Science Operations, which are also shown in
Table 1 and summarized in Section 2.5 below.

Table 1: Relations among the lessons learned from ASCA, Suzaku, and Hitomi, summarized by category (SO1, SO2, SO3, and SO4; Section 2.1). Identifiers such as 1a-ASCA+ and 2ab-Suzaku+ are defined in Sections 2.2, 2.3, and 2.4. The ”$\rightarrow$” mark represents that the following mission continued the activities in the column on the left. Category | ASCA | Suzaku | Hitomi | XRISM
---|---|---|---|---
SO1 | 1a-ASCA+ | $\rightarrow$ | 1a-Hitomi± | $\rightarrow$
| (GO program) |  | (not activated) |
| 1b-ASCA+ | $\rightarrow$ | $\rightarrow$ | $\rightarrow$
| (data distribution) |  |  |
SO2 | 2a-ASCA+ | 2ab-Suzaku+ | $\rightarrow$ | $\rightarrow$
| (tool verification) | (verification, |  |
| 2b-ASCA- | public tools) | $\rightarrow$ | $\rightarrow$
| (non-public tools) |  |  |
|  | 2c-Suzaku- | 2c-Hitomi+ | $\rightarrow$
|  | (software development) | (specific team) |
|  |  | 2d-Hitomi- | to be fixed
|  |  | (management) |
SO3 | 3a-ASCA+ | $\rightarrow$ |  | $\rightarrow$
| (GO support) |  |  |
| 3b-ASCA- | 3b-Suzaku+ | 3-Hitomi± | $\rightarrow$
| (support center) | (Help desk Japan) | (not activated) |
|  | 3c-Suzaku- |  | to be fixed
|  | (data access rights) |  |
SO4 | 4a-ASCA+ | $\rightarrow$ | $\rightarrow$ | $\rightarrow$
| (communication |  | (communication |
| Japan/US) |  | Japan/US/ESA) |
| 4b-ASCA+ | $\rightarrow$ | 4b-Hitomi± | $\rightarrow$
| (calibration) |  | (limited calibration) |
|  | 4c-Suzaku+ | $\rightarrow$ | $\rightarrow$
|  | (IACHEC) |  |

###  2.2 Lessons Learned from ASCA

The ASCA mission was the fourth in the series of Japanese X-ray satellites [7]
and was launched in 1993 carrying a Gas Imaging Spectrometer and Solid-state
Imaging Spectrometer X-ray CCD cameras to observe astrophysical objects in the
0.5–10 keV band. The science operations activities as a public observatory
were well established in almost the first collaboration between the Institute
of Space and Astronautical Science (ISAS) at current JAXA and NASA/GSFC in the
GO program (1a-ASCA+) and the distribution of observation data (1b-ASCA+),
analysis software (2a-ASCA+), GO support (3a-ASCA+), international
collaboration (4a-ASCA+), and calibration of instruments (4b-ASCA+), but there
were also two negative items (2b-ASCA- and 3b-ASCA-). The successful parts of
ASCA are summarized below.

> 1a-ASCA+:
>
>
> The GO program worked well both in Japan and the US. GOs were able to submit
> their proposals to agencies, which were reviewed by the scientists and
> selected based on priorities, and information regarding the approved targets
> were used by mission operations in Japan. The basic procedures of the GO
> program were established.
>
> 1b-ASCA+:
>
>
> All data products were well managed and distributed to GOs. The backbone of
> the procedure for processing and archiving observation data was established.
>
> 2a-ASCA+:
>
>
> The core algorithms in the analysis software were well verified via on-
> ground calibration measurements before launch by instrument teams (ITs).
> Analysis tools using these algorithms and the calibration database were
> delivered to GOs. The concept was established in this mission.
>
> 3a-ASCA+:
>
>
> As a part of the GO program, user support activities were established.
>
> 4a-ASCA+:
>
>
> Collaboration between Japan and US was established on the ASCA science
> operations and was well organized especially on the development of the
> public software and the calibration database.
>
> 4b-ASCA+:
>
>
> The instrument team members in Japan performed ground calibrations while
> scientists both in Japan and US performed continuous in-orbit calibrations,
> which delivered good calibration accuracy.

However, the following items can be regarded as negative lessons provided by
this past mission.

> 2b-ASCA-:
>
>
> The instrument team members developed their own tools for analyzing the
> ground calibration data, which sometimes provide better results than the
> public analysis software released as part of item 2a-ASCA+ in the early GO
> phase. Since these tools were not initially made public, they became
> referred to as ”animal software” because of the unfairness of the analyses
> from the viewpoint of a public observatory, although this unfair situation
> was resolved in the final version of the products.
>
> 3b-ASCA-:
>
>
> GO support was provided only in the US by the US ASCA Guest Observer
> Facility (GOF), but not in Japan, although instrument teams in Japan
> provided deep support for the GOF activity. The interface to GOs existed
> only in the US.

###  2.3 Lessons Learned from Suzaku

The Suzaku mission was the fifth in the series of the Japanese X-ray
satellites in collaboration between JAXA and NASA [8], and was launched in
2005 carrying the High-throughput X-ray Telescope, X-ray Imaging Spectrometer
CCD cameras, and non-imaging Hard X-ray Detector. The science operations
members of Suzaku tried to utilize the positive lessons from ASCA (i.e., on
the GO program and data distribution 1a-ASCA+ and 1b-ASCA+, the software
development 2a-ASCA+, the GO support 3a-ASCA+, and the PVO activities 4a-ASCA+
and 4b-ASCA+) and fix the negative situations (2b-ASCA- and 3b-ASCA-). They
successfully fixed these using 2ab-Suzaku+ and 3b-Suzaku+, respectively, which
are summarized below.

> 2ab-Suzaku+:
>
>
> In order to keep the positive situation 2a-ASCA+ and solve negative
> situation 2b-ASCA- (i.e., avoiding animal software), the public tools and
> tools for ground calibration measurements were designed to have the core
> algorithms for calculating variables such as time, pulse-height invariant
> (PI[9]), and grade, shared as software libraries. The instrument team
> members developed and verified these core libraries via hardware
> development, and the same libraries were smoothly exported into public tools
> that could be used by the GOs. Therefore, the public tools were well
> verified and well calibrated.
>
> 3b-Suzaku+:
>
>
> In order to improve situation 3b-ASCA-, a Suzaku Help Desk in Japan
> (RIKEN)[10] were operated in addition to the Suzaku GOF in the US. Since
> several members of the Suzaku Help Desk also belong to the instrument teams
> in Japan, these two bodies had a strong potential for solving questions from
> GOs very quickly because of their tight connection to the operation team and
> developers of instruments.

In addition, starting from Suzaku, communication was established with other
X-ray missions in terms of calibration activities (keeping 4b-ASCA+), as
indicated as 4c-Suzaku+ below.

> 4c-Suzaku+:
>
>
> The Suzaku instrument members participated from the beginning in cross-
> calibration activities in the International Astrophysical Consortium for
> High Energy Calibration (IACHEC) [11].

However, the following two negative points related to 2ab-Suzaku+ and
3b-Suzaku+ arose in the Suzaku Science Operations, and were left as open
issues for the next mission (Hitomi).

> 2c-Suzaku-:
>
>
> Software development by the instrument teams in 2ab-Suzaku+ caused a)
> unexpected software freezes and b) delays in the delivery schedule, because
> a) not all the instrument members were experts on programming, and b) the
> first priority of the instrument teams was the delivery and maintenance of
> the detector itself, with software development having a lower priority.
>
> 3c-Suzaku-:
>
>
> The members of the Suzaku Help Desk in Japan (3b-Suzaku+) were not appointed
> by the agency and had no special data-access permission. Therefore, the
> tasks were performed on a best-effort basis and sometimes activity stopped
> because of other business.

###  2.4 Lessons Learned from Hitomi

The Hitomi mission was the sixth in the series of Japanese X-ray satellites
developed at JAXA in collaboration with NASA and Japanese and Canadian
institutions with contribution from the ESA [2], and carried an micro X-ray
calorimeter array and X-ray CCD cameras on the focal plane of X-ray mirrors,
as well as hard X-ray instruments with hard X-ray mirrors and a soft gamma-ray
detector. The satellite was successfully launched in February 2016, but
contact with the spacecraft was lost in March 2016 owing to problems in the
bus system before the performance verification (PV) phase. Therefore, most of
the science operations after launch were not activated, as indicated by
1a-Hitomi±, 3-Hitomi±, and 4-Hitomi± below.

> 1a-Hitomi±:
>
>
> Opportunity for calling for GO proposals was canceled, although the
> distribution of the in-orbit data was completed (keeping 1b-ASCA+).
>
> 3-Hitomi±:
>
>
> GO support helpdesks were prepared but not activated.
>
> 4b-Hitomi±:
>
>
> Calibration of instruments was performed on the ground and partially in
> orbit during the commissioning phase, and the results were released in the
> calibration database.

In the science operations of Hitomi, positive lessons from ASCA and Suzaku
were kept in the activities under the above situations, including
participation in the IACHEC (4c-Suzaku+). In parallel, the science operations
members tried to solve negative item 2c-Suzaku- (leaving 3c-Suzaku- open when
the mission terminated), and it was solved as 2c-Hitomi+ with 2d-Hitomi- left
as an open issue as shown below.

> 2c-Hitomi+:
>
>
> In order to avoid 2c-Suzaku- (unexpected software freeze and schedule
> delay), a specific team was defined for the development of software and the
> calibration database. The team was the Hitomi software and calibration team
> (SCT) and was independent from the instrument teams and consisted of
> scientists and programmers. As a result, there were no delays in the
> schedule of software preparation and no delay in the release of tools and
> the calibration database. The products were well calibrated using
> instrument-specific methods[12, 13, 14, 15, 16, 17, 18, 19, 20, 21], because
> all the algorithms were imported into the analysis software and the latest
> calibration information was quickly released in the database[5].
>
> 2d-Hitomi-:
>
>
> In order to achieve 2c-Hitomi+, there were many interactions among the
> software and calibration team, instrument teams, and operation teams, which
> were spread across multiple agencies. Therefore, many more tasks than
> expected were required in order to manage tasks for science operations such
> as the schedule, manpower of activities, and interfaces.

###  2.5 Recommendations for XRISM Science Operations

In summary, from the lessons learned from ASCA, Suzaku, and Hitomi missions,
two items labeled 2d-Hitomi- (team management issues) and 3c-Suzaku- (data
access rights in users support) remain as open items for the XRISM Science
Operations, with the remaining items recommended to remain unchanged. The two
items are related to the management of manpower and the preparation of science
operations before launch.

##  3 Concept for the XRISM Science Operations

Based on the lessons learned from the past X-ray missions and recommendations
for XRISM operations (Section 2), we established the XRISM Operations Concept,
as described in this section.

###  3.1 Key points of the XRISM Operations Concept

Considering the recommendations from lessons learned from past X-ray missions
(Section 2), the key points of the XRISM Operations Concept can be summarized
into the following three items.

>   1. OC01
>
> Clear division between spacecraft operations (hereafter, ”mission
> operations”) and science operations is required so that the scientists can
> concentrate on science operations.
>
>   2. OC02
>
> The plans for the operations (both mission and science operations),
> including the team structure and interfaces, should be defined in an early
> phase before launch. Similarly, training and actual operations should start
> before launch.
>
>   3. OC03
>
> All members of the operations (both mission and science operations) should
> be appointed by the agencies, and all activities, except for PVO activities
> (see Section 2.1), should be performed as well-defined tasks with clear due
> dates, that continue to work until the end of the mission.
>
>

###  3.2 Task Division between Mission Operations and Science Operations

In operations concept OC01, operations tasks that require scientific decisions
from the viewpoints of scientists are all assigned as XRISM Science
Operations, and all other tasks are assigned as XRISM Mission Operations. For
example, the weekly negotiation of contact passes of ground stations,
generation of daily operation commands, execution of the spacecraft simulator,
down-link-and-command operations at ground stations, and checking the house-
keeping telemetries from the spacecraft in real-time and off-line are assigned
as tasks for mission operations. On the other hand, the handling of proposals
from GOs, scientific scheduling of astrophysical objects, quick-look checks of
science telemetries, instrument calibration activities111However, the
calibration activities were shared among XRISM-internal groups, as described
later in Section 5., and the management of daily data process and archive
operations and user support activities are considered tasks for science
operations.

We defined individual teams for performing XRISM Mission Operations and XRISM
Science Operations separately, which are called the Mission Operations Team
(MOT) and Science Operations Team (SOT), respectively. Following the lessons
of 2c-Hitomi+, these teams also need to be independent from the instrument
teams. In addition, we defined the Science Management Office (SMO) to manage
the overall XRISM Science Operations for deciding items regarding science
operations. For example, the SMO handles activities such as calling for,
reviewing, and selecting GO proposals, and approving targets for director
time-of-opportunity (ToO) observations.

###  3.3 Operation Phases and Team Structure

In operations concept OC02, the operation phases of the XRISM are defined as
follows.

>   1. 1.
>
> Before Proto-Flight Test (PFT) Phase
>
>   2. 2.
>
> PFT Phase
>
>   3. 3.
>
> Launch Preparation Phase & Launch Phase
>
>   4. 4.
>
> Initial Phase, which consists of the Critical Operations Period and
> Commissioning Period
>
>   5. 5.
>
> Nominal Operations Phase, which consists of the Initial-Calibration and
> Performance-Verification (PV) Period and Nominal Observation Period
>
>   6. 6.
>
> Latter Phase, which consists of the Latter Observation Period and Completion
> of Operations
>
>

Based on concept C02, the phase in which science operations starts should be
the 2) PFT Phase, not the 5) Nominal Operations Phase after launch. In other
words, preparation of the mission and science operations should be completed
before the PFT Phase on the ground, and the MOT and SOT shall start from the
PFT Phase. Therefore, we define the Mission Operations Planning Team (MOPT)
for preparing the mission and science operations (e.g., construction of the
detailed operations plan, preparation of the OTs and ground system), which is
active from the 1) Before PFT Phase.

According to operations concept OC03, all members of the MOT and SOT should be
appointed by the agencies (JAXA or NASA), and work on well-defined tasks under
a managed schedule until the end of the mission, although members of the MOPT
may be non-appointed members from universities as developers of tools in the
Before PFT Phase. The SOT members consist of not only leader(s) and senior
scientists, but also young scientists, referred to as Duty Scientists, who
perform the actual XRISM Science Operations and are appointed by the agencies.
In our concept, the tasks of the Duty Scientists should be defined such as to
provide a good career path for young scientists. Note that the concept of the
Duty Scientists is applied only on the science operations center in Japan as
described in the next section 4.

##  4 Design of the Team and Management Structure of the XRISM Science
Operations

Based on operations concepts OC01, OC02, and OC03 in Section 3, we defined the
details of the structure of the SOT and the interfaces and task divisions
among the subgroups of the XRISM team, which are described in this section.

###  4.1 Interface Structure Between Subgroups and the SOT

In addition to the SOT, MOT, and SMO described in Section 3, the XRISM team
consists of the instrument teams, namely, the Resolve and Xtend teams, and the
In-flight Calibration Planning Team (IFCP)[22], which provides the detailed
plans for in-orbit calibration observations before launch. Interactions
between these subgroups after the PFT Phase (Section 3.3) are summarized in
Figure 1.

Figure 1: Interactions between the SOT and other internal subgroups in the
XRISM team after the PFT Phase. The scientific community and public are also
shown on the right.

The SOT is directed by the Project Manager (PM), and works with the SMO, which
provides recommendations and specifications for science operations as
established in the concept study in Section 3. The SOT does not communicate
directly with the Science Advisory Committee, but rather through the SMO. The
SOT communicates with the MOT regarding tasks such as the planning of
spacecraft operation, verification of telemetry data, and reports. Several
tasks such as in-orbit calibration observations and instrument performance
monitoring are performed in collaboration with the Resolve, Xtend, and IFCP
teams. The SOT communicates with GOs directly regarding the acceptance of
proposals and user support activities. Communications with other observatories
(i.e., X-ray missions and/or observatories in other wavelengths), negotiations
for in-orbit calibration campaigns, and/or multi-wavelength scientific
programs are handled by the SMO, and communications on actual observation
plans/schedules are handled by the SOT. Similarly, decisions regarding press
releases of scientific outputs or mission status are made by the SMO, and the
actual work of the publications is done by the SOT members under the direction
of the SMO.

The calibration activities of payload instruments consist of many steps, and
the task divisions among the SOT, instrument teams, and IFCP team are defined
as Table 2.

Table 2: Task division among the SOT, ITs, and IFCP team on the calibration
activities.

Calibration Tasks | Subgroup Name
---|---
Preparation of calibration requirement | ITs
Review calibration requirement from viewpoint | SOT
of flow down from mission science objectives |
Ground calibration tests | ITs and
| bus company on timing system
Analyses of ground calibration data | ITs with SOT and
| bus company with SOT on timing system
Prepare pre-launch calibration database | ITs with SOT
Release pre-launch calibration database | SOT
Preparation of in-orbit calibration plan | IFCP†
Schedule of in-orbit calibration targets | SOT
Spacecraft operation of in-orbit calibration targets | MOT
Analyses of in-orbit calibration data | SOT and ITs
| with PVO contributor‡
Preparation of calibration database | SOT and ITs
Release calibration database | SOT

$\dagger$ The IFCP team is active before launch. $\ddagger$ The PVO
contributors are defined in Figure 3 in Section 4.3.

###  4.2 SOT Structure and Task Divisions

The XRISM Science Operations are covered by JAXA, NASA, and the ESA. Since
tight collaboration between JAXA and NASA is required for preparation and
maintenance of the data distribution (SO1 in Section 2.1) and analysis
software and the calibration database (SO2 in Section 2.1), the SOT is
designed to operate at the Science Operations Center (SOC) at JAXA and the
Science Data Center (SDC)[23] at NASA, as shown in Figure 2. Each center is
made up of leads, scientists, and software engineers. In the SDC, the Product
Development Lead and the Science Lead direct the SDC internal groups, namely,
the Data Center Team, Guest Observer Facility, and User Support Group. The SOC
is activated after the PFT Test Phase (see Section 3.3), and the SOC Lead
directs SOC members, such as the Duty Scientists defined in Section 3.3 and
Supporting Scientists from the MOPT, after the PFT Phase. Before launch, the
science part of the MOPT is also under the direction of the SOC Lead, and
consists of three groups: the Process and Plan Group, the PVO Group, and the
User Support Group. In addition, the ESA operates the European Space Astronomy
Center (ESAC) from the Before-PFT Phase and communicates with the SOC and SDC
for the science operations in Europe.

Figure 2: Structure of the SOT.

The task divisions among the three centers in JAXA, NASA, and ESA are defined
in Table 3, summarized into four categories (SO1, SO2, SO3, and SO4 in Section
2.1). The details of these tasks are described in the next Section 5.

Table 3: Task Division between SOC, SDC, and ESAC in the XRISM Science Operations. Category | SOC | SDC | ESAC
---|---|---|---
SO1 | Proposal support (JAXA) | Proposal support (US/Canada) | Proposal support (Europe)
(GO) | \- Web pages for Submission | \- Web pages for submission | \- Web pages for submission
|  | \- Proposing systems and tools |
| \- Proposal support | \- Proposal support | \- Proposal support
| \- Supporting review process | \- Supporting review process | \- Supporting review process
(Data) | Observation scheduling | Observation scheduling |
| \- Planning operations | \- Planning operations |
| \- ToO handling | \- Planning tools and personnel |
| \- MOT interface |  |
| Pre-pipeline process | Pipeline process | (No data process)
| \- Format conversion | \- Filling calibrated columns |
| \- Observation database | \- Metadata for archive |
| Archiving at JAXA | Archiving at NASA | (no archive)
| \- Data archive | \- Data archive |
| \- Quick-viewing tool | \- Calibration database release | \- Quick-viewing tool
SO2 | Telemetry check | Development of analysis tools |
(Software) | \- Health check |  |
| \- Performance check | Maintenance of calibration database |
SO3 | User support (JAXA) | User support (US/Canada) | User support (Europe)
(Support) | \- Handling GO questions | \- Handling GO questions | \- Handling GO questions
| \- User guide documents | \- User guide documents | \- Documentation
| \- Researcher webpages | \- HEASARC webpage | \- ESAC webpage
| \- EPO support | \- EPO support | \- EPO support
SO4 | Calibration operations | Calibration support |
(PVO) | \- Analyses | \- Analyses |
| \- Monthly plan |  |
| Performance check | Performance check | Performance check
| \- Analysis threads | \- Analysis threads | \- Analysis threads
| \- Daily/monthly check | \- Post-process development |
| Xtend Transient search |  |

###  4.3 Management structure

All specifications of the overall science operations are handled by the SMO,
and therefore, the Mission Principal Investigator (PI) and co-PI (JAXA/NASA),
Project Scientists (JAXA/NASA/ESA), Deputy Project Scientists (JAXA/NASA/ESA),
and all the leaders of the internal subgroups (SOT and Resolve and Xtend
teams) except for the MOT and IFCP team in Figure 1 belong to the SMO
Committee in the Nominal Operations Phase, as shown in Figure 3. The Project
Managers both in JAXA and NASA also belong to the SMO as ex officio. Before
launch, leaders of the IFCP team, chairs of the science categories, which are
active in the selection of the PV targets, and chairs of subgroups for
specific topics such as atomic physics also participate in the SMO.

Following concept C03 in Section 3.1, the members marked by $\largestar$ in
Figure 3 are appointed by the agencies, namely, JAXA or NASA (or the ESA).
Within the SOT, the SOC plans to have eight Duty Scientists (see Section 3.3),
as well as Support Scientists from the MOPT (Section 3.3) and/or JAXA academic
positions to help the Duty Scientists, as described in Section 4.2. In the
SDC, more than seven staff scientists and more than four software engineers
will perform the science operations at NASA. All members of the SOC and SDC
are appointed by JAXA and NASA, respectively, in the Nominal Operations Phase.

Figure 3: Organization structure of the XRISM Science Operations during the
Nominal Operation Phase.

###  4.4 Data Access Policy for Science Operations

In the science operations performed by the SOT, SOT members have access to all
of the telemetry items including scientific properties in order to check the
performance of the instruments and to make quick-look reports to GOs. These
activities are limited to monitoring or checking of the instrument health and
performance, and do not extend to performing scientific analyses of the
scientific interests of scientists. The SOT members also check all the
proposals approved by the SMO and their scientific justifications. While SOT
members can access all the data and products to the extent that such access is
required to perform their duties, they are required to maintain
confidentiality of all scientific knowledge obtained in this context. The SOT
members shall understand this data access policy in all of the science
operations.

##  5 Details of the Science Operations Plan

Following operations concept OC02 (Section 3.1), the detailed plan of the
XRISM Science Operations is constructed as described in the following sections
under the team structure defined in Section 4, well before the launch during
the Before PFT Phase (Section 3.3).

###  5.1 Summary of the Science Operations Scenario

All the tasks for the XRISM Science Operations are defined in terms of the
four types of science operations defined in Section 2.1 (i.e., SO1, SO2, SO3,
and SO4) in Table 4, which can be categorized into the following three-step
operation flow.

>   1. Step-1:
>
> Proposal and Planning Step (before observation)
>
>   2. Step-2:
>
> Telemetry Check and Data Processing and Archive Step (after daily spacecraft
> observation)
>
>   3. Step-3:
>
> User Support and PVO Step (after distribution of the observational data to
> GOs)
>
>

These steps are performed in parallel with observations, and are operated both
by the SOT and MOT, with various timescales (once per year, monthly, weekly,
daily, and continuous), as also shown in Table 4. The tasks for the SOC in
Japan (see Table 3) are shared among the Duty Scientists and the SOT Scheduler
(who works on planning as a contribution from SDC staying at SOC; Figures 2
and 3) evenly by week or month. For example, one Duty Scientist performs the
first task, which is then performed by another Duty Scientist the following
week.

Table 4: Tasks for XRISM Science Operations for the steps of observations (see the text in Section 5.1) in the Nominal Operations Phase. Types of science operations are defined in Section 2.1. Step | Type | Category (frequency) | Tasks | Subgroup
---|---|---|---|---
Step-1 | SO1 | GO proposal support | Call for proposals, receive proposals | SOT
| SO3 | (year) | Support GO proposals | SOT
| SO1 |  | Support review process | SOT
| SO1 |  | Handle approved target | SOT
| SO1 | Observation planning | In-orbit calibration planning | SOT
| SO1 | (monthly, weekly, daily) | ToO handling | SOT
| SO1 |  | Observation scheduling (long/short term) | SOT
| SO1 |  | Coordination of observations | SOT
| SO1 |  | MOT I/F for operation command generation | SOT
NA | N/A | Spacecraft operation |  | MOT
|  | (daily) |  |
Step-2 | SO2 | Telemetry check | Process quick-look data (QLDP) | MOT
| SO2 | (daily) | Health check | MOT,SOT
| SO4 |  | Instrument performance check | SOT
| SO1 | Data process | Format conversion (PPL) | MOT, SOT
| SO1 | (daily) | Calculate calibrated columns (PL) | SOT
| SO1 |  | Observation database/Archive Metadata | MOT,SOT
| SO2 |  | Product check | SOT
| SO1 |  | User notification | MOT
| SO1 | Archive | Data publication | MOT
|  | (daily) |  |
Step-3 | SO3 | User support | Researcher webpages | SOT
| SO3 | (daily) | Handling GO questions | SOT
| SO3 |  | Documentation | SOT
| SO3 |  | EPO support | SOT
| SO4 | PVO activities | Calibration analyses | SOT
| SO4 | (daily) | Monthly performance check | SOT
| SO4 |  | Development of Analysis threads and tools | SOT
| SO4 |  | Xtend transient search | SOT

###  5.2 Details of Tasks in Step-1 Before Observation

Most of the tasks in Step-1 before observation are of Type SO1 (defined in
Section 2.1), which can be divided into the following two categories. The
details are as follows.

>   * •
>
> GO proposal support
>
>     * –
>
> The SOT supports calls for proposals by the SMO and receives proposals from
> GOs. During proposal acceptance, the SOT supports GOs with preparation of
> proposals (SO3).
>
>     * –
>
> After the review process, the SOT gets a prioritized approved target list
> from the SMO. In parallel, the in-orbit calibration objects are merged into
> the list. The SOT puts information regarding approved targets and
> calibration objects into the observation database and opens the list via the
> webpages of the researchers.
>
>   * •
>
> Observation planning
>
>     * –
>
> After the SOT obtains the list of targets, the SOT Scheduler (defined in
> section 5.1) generates a long-term operation plan taking into account the
> spacecraft operational constraints.
>
>     * –
>
> Using the long-term plan as a guide, the SOT generates a more detailed
> short-term observation schedule weekly and prepares the detailed plans for
> observations for the week.
>
>     * –
>
> During preparation of the observation details, the SOT notifies the
> observation PI of the plan, and negotiates the hardware configuration with
> the instrument teams.
>
>     * –
>
> In addition to the planned objects, the SOT handles ToO proposals from GOs.
> If the SMO approves a ToO proposal, the short-term operation plan is quickly
> updated and used for the spacecraft operations.
>
>     * –
>
> Before spacecraft operations, the SOT acts as an interface to the MOT from
> SMO and GOs on scientific topics for the generation of operation commands to
> the spacecraft.
>
>

###  5.3 Details of Tasks in Step-2 After Observation

The tasks in Step-2 after observation are a mixture of Types SO1, SO2, and SO4
(Section 2.1), and can be divided into the following three categories. The
details are as follows.

>   * •
>
> Telemetry check
>
>     * –
>
> Telemetry from the spacecraft needs to be checked quickly after spacecraft
> operation. In order to quickly perform these telemetry checks before the
> official data processing for GOs, which takes about one or two weeks in
> total, a Quick-look Data Process (QLDP) is defined to generate products
> quickly. The QLDP simplifies the timing calibration, orbit determination,
> and attitude determination processes from the official data processing.
>
>     * –
>
> The MOT executes the QLDP and performs the quick health checks of
> instruments using the housekeeping telemetry semi-automatically. This
> function checks every value of the attribute in the engineering housekeeping
> telemetry to verify the proper and safe operation of the observatory at all
> times. If the MOT find an anomalous telemetry for the spacecraft safety from
> this limit checks, they will respond immediately as an emergency operation.
> In any cases, the MOT reports the results to the SOT daily after the
> spacecraft operation.
>
>     * –
>
> In addition to the engineering health checks by the MOT, further checks of
> the performance of payload instruments from a scientific viewpoint are also
> required for the SOT. The SOT uses the products from QLDP of not only the
> housekeeping telemetry but also the science telemetry, performs the
> pipeline-equivalent process to calculate data such as the time, coordinate,
> and energy information, and then checks the instrument performance.
>
>   * •
>
> Data Process
>
>     * –
>
> After spacecraft operation, telemetry is converted into the FITS format[4]
> by the pre-pipeline (PPL) process, and then higher-level calculations, such
> as filling time, coordinate, and energy information (PI[9]), are performed
> by the pipeline (PL) process. (Note that the details of the PPL and PL are
> described later in Section 6.2.1.) The products of the PPL and PL are
> archived and distributed to GOs. Execution of the PPL and PL is performed by
> the MOT supported by the SOC and the SDC, respectively.
>
>     * –
>
> The products of the PPL and PL are checked by the SOT before the
> distribution to GOs.
>
>   * •
>
> Archive
>
>     * –
>
> The products of the PPL and PL checked by the SOT are archived both in JAXA
> and NASA archive centers, the Data ARchive and Transmission System (DARTS)
> and the High Energy Astrophysics Science Archive Research Center (HEASARC),
> respectively.
>
>     * –
>
> When the products are archived, the SOT notifies the readiness to GOs.
>
>

###  5.4 Details of Tasks in Step-3 After Data Distribution

The tasks in Step-3 after data distribution are of Types SO3 and SO4 (defined
in Section 2.1), which can be divided into the following two categories. The
details are as follows.

>   * •
>
> User Support
>
>     * –
>
> The SOT prepares and operates webpages for GOs to provide information on GO
> proposals, operation schedules and logs, analysis documents, etc. Such
> researcher website for XRISM are prepared at the three agencies, JAXA, NASA,
> and ESA, separately but main contents are synchronized.
>
>     * –
>
> The SOT operates the agency Help Desks for handling questions from GOs.
>
>     * –
>
> The SOT prepares documents related to the data analyses of XRISM, such as
> analysis walkthrough, analysis manuals, and descriptions of instruments.
>
>     * –
>
> Education and Public Outreach (EPO) activities are performed by other
> institutes in JAXA or NASA, and the SOT supports such activities for XRISM.
>
>   * •
>
> PVO Activities
>
>     * –
>
> As defined in the task division of the calibration activities in Table 2,
> the SOT calibrates payload instruments regularly with the instrument teams
> using the in-orbit calibration targets or trend archive data (i.e., non-
> scientific data obtained for performance monitoring, such as data during
> earth occultation of normal operations).
>
>     * –
>
> In addition to the daily performance checks in Step-2, the SOT also monitors
> the instrument performance monthly.
>
>     * –
>
> The SOT enhances the instrument performance (such as improving the pointing
> accuracy and tuning of the good time interval) by checking short-/long-term
> trends and correlations between telemetry items and performance parameters.
> The output of such performance enhancement activities are implemented as an
> analysis thread or a new analysis tool, which is provided to GOs via the
> researcher website or the software archive.
>
>     * –
>
> During the daily data checks in Step-2 (Table 4), the SOT carries out
> further analyses to search for possible new transient objects within the
> field of view of Xtend using the quick-look data products and the final
> products. If the SOT finds a transient and the SMO considers it worth
> reporting, the SOT posts a quick report to the Astronomers Telegram (ATEL,
> http://www.astronomerstelegram.org/) from quick-look data products, and
> updates the detailed information using the final products if necessary. This
> activity is performed during the Initial-Calibration and PV Period under the
> permission of the observation PI during the Nominal Observation Period.
>
>

##  6 Preparation for Science Operations on the Ground

Following the science operations plan in Section 5, the MOPT (Section 3.2)
prepares science operations well before the launch along the timeline of the
science operations (Section 6.1): i.e., the MOPT prepares the OTs for science
operations (Section 6.2), science operations manual, and website (Section
6.3), and performs the PVO activities from before launch (Section 6.4). This
section describes the preparation status at the end of the 1) Before PFT Phase
(see Section 3.3) on the ground.

###  6.1 Timeline of the Science Operations Preparation

In each operation phases defined in Section 3.3, the MOPT and SOT prepare
and/or perform the science operations following the timeline shown in Table 5,
respectively. Before the launch, on ground, the MOPT prepares the descriptions
for science operations, such as the science operations plan, the manual for
the science operations, and the detail design of the operation tools (OTs),
etc, and develops and verifies the OTs in the before PFT phase, and then
trains the SOT members in the PFT phase. The list of targets to be observed
during the initial-calibration and PV period in the Nominal operations phase
(Section 3.3) is released during this phase. The PV target list has been
released on Febrary 15, 2021. After the launch of the satellite, the SOT
supports the critical and commissioning operations by the MOT and ITs during
the initial operations phase, prepares the data process of the first light
object at the end of commissioning period, and, after that, performs the
nominal operations in the Normal Operations Phase.

Table 5: Timeline of the XRISM science operations.The type of science operations are defined in Section 2.1, the CP($\dagger$) represents the calibration plan of instruments, and the ERS($\ddagger$) represents the Early Release Science targets in the PV period.) Phase | XRISM | SO1 | SO2 | SO3 | SO4
---|---|---|---|---|---
|  | (GO, Data) | (Software) | (Support) | (PVO)
Before PFT | Reviews | Prepare science operations plan and manual | Review CP†
|  | Design and verify OTs |
| Release PV list |  |  | Open webpages | Prepare IFCP
PFT | PFT | Verify OTs | Process data | Update Guide |
|  | Update manual | Update manual | Update webpages |
| end-to-end | Training Science Operations members
Launch | launch | Preparation of call for proposal | Update webpages
|  | Release 1st version of XRISM software and calibration database
Initial | Critical Period
|  | Internal process | Check telemetry |  |
| Commissioning Period
|  | Internal process | Check telemetry | Update webpages |
| First light | Process data |  | Press release |
Nominal | Initial-Calibration/PV Period
|  | Process data | Check telemetry |  | Calibration
| ERS‡ | Release ERS |  | Update webpages | Search transient
|  | Preparation of call for proposal
| Nominal observation Period
|  | Process data | Check telemetry | Update webpages | Calibration
|  | Release PV/GO |  | Helpdesk | Search transient
|  | Preparation of call for proposal
|  | Release/update XRISM software and calibration database
Latter | Observation | Continue the activities in the Nominal Operations Phase

###  6.2 Tools for Science Operations and Detailed Designs

The tools and database required for the XRISM Science Operations by the steps
(Section 5.1) are summarized in Table 6. The responsibilities for these
tools/databases in the subgroups within the SOT are also shown. Among these 10
OTs, the proposal submission tools, planning tool, PL, calibration database,
and analysis tools (OT01, OT02, OT06, OT07, and OT09, respectively) are
developed by the SDC, and the details of these OTs are described in
Loewenstein et. al. (2020)[23]. Hereafter, this paper describes the details of
the observation database, QLDP, PPL, and archive quick-viewing tool (OT03,
OT04, OT05, OT08) in Section 6.2 and the conversion tools for the researcher
webpages (OT10) in Section 6.3.

Table 6: List of science OTs and databases. Steps are defined in Section 5.1. Tool ID | Tools/Database | Step | SOT subgroups
---|---|---|---
OT01 | Proposal submission tools | Step-1 | SDC, (SOC, ESAC as users)
OT02 | Planning tools | Step-1 | SDC with SOC
OT03 | Observation database | Step-1,2 | SOC
OT04 | QLDP | Step-2 | SOC
OT05 | PPL | Step-2 | SOC
OT06 | PL | Step-2 | SDC
OT07 | Calibration database | Step-2,3 | SDC with SOC and ITs
OT08 | Archive quick-viewing tool | Step-2 | SOC, (HEASARC), ESAC
OT09 | Analysis tools | Step-3 | SDC with SOC and ITs
OT10 | Conversion tools for researcher webpages | Step-1, -2, and -3 | SOC

####  6.2.1 Pre-pipeline and Pipeline Process

Since the raw telemetry from the spacecraft is a collection of space packets,
which are unreadable by the standard analyis tools used in high energy
astronomy, they need to be converted into the standard FITS format[4] for
distribution to GOs, as described in Angelini et al. (2018)[5]. In addition,
the GOs need calibrated information on variables such as time, coordinates,
and pulse height invariant (PI[9]; energy information), which are filled in by
the data processing. This corresponds to the the functions of a) format
conversion and b) filling calibration columns. The data processing is divided
into two steps, PPL (OT05 in Table 6) and PL (OT06 in Table 6) as shown in
Figure 4 to archive functions a) and b), respectively.

Figure 4: Schematic view of the flow of data processing for the XRISM (and
Hitomi, Suzaku) in Step-2 and OTs.

The raw telemetry from the spacecraft is stored in the SIRIUS database, and
all the information regarding approved targets, instrument configuration, and
other spacecraft information are stored in the observation database (ODB in
Figure 4; OT03 in Table 6). The PPL accesses the SIRIUS database via the Space
Data Transfer Protocol (SDTP) to retrieve the telemetry and first converts the
telemetries into a raw packet telemetry (RPT) file, which is a simple dump of
the series of space packets in the variable-length FITS format, using the
information from the observation database (OT03). In the second step, the PPL
interprets the telemetry attributes in the space packets using the telemetry-
description database, shown as the ”Spacecraft Information Base version 2
(SIB2) database” in Figure 4, and converts the RPT into the First FITS Files
(FFFs) with meaningful columns. The FITS header keywords of FFFs represent the
instrument configurations as identified from the telemetry and observation
database. After the PPL process, the PL fills in the calibration columns, such
as time, coordinates, and PI, using the FITS tools called ftools in the
HEASOFT XRISM package released from HEASARC by using the calibration database
(denoted as ”CALDB” in Figure 4; OT07 in Table 6), and stores them in the
Second FITS Files (SFFs). The PL process then continues to extract the
cleaned-event FITS for analyses from the SFFs by deleting low-quality events
and by selecting good-time intervals.

The key point of this procedure is that the FFFs have the same format as SFFs
(i.e., FITS columns for time, coordinates, and PI are already prepared as
blank columns in the FFF stage) and the CALDB and ftools are all distributed
to GOs (i.e., public), so that the GOs can reprocess the SFF with the latest
calibration information by themselves. This concept was established in the
Suzaku Science Operations, and also used in Hitomi successfully. The XRISM
data process also follows this procedure.

The PPL requires inputs from the mission operation information, such as the
definition of the telemetry format, the orbital estimation, the attitude
determination, the time calibration, etc. Therefore, the PPL software for the
XRISM mission is prepared and executed by the SOC at JAXA, where the mission
operations are performed and the operation information is easily accessible
from the SOT. The FFFs are then sent via a data transfer system protocol
(DTS222https://heasarc.gsfc.nasa.gov/dts/) to the SDC, and processed in the PL
at the SDC, as already described in the task division (Section 4.2).

####  6.2.2 Design of Tools for the PPL and QLDP

Since the QLDP (OT04 in Table 6; Section 5.3) is the simplified version of the
PPL (OT05), these tools can be shared with each other just by switching the
execution mode. The PPL and QLDP are designed to have a structure consisting
of three stages of tool, modules, and top-level script, as shown in Figure 5,
and the difference between PPL and QLDP is designed to be absorbed in the top-
level script. A tool is the smallest unit of the software code, and a module
is a collection of tools to achieve one function (for example, generation of
RPT, generation of time calibration fits, etc.). The top-level script controls
the process flow of multiple modules using the configuration files, with which
the detail flow of PPL or QLDP are described.

Since XRISM is a recovery mission for Hitomi, the tools have already been
developed and verified and can be reused for XRISM. However, the Hitomi PPL is
not easy to maintain because the Hitomi SCT (section 2.4) were forced to use
it during the Commissioning Phase for the unplanned spacecraft problems, and
the Hitomi PPL has many patches for complex hardware modes during the
Commissioning Phase, even though it was well designed for use in the Nominal
Operations Phase. Therefore, the MOPT decided to re-organize the PPL flow
diagram and to newly develop the modules and top-level script for XRISM.

Figure 5: Structure of the PPL and QLDP.

In detail, the MOPT defined the following 14 modules corresponding to the 14
functions required for the PPL and QLDP of XRISM. Using these modules, the
typical flow diagrams for the PPL and QLDP in the flight configuration are
designed as shown in Figures 6 and 7, respectively. In the QLDP, several
modules are omitted in the process flow and the access point to retrieve the
telemetry via SDTP is different from that for PPL, as well as the inputs for
the time assignment tool and orbital-file generation tool. In addition, the
MOPT also identified 17 use cases for the ground tests and operations in
orbit.

>   1. 1.
>
> Data processing setup module
>
>   2. 2.
>
> Raw telemetry packet processing module
>
>   3. 3.
>
> Spacecraft-bus data processing module
>
>   4. 4.
>
> Resolve data processing module
>
>   5. 5.
>
> Xtend data processing module
>
>   6. 6.
>
> Time calibration data processing module
>
>   7. 7.
>
> Time assignment process/library
>
>   8. 8.
>
> Attitude data processing module
>
>   9. 9.
>
> Orbit data processing module
>
>   10. 10.
>
> Operation-command data processing module
>
>   11. 11.
>
> Good-time-interval generation and slew/pointing-division module
>
>   12. 12.
>
> Common header management module
>
>   13. 13.
>
> Observation-database interface module
>
>   14. 14.
>
> Package processing module
>
>

Figure 6: Flowchart of the PPL process. Figure 7: Same as Figure 6, but for
QLDP. Gray colors represent the modules omitted in QLDP.

####  6.2.3 Design of Tools for the Archive Quick-Viewing Tool

For the data archive for XRISM, it is important to define the division of
tasks between the XRISM project and the archive centers of the agencies. In
the archive activity at JAXA, the MOPT defined the task division between the
SOT and the Center for Science Satellite Operation and Data Archive (C-SODA)
at ISAS/JAXA, as summarized in Table 7. In principle, content is provided by
the XRISM project and preparation of mission-independent systems and
infrastructure are made by C-SODA at ISAS/JAXA.

Table 7: List of tasks for archiving XRISM data at JAXA and division of tasks between SOT and C-SODA. Category | Task | Team
---|---|---
Data preparation | Preparation of FITS files |  XRISM SOT
| Definition of distribution range |  XRISM SOT
| Definition of proprietary period |  XRISM SOT
| Preparation of data storage | JAXA C-SODA
| Preparation of public webpages | JAXA C-SODA
Project introduction page | Preparation of contents |  XRISM SOT
| Preparation of public webpages | JAXA C-SODA
Public data list | Generation of the list and HTML |  XRISM SOT
| Preparation of public webpages | JAXA C-SODA
Data search | Preparation of search engine | JAXA C-SODA
| Preparation of metadata |  XRISM SOT
| Installation of XRISM metadata | JAXA C-SODA
Quick viewing | Preparation of quick-viewing system | JAXA C-SODA
| Preparation of HiPS format of XRISM |  XRISM SOT
| Installation of XRISM HiPS data | JAXA C-SODA

Among the tasks by the SOT, the data preparation tasks (Table 7) are performed
by the PPL and related tools OT05, and the project introduction page and
public data list tasks are performed manually by the SOT. Therefore,
additional OTs for the archive activity are a generation tool for a) metadata
for data search and b) hierarchical progressive survey (HiPS) data for quick
viewing, which are identified as OT08 in Table 6.

###  6.3 Preparation for User Support

Researcher webpages are required for communication with GOs for the user-
support activities listed in Table 4, and are planned to be operated in three
centers: SOC, SDC, and ESAC. In the first version, the following content is
listed on the researcher webpages at SOC.

>   * •
>
> Top page, News and Announcements
>
>   * •
>
> About XRISM (XRISM documents, workshops, publication list, resources)
>
>   * •
>
> Proposer (GO proposal documents, response files, generic ToO request,
> approved target list)
>
>   * •
>
> Observers (short-term/long-term operation plan, spacecraft operation log)
>
>   * •
>
> Analysis (manual, link to archive web, link to download page for
> software/calibration database)
>
>   * •
>
> XRISM Help Desk (FAQ, proposal plan support, analysis questions, XRISM
> workshops)
>
>   * •
>
> Useful links (link to general public XRISM website, DARTS archive website,
> HEASARC website, ESAC website)
>
>

The MOPT prepares the web servers and the tools for filling the content of the
pages of observation plan and spacecraft operation log semi-automatically.
These tools are identified as OT10 in Table 6. The JAXA researcher webpage was
opened on 1 November 2020 at https://xrism.isas.jaxa.jp/research/ and is used
for announcements to GOs before launch and is to be activated on the science
operations after launch.

###  6.4 Preparation of PVO Activities

In principle, all the PVO activities are performed by all the science members
of the XRISM team before launch. The items for the MOPT to prepare for the
science operations in orbit are the detailed procedure for opening these
efforts to GOs via the XRISM software, calibration database, and the analysis
method, which are already covered in Section 5.

In order to perform the four science operations tasks described in Section 5.4
and Table 4, the MOPT prepares the following items.

>   * •
>
> Calibration analyses
> As defined in Table 2, the in-orbit calibration items and procedures are
> prepared by the instrument teams, and the SOT also analyzes the plan to
> observe the in-orbit calibration observations with the instrument teams.
> Therefore, the MOPT prepares the training procedure for understanding the
> calibration items and procedures with the instrument team before launch.
>
>   * •
>
> Monthly performance check
> Daily and monthly checks of instrument performance require no special tools
> other than the standard analysis software. In this sense, the MOPT has no
> plan to prepare the tools before launch. If the MOPT identifies additional
> tools during the rehearsal of instrument operation on ground in the PFT
> Phase (Section 3.3), the MOPT will prepare the tools from this phase.
>
>   * •
>
> Development of analysis threads and tools
> All of the standard analysis will be performed using the public standard
> tools (OT08 in Table 6). For the monthly or daily performance checks, the
> SOT tries to study and identify new proprieties or behaviors of instruments
> which affect the instrument performance. If the SOT finds a way to enhance
> the instrument performance using these items, the SOT will implement the
> procedure as an analysis thread or prepare a new public tool using the newly
> found algorithm.
>
>   * •
>
> Xtend transient search
> As described in Section 5.4, the SOT carries out further analyses to search
> for possible new transient objects within the field of view of the Xtend and
> posts a quick report to the ATEL, under the permission of the observation
> PIs. The MOPT prepares the detailed procedure for this operation to obtain a
> quick response and the automatic search tool of transients.
>
>

##  7 Summary

In preparation of science operations of the XRISM mission, we reviewed the
lessons learned from past X-ray missions in Section 2 to identify
recommendations for the XRISM Science Operations (Section 2.5), which are
considered as part of the operations concept (Section 3). Based on the
operations concepts, we designed the structure of the SOT, interfaces among
subgroups, management structure, and data policy in Section 4 and established
a detailed plan of the science operations as described in Section 5. As the
final step of preparation of science operations, we identified 10 OTs and
developed them as summarized in Section 6 before launch.

###### Acknowledgements.

We thank Dr. Masa Sakano (WiseBabel Ltd.) for the detail design and
implementation of the top-level-script of PPL, and Ms. Seiko Sakurai (Saitama
University) for the implementation of the XRISM researcher’s website. We also
thank Dr. Matteo Guainazzi, Dr. Jan-Uwe Ness (ESA), Dr. Katja Pottschmidt, and
Dr. Tess Jaffe (NASA/GSFC) for the discussions on the user support activities
in collaboration with ESAC and HEASARC. This work was supported in part by
JSPS KAKENHI Grant Numbers JP18H04571 and JP20K04009 (YT), JP19K14762 (MS),
JP17K14289 (MN), JP20K20935 (SK and MT), 17K05392 (Y. T.), and NASA Grant
Number 80NSSC20K0737.

##  Appendix A Acronyms and Abbreviations

Abbreviations and acronyms used in this paper are shown in Table 8.

Table 8: Acronyms and Abbreviations. Abbreviations | Acronyms | Section
---|---|---
ATEL | the Astronomers Telegram | section 5.4
C-SODA | the Center for Science Satellite Operation and Data Archive | section 6.2.3
CALDB | the calibration database | section 6.2.1
DARTS | the Data ARchive and Transmission System | section 5.3
EPO | Education and Public Outreach | section 5.4
ESA | the European Space Agency | section 1
ESAC | the European Space Astronomy Center | section 4.2
FFF | the First FITS Files | section 6.2.1
FITS | flexible image transport system | section 1
FOV | field of view | section 1
GO | Guest Observer | section 2.1
GOF | Guest Observer Facility | section 2.2
HEASARC | the High Energy Astrophysics Science Archive Research Center | section 5.3
IACHEC | the International Astrophysical Consortium for High Energy Calibration | section 2.3
ISAS | the Institute of Space and Astronautical Science | section 2.2
IFCP | the In-flight Calibration Planning | section 4
IT | instrument team | section 2.2
JAXA | the Japan Aerospace Exploration Agency | section 1
MOPT | the Mission Operations Planning Team | section 3.3
MOT | the Mission Operations Team | section 3.2
NASA | National Aeronautics and Space Administration | section 1
OC1 – OC3 |  XRISM Operations Concept | section 3.1
OT | operations tool | section 6.1
PFT | Proto-Flight Test | section 3.3
PI | pulse-height invariant | section 2.3
PL | pipeline | section 5.3
PM | the Project Manager | section 4.1
PPL | the pre-pipeline | section 5.3
PV | the performance verification | section 2.4
PVO | Performance verification and optimization | section 2.1
QLDP | Quick-look Data Process | section 5.3
RPT | raw packet telemetry | section 6.2.1
SCT |  Hitomi software and calibration team | section 2.4
SDTP | the Space Data Transfer Protocol | section 6.2.1
SFF | the Second FITS Files | section 6.2.1
SIB2 | Spacecraft Information Base version 2 | section 6.2.1
SMO | the Science Management Office | section 3.2
SO1 – SO5 | Science Operations Type | section 2.1
SOT | the Science Operations Team | section 3.2
ToO | time of opportunity | section 3.2
XRISM | The X-Ray Imaging and Spectroscopy Mission | section 1

## References

  * [1] M. Tashiro, H. Maejima, K. Toda, et al., “Concept of the X-ray Astronomy Recovery Mission,” in Space Telescopes and Instrumentation 2018: Ultraviolet to Gamma Ray, J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10699, 1069922 (2018).
  * [2] T. Takahashi, M. Kokubun, K. Mitsuda, et al., “Hitomi (ASTRO-H) X-ray Astronomy Satellite,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 021402 (2018).
  * [3] T. Makoto, M. Hironori, T. Kenichi, et al., “Status of x-ray imaging and spectroscopy mission (XRISM),” in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., 11444, 293 – 303, International Society for Optics and Photonics, SPIE (2020).
  * [4] R. J. Hanisch, A. Farris, E. W. Greisen, et al., “Definition of the Flexible Image Transport System (FITS),” A&A 376, 359–380 (2001).
  * [5] L. Angelini, Y. Terada, M. Dutka, et al., “Astro-H/Hitomi data analysis, processing, and archive,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 011207 (2018).
  * [6] Y. Terada, M. Holland, M. Loewenstein, et al., “Detail plans and preparations for the science operations of the XRISM mission,” in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., 11444, 926 – 945, International Society for Optics and Photonics, SPIE (2020).
  * [7] Y. Tanaka, H. Inoue, and S. S. Holt, “The X-Ray Astronomy Satellite ASCA,” PASJ 46, L37–L41 (1994).
  * [8] K. Mitsuda, M. Bautz, H. Inoue, et al., “The X-Ray Observatory Suzaku,” PASJ 59, S1–S7 (2007).
  * [9] Y. Terada, S. Watanabe, M. Ohno, et al., “Development of a Monte Carlo Simulator for the Astro-E2 Hard X-Ray Detector (HXD-II),” IEEE Transactions on Nuclear Science 52, 902–909 (2005).
  * [10] Y. Terada, A. Bamba, J. S. Hiraga, et al., “The 7-Steps of the Data Analysis,” Progress of Theoretical Physics Supplement 169, 312–315 (2007).
  * [11] K. Madsen et al., “IACHEC: The International Astrophysical Consortium of High Energy Calibration,” in ”SPIE Astronomical Telescopes + Instrumentation (AS20)”, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 000, 11444–22 (2020).
  * [12] Y. Terada, S. Yamaguchi, S. Sugimoto, et al., “Time assignment system and its performance aboard the Hitomi satellite,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 011206 (2018).
  * [13] M. A. Leutenegger, M. Audard, K. R. Boyce, et al., “In-flight verification of the calibration and performance of the ASTRO-H (Hitomi) Soft X-ray Spectrometer,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 021407 (2018).
  * [14] M. E. Eckart, J. S. Adams, K. R. Boyce, et al., “Ground calibration of the Astro-H (Hitomi) soft x-ray spectrometer,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 021406 (2018).
  * [15] M. Tsujimoto, T. Okajima, M. E. Eckart, et al., “In-flight calibration of Hitomi Soft X-ray Spectrometer. (3) Effective area,” PASJ 70, 20 (2018).
  * [16] C. A. Kilbourne, M. Sawada, M. Tsujimoto, et al., “In-flight calibration of Hitomi Soft X-ray Spectrometer. (1) Background,” PASJ 70, 18 (2018).
  * [17] M. E. Eckart, K. R. Boyce, G. V. Brown, et al., “Calibration of the microcalorimeter spectrometer on-board the Hitomi (Astro-H) observatory (invited),” Review of Scientific Instruments 87, 11D503 (2016).
  * [18] R. Iizuka, T. Hayashi, Y. Maeda, et al., “Ground-based x-ray calibration of the Astro-H/Hitomi soft x-ray telescopes,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 011213 (2018).
  * [19] Y. Maeda, T. Sato, T. Hayashi, et al., “In-flight calibration of the Hitomi Soft X-ray Spectrometer. (2) Point spread function,” PASJ 70, 19 (2018).
  * [20] H. Mori, T. Miyazawa, H. Awaki, et al., “On-ground calibration of the Hitomi Hard X-ray Telescopes,” Journal of Astronomical Telescopes, Instruments, and Systems 4, 011210 (2018).
  * [21] K. Hagino, K. Nakazawa, G. Sato, et al., “In-orbit performance and calibration of the Hard X-ray Imager onboard Hitomi (ASTRO-H),” Journal of Astronomical Telescopes, Instruments, and Systems 4, 021409 (2018).
  * [22] E. D. Miller, M. Sawada, M. Guainazzi, et al., “Planning in-flight calibration for XRISM,” in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., 11444, 325 – 344, International Society for Optics and Photonics, SPIE (2020).
  * [23] M. Loewenstein, R. S. Hill, M. P. Holland, et al., “The XRISM science data center: optimizing the scientific return from a unique x-ray observatory,” in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, J.-W. A. den Herder, S. Nikzad, and K. Nakazawa, Eds., 11444, 910 – 925, International Society for Optics and Photonics, SPIE (2020).

Yukikatsu Terada is an associated professor at Saitama University, cross-
appointed with Japan Aerospace Exploration Agency. He received his BS and MS
degrees in physics, and a PhD degree in science from the University of Tokyo
in 1997, 1999, and 2002, respectively. He is the leader of the Science
Operations Team of the X-Ray Imaging and Spectroscopy Mission (XRISM) project
now.

Biographies and photographs of the other authors are not available.


Social Media |  0.45% |  twitter, facebook, social, media, instagram, post, people, account, also, pic, tweet, share, news, online, posted, video, users, page, wrote, shared
Comparative Analysis |  0.42% |  one, also, however, two, may, different, many, used, example, well, often, first, part, although, another, time, known, fact, various, number

###  A.3 Ethical discussion

At the beginning of the project, we reflected on ethical principles111111
https://huggingface.co/blog/ethical-charter-multimodal guiding the project,
including the creation of the dataset, in order to incorporate ethical values
we agreed on. These values motivated the careful crafting of the content
filters. For instance, we used the Spawning API to respect as much as possible
the consent decisions of content creators or iterated significantly on filters
around pornographic content.

Exploring large-scale corpora is often a tedious process which contributes to
the lack of transparency and lack of documentation around these artifacts.
With that in mind, we built an interactive visualization121212
https://atlas.nomic.ai/map/f2fba2aa-3647-4f49-a0f3-9347daeee499/ee4a84bd-f125-4bcc-a683-1b4e231cb10f
of OBELICS which allows browsing through a subset (11M documents) of the
dataset and navigate the different topics covered. Yet, we note that despite
our efforts, OBELICS contains a small proportion of documents that are not
suitable for all audiences. For instance, one might find the cluster named
“Sex” which predominantly contains descriptions of pornographic movies along
with pornographic images. Other clusters would contain advertising for sex
workers, or reports of violent shootings. In our experience, these documents
represent a small proportion of all the documents.

Due to the nature of our dataset (multimodal documents extracted from the
web), OBELICS inherits the same ethical concerns of unlabeled text corpora
crawled from the web: difficulty to document/inspect, presence of unintended
biases, under-representation of certain demographics, etc. These concerns have
been well documented for text corpora (Biderman and Scheirer, 2020; Bender et
al., 2021). Data audits have shed light on the some limitations and unintended
biases contained in these text corpora (Caswell et al., 2020; Dodge et al.,
2021). The augmentation of text corpora with interleaved images is a recent
development of multimodal machine learning. We hope that our dataset along
with exploration tools will serve as a solid ground for endeavors such as data
audits. Existing works auditing large-scale multimodal datasets have focused
on image-text pairs datasets (Birhane et al., 2021) and highlight how curation
and filtering decisions lead to biases (including racism and misogyny) in the
resulting pairs. We believe that interleaved image-text datasets will play a
significant role in the development of increasingly more capable multimodal
models, and having large-scale versions of these datasets that are
transparent, maintained and in open-access is critical.

We also have evaluated the trained models as part of a red-teaming effort and
a systematic evaluation of the generations produced by the model compared
across the axis of gender and race. More specifically, the model was
separately prompted to write a resume, a dating profile, and a headline about
a person’s recent arrest based on their appearance. We studied the generations
and analyzed the trends for each protected characteristic using FairFace
(Kärkkäinen and Joo, 2021) and StableBias (Luccioni et al., 2023). The details
of these evaluations and insights are made public as part of the model
release. As an example, the model trained on OBELICS associates men more
frequently than women with terms like “financial”, “development”, “product”,
and “software”.

###  A.4 Building the Model

####  A.4.1 Architecture Details

We closely follow the Flamingo architecture introduced in Alayrac et al.
(2022). To form the model, we combine a pre-trained image encoder, a pre-
trained language model, and add newly initialized parameters of the form of
Perceiver blocks (Jaegle et al., 2021) and Transformer-based cross-attentions
blocks inserted within the language model every 4 layers.

The pre-trained backbones are frozen during the training, and only the new
parameters are updated along with the embeddings of additional tokens.

Following Dehghani et al. (2023), we apply a layer normalization on the
projected queries and keys of both the Perceiver and cross-attention blocks,
which improved training stability in our early experiments. We use the RMSNorm
implementation (Zhang and Sennrich, 2019) for the layer normalization.

Total | Trainable | Language Model | Vision Model | Perceiver | Cross-Attentions
---|---|---|---|---|---
9B | 1.5B | 7B | 630M | 126M | 1.4B
80B | 14B | 65B | 630M | 126M | 13.9B

Table 6: Breakdown of model parameters. We use LLaMA (Touvron et al., 2023)
for the language backbone and OpenCLIP (https://laion.ai/blog/large-openclip/)
for the vision backbone.

####  A.4.2 Training Details

We roughly use the same set hyper-parameters for all the runs presented in
Figure 7 and Table 1, as detailed in Table 7. The training of IDEFICS uses a
larger batch size and examples of longer sequence length. In all experimental
runs, we employ the AdamW optimizer (Loshchilov and Hutter, 2017) and
incorporate an auxiliary loss, denoted as $z\\_loss=10^{-3}\times log^{2}(Z)$,
to encourage the softmax normalizer $log(Z)$ to get closer to 0 (Chowdhery et
al., 2022). We use gradient clipping of $1.0$.

|  Parameters |  IDEFICS-80B |  IDEFICS-9B
---|---|---|---
Perceiver Resampler |  Number of Layers |  6 |  6
|  Number of Latents |  64 |  64
|  Number of Heads |  16 |  16
|  Resampler Head Dimension |  96 |  96
Model |  Language Model Backbone |  Llama-65b |  Llama-7b
|  Vision Model Backbone |  laion/CLIP-ViT-H-14-laion2B-s32B-b79K |  laion/CLIP-ViT-H-14-laion2B-s32B-b79K
|  Cross-Layer Interval |  4 |  4
Training |  Sequence Length |  1024 |  1024
|  Effective Batch Size (# of tokens) |  3.67M |  1.31M
|  Max Training Steps |  200K |  200K
|  Weight Decay |  0.1 |  0.1
|  Optimizer |  Adam(0.9, 0.999) |  Adam(0.9, 0.999)
|  Gradient Clipping |  1.0 |  1.0
|  Z-loss weight |  1e-3 |  1e-3
Learning Rate |  Initial Max |  5e-5 |  1e-5
|  Initial Final |  3e-5 |  6e-6
|  Decay Schedule |  Linear |  Linear
|  Linear warmup Steps |  2K |  2K
Large-scale Optim. |  Gradient Checkpointing |  True |  True
|  Precision |  Mixed-pres bf16 |  Mixed-pres bf16
|  ZeRO Optimization |  Stage 3 |  Stage 3
Table 7: Training Hyper-Parameters

During the training, two models – IDEFICS and the 9B-parameter model trained
on LAION + OBELICS – encountered unrecoverable loss spikes. As a remedial
measure, we restarted the training from a checkpoint before the spike,
shuffled the data and optionally reduced the learning rate. Both models
underwent exactly three restarts within the training duration.

The four runs conducted have distinct data mixtures as detailed in Table 9,
and Tabel 8 gives the number of tokens and images in the different datasets.
Each run involves training on a mixture of web documents and image-text pairs.
A sampling probability $p$ determines the mixture of these two data sources,
which influences the frequency of batches originating from web documents
versus those from image-text pairs.

For IDEFICS and IDEFICS-9B, the web-document dataset includes both OBELICS and
Wikipedia, and the image-text pair dataset included LAION and Public
Multimodal Dataset (PMD) (Singh et al., 2022). Given Wikipedia and PMD’s
higher quality but lower number of examples, we repeat PMD three times and
Wikipedia three times.

We used a deduplicated version of LAION (Webster et al., 2023) for all the
runs where this dataset was used.

Data Source |  Data Type |  # Tokens in Source |  # Images in Source |  Epochs
---|---|---|---|---
OBELICS |  Unstructured Multimodal Web Documents |  114.9B |  353M |  1
Wikipedia |  Unstructured Multimodal Web Documents |  3.192B |  39M |  3
LAION |  Image-Text Pairs |  29.9B |  1.120B |  1
PMD |  Image-Text Pairs |  1.6B |  70M |  3
Table 8: Number of tokens and images in the different datasets used for the training of IDEFICS. Model | OBELICS | Wikipedia | LAION |  PMD
---|---|---|---|---
9B-parameter model, OBELICS + LAION | 50% | 0% | 50% | 0%
9B-parameter model, OBELICS only | 100% | 0% | 0% | 0%
9B-parameter model, LAION only | 0% | 0% | 100% | 0%
IDEFICS-9B | 73.85% | 6.15% | 17.18% | 2.82%
IDEFICS | 73.85% | 6.15% | 17.18% | 2.82%
Table 9: Breakdown of the dataset mixtures used. Percentages correspond to the
effective number of tokens seen from each dataset.

####  A.4.3 Compute Details

We train the 9B-parameter models on OBELICS-only and LAION-only on 32 80GB
A100 GPUs, and on OBELICS + LAION on 64 80GB A100s, for approximately 6 days.
These 3 trainings have the same effective batch size. We train IDEFICS on 512
80GB A100 GPUs and IDEFICS-9B on 128 80GB A100 GPUs for about 14 days each.
The compute infrastructure is hosted on an AWS cluster located in Oregon.

####  A.4.4 Evaluation

To ensure fair comparisons against Flamingo (Alayrac et al., 2022), we make
sure that we are using the same evaluation splits for each benchmark. We
evaluate the models using an in-context learning approach (Brown et al.,
2020), with random in-context examples. For the 0-shot evaluations, as in
Alayrac et al. (2022), we use 2 random priming in-context examples but without
passing the associated images. We systematically use different data splits to
select the best-performing prompt (which involves creating validation sets
from the training sets, following the methodology proposed by Alayrac et al.
(2022)). Table 10 lists the prompts used for each model and task.

For the classification tasks (HatefulMeme (Kiela et al., 2020), IIIT-5k
(Mishra et al., 2012)), we use rank classification, i.e. we compute the log
probability of the prompt followed by each of the labels individually, and
select as the predicted label the one with the highest probability.

For the image captioning (COCO (Lin et al., 2014), Flickr30k (Young et al.,
2014)) and visual question answering tasks (VQAv2 (Antol et al., 2015), OKVQA
(Marino et al., 2019), TextVQA (Singh et al., 2019), VizWiz (Gurari et al.,
2018)), we report evaluation in the open-ended setup. We use the greedy
decoding as we found that it increased the performance. However, we observe
that the models tend to generate long answers. To truncate the generated
caption or answer, unless specified otherwise, we use a list of manually
selected stop words. For VisDial, since the evaluation metric is NDCG, we
instead rank the possible candidates for each question.

The VQA tasks comporting a high proportion of questions with a single-word
answer, it was beneficial for the 9B-parameter model trained on LAION only to
keep the first word of the generated answer as the prediction to boost its
performance.

Task |  Model |  Prefix prompt |  Example prompt |  Stop words
---|---|---|---|---
VQAv2
OKVQA
TextVQA |  IDEFICS
IDEFICS-9B
9B LAION only
9B OBELICS only
9B LAION \+ OBELICS |  {bos_token}Instruction: provide an answer to the question. Use the image to answer.\n |  Image:{token_around_image}{image_token}{token_around_image}Question: {question} Answer: {answer}\n |  "Question", "User", "Image", "task", "What", "Who", "When", "Where", "Why", "How"
COCO
Flickr30k |  IDEFICS
IDEFICS-9B
9B OBELICS only
9B LAION \+ OBELICS |  {bos_token} |  Image:{token_around_image}{image_token}{token_around_image}Caption: {caption}\n |  "Caption", "Description", "User", "Image", "task"
COCO
Flickr30k |  9B LAION only |  {bos_token}Instruction: provide a short caption of the input image.\n |  Image:{token_around_image}{image_token}{token_around_image}Image description: {caption}\n |  "Caption", "Description", "User", "Image", "task"
Hateful-Memes |  IDEFICS
IDEFICS-9B
9B LAION only
9B OBELICS only
9B LAION \+ OBELICS |  It’s a conversation between a human, the user, and an intelligent visual AI, Bot. The user sends memes with text written on them, and Bot has to say whether the meme is hateful or not. |  {token_around_image}{image_token}{token_around_image}is an image with written "{context}" on it. Is it hateful? Answer: {class_name} |  ✗
IIIT5k |  9B LAION only
9B OBELICS only
9B LAION \+ OBELICS |  ✗ |  {token_around_image}{image_token}{token_around_image}"{class_name}" is written on the picture. |  ✗
VizWiz |  IDEFICS
IDEFICS-9B |  {bos_token}Task: Answer the questions based on the image when possible, otherwise say unanswerable.\n |  Image:{token_around_image}{image_token}{token_around_image}Question: {question} Answer: {answer}\n |  "Question", "User", "Image", "task", "What", "Who", "When", "Where", "Why", "How"
VisDial |  IDEFICS
IDEFICS-9B |  ✗ |  {token_around_image}{image_token}{token_around_image}{caption}. {context}{class_name}. |  ✗
Table 10: We select the prompts from a pool of candidates by evaluating 5
intermediate checkpoints on the query and support validation task sets. To
form the prompt with $N$ priming examples, we concatenate the prefix prompt,
followed by $N$ example prompts filled with data from the priming examples,
and finally the example prompt filled with data from the example to be
evaluated. The data to be replaced is between curly brackets.

####  A.4.5 Additional Experimental Results

0.40.5VQAv2

0.40.5

$0.3$$0.4$OK-VQA

$0.3$$0.4$

$0.1$$0.2$$0.3$TextVQA

$0.1$$0.2$$0.3$

$0.6$$0.8$COCO

$0.6$$0.8$

$0.2$$0.4$$0.6$Flickr30k

$0.2$$0.4$$0.6$

$0.4$$0.6$$0.8$$1$IIIT5K

$0.4$$0.6$$0.8$$1$

$10^{8}$$10^{9}$$10^{10}$0.5# of training tokensHateful Memes

$10^{6}$$10^{7}$$10^{8}$0.5# of training imagesLAION onlyOBELICS onlyOBELICS
\+ LAION

Figure 12: 4-shot performance through the training using LAION only, OBELICS
only and a mixture of both. The training sequences from multimodal documents
and the packed sequences obtained from image-text pairs have different numbers
of images but the same number of tokens. Thus, we plot the performance over
two log x-axes.

In Figure 12, we plot the performance per benchmark for the 9B-parameter
models trained on LAION only, OBELICS only, and a mixture of OBELICS and
LAION. We notice that, even if the training on LAION only is smooth and the
loss keeps decreasing (there are no spikes nor instabilities), performance
starts to decrease after a certain point on visual question answering
benchmarks. We hypothesize that training on image-text pairs can allow a fast
association of concepts between images and texts, but fails to teach the model
more complex reasoning skills required to solve visual question answering. We
tried many different prompt candidates in order to boost the performance of
the model trained on LAION only for the VQA tasks, without much success.

On the other hand, we note that training on image-text pairs yield stronger
performance on image captioning tasks than on multimodal documents only. This
is expected since training and evaluation correspond to the exact same task.

###  A.5 License and Author Statement

We release the dataset under a CC-BY license and Terms of Use that require
disclosure of when the dataset is used for the purpose of training models.
This license is not intended to replace the licenses of the source content,
and any use of content included in the dataset must comply with the original
licenses and applicable rights of its data subjects.

The purpose of this statement is to clarify the responsibilities and
liabilities associated with the use of this dataset. While we have made every
effort to ensure the accuracy and legality of the data contained within this
dataset, we cannot guarantee its absolute completeness or correctness.

Therefore, if any rights, legal or otherwise, are violated through this
dataset, including but not limited to copyright infringement, privacy
violations, or misuse of sensitive information, we, the authors, assume no
liability for such violations.

By utilizing this dataset, you agree that any consequences, legal or
otherwise, arising from using this dataset will be the user’s sole
responsibility. You acknowledge that you will exercise due diligence and
adhere to all applicable laws, regulations, and ethical guidelines when using
the dataset.

By accessing, downloading, or using this dataset, you signify your acceptance
of this statement and your commitment to abide by the terms and conditions of
the CC-BY license.

If you disagree with the terms of this statement or the CC-BY license, you are
not authorized to use this dataset.

The dataset will be hosted and maintained on the Hugging Face Hub.