declare autowrapped toks=pic me, pic dir tree/.style=for
tree=folder,font=,grow’=0,, before typesetting nodes=for tree=edge
label+/.option=pic me,,, , pic me set/.code n args=2#1/.style=inner xsep=2pic
me=pic #2, , pic me set=directoryfolder, pic me set=filefile,

FLAIR: French Land cover from Aerospace ImageRy.

Challenge FLAIR #2:
textural and temporal information for semantic segmentation
from multi-source optical imagery
Anatol Garioud, Apolline De Wit, Marc Poupée, Marion Valette, Sébastien
Giordano, Boris Wattrelos
Institut national de l’information géographique et forestière (IGN), France
<EMAIL_ADDRESS>
## Dataset overview

Figures
---

➔ 20,384,841,728 pixels annotated at 0.20 m spatial resolution
---
➔ 77,762 patches (512$\times$512)
➔ 51,244 satellite acquisitions with broader spatial context
➔ 50 spatio-temporal domains and 916 areas covering 817 km²
➔ 13 semantic classes (+6 optional ones)

Structure
---

pic dir tree, where level=0directory,, for tree= s sep=0.1cm, l sep=1cm, font=
[Dataset [aerial train\val [domain _ year [area [img [IMG_ID.tif, file]] ] ] ]
[sen train\val [domain _ year [area [sen [SEN2_data.npy, file]
[SEN2_masks.npy, file] [SEN2_products.txt, file]] ] ] ] [labels train
[domain_year [area [msk [MSK_ID.tif, file]] ] ] ] [metadata_aerial.json, file]
[centroids_sp_to_patch.json, file] ]
---

## Context

According to a report by the Food and Agriculture Organization of the United
Nations (FAO) in 2015 [1], a significant portion of the world’s soil resources
are in a condition that can be classified as fair, poor, or very poor. This
degradation of soils, coupled with the loss of biodiversity, has far-reaching
implications for the state of ecosystems and their long-term sustainability.
Soils play a vital role in providing a range of ecosystem services. They serve
as natural habitats for numerous plant and animal species, act as a crucial
carbon sink by absorbing CO2 (to the extent that they are the largest carbon
sink, surpassing the atmosphere and all vegetation and animals on Earth’s
surface), filter rainwater, support food production, and function as the
planet’s largest water reservoir. The degradation of soils and biodiversity
can be attributed in large part to the process of land artificialization, with
urban sprawl being a significant contributing factor. This growing phenomenon
has raised concerns among public authorities, who recognize the importance of
monitoring the state of territories. Artificialization is defined as the long-
term deterioration of the ecological functions of soil, including its
biological, hydrological, climatic, and agronomic functions, resulting from
its occupation or use [2].

The French National Institute of Geographical and Forest Information (IGN)[3],
in response to the growing availability of high-quality Earth Observation (EO)
data, is actively exploring innovative strategies to integrate these data with
heterogeneous characteristics. As part of their initiatives, the institute
employs artificial intelligence (AI) tools to monitor land cover across the
territory of France and provides reliable and up-to-date geographical
reference datasets.

The FLAIR #1 dataset, which focused on aerial imagery for semantic
segmentation, was released to facilitate research in the field. Building upon
this datset, the FLAIR #2 dataset extends the capabilities by incorporating a
new input modality, namely Sentinel-2 satellite image time series, and
introduces a new test dataset Both FLAIR #1 and #2 datasets are part of the
currently explored or exploited resources by IGN to produce the French
national land cover map reference Occupation du sol à grande échelle (OCS-GE).

## Multi-modality fusion challenge

The growing importance of EO in the monitoring and understanding of Earth’s
physical processes, and the diversity of data now publicly available naturally
favours multi-modal approaches that take advantage of the distinct strengths
of this data pool. Remote sensing data have several main characteristics that
are of crucial importance depending on the intended purpose. Spatial, temporal
and spectral resolutions will influence the choice of data and their
importance in a process. The complexity of integrating these different data
tend to promotes the use of machine learning for their exploitation.

This FLAIR #2 challenge organized by IGN proposes the development of multi-
resolution, multi-sensor and multi-temporal aerospace data fusion methods,
exploiting deep learning computer vision techniques.

The FLAIR #2 dataset hereby presented includes two very distinct types of
data, which are exploited for a semantic segmentation task aimed at mapping
land cover. The data fusion workflow proposes the exploitation of the fine
spatial and textural information of very high spatial resolution (VHR) mono-
temporal aerial imagery and the temporal and spectral richness of high spatial
resolution (HR) time series of Copernicus Sentinel-2 [4] satellite images, one
of the most prominent EO mission. Although less spatially detailed, the
information contained in satellite time series can be helpful in improving the
inter-class distinction by analyzing their temporal profile and different
responses in parts of the electromagnetic (EM) spectrum.

## Spatial and temporal domains definition

Spatial domains and divisions: as for the FLAIR #1 dataset, a spatial domain
is equivalent to a French ’département’ which is a french sub-region
administrative division. While the spatial domains can be geographically
close, heavy pre-processing of the radiometry of aerial images independently
per ’département’ create important differences (see [5]). Each domain has a
varying number of areas subdivided in patches of same size across the dataset.

While these areas were initially defined to contain sufficient spatial context
by taking into account aerial imagery, the strong difference in spatial
resolution with satellite data means that they consist of few Sentinel-2
pixels. Therefore, in order to also provide a minimum of context from the
satellite data, a buffer was applied to create super-areas. This allows, for
every patch of the dataset to be associated to a super-patch of Sentinel-2
data with sufficient size through a large footprint. Figure 1 illustrates the
different spatial units of the dataset.

Figure 1: Spatial definitions of the FLAIR #2 dataset: HR Sentinel-2 super-
area, VHR aerial area, HR Sentinel-2 super-patch and VHR aerial patch. Figure
2: The 50 spatial domains in France of the FLAIR #2 dataset and the train/test
split.

Temporal domains: they are twofold, on the one hand the date of acquisition of
the aerial imagery (which varies in terms of year, month, days) and on the
other hand by the satellite acquisitions, varying in terms of months and days.

Dataset extent: The dataset includes 50 spatial domains (Figure 2)
representing the different landscapes and climates of metropolitan France. The
train dataset constitute 4/5 of the spatial domains (40) while the remaining
1/5 domains (10) are kept for testing. This test dataset introduces new
domains compared to the FLAIR #1 test dataset. Some domain are in common but
areas within those domains are distinct. The FLAIR #2 dataset covers
approximately 817 km ² of the French metropolitan territory.

Figure 3: Example of mono-temporal aerial acquisition and annotations over an
area (bottom) and Sentinel-2 time series acquisitions of the corresponding
super-area (top) with four acquisitions examples.

## Dataset resolutions

We hereby define different resolutions of the data used in the FLAIR #2
dataset.

Spatial resolution: spatial resolution of aerial images can slightly vary
depending on the camera used (refer to [5] for more information), but all
images are resampled to a 0.2 m spatial resolution. In contrast, the
Sentinel-2 MultiSpectral Instrument (MSI) sensor acquires images at 10, 20 and
60 m spatial resolutions. The 60 m bands mainly intended for atmospheric
corrections are not taken into account.

Spectral resolution: aerial images acquire 4 spectral bands, namely blue,
green, red and near-infrared [5]. Satellite images have more spectral depth
with 10 bands, ranging from the visible to the medium infrared parts of the EM
spectrum. Sentinel-2 original bands are described in Table I.

Temporal resolution: aerial images are usually acquired between the months of
April and November and it takes three years to cover the entire French
territory (see temporal domains in [5]). Therefore, over an area, a single
aerial image is available in the FLAIR #2 dataset. While orbiting, the
Sentinel-2 constellation composed of two satellites (Sentinel-2 A&B) has a
revisit frequency of 5 days at the equator, less as you move towards the
poles. In the case of the FLAIR #2 dataset, for each area of the dataset, all
images acquired by Sentinel during the same acquisition year are considered,
including cloudy dates. Nonetheless, for each area, the number of acquisitions
varies because of relative orbits and orbit overlaps. An example of aerial and
satellite acquisition over an area of the dataset is illustrated in Figure 3.

Band |  Central wavelength (nm) |  Bandwidth (nm) |  Spatial resolution (m)
---|---|---|---
2 | 490 | 65 | 10
3 | 560 | 35 | 10
4 | 665 | 30 | 10
5 | 705 | 15 | 20
6 | 740 | 15 | 20
7 | 783 | 20 | 20
8 | 842 | 115 | 10
8a | 865 | 20 | 20
11 | 1610 | 90 | 20
12 | 2190 | 180 | 20
TABLE I: Original spatial and spectral resolutions of Sentinel-2 images.

In summary, both aerial and satellite data-types included in the FLAIR #2
dataset have strong heterogeneity in terms of spatial (factor of 50), spectral
(4 bands versus 10) and temporal (mono-temporal versus time series)
resolutions. This opens up challenging perspectives for data fusion schemes in
both technical and thematic aspects.

## Data sources and pre-processing

For details about aerial images (ORTHO HR®) and associated elevation data, as
well as pre-processing, refer to the FLAIR #1 datapaper [5].

Technical details about Sentinel-2 can be found in [4]. The images were
downloaded from the Sinergise API [6] as Level-2A products (L2A) which are
atmospherically corrected using the Sen2Cor algorithm [7]. L2A products
provide Bottom-Of-the-Atmosphere (BOA) reflectances, corresponding to a
percentage of the energy the surface reflects. L2A products also deliver
pixel-based cloud (CLD) and snow (SNW) masks at 20 m spatial resolution.
Sentinel-2 images are typically provided as 110$\times$110 km (with 10 km
overlay) squared ortho-image in UTM/WGS84 projection. However, in order to
limit the size of the data and due to the wide extent of the dataset, only the
super-areas were downloaded. Concerning Sentinel-2 pre-processing, the 20 m
spatial resolution bands are first resampled during data retrieval to 10 m by
the nearest interpolation method. Same approach is adopted for the cloud and
snow masks. Due to the relative orbits of Sentinel-2 some images contain
nodata pixels (reflectances at 0). As all Sentinel-2 images during the aerial
image acquisition year are gathered all dates containing such nodata were
removed. It must be remarked that the length of time series and the
acquisition dates thus varies for each super-area. Table II provides
information about the number of dates included in the filtered Sentinel-2 time
series for the train and test datasets. In average, each area is acquired on
55 dates over the course of a year by the satellite imagery.

| acquisitions per super-area |
---|---|---
Sentinel-2 time series (1 year) | min | max | mean | total
train dataset | 20 | 100 | 55 | 757
test dataset | 20 | 114 | 55 | 193
TABLE II: Number of acquisitions (dates) in the Sentinel-2 times series of one
year (corresponding to the year of aerial imagery acquisition).

Note that cloudy dates are not suppressed from the time series. Instead, the
masks are provided and can be used to filter the cloudy dates if needed. The
resulting Sentinel-2 time series are subsequently reprojected into the
Lambert-93 projection (EPSG:2154) which is the one of the aerial imagery.

## Data description, naming conventions and usage

The FLAIR #2 dataset is composed of 77,762 aerial imagery patches, each
512$\times$512 pixels, along with corresponding annotations, resulting in a
total of over 20 billion pixels. The patches correspond to 916 areas
distributed across 50 domains and cover approximately 817 km2. The area sizes
and the number of patches per area vary but are always a multiple of 512
pixels at a resolution of 0.20 meters. Additionally, the dataset includes
55,244 satellite super-areas acquisitions that have a buffer of 5 aerial
patches (512 m) surrounding each aerial area. Description of the data is
provided bellow:

Figure 4: Example of input and supervision data: true color composition, near-
infrared color composition, elevation band, Sentinel-2 true color composition
super-patch and supervision masks. The data from the first three columns are
retrieved from the IMG files, the super-patch from SEN numpy files while the
last column corresponds to the MSK files.

  * $\blacktriangleright$

The aerial input patches (IMG) consist of 5 channels, similar to the FLAIR #1
dataset. These channels include blue, green, red, near-infrared, and elevation
bands, all encoded as 8-bit unsigned integer datatype. The aerial patches are
named as IMG_ID, with a unique identifier (ID) across the dataset assigned to
each patch.

  * A file named flair_aerial_metadata.json contains metadata for each of the aerial patches. This JSON file provides detailed information such as the date and time of acquisition, the geographical location of the patch centroid (x, y), the mean altitude of the patch (z), and the type of camera used. For more in-depth descriptions of these metadata attributes, please refer to the documentation provided in [5].

  * $\blacktriangleright$

The Sentinel-2 super-areas (SEN2) data is composed of several elements - data,
masks, products and a JSON file to match aerial and satellite imagery - :

    * –

the super-area reflectance time series is stored in the SEN2_xxxx_data.npy
files. These files contain 4D NumPy arrays with a shape of
T$\times$C$\times$H$\times$W, where T represents the acquisition dates (which
can vary for each file), C represents the 10 spectral bands of Sentinel-2, and
H and W denote the height and width dimensions of the data, respectively. The
data is stored as uint16 datatype, which differs from the acquisition datatype
mentioned in the SenHub reference provided [6]. It’s important to note that
the data in these files is provided without any additional processing or
modifications.

    * –

the super-area cloud and snow masks are stored in the SEN2_xxxx_masks.npy
files. These files have a similar shape as the data files, with a 4D array
format of T$\times$C$\times$H$\times$W. However, they consist of only two
channels, representing the snow masks and cloud masks, respectively, in that
order. The values in the masks range from 0 to 100 and indicate the
probability of cloud or snow occurrence for each pixel. A value of 100
indicates a high probability.

    * –

the names of the Sentinel-2 time series products are listed in the
SEN2_xxxx_products.txt file. This file provides additional information for
each acquisition, including the Sentinel-2 platform (S2A or S2B), the
acquisition date (which corresponds to the first date mentioned in the product
name), the acquisition time, the orbit number and tile name associated with
the product. These details help identify and differentiate the specific
products within the Sentinel-2 time series dataset.

Additionally, flair-2_centroids_sp_to_patch.json file is provided alongside
the data. This file plays a role in dynamically cropping the satellite super-
areas into super-patches during the data loading process. The JSON file uses
the aerial patch name (e.g., IMG_077413) as the key and provides a list of two
indexes (e.g., [13,25]) that represent the data-coordinates of the aerial
patch centroids. Using these coordinates and a specified number of pixels
(referred to as sat_superpatch_size), super-patches are extracted from the
satellite data. For the experiments, the default sat_superpatch_size is set to
40, resulting in super-patches with a spatial size of 40*40 pixels. This size
corresponds approximately to two aerial patches on each side of the centroid.

The pattern xxxx in the file names corresponds to the format domain_year-
areanumber_arealandcoverletters (e.g., D077_2021-Z9_AF). The
arealandcoverletters represent the two broad types of land cover present in
the area. For more detailed information about the specific land cover types,
please refer to [5].

  * $\blacktriangleright$

The annotation patches (MSK) consist of a single channel with values ranging
from 1 to 19, encoded as an 8-bit unsigned integer datatype. These files are
named as MSK_ID, where ID corresponds to the same identifier used for the
aerial imagery patches.

It is important to note that annotations are limited to the boundaries of
aerial imagery areas and do not extend to satellite super-areas. In addition,
annotations derived from aerial imagery correspond to the specific date the
images were captured. However, certain evolving classes may not accurately
reflect the current state of the features as observed in Sentinel imagery. For
instance, the banks of a watercourse, delineated based on aerial imagery, may
undergo changes over time, spanning a year. These changes can result from
various factors such as natural processes or human activities, causing the
banks to shift or erode. Consequently, the annotations based on older aerial
imagery may not capture these temporal variations.

Figure 4 gives an example of aerial patches, corresponding extracted super-
patch (with the aerial patch footprint in black outlines) and annotation
patches. The interest of the extended spatial information provided by the
Sentinel-2 super-patches is particularly visible in the last two rows of
Figure 4. Indeed, the location on a beach or on a lake is difficult to
determine from the aerial image alone, and could easily be confused with the
sea for example in the last row.

## Semantic classes and frequency

For a complete description of the 18 semantic classes (+1 other class
corresponding to unknown land cover) that are available in the annotations
patches and information about the annotation process, see [5]. As explained in
[5], 5 from the 18 classes were merged into the other class, due to strong
under-representation ($<$ 1% of the complete dataset). This results in a
nomenclature of 12 classes plus the other class. Table III indicates the
resulting classes, value in the MSK patches and frequency across the entire
FLAIR #2 dataset. The class distribution in percentages of the train and test
datasets are presented in Figure 5.

| Class | MSK | Pixels | %
---|---|---|---|---
| building | 1 | 1,453,245,093 | 7.13
| pervious surface | 2 | 1,495,168,513 | 7.33
| impervious surface | 3 | 2,467,133,374 | 12.1
| bare soil | 4 | 629,187,886 | 3.09
| water | 5 | 922,004,548 | 4.52
| coniferous | 6 | 873,397,479 | 4.28
| deciduous | 7 | 3,531,567,944 | 17.32
| brushwood | 8 | 1,284,640,813 | 6.3
| vineyard | 9 | 612,965,642 | 3.01
| herbaceous vegetation | 10 | 3,717,682,095 | 18.24
| agricultural land | 11 | 2,541,274,397 | 12.47
| plowed land | 12 | 703,518,642 | 3.45
| other | >13 | 153,055,302 | 0.75
TABLE III: Semantic classes of the main nomenclature of the FLAIR #2 dataset
and their corresponding MSK values, frequency in pixels and percentage among
the entire dataset. Figure 5: Class distribution of the train dataset (top)
and test dataset (bottom).

The current test dataset has a different sampling than FLAIR #1. The use of
satellite time series to inject temporal information is especially relevant
for natural surfaces with e.g. a seasonal variation. Therefore, the classes of
forests (coniferous and deciduous), agricultural land and herbaceous cover
were favored, accounting for 72.98% of the test dataset.

Figure 6: Texture and Time extraction network including two branches: i) a
U-TAE network applied to the Sentinel-2 super-patch time series and ii) a
U-Net network applied to the mono-date aerial imagery patch. The last decoder
layer yielded features from the U-TAE branch are used as embeddings added to
the features of the U-Net branch, integrating temporal information from the
time series and spatial information from the extended super-patch. The light-
blue fusion type modules are enabled or not and varying according to the
fusion method. Figure 7: Fusion module taking as input the last U-TAE
embeddings. This module is applied to each stages of the U-Net encoder feature
maps. out corresponds to the channel size of the U-Net encoder feature map and
H and W to the corresponding spatial dimensions.

## Benchmark architecture

Network definition: to capture both spatial and temporal information from very
high resolution aerial images and high-resolution satellite images, we propose
a two-branch architecture called U-T&T, for Textural and Temporal information.
The model allows enables the fusion of learned time series-related information
with the low-level representations of mono-date learned information. The U-T&T
model combines two commonly used architectures:

  * $\blacktriangleright$

U-Net (spatial/texture branch): to handle the aerial imagery patches, a U-Net
architecture [8] is adopted. The encoder is using a ResNet34 backbone model
[9] which has been pre-trained on the ImageNet dataset [10]. The U-Net branch
has $\approx$ 24.4 M parameters. Ith closely resembles to the model described
in the FLAIR #1 datapaper [5], ensuring consistency and comparability with
prior work.

  * $\blacktriangleright$

U-TAE (spatio-temporal branch): a U-TAE [11] architecture focuses on
extracting and incorporating both spatial and temporal information from the
Sentinel-2 time series data. This architecture is based on U-Net but
incorporates a Temporal self-Attention Encoder (TAE) component taking as input
the lowest resolution features of the convolutional encoder to generate set of
attention masks that capture the temporal dependencies within the time series
data. These attention masks are then applied at all resolutions upon the
decoding process, enabling the model to capture spatio-temporal patterns in
the data.

Figure 6 provides an overview of the proposed method, which combines the U-TAE
and U-Net architectures. The main idea behind this approach is to incorporate
features learned by the U-TAE branch, which considers the temporal dimension
and a wider spatial context, into the U-Net branch, which focuses on aerial
imagery. However, a key constraint is the significant difference in spatial
resolution between the satellite and aerial data. With the satellite imagery
having a spatial resolution 50 times lower than the aerial imagery (10 m
versus 0.2 m), early and late fusion strategies (i.e., fusion at input or
prediction levels) are not viable due to the large size disparity. To address
this, a Fusion Module is introduced, depicted in Figure 7, which enables mid-
stage fusion of features from both branches:

  * $\blacktriangleright$

Fusion Module: the fusion module takes as input the U-TAE embedding (last
feature maps of the U-TAE decoder, shown in blue in Figure 6) and is applied
to each stage of the U-Net branch. Within the Fusion Module, two sub-modules
have different purposes and focus on distinct aspects:

    * –

Cropped: this sub-module aims at incorporating information from the U-TAE
super-patch embedding into the spatial extent of the aerial parches. The U-TAE
embedding is first cropped to match the extent of the aerial patch. This
cropped embedding is then fed to a single convolution layer, which produces a
new channel dimension size that aligns with the one of the U-Net encoder
feature maps channel size. The output of this convolutional layer is then
passed through an interpolation layer that utilizes bilinear resampling. This
interpolation ensures that the spatial dimensions matches those of the U-Net
feature maps.

    * –

Collapsed: this sub-module is designed to preserve spatial information from
the extended super-patch, which will be integrated into the U-Net feature
maps. Initially, the spatial dimension of the U-TAE is collapsed into a single
value per channel, typically by taking the mean. The resulting vector is then
fed into a shallow Multi-Layer Perceptron (MLP) consisting of three linear
layers with dropout regularization and Rectified Linear Unit (ReLU)
activation. The output size of the MLP is adjusted to match one of the U-Net
encoder feature maps channel size.Subsequently, for each value in the obtained
vector, the value is duplicated across the spatial dimension of the
corresponding U-Net encoder feature maps.

  * Both the cropped and collapsed sub-modules produce a mask of size out$\times$H$\times$W, where out, H, and W correspond to the targeted feature map dimensions of the U-Net model. These masks, generated separately, are initially added together to integrate spatio-temporal information from the Sentinel-2 satellite time series. The resulting combined mask is added to the feature maps of the U-Net model. This integration step allows the spatio-temporal information captured by the cropped and collapsed sub-modules from the Sentinel-2 satellite time series to be incorporated into the U-Net’s feature representation.

Network supervision: a single $\mathcal{L_{T\&T}}$ loss is used to monitor the
training, which is the sum of two auxiliary losses $\mathcal{L}$sat and
$\mathcal{L}$aerial, obtained respectively from the U-TAE and U-Net branches.
The two branches are using a categorical Cross Entropy (CE) cost-function,
suitable for multi-class supervised classification task :

$\mathcal{L_{CE}}=-\sum_{i=1}^{n}t_{i}\log(p_{i})\quad,$
$\mathcal{L}_{T\&T}=\mathcal{L}_{CE\;aerial}+\mathcal{L}_{CE\;sat}$

where $t_{i}$ is the MSK label and $p_{i}$ the Softmax probability of the
$i^{th}$ class.

The MSK files in the FLAIR #2 dataset are provided at a spatial resolution of
0.2 m. The output of the U-TAE branch corresponds to a super-patch, which
lacks annotations for most of its parts. To address this, the U-TAE outputs
are initially cropped to match the extent of the corresponding aerial patch.
Subsequently, they are interpolated to fit the spatial dimensions of the MSK
files (512$\times$512 pixels). This interpolation ensures compatibility before
calculating the $\mathcal{L}$sat loss.

## Benchmark metric

The evaluation methodology for the semantic segmentation task follows the
approach used in the FLAIR #1 challenge [5]. Initially, confusion matrices are
calculated per patch, and then aggregated across the test dataset to create a
single confusion matrix. To assess the performance of each semantic class, the
Intersection over Union (IoU) metric, also known as the Jaccard Index, is
computed. The IoU is calculated using the formula:

$IoU=\frac{|U\cap V|}{|U\cup V|}=\frac{TP}{TP+FP+FN}$

where U denotes the intersection, V the union, TP the true positives, FP the
false positives and FN the false negatives.

The mean Intersection over Union (mIoU) is then determined by taking the
average of the per-class IoU values. However, since the other class is not
well-defined and is equivalent to void, it is excluded from the IoU
calculations. Consequently, the mIoU is computed as the average of the IoUs
from the remaining 12 classes.

## Benchmark framework and settings

The baselines are calculated using the efficient PyTorch Lightning framework
[12]. For the implementation of the U-Net model, the segmentation-models-
pytorch library [13] is exploited, while the U-TAE network is obtained from
[11]. The U-TAE parameters are kept at their default values (as provided in
the GitHub implementation), except for the encoder and decoder widths.

For the training process, the train dataset consists of 40 domains, out of
which 32 are used for training the model, while the remaining 8 domains are
used for validation. The optimization technique employed is the stochastic
gradient descent (SGD) with a learning rate of 0.001. A reduction strategy is
implemented with a patience value of 10, allowing for adaptive adjustments to
the learning rate during training. The maximum number of epochs is set to 100,
but to prevent overfitting and save computational resources, an early stopping
method is utilized with a patience of 30 epochs. A batch size of 10 is used
for the baselines.

To ensure reproducibility and consistent results, all randomness is controlled
by fixing the seed using the seed_everything function from the PyTorch
library, with the seed value set to 2022. Twelve NVIDIA Tesla V100 GPUs with
32 GB memory each, located on a High-Performance Computing (HPC) cluster, are
used to speed up experiments. The distributed data parallel (ddp) strategy is
employed to leverage these computational resources efficiently, allowing for
parallel training across multiple GPUs.

In the context of the U-TAE and U-Net models, both of which utilize CE loss,
per class weighting is employed. When assigning weights to the classes, the
other class is explicitly set to 0, indicating that it does not contribute to
the loss calculation. The remaining classes are assigned a weight of 1.
However, in the case of the U-TAE model, the plowed land class is also
assigned a weight of 0 for the U-TAE CE loss. This decision is made because
the plowed land class is specifically designed for mono-temporal data. The
inclusion of time series data introduces ambiguity with agricultural land, and
therefore, setting the weight of the plowed land class to 0 helps to mitigate
this confusion.

In addition to these general hyperparameters, there are several other
parameters and strategies that have been or could be explored further:

  * ➩

the size of super-patches refers to the dimensions, in terms of pixels, of the
patches that are cropped from the super-areas. Different sizes can be tested,
allowing for experimentation with smaller or larger super-patch sizes.
However, it is important to note that there is a limit of 110 pixels for edge
patches. The choice of super-patch size has an impact on the spatial context
provided to both the U-TAE and U-Net branches through the collapsed fusion
sub-module.

Baselines:  the number 40 has been empirically determined and set as the
baseline for this specific parameter.

  * ➩

with the exception of the other and plowed land classes, no specific
distinction or weighting has been applied during training between the classes
and the network branches. However, it is possible to introduce per-class
weights for both the $\mathcal{L}$${sat}$ and $\mathcal{L}$${aerial}$ losses.
These weights can be determined based on expert knowledge to encourage
specialization of one branch or the other on certain classes. Another approach
is to apply weights during the summation of both losses to obtain
$\mathcal{L}$T&T.

Baselines:  the other class is assigned a weight of 0 for both branches, and
the plowed land class is assigned a weight of 0 for the U-TAE branch. The
remaining classes are assigned a weight of 1. Additionally, no weights are
applied during the summation of the $\mathcal{L}$${sat}$ and
$\mathcal{L}$${aerial}$ losses.

  * ➩

to prevent overfitting of the U-TAE branch and enhance the learned aerial
features, we incorporate a modality dropout mechanism. This involves
generating a random single value for each batch. If the generated value
exceeds a specified threshold, provided as an input parameter, the U-TAE
modality is dropped out, and only the U-Net branch is used for that particular
batch.

Baselines:  considering the coarse spatial resolution of Sentinel-2 data, we
set the modality dropout threshold relatively high, at a value of 0.5. This
ensures that a significant portion of the batches will exclusively utilize the
U-Net branch, thereby emphasizing the importance of the aerial imagery.

  * ➩

to address the potential impact of cloud or snow in the Sentinel-2 time
series, two strategies are implemented using the provided masks files. The
first strategy, called filter clouds, involves examining the probability of
cloud occurrence in the masks. If the number of pixels above a certain
probability threshold exceeds a specified percentage of all pixels in the
image, that particular date is excluded from the training process. This helps
to mitigate the influence of cloudy or snowy images on the training data. The
second strategy, known as monthly average, is specifically implemented to
alleviate potential challenges faced by the U-TAE branch due to a large number
of dates in the time series. In this strategy, a monthly average is computed
using cloudless dates. If no cloudless dates are available for a specific
month, fewer than 12 images may be used as input to the U-TAE branch.

Baselines:  a probability threshold of 0.5 is employed for filtering clouds or
snow in the masks. Additionally, to be considered for exclusion, the clouds or
snow must cover at least 60% of the super-patch.

  * ➩

similar to the FLAIR #1 approach, metadata associated with each aerial patch
are integrated into the model. These metadata are encoded using positional
encoding or one-hot encoding techniques (see [5]). The encoded metadata are
then passed through a MLP before being added to each U-Net encoder feature
map.

Baselines:  a positional encoding of size 32 is used specifically for encoding
the geographical location information.

  * ➩

data augmentation techniques usually prevent overfitting and help
generalization capabilities of a network. Simple geometric transformations are
applied during the training process. These transformations include vertical
and horizontal flips as well as random rotations of 0, 90, 180, and 270
degrees. This approach aligns with the methodology used in the FLAIR #1
challenge.

Baselines:  a data augmentation probability of 0.5 is used.

| INPUT | FILT. | AVG M. | M.DR | MTD | AUG | PARA. | EP. | mIoU
---|---|---|---|---|---|---|---|---|---
U-Net | aerial | - | - | - | ✗ | ✗ | 24.4 | 62 | 0.5467$\pm$0.0009
+MTD | aerial | - | - | - | ✔ | ✗ | 24.4 | 59 |  0.5473$\pm$0.0017
+MTD +AUG | aerial | - | - | - | ✔ | ✔ | 24.4 | 52 |  0.5517$\pm$0.0013
U-T&T | aerial+sat | ✗ | ✗ | ✗ | ✗ | ✗ | 27.3 | 9 | 0.5490$\pm$0.0072
+FILT | aerial+sat | ✔ | ✗ | ✗ | ✗ | ✗ | 27.3 | 11 |  0.5517$\pm$0.0135
+AVG M | aerial+sat | ✗ | ✔ | ✗ | ✗ | ✗ | 27.3 | 10 |  0.5504$\pm$0.0067
+M DR | aerial+sat | ✗ | ✗ | ✔ | ✗ | ✗ | 27.3 | 27 |  0.5354$\pm$0.0104
+MTD | aerial+sat | ✗ | ✗ | ✗ | ✔ | ✗ | 27.3 | 7 |  0.5494$\pm$0.0064
+AUG | aerial+sat | ✗ | ✗ | ✗ | ✗ | ✔ | 27.3 | 22 |  0.5554$\pm$0.0146
+FILT +AVG M +M DR +MTD +AUG | aerial+sat | ✔ | ✔ | ✔ | ✔ | ✔ | 27.3 | 36 |  0.5323$\pm$0.0016
+FILT +AVG M +AUG | aerial+sat | ✔ | ✔ | ✗ | ✗ | ✔ | 27.3 | 26 | 0.5623$\pm$0.0056
TABLE IV: Baseline results of ResNet34/U-Net architecture with aerial imagery
only and U-T&T with aerial and satellite imagery on the FLAIR #2 test set.
Results are averages of 5 runs of each configuration. FILT: filter Sentinel-2
acquisition with masks (clouds & snow); AVG M.: monthly average of all
Sentinel-2 acquisitions; M.DR: modality dropout of the U-TAE branch; MTD:
metadata for aerial imagery added; AUG: geometric data augmentation for aerial
imagery; PARA.: number of parameters of the network; EP.: best validation loss
epoch.

## Benchmark results

Firstly, an evaluation is conducted on a U-Net model that incorporates only
aerial imagery, resembling the approach used in the FLAIR #1 challenge. The
evaluation involves assessing the model’s performance using the code provided
in the GitHub repository (accessible at [14]). Following this, the results
obtained from applying the two-branches U-T&T model are reported.
Additionally, various parameters and strategies mentioned earlier are tested.

The models used in the evaluation were trained using a consistent
train/validation/test split and the parameters previously specified. The
training dataset consisted of 61,712 aerial imagery patches, and for the U-T&T
approach, an additional 41,029 (unfiltered) Sentinel-2 acquisitions are
included. During the inference phase, the models were applied to 16,050
patches of aerial imagery and 10,215 (unfiltered) satellite acquisitions from
the test dataset. The reported results represent the average mIoU scores
obtained from five separate runs of each model configuration. Additionally,
the standard deviation of the mIoU scores across the five runs is provided,
indicating the degree of variability in the performance of the models.

The results obtained from the different experiments are presented in Table IV.
When using only aerial imagery and a U-Net model, the highest mIoU score of
0.5517 is achieved by integrating aerial metadata and employing data
augmentation techniques. In the case of jointly utilizing aerial and satellite
imagery with the U-T&T model, the baseline model yields a slightly better mIoU
score compared to the aerial-only baseline (0.5490 versus 0.5467), but it also
exhibits a higher standard deviation in the results.

Table IV also includes the results obtained when implementing additional
strategies individually, as described in the Benchmark framework and settings
section. It is observed that using modality dropout leads to a decrease in the
mIoU score. Integrating aerial metadata into the U-Net branch only marginally
improves the results. However, for the remaining three strategies, namely
filtering the dates using cloud and snow masks, performing a monthly average
of Sentinel-2 acquisitions, and applying data augmentation, the mIoU scores
improve. By combining these three strategies, a mIoU score of 0.5623 is
achieved, corresponding to a 2.85% increase compared to the U-Net baseline.

|  mIoU |  building |  pervious surface |  impervious surface |  bare soil |  water |  coniferous |  deciduous |  brushwood |  vineyard |  herbaceous vegetation |  agricultural land |  plowed land
---|---|---|---|---|---|---|---|---|---|---|---|---|---
U-Net | 0.5470 | 0.8009 | 0.4727 | 0.6988 | 0.3076 | 0.7985 | 0.5758 | 0.7014 | 0.2392 | 0.6012 | 0.4653 | 0.5449 | 0.3583
U-T&T | 0.5594 | 0.8285 | 0.4980 | 0.7204 | 0.2982 | 0.8009 | 0.6041 | 0.7189 | 0.2541 | 0.6580 | 0.4684 | 0.5478 | 0.3157
U-T&T best | 0.5758 | 0.8368 | 0.4995 | 0.7446 | 0.3959 | 0.7952 | 0.6339 | 0.7239 | 0.2485 | 0.6678 | 0.4750 | 0.5513 | 0.3381
TABLE V: Per-class IoU results of the U-Net baseline (aerial imagery), the
U-T&T baseline (aerial and satellite imagery) and the best U-T&T
configuration.

The per-class IoU scores for three models are provided in Table V. The three
models considered are the U-Net baseline, the U-T&T baseline, and the U-T&T
model with dates filtering of Sentinel-2, monthly average, and data
augmentation. These models were selected based on achieving the highest mIoU
scores among the five runs. Among the 12 classes, the U-Net baseline
outperforms the other models by having a higher IoU score only for the plowed
land class, with a marginal improvement of 0.02 points compared to the U-T&T
best model. On the other hand, the U-T&T baseline model performs better in
predicting the water and brushwood classes, but the differences in IoU scores
are quite close to the other models. For the remaining nine classes, the U-T&T
best model surpasses the other models, exhibiting notable improvements in
classes such as buildings, impervious surfaces, bare soil, coniferous, and
vineyards. These improvements highlight the effectiveness of the U-T&T model
with the integrated strategies of dates filtering, monthly average, and data
augmentation.

Figure 8 illustrates the confusion matrix of the best U-T&T model. This
confusion matrix is derived by combining all individual confusion matrices per
patch and is normalized by rows. The analysis of the confusion matrix shows
that the best U-T&T model achieves accurate predictions with minimal confusion
in the majority of classes. However, when it comes to natural areas such as
bare soil and brushwood, although there is improvement due to the use of
Sentinel-2 time series data, a certain level of uncertainty remains. These
classes exhibit some confusion with semantically similar classes, indicating
the challenge of accurately distinguishing them.

Figure 8: U-T&T best model confusion matrix of the test dataset normalized by
rows.

Figure 9 showcases an example that illustrates the results of both the U-net
baseline andU-T&T baseline models in relation to the aerial imagery and the
corresponding annotations.

## Acknowledgment

The experiments conducted in this study were performed using HPC/AI resources
provided by GENCI-IDRIS (Grant 2022-A0131013803). This work was supported by
the European Union through the project ”Copernicus / FPCUP,” as well as by the
French Space Agency (CNES) and Connect by CNES. The authors would like to
acknowledge the valuable support and resources provided by these
organizations.

## Data access

The dataset and codes used in this study will be made available after the
completion of the FLAIR #2 challenge at the following website:
https://ignf.github.io/FLAIR/.

## References

  * [1] Food and FAO Agriculture Organization.  Status of the World’s Soil Resources: Main Report.  https://www.fao.org/3/i5199e/I5199E.pdf, 2015.  [Online; accessed 14-April-2023].
  * [2] Loi n° 2021-1104 du 22 août 2021 portant lutte contre le dérèglement climatique et renforcement de la résilience face à ses effets (1).  https://www.legifrance.gouv.fr/loda/article_lc/JORFARTI000043957221, 2021.  [Online; accessed 16-May-2023].
  * [3] Institut national de l’information géographique et forestière.  https://www.ign.fr.  [Online; Accessed: 02-May-2023].
  * [4] M. Drusch, U. Del Bello, S. Carlier, O. Colin, V. Fernandez, F. Gascon, B. Hoersch, C. Isola, P. Laberinti, P. Martimort, A. Meygret, F. Spoto, O. Sy, F. Marchese, and P. Bargellini.  Sentinel-2: Esa’s optical high-resolution mission for gmes operational services.  Remote Sensing of Environment, 120:25–36, 2012.  The Sentinel Missions - New Opportunities for Science.
  * [5] Anatol Garioud, Stéphane Peillet, Eva Bookjans, Sébastien Giordano, and Boris Wattrelos.  Flair #1: semantic segmentation and domain adaptation dataset.  arXiv preprint arXiv:2211.12979v5, 2022.
  * [6] Sentinel Hub, Sinergise Ltd.  https://www.sentinel-hub.com.  [Online; Accessed: 02-May-2023].
  * [7] Magdalena Main-Knorn, Bringfried Pflug, Jerome Louis, Vincent Debaecker, Uwe Müller-Wilm, and Ferran Gascon.  Sen2Cor for Sentinel-2.  In Image and Signal Processing for Remote Sensing XXIII, volume 10427, page 1042704. SPIE, 2017.
  * [8] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241. Springer International Publishing, 2015.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
  * [11] Vivien Sainte Fare Garnot and Loic Landrieu.  Panoptic segmentation of satellite image time series with convolutional temporal attention networks.  ICCV, 2021.
  * [12] PyTorch Lightning.  https://www.pytorchlightning.ai.  [Online; Accessed: 10-October-2022].
  * [13] Pavel Iakubovskii.  Segmentation models pytorch.  https://github.com/qubvel/segmentation_models.pytorch, 2019.
  * [14] French Land cover from Aerospace ImageRy, FLAIR. IGNF.  https://ignf.github.io/FLAIR/.  [Online; Accessed: 02-May-2023].

Figure 9: Comparison of results obtained on an area of the test dataset.
Top row: very high spatial resolution aerial imagery and corresponding labels.
Bottom row: U-Net baseline from aerial imagery and U-T&T baseline from aerial
and satellite imagery.