# Cross-attention Spatio-temporal Context Transformer for Semantic
Segmentation of Historical Maps

Sidi Wu<EMAIL_ADDRESS>Institute of Cartography and Geoinformation, ETH
ZurichSwitzerland ,  Yizi Chen<EMAIL_ADDRESS>Institute of Cartography and
Geoinformation, ETH ZurichSwitzerland ,  Konrad Schindler<EMAIL_ADDRESS>Photogrammetry and Remote Sensing, ETH ZurichSwitzerland and  Lorenz Hurni
<EMAIL_ADDRESS>Institute of Cartography and Geoinformation, ETH
ZurichSwitzerland

###### Abstract.

Historical maps provide useful spatio-temporal information on the Earth’s
surface before modern earth observation techniques came into being. To extract
information from maps, neural networks, which gain wide popularity in recent
years, have replaced hand-crafted map processing methods and tedious manual
labor. However, aleatoric uncertainty, known as data-dependent uncertainty,
inherent in the drawing/scanning/fading defects of the original map sheets and
inadequate contexts when cropping maps into small tiles considering the memory
limits of the training process, challenges the model to make correct
predictions. As aleatoric uncertainty cannot be reduced even with more
training data collected, we argue that complementary spatio-temporal contexts
can be helpful. To achieve this, we propose a U-Net-based network that fuses
spatio-temporal features with cross-attention transformers (U-SpaTem),
aggregating information at a larger spatial range as well as through a
temporal sequence of images. Our model achieves a better performance than
other state-or-art models that use either temporal or spatial contexts.
Compared with pure vision transformers, our model is more lightweight and
effective. To the best of our knowledge, leveraging both spatial and temporal
contexts have been rarely explored before in the segmentation task. Even
though our application is on segmenting historical maps, we believe that the
method can be transferred into other fields with similar problems like
temporal sequences of satellite images. Our code is freely accessible at
https://github.com/chenyizi086/wu.2023.sigspatial.git.

semantic segmentation, transformer, convolutional neural network, historical
map, aleatoric uncertainty

##  1\. Introduction

Before the invention of modern air- and space-borne earth observation
techniques, historical maps are almost the exclusive and comprehensive source
to depict the spatio-temporal characteristics of the Earth’s surface. The
topographic maps made two centuries ago can already have plausible accuracy
thanks to the advances of surveying methods on a large scale. They can be used
to study the past states or changes of the Earth’s surface (Burghardt et al.,
2022a; Levin et al., 2010; San-Antonio-Gómez et al., 2014). Instead of
manually retrieving information, which is time-consuming and tedious, recently
Convolutional Neural Networks (CNNs)have been favored to automate and
accelerate the extraction process (Heitzler and Hurni, 2020; Wu et al.,
2022b). However, extracting information from historical maps have inevitably
aleatoric uncertainty (Kendall and Gal, 2017; Wu et al., 2022a, b), known as
data-dependent uncertainty, which comes from mainly two folds:

  * •

noise inherent in scanning artifacts, drawing and printing defects as well as
fading papers of the map sheets, illustrated in Figure 1;

  * •

insufficient context when cropping an entire map sheet into small training
tiles (commonly used for deep learning), within which distinguishing objects
with similar symbols is nearly impossible, shown in Figure 2.

Implicated by its definition, aleatoric uncertainty cannot be reduced even
with more training data collected.

(a)

(b)

(c)

Figure 1. Noise inherent in printing/scanning artifacts LABEL:sub@fig:1a,
defects of painting strokes LABEL:sub@fig:1b and printing errors
LABEL:sub@fig:1c where lake texture (dense dots) spreads out and mixes with
wetland texture (short strokes). Figure 2. Limited contexts when cropping an
entire map sheet into small training tiles. Within these contexts
distinguishing objects with similar symbols like a lake (left on the big
image) and a river (right on the big image), is nearly impossible.

One possible way to alleviate the aleatoric uncertainty is to add extra
spatio-temporal contexts – a larger temporal context is likely to include
noise-less instances of the same object and a larger spatial context can
provide a longer range of information to better differentiate objects.
Unlocking a larger spatio-temporal context is feasible because it’s a general
practice for cartographers to update maps regularly and usually the whole map
was scanned and stored in the digital archive.
A considerable amount of work has been done to incorporate spatial contexts
into vision tasks (Chen et al., 2018; Ding et al., 2018; He et al., 2019; Jin
et al., 2021; Li et al., 2020; Liu et al., 2020a; Yuan et al., 2020). While
these works explore contextual modules within the input image, works like
(Ding et al., [n. d.]; Schmitz et al., 2021; Yan et al., 2021) integrate
larger contextual information outside the input image for applications like
whole slide images (WSI) or high-resolution satellite images. However, the
straightforward down-sampling strategy for the contextual images in these
works might have aliasing effects for maps with symbolic representations
rather than natural images. Temporal contexts have been widely researched in
video segmentation (Jin et al., 2017; Liu et al., 2020b; Nilsson and
Sminchisescu, 2018) and classification/segmentation of satellite image time
series (Garnot and Landrieu, 2021; Garnot et al., 2020; Rußwurm and Körner,
2018). While either spatial- or temporal- contexts have been well studied for
the vision tasks, the benefits of incorporating both have been rarely
explored.
To aggregate information from a larger spatial context, one might simply
enlarge the input size to CNNs, which, however, will increase the memory
consumption and does not consider the long-range dependency between pixels,
due to the intrinsic locality of the convolution. Transformers, which gained
wide interest nowadays, have the powerful ability to model global
relationships within all the input tokens based on the self-attention
mechanism. They have been widely adopted from Natural Language Processing
(NLP)to computer vision (Burghardt et al., 2022b; Liu et al., 2021a; Wang et
al., 2021) as an alternative to CNNs. However, applying transformers directly
on the input image as well as its spatio-temporal contextual images is
computationally expensive, considering the total number of pixels. Some works
(Ding et al., [n. d.]; Garnot and Landrieu, 2021; Liu et al., 2021b) take
advantage of both CNNs and transformers – the former for better feature down-
sampling and representation and the latter for long-range dependencies. Our
work follows this direction of research. Besides spatial contexts, our model
also aggregates temporal contexts in one go. Different from existing
approaches that aggregate temporal information at the same location (Çiçek et
al., 2016; Garnot and Landrieu, 2021), our method relates all the pixels
between temporally-neighboring samples. This accounts for the shift of
geographical objects at different years due to surveying/map-making/geo-
referencing bias on the one hand and integrates additional spatial information
of temporal neighbors on the other hand.
Our main contributions can be summarized as follows:

  * •

We integrate spatial and temporal contexts simultaneously for better
historical map segmentation in the presence of aleatoric uncertainty.

  * •

Our proposed simple yet effective network U-SpaTem takes advantage of both
CNNs and transformers with a cross-attention mechanism instead of the
commonly-used self-attention mechanism.

##  2\. Related Work

###  2.1. Historical Map Segmentation

To unlock the spatial information of the Earth’s surface in the past and track
the long-term evolution of natural and man-made objects, historical maps serve
as useful sources. Tremendous scanned historical maps stored in digital
archives have sparked automatic map processing methods in the past two decades
(Bin and Cheong, 1998; Chiang and Knoblock, 2010; Leyk, 2009).

With the advances of neural networks for the computer vision tasks, automatic
map processing has shifted towards CNNs-based methods (Chen et al., 2016,
2018; Ronneberger et al., 2015; Visin et al., 2016), for extracting buildings
(Heitzler and Hurni, 2020; Uhl et al., 2019), building blocks (Chen et al.,
2021; Wu et al., 2021), roads (Ekim et al., 2021), water bodies (Wu et al.,
2022b, 2023), and texts (Arundel et al., 2022; Li et al., 2021).

###  2.2. Transformer in Vision Tasks

Firstly introduced for NLP (Vaswani et al., 2017), transformers account for
long-range dependencies of sequence data based on the self-attention mechanism
within all the input tokens. With this mechanism, the model learns to focus on
the most salient tokens. The Vision Transformer (ViT), proposed by (Burghardt
et al., 2022b), applies the transformer to images by splitting images into
patches and treating feature vectors of these patches as tokens. Despite the
state-of-art performance of transformers in segmentation (Xie et al., 2021;
Zheng et al., 2021), the use of transformers in a large spatial-temporal
context is non-trivial if we regard each patch from the target image as well
as from the spatio-temporal contextual images as a token and apply self-
attention to all tokens. To reduce the computational complexity, (Sun et al.,
2022) proposed a cross-attention mechanism between the central feature and its
contexts. Works like (Ding et al., [n. d.]; Garnot and Landrieu, 2021; Liu et
al., 2021b) combine transformers and CNNs, which not only improves the
training efficiency but also takes both advantages of CNNs for locality
modeling and transformers for global dependencies, arguably both essential for
vision tasks (Liu et al., 2021b). Our work follows this line of research and
extends the work of (Garnot and Landrieu, 2021) for integrating both spatio-
temporal contexts with a cross-attention mechanism.

##  3\. Methodology

###  3.1. Network Architecture

Our proposed architecture U-SpaTem (U-Net with Spatio-Temporal Context
Transformer) is illustrated in Figure 3. Given a map tile, we get its adjacent
tiles in the same map as spatial contexts and tiles at the same location from
its temporally consecutive maps as temporal contexts. The central tile,
spatial tiles, as well as temporal tiles, are processed simultaneously and
independently by the encoder blocks, each of which is made up of down-sampling
(strided convolution) and a sequence of convolutions, ReLU activation, and
batch normalization, to gradually reduce the resolution of the input images.
The In_Conv in the first level differs from other encoder blocks in that no
down-sampling is used. No cross-tile convolution is used like 3D U-Net (Çiçek
et al., 2016).
The bottleneck features at the end of encoder blocks are passed through a
Spatio-Temporal Context Transformer, to get the fused features weighted by the
spatio-temporal attention masks. The masks will then be upsampled to fuse
spatio-temporal features at different depths. The details will be introduced
later. The fused bottleneck features are then passed to the decoder blocks,
each of which is made up of up-sampling (strided transposed convolution) and a
sequence of convolutions, ReLU activation, and batch normalization. The fused
spatio-temporal features from the encoder are passed to the decoder block at
the same resolution through skip connections. The Out_Conv differs from other
decoder blocks as it’s a single convolution operation to map feature channels
to the number of classes, followed by Sigmoid activation to generate a per-
class segmentation map for the central tile.

Figure 3. The proposed architecture (U-SpaTem).

###  3.2. Spatio-temporal Context Transformer

The bottleneck features at the end of the encoder are processed independently
for each temporal and spatial tile through one multi-head spatio-temporal
cross-attention block. Instead of adding an attention block at each depth,
which will increase the memory drastically, we only conduct cross-attention on
bottleneck features and up-sample the attention masks to all resolutions, to
fuse spatio-temporal features at different depths, following (Garnot and
Landrieu, 2021). We apply layer normalization(LN) (Xiong et al., 2020) to all
the features before the attention blocks.
For transformers, it is important to let the network be aware of the sequence
order by adding positional encoding to inject relative or absolution positions
into tokens of a sentence (Vaswani et al., 2017). Instead of hard-set
positional encoding, we add to the features the learnable positional
embeddings as (Fan et al., 2021) before the attention blocks.

####  3.2.1. Multi-head Cross-attention

Different from the self-attention mechanism, where the query, value, and key
come from a single embedding sequence, cross-attention combines two separate
embedding sequences, where the key and value come from one sequence and the
query comes from the other. Let $t_{1}\in\mathbb{R}^{N\times D}$ and
$t_{2}\in\mathbb{R}^{M\times D}$ denote two separate embedding sequences,
where $D$ denotes the feature dimensions, and $M,N$ denote the lengths of the
sequences, respectively. Then the query $\mathbf{Q}$ is derived from $t_{1}$
and the key $\mathbf{K}$ and value $\mathbf{V}$ are from $t_{2}$:

(1)
$\mathbf{Q}=t_{1}\mathbf{W}^{Q},\mathbf{K}=t_{2}\mathbf{W}^{K},\mathbf{V}=t_{2}\mathbf{W}^{V},$

where the $\mathbf{W}^{Q}\in\mathbb{R}^{D\times D}$,
$\mathbf{W}^{K}\in\mathbb{R}^{D\times D}$ and
$\mathbf{W}^{V}\in\mathbb{R}^{D\times D}$ are parameters for linear
projection. Then we have the attention
$\mathbf{A}=Softmax(\frac{QK^{T}}{\sqrt{D}})\in\mathbb{R}^{N\times M}$, and
$t_{1}$ will be updated by:

(2) $\hat{t}_{1}=AV\mathbf{W}^{P}+t_{1},$

with the residual connection. $\mathbf{W}^{P}\in\mathbb{R}^{D\times D}$ is a
fully connected layer for feature projection. We use multi-head attention with
$n$ heads, which divides features into $n$ groups along the channel dimension
$D$, and thus runs $n$ multiple cross-attention in parallel. Each group $g$
has then the channel dimension of $D_{head}=\frac{D}{n}$. Each attention can
be denoted as $A_{g}=Softmax(\frac{Q_{g}K_{g}^{T}}{\sqrt{D_{head}}})$ where
$Q_{g}\in\mathbb{R}^{N\times D_{head}}$ and $K_{g}\in\mathbb{R}^{M\times
D_{head}}$. We then concatenate features obtained from each attention head
along the channel dimension:

(3) $\hat{t}_{1}=Concat(A_{g}V_{g})^{n}_{g=1}\mathbf{W}^{P}+t_{1},$

where $V_{g}\in\mathbb{R}^{M\times D_{head}}$.
The output $\hat{t}_{1}\in\mathbb{R}^{N\times D}$ is passed to a feed-forward
module $FFM$ consisting of LN and multi-layer perceptron (MLP), with the
residual connection:

(4) $\displaystyle FFM(\hat{t}_{1})=MLP(LN(\hat{t}_{1})),$
$\displaystyle\hat{t}_{1}=FFM(\hat{t}_{1})+\hat{t}_{1}.$

####  3.2.2. Multi-head Spatio-Temporal Cross-attention

We use cross-attention between the central feature
$f\in\mathbb{R}^{H_{L}\times W_{L}\times D}$ and its contextual features
$f_{c}\in$ $\mathbb{R}^{I\times H_{L}\times W_{L}\times D}$, including $S$
spatial tiles and $T$ temporal tiles, in total $I=S+T$ tiles, to fuse the
spatio-temporal contexts at the depth $L$. The multi-head cross-attention is
carried out between the central tile and each contextual tile $i$
independently with shared weights $\mathbf{W}^{Q}_{c}\in\mathbb{R}^{D\times
D},\mathbf{W}^{K}_{c}\in\mathbb{R}^{D\times
D},\mathbf{W}^{V}_{c}\in\mathbb{R}^{D\times D}$:

(5) $\displaystyle
Q_{c}=f{\mathbf{W}^{Q}_{c}},K^{i}_{c}=f^{i}_{c}{\mathbf{W}^{K}_{c}},V^{i}_{c}=f^{i}_{c}{\mathbf{W}^{V}_{c}};i\in[1,\dots,I].$

To save memory cost of attention operation, we make use of spatial-reduction
attention (SRA) proposed by (Wang et al., 2021), which uses a reduction ratio
$R$ to reduce image resolution, illustrated in Figure 4:

(6) $SR(f^{i}_{c})=Reshape(f^{i}_{c},R)W^{SR}.$

Figure 4. Spatial reduction attention (SRA) (Wang et al., 2021).

$Reshape(f^{i}_{c},R)$ is to reshape $f^{i}_{c}$ of shape $H_{L}\times
W_{L}\times D$ to $\frac{H_{L}}{R}\times\frac{W_{L}}{R}\times(R^{2}D)$. It is
followed by a linear projection layer $W^{SR}\in\mathbb{R}^{(R^{2}D)\times D}$
to reduce the channel dimension from $(R^{2}D)$ to $D$. Then we have:

(7)
$Q_{c}=f{\mathbf{W}^{Q}_{c}},K^{i}_{c}=SR(f^{i}_{c}){\mathbf{W}^{K}_{c}},V^{i}_{c}=SR(f^{i}_{c}){\mathbf{W}^{V}_{c}};i\in[1,\dots,I].$

We derive an attention map for each contextual tile $i$:

(8)
$A^{i}_{c}=Softmax(\frac{Q_{c}{K^{i}_{c}}^{T}}{\sqrt{D_{head}}});A^{i}_{c}\in\mathbb{R}^{(H_{L}\times
W_{L})\times(h_{L}\times w_{L})},$

with $h_{L}=\frac{H_{L}}{R}$ and $w_{L}=\frac{W_{L}}{R}$. Here we omit the
formulation of multi-head attention as in Equation 3 for simplicity. We then
have updated spatial and temporal features $\hat{f}^{i}_{c}$ weighted by the
attention maps:

(9) $\hat{f}^{i}_{c}=A^{i}_{c}V^{i}_{c}\mathbf{W}^{P}_{c};i\in[1,\dots,I].$

The computation complexity for Equation 9 is reduced by $R^{2}$ compared with
no spatial reduction.

####  3.2.3. Spatio-temporal Feature Fusion

We concatenate $\hat{f}^{i}_{c}$ among I tiles to the shape of $I\times
H_{L}\times W_{L}\times D$. We use a 1 $\times$ 1 convolution operation to
weight and fuse $I$ spatial/temporal features to one spatio-temporal feature
(attentive feature pooling). We obtain the resulting feature:

(10) $\hat{f}_{c}=Conv_{1\times
1}(Concat(\hat{f}^{i}_{c})^{I}_{i=1});\hat{f}_{c}\in H_{L}\times W_{L}\times
D.$

Then we update the center feature:

(11) $\hat{f}=f+\hat{f}_{c}.$

This is followed by Layer Normalization and a feed-forward module ($FFM$) with
a residual connection. The $FFM$ consists of two linear transformations with
GeLU (Hendrycks and Gimpel, 2016) activation and dropouts in between as in
(Wang et al., 2021):

(12) $\hat{f}=FFM(\hat{f})+\hat{f}.$

Table 1. Comparison with other methods. We report mean Intersection over Union (mIoU), mean F1 and per-class F1 as well as parameters (million(M)) and inference time (for 100 samples); $\bullet$: U-SpaTem uses spatial context as input; $\bullet$: U-SpaTem uses temporal context as input; ($\bullet$, $\bullet$): U-SpaTem uses both spatial and temporal context as input. As (Wu et al., 2022b) was implemented in Keras (Chollet et al., 2015) while others were in Pytorch (Paszke et al., 2017), we omit parameters and inference time for (Wu et al., 2022b) for comparison. Method |  Spatial context |  Temporal context | Per-class F1(%) $\uparrow$ | mean F1(%) $\uparrow$ | mIoU(%) $\uparrow$ | Params(M) $\downarrow$ | IT(s) $\downarrow$
---|---|---|---|---|---|---|---
Stream | Wetland | River | Lake
U-Net (Ronneberger et al., 2015) | - | - | 96.44 | 88.30 | 64.22 | 51.80 | 75.19 | 74.34 | 1.08M | 0.97s
3D U-Net (Çiçek et al., 2016) | - | ✓ | 95.55 | 88.54 | 64.58 | 54.37 | 75.76 | 74.63 | 1.54M | 1.94s
U-TAE (Garnot and Landrieu, 2021) | - | ✓ | 89.40 | 88.41 | 62.98 | 55.80 | 74.15 | 73.08 | 1.06M | 2.08s
U-ASPP-LCE (Wu et al., 2022b) | - | - | 96.35 | 88.50 | 65.49 | 53.65 | 76.00 | 75.12 | – | –
Segformer (Xie et al., 2021) | ✓ | - | 87.72 | 68.16 | 41.65 | 58.79 | 64.08 | 61.40 | 3.72M | 1.63s
U-SpaTem (Ours) $\bullet$ | ✓ | - | 94.95 | 89.49 | 65.47 | 65.15 | 78.77 | 77.55 | 1.98M | 1.63s
U-SpaTem (Ours) $\bullet$ | - | ✓ | 96.20 | 90.82 | 65.68 | 62.24 | 78.74 | 77.84 | 1.98M | 1.19s
U-SpaTem (Ours) ($\bullet$, $\bullet$) | ✓ | ✓ | 94.93 | 91.31 | 64.61 | 63.69 | 78.64 | 77.54 | 2.44M | 1.94s

###  3.3. Aggregating Spatio-temporal Attention At All Depths

Similar to (Garnot and Landrieu, 2021), we up-sample spatial and temporal
attention masks and use them to fuse spatio-temporal features at different
depths.
We concatenate $A^{i}_{c}$ from Equation 8 among $I$ tiles and we have
$A^{L}_{c}$ of the size $H_{L}\times W_{L}\times(I\times h_{L}\times h_{L})$
at the depth $L$. For each depth $l=1,...,L$ of the encoder, we up-sample the
attention map $A^{L}_{c}$ to $A^{l}_{c}$ of the size $H_{l}\times
W_{l}\times(I\times h_{L}\times h_{L})$, respectively.
For spatial/temporal features $f^{l}_{c}$ of the shape $I\times H_{l}\times
W_{l}\times C_{l}$, we first down-sample them into $(I\times h_{L}\times
w_{L})\times C_{l}$ using max pooling and multiply them with $A^{l}_{c}$ to
obtain the fused spatio-temporal contexts of the size $H_{l}\times W_{l}\times
C_{l}$ at depth $l$. Since we have $n$ attention maps out of $n$ heads used in
the transformer, we split $f^{l}_{c}$ into $n$ groups along the channel
dimension $C_{l}$ and the resulting maps are concatenated back along the
channel dimension:

(13) $f^{l}_{c}=Concat(A^{l,g}_{c}f^{l,g}_{c})^{n}_{g=1}.$

We combine the central features with the spatio-temporal contextual features
derived above to get the fused features at depth $l$:

(14) $\hat{f}^{l}=Conv_{1\times 1}(Concat(f^{l},f^{l}_{c})).$

with a $1\times 1$ convolution layer of $C_{l}$ channels. $\hat{f}^{l}$ is
passed to the decoder at the same depth (skip connection).

Input | Annotation | U-Net | 3D U-Net | U-TAE | U-ASPP-LCE | Segformer | U-SpaTem (ours)
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Figure 5. Qualitative comparison between different methods. In both
predictions and annotations, red, green, blue, and pink represent streams,
wetlands, river, and lakes, respectively.

##  4\. Experiments and Results

###  4.1. Data Preparation

We make use of Siegfried
maps111https://www.swisstopo.admin.ch/en/geodata/maps/historical/siegfried25.html,
a Swiss national topographic map series that was published between 1870 and
1949, at the scale of 1:25000 for the Swiss plateau and 1:50000 for the Swiss
Alps. The map sheets were scanned and archived by the Federal Office of
Topography (Swisstopo). We used 60 maps and their consecutive temporal sheets
at the scale of 1:25000 around the year 1880 in our experiment. Each map sheet
is 7000 $\times$ 4800 pixels with a spatial resolution of 1.25m and has the
ground-truth annotation maps for streams, wetlands, rivers, and lakes. We
split the map sheets into training (40), validation (10), and testing (10).
For training and validation, we randomly sample tiles of 256 $\times$ 256
pixels and their spatio-temporal contextual tiles of the same size, for each
class equally. We empirically fix the ratio between the number of positive
samples (containing objects of interest) and negative samples (not containing
objects of interest) as 4:1. In total, we have 2793 sequences for training and
671 sequences for validation. Each sequence has a length of 13, including one
central tile, four temporally-consecutive tiles at the same location, and
eight spatially-adjacent tiles in the 3 $\times$ 3 neighborhood. We evaluate
our model on the entire 10 testing sheets.

###  4.2. Implementation Details

Our U-SpaTem has $L=5$ depths. We use $n=16$ heads for our Spatio-temporal
Context Transformer and $R=4$ for SRA. For our experiments, we use Adam
optimizer (Kingma and Ba, 2014) with an initial learning rate of 0.001 and a
decay of 0.5 if the validation loss does not decrease for 5 epochs, until
0.00001. We use dice loss for training, considering the feature sparsity. The
batch size in all our experiments is fixed as 32 and we train each model for
50 epochs. We compare our model with other promising models:

  * •

U-Net (Ronneberger et al., 2015), which is our baseline model without
introducing any spatial/temporal contexts.

  * •

3D U-Net (Çiçek et al., 2016), an extended U-Net model with 3D convolutions in
both encoder and decoder blocks to additionally aggregate temporal
information.

  * •

U-TAE (Garnot and Landrieu, 2021), a U-Net-based model with a transformer
between the encoder and decoder blocks to fuse the temporal features with
attention.

  * •

U-ASPP-LCE (Wu et al., 2022b), a state-of-art U-Net model for hydrology
segmentation in historical maps with Atrous Spatial Pyramid Pooling (ASPP) and
uncertainty estimation.

  * •

Segformer (Xie et al., 2021), a state-of-art transformer-based segmentation
model.

For Segformer (Xie et al., 2021), we initialize the network with the provided
pre-trained weights. An image of $768\times 768$ is input directly to the
Segformer so that the equivalent spatial context as our model is included. By
default, the image will be reshaped to 512 before the transformer blocks.
Finally, the output of $128\times 128$ is up-sampled to restore the original
resolution.

###  4.3. Results and Discussion

We compare the segmentation results of our proposed U-SpaTem and other
methods, shown in Table 1 (the upper part). We mark which type(s) of context
each method utilizes. We conduct ablation studies on spatial and temporal
contexts and our proposed architecture is equally applicable no matter if only
one or both contexts are available. The results are shown in Table 1 (the
lower part).
As we can see, adding spatial contexts largely helps to refine the predictions
of lakes, to which contexts can often be vital. Compared with Segformer, our
spatial-context-only model has half the number of parameters and much higher
accuracy. Compared with U-TAE and 3D U-Net which fuses temporal information
only at the same location, our temporal-context-only model also takes the
spatial information of temporal neighbors into account. Our model has a
slightly higher number of parameters but a much shorter inference time and
better segmentation accuracy. From our ablation studies, we see that adding
temporal information improves the segmentation of wetlands that may have faded
strokes/scanning artifacts occasionally. While adding either temporal or
spatial context leads to an obvious improvement for certain classes, fusing
both spatial and temporal contexts does not yield a better result in our
experiment.
Figure 5 shows the qualitative comparison, for lakes in the first two
examples, streams and rivers with unstable painting strokes in the last two
examples, and scanning artifacts in the last example. Our model has achieved a
better overall performance in front of all these scenarios/challenges.
We visualize spatial attention maps in Figure 6, where the center tile (marked
in red) has an inadequate context to differentiate between rivers and lakes
that have the same symbolization, as illustrated in Figure 2. Higher values
indicate more salient/important features the model pays attention to. We see
that the attention maps have strong activation on the neighboring tiles, which
means that the model tries to collect information about the shape and spatial
extent of the feature. In Figure 7, where the center tile has wetlands of
faded strokes, the model pays attention to other temporal instances of the
same wetland to gather temporal information, while ignoring the changes (in
the last two images of the second example).

##  5\. Conclusion

We have proposed a simple yet effective model U-SpaTem that leverages the
advantages of both CNNs, which can represent features and preserve spatial
information effectively, and transformers, which can aggregate information
from a larger spatio-temporal context. The additional spatio-temporal
information is especially useful for segmenting historical maps where
aleatoric uncertainty is present due to defects of original map sheets or
limited contexts caused by tile cropping during training. Our model shows a
better performance compared with other models that use either temporal or
spatial context. Also, our model is more lightweight and effective than pure
transformer-based models and does not need to sacrifice the resolution of
predictions. In our experiment, while temporal context is more important to
wetlands that sometimes have faded strokes, spatial context is more vital to
lakes that can have inadequate context due to tile cropping. However, fusing
both spatial and temporal contexts does not lead to an obvious improvement,
which means that it might not be necessary to always involve both contexts in
our case. To find out which type(s) of context is most suitable for each
class, it is worth investigating all different possibilities, where our
proposed architecture is equally applicable. A better strategy for combining
both contexts might be further investigated. Despite that our use case is
historical maps, the proposed model is general enough for other geospatial
applications like segmenting satellite images given a temporal sequence.

|
---|---
|

Figure 6. Visualization of spatial attention. Blue to red represents values
from low to high. We visualize the maximum value among all the attention
heads. The input tiles are on the left side, and the attention maps are on the
right side. The central tile is marked as a red square.

---

Figure 7. Visualization of temporal attention. Blue to red represents values
from low to high. We visualize the maximum value among all the attention
heads. The center tile is marked as a red square and the tiles are connected
chronologically from the left to right.

## References

  * (1)
  * Arundel et al. (2022) Samantha T Arundel, Trenton P Morgan, and Phillip T Thiem. 2022\.  Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps.  _Frontiers in Environmental Science_ (2022), 117.
  * Bin and Cheong (1998) Ding Bin and Wong Kok Cheong. 1998.  A system for automatic extraction of road network from maps. In _International Joint Symposia on Intelligence and Systems_. 359–366.
  * Burghardt et al. (2022a) Keith Burghardt, Johannes H Uhl, Kristina Lerman, and Stefan Leyk. 2022a.  Road network evolution in the urban and rural United States since 1900.  _Computers, Environment and Urban Systems_ 95 (2022), 101803.
  * Burghardt et al. (2022b) Keith Burghardt, Johannes H Uhl, Kristina Lerman, and Stefan Leyk. 2022b.  Road network evolution in the urban and rural United States since 1900.  _Computers, Environment and Urban Systems_ 95 (2022), 101803.
  * Chen et al. (2016) Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. 2016.  Attention to scale: Scale-aware semantic image segmentation. In _IEEE Conference on Computer Vision and Pattern Recognition_. 3640–3649.
  * Chen et al. (2018) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018.  Encoder-decoder with atrous separable convolution for semantic image segmentation. In _Proceedings of the European conference on computer vision (ECCV)_. 801–818.
  * Chen et al. (2021) Yizi Chen, Edwin Carlinet, Joseph Chazalon, Clément Mallet, Bertrand Duménieu, and Julien Perret. 2021.  Combining deep learning and mathematical morphology for historical map segmentation. In _Discrete Geometry and Mathematical Morphology: First International Joint Conference, DGMM 2021, Uppsala, Sweden, May 24–27, 2021, Proceedings_. Springer, 79–92.
  * Chiang and Knoblock (2010) Yao-Yi Chiang and Craig A Knoblock. 2010.  Extracting road vector data from raster maps. In _Graphics Recognition. Achievements, Challenges, and Evolution: 8th International Workshop, GREC 2009, La Rochelle, France, July 22-23, 2009. Selected Papers 8_. Springer, 93–105.
  * Chollet et al. (2015) François Chollet et al. 2015\.  Keras.  https://keras.io.
  * Çiçek et al. (2016) Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016\.  3D U-Net: learning dense volumetric segmentation from sparse annotation. In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19_. Springer, 424–432.
  * Ding et al. (2018) Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. 2018.  Context contrasted feature and gated multi-scale aggregation for scene segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2393–2402.
  * Ding et al. ([n. d.]) L Ding, D Lin, S Lin, J Zhang, X Cui, Y Wang, H Tang, and L Bruzzone. [n. d.].  Looking outside the window: Wide-context transformer for the semantic segmentation of high-resolution remote sensing images. arXiv 2021.  _arXiv preprint arXiv:2106.15754_ ([n. d.]).
  * Ekim et al. (2021) Burak Ekim, Elif Sertel, and M Erdem Kabadayı. 2021\.  Automatic road extraction from historical maps using deep learning techniques: A regional case study of turkey in a German World War II Map.  _ISPRS International Journal of Geo-Information_ 10, 8 (2021), 492.
  * Fan et al. (2021) Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021.  Multiscale vision transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 6824–6835.
  * Garnot and Landrieu (2021) Vivien Sainte Fare Garnot and Loic Landrieu. 2021.  Panoptic segmentation of satellite image time series with convolutional temporal attention networks. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4872–4881.
  * Garnot et al. (2020) Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, and Nesrine Chehata. 2020.  Satellite image time series classification with pixel-set encoders and temporal self-attention. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12325–12334.
  * He et al. (2019) Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. 2019.  Adaptive pyramid context network for semantic segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7519–7528.
  * Heitzler and Hurni (2020) Magnus Heitzler and Lorenz Hurni. 2020.  Cartographic reconstruction of building footprints from historical maps: A study on the Swiss Siegfried map.  _Transactions in GIS_ 24, 2 (2020), 442–461.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ (2016).
  * Jin et al. (2017) Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, et al. 2017\.  Video scene parsing with predictive feature learning. In _Proceedings of the IEEE International Conference on Computer Vision_. 5580–5588.
  * Jin et al. (2021) Zhenchao Jin, Tao Gong, Dongdong Yu, Qi Chu, Jian Wang, Changhu Wang, and Jie Shao. 2021.  Mining contextual information beyond image for semantic segmentation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7231–7241.
  * Kendall and Gal (2017) Alex Kendall and Yarin Gal. 2017.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ 30 (2017).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.
  * Levin et al. (2010) Noam Levin, Ruth Kark, and Emir Galilee. 2010.  Maps and the settlement of southern Palestine, 1799–1948: an historical/GIS analysis.  _Journal of Historical Geography_ 36, 1 (2010), 1–18.
  * Leyk (2009) Stefan Leyk. 2009\.  Segmentation of colour layers in historical maps based on hierarchical colour sampling. In _International Workshop on Graphics Recognition_. 231–241.
  * Li et al. (2020) Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, and Hong Liu. 2020\.  Spatial pyramid based graph reasoning for semantic segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8950–8959.
  * Li et al. (2021) Zekun Li, Runyu Guan, Qianmu Yu, Yao-Yi Chiang, and Craig A Knoblock. 2021.  Synthetic map generation to provide unlimited training data for historical map text detection. In _Proceedings of the 4th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery_. 17–26.
  * Liu et al. (2020a) Jianbo Liu, Junjun He, Yu Qiao, Jimmy S Ren, and Hongsheng Li. 2020a.  Learning to predict context-adaptive convolution for semantic segmentation. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16_. Springer, 769–786.
  * Liu et al. (2020b) Yifan Liu, Chunhua Shen, Changqian Yu, and Jingdong Wang. 2020b.  Efficient semantic video segmentation with per-frame inference. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16_. Springer, 352–368.
  * Liu et al. (2021b) Yun Liu, Guolei Sun, Yu Qiu, Le Zhang, Ajad Chhatkuli, and Luc Van Gool. 2021b.  Transformer in convolutional neural networks.  _arXiv preprint arXiv:2106.03180_ 3 (2021).
  * Liu et al. (2021a) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF international conference on computer vision_. 10012–10022.
  * Nilsson and Sminchisescu (2018) David Nilsson and Cristian Sminchisescu. 2018.  Semantic video segmentation by gated recurrent flow propagation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6819–6828.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in PyTorch.  (2017).
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 234–241.
  * Rußwurm and Körner (2018) Marc Rußwurm and Marco Körner. 2018.  Convolutional LSTMs for cloud-robust segmentation of remote sensing imagery.  _arXiv preprint arXiv:1811.02471_ (2018).
  * San-Antonio-Gómez et al. (2014) Carlos San-Antonio-Gómez, Cristina Velilla, and Francisco Manzano-Agugliaro. 2014.  Urban and landscape changes through historical maps: The Real Sitio of Aranjuez (1775–2005), a case study.  _Computers, environment and urban systems_ 44 (2014), 47–58.
  * Schmitz et al. (2021) Rüdiger Schmitz, Frederic Madesta, Maximilian Nielsen, Jenny Krause, Stefan Steurer, René Werner, and Thomas Rösch. 2021\.  Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture.  _Medical image analysis_ 70 (2021), 101996.
  * Sun et al. (2022) Guolei Sun, Yun Liu, Henghui Ding, Thomas Probst, and Luc Van Gool. 2022.  Coarse-to-fine feature mining for video semantic segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3126–3137.
  * Uhl et al. (2019) Johannes H Uhl, Stefan Leyk, Yao-Yi Chiang, Weiwei Duan, and Craig A Knoblock. 2019.  Automated extraction of human settlement patterns from historical topographic map series using weakly supervised convolutional neural networks.  _IEEE Access_ 8 (2019), 6978–6996.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30\.
  * Visin et al. (2016) Francesco Visin, Marco Ciccone, Adriana Romero, Kyle Kastner, Kyunghyun Cho, Yoshua Bengio, Matteo Matteucci, and Aaron Courville. 2016\.  ReSeg: A recurrent neural network-based model for semantic segmentation. In _IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 41–48.
  * Wang et al. (2021) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In _Proceedings of the IEEE/CVF international conference on computer vision_. 568–578.
  * Wu et al. (2021) Jianhua Wu, Jiaqi Xiong, Yu Zhao, and Xiang Hu. 2021\.  An Automatic Extraction Method for Hatched Residential Areas in Raster Maps Based on Multi-Scale Feature Fusion.  _ISPRS International Journal of Geo-Information_ 10, 12 (2021), 831.
  * Wu et al. (2022a) Sidi Wu, Magnus Heitzler, and Lorenz Hurni. 2022a.  a Closer Look at Segmentation Uncertainty of Scanned Historical Maps.  _International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences-ISPRS Archives_ 43, B4-2022 (2022), 189–194.
  * Wu et al. (2022b) Sidi Wu, Magnus Heitzler, and Lorenz Hurni. 2022b.  Leveraging uncertainty estimation and spatial pyramid pooling for extracting hydrological features from scanned historical topographic maps.  _GIScience & Remote Sensing_ 59, 1 (2022), 200–214.
  * Wu et al. (2023) Sidi Wu, Konrad Schindler, Magnus Heitzler, and Lorenz Hurni. 2023.  Domain adaptation in segmenting historical maps: A weakly supervised approach through spatial co-occurrence.  _ISPRS Journal of Photogrammetry and Remote Sensing_ 197 (2023), 199–211.
  * Xie et al. (2021) Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. 2021\.  SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. In _Neural Information Processing Systems (NeurIPS)_.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020\.  On layer normalization in the transformer architecture. In _International Conference on Machine Learning_. PMLR, 10524–10533.
  * Yan et al. (2021) Jiangpeng Yan, Hanbo Chen, Kang Wang, Yan Ji, Yuyao Zhu, Jingjing Li, Dong Xie, Zhe Xu, Junzhou Huang, Shuqun Cheng, et al. 2021\.  Hierarchical attention guided framework for multi-resolution collaborative whole slide image segmentation.  (2021), 153–163.
  * Yuan et al. (2020) Yuhui Yuan, Xilin Chen, and Jingdong Wang. 2020.  Object-contextual representations for semantic segmentation. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_. Springer, 173–190.
  * Zheng et al. (2021) Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. 2021\.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 6881–6890.