# Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing

Yating Xu Conghui Hu Gim Hee Lee
Department of Computer Science, National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Existing works on weakly-supervised audio-visual video parsing adopt hybrid
attention network (HAN) as the multi-modal embedding to capture the cross-
modal context. It embeds the audio and visual modalities with a shared
network, where the cross-attention is performed at the input. However, such an
early fusion method highly entangles the two non-fully correlated modalities
and leads to sub-optimal performance in detecting single-modality events. To
deal with this problem, we propose the messenger-guided mid-fusion transformer
to reduce the uncorrelated cross-modal context in the fusion. The messengers
condense the full cross-modal context into a compact representation to only
preserve useful cross-modal information. Furthermore, due to the fact that
microphones capture audio events from all directions, while cameras only
record visual events within a restricted field of view, there is a more
frequent occurrence of unaligned cross-modal context from audio for visual
event predictions. We thus propose cross-audio prediction consistency to
suppress the impact of irrelevant audio information on visual event
prediction. Experiments consistently illustrate the superior performance of
our framework compared to existing state-of-the-art methods.

##  1 Introduction

With the ultimate goal of understanding both audio and visual content in
video, multimodal video understanding finds a variety of applications in video
retrieval [11], video surveillance [38] _etc_. As video is naturally equipped
with both audio and visual signals, many prior works have incorporated audio
modality into the analyses and shown its benefits to several emerging visual
tasks [14, 30, 35, 10, 17]. Audio-Visual Video Parsing (AVVP) [34] is one of
the most challenging tasks which aims at classifying and localizing the
temporal event segments in the audio and visual streams respectively. The task
requires the model to fully understand video content in both audio and visual
streams while only video-level label is provided, as the fine-grained event
labels for the two modalities are labour-intensive to source. Since there
exists an intractable barrier to access full supervision, models must resort
to a weakly-supervised paradigm by learning from the union of all events in a
video without any modality and time indication.

Figure 1: Illustration of audio-visual video parsing task. Given a video, it
classifies events and detects their temporal locations in the audio and visual
streams, respectively. During training, only video-level label is provided.

To address the challenging problem, the state-of-the-art approaches utilize
the correlation between the audio and visual streams to guide the model
training. For example, Hybrid Attention Network (HAN) [34] captures the cross-
modal context by fusing the pre-extracted audio and visual features directly
from the first layer of the network using cross-attention (as illustrated in
Fig. 2(a)). However, this strong entanglement could undesirably mix
uncorrelated information from audio and visual streams.  Real scenes may
include occlusion or may be captured by camera of limited field of view, which
causes the audio and visual streams not fully correlated. For instance, in
Fig. 1, the silent dog and off-camera human speech only appear in one single
modality. Accordingly, we analyze the performance of HAN in detecting single-
modality and multi-modality events as shown in Tab. 1. Single-modality events
denote events only happening in audio or visual modality, while multi-modality
events refer to events appearing with temporal overlap on audio and visual
streams. The results are averaged F-scores per event. Compared with the
original HAN, the HAN-CA111We change the input of the cross-attention module
to the modality itself so as to keep the model size unchanged., when excluding
cross-modal fusion, exhibits a significant decrease in predicting multi-
modality events. However, the prediction performance for single-modality
events experiences an enhancement. It suggests that strong entanglement with
another non-fully correlated modality is harmful in detecting its own
exclusive events while the absence of fusion negatively affects the detection
of audio-visual events. As either fully entangled fusion or complete
independence of the two modalities can hurt the performance badly, it is
imperative to design a better fusion strategy for the two partially correlated
modalities.

Model | Single-modality Event | Multi-modality Event
---|---|---
HAN | 44.0 | 67.2
HAN-CA | 47.0 | 58.8
Ours | 50.6 | 66.1

Table 1: Analysis of HAN [34]. ‘HAN-CA’ denotes HAN without cross-attention.
Segment-level evaluation is conducted.

To solve this problem, we propose messenger-guided mid-fusion transformer
(MMT) to suppress the uncorrelated information exchange during fusion.
Compared to the early fusion in HAN, the mid-fusion is more flexible in
controlling the flow of the cross-modal context. It can first aggregate a
clearer global understanding of the raw input sequence, which helps identify
the useful cross-modal context in the fusion module. The messengers are the
core of MMT, which serves as the compact cross-modal representation during
fusion. (Fig. 2(b)). Due to their small capacity, they can help amplify the
most relevant cross-modal information that agrees best with the clean labels
while suppressing the noisy information that causes disagreement. Our MMT is
able to largely improve the performance of detecting single-modality events
while maintaining a relatively high performance of detecting multi-modality
events.

We further propose cross-audio prediction consistency (CAPC) to suppress the
undesired predictions in the visual stream caused by mismatched audio
information. As pointed out by [39], the “audible but not visible events” are
more common than “visible but not audible events” since the camera only
captures the scene of limited view, while the microphones capture events from
all directions. Thus, the visual modality is more likely to encounter the non-
correlated cross-modal context. To alleviate such situations, we introduce
CAPC. Our idea is to allow the visual modality to learn from beyond its paired
audio, and induce it to have consistent visual event predictions as learning
with its original pair. As such, the visual stream learns to only fuse the
audio context that is correlated with itself and ignores other unrelated
information to maintain the same visual event detection under different audio
contexts.

Figure 2: Network comparison: (a) HAN embedding and (b) our embedding. A and V
denote the pre-extracted audio and visual features, respectively.
$\overline{\text{A}}$ and $\overline{\text{V}}$ denotes the self-refined audio
and visual features, respectively. $\tilde{\text{A}}$ and $\tilde{\text{V}}$
denote the audio and visual representation after cross-modal fusion,
respectively. Audio-exclusive (visual-exclusive) events denotes events only
happening on audio (visual) modality. Each circle represents one event.

In summary, our contributions are as follows:

  * •

We propose messenger-guided mid-fusion transformer to reduce the uncorrelated
cross-modal context in the audio-visual fusion. The messengers serve as a
compact representation to amplify the most relevant cross-modal information
under weak supervision.

  * •

We propose cross-audio prediction consistency to calibrate the visual event
prediction under interference from the unmatched audio context. The visual
event prediction is forced to remain unchanged when pairing with different
audios so as to ignore the irrelevant audio information.

  * •

We conduct extensive qualitative and quantitative experiments to analyze the
effectiveness of our approach. Our method also achieves the state-of-the-art
performance on AVVP benchmark.

##  2 Related Work

###  2.1 Audio-Visual Representation Learning

The natural correspondence between audio and visual modalities overcomes the
limitations of perception tasks in single modality and introduces a series of
new applications. Semantic similarity is the most commonly used audio-visual
correlation [2, 5, 3, 20, 28]. The shared semantic information in both audio
and visual modality is a valuable free-source supervision. SoundNet [5] learns
sound representation by distilling knowledge from the pre-trained visual
models. Morgado _et al_. [28] propose cross-modal contrastive learning, where
negative and positive samples of visual frames are drawn from audio samples
and vice-versa. Besides semantic correlation, other works utilize temporal
synchronization [30, 22, 1, 32], motion correlation [12, 41] and spatial
correspondence [40, 13, 26]. However, self-supervised learning from natural
videos is potentially noisy as the audio and visual modalities are not always
correlated. Recently, Morgado _et al_. [27] propose to learn robust audio-
visual representation by correcting the false alignment in the contrastive
loss. Our task shares similar motivation with [27] as some events only happen
in single modality leaving no audio-visual correspondence.

###  2.2 Audio-Visual Video Parsing

Early works [35, 23] only detect events that is both audible and visible.
Based on the strong assumption that the audio and visual information are
aligned at each time step, [35, 23] fuse the audio and visual features at the
same time step. However, the events happening on the two modalities are not
always the same since the audio and vision are inherently different sensors.
To fully understand the content in the multimodal videos, Tian _et al_. [34]
introduced the task of audio-visual video parsing (AVVP). It classifies and
localizes all the events happening on the audio and visual streams in a
weakly-supervised manner. They design a hybrid attention network (HAN) to
capture the uni-modal and cross-modal temporal contexts simultaneously. The
audio and visual features are fused at the start of the network, where the
self-attention and cross-attention are performed in parallel. Since then, HAN
serves as the state-of-the-art audio-visual embedding and is widely adopted in
follow-up works. MA [39] generates reliable event labels for each modality by
exchanging the audio and visual tracks of a training video with another
unrelated video. JoMoLD [8] leverage audio and visual loss patterns to remove
modality-specific noisy labels for each modality. Lin _et al_. [24] explore
the cross-modality co-occurrence and shared cross-modality semantics across-
videos. Although HAN embedding shows promising performance, the full
entanglement of two non-fully correlated modalities is not ideal for the task
of AVVP. Therefore, we propose messenger-guided mid-fusion transformer and
cross-audio prediction consistency to reduce the uncorrelated cross-modal
context in the audio-visual fusion.

###  2.3 Multimodal Transformer

The attention module in the transformer [37] is effective in capturing the
global context among the input tokens, and is widely adopted in the multimodal
task [25, 11, 16, 21, 29]. Gabeur _et al_. [11] use transformer to capture
cross-modal cues and temporal information in the video. OMNIVORE [16] proposes
a modality-agnostic visual model that can perform classification on image,
video, and single-view 3D modalities using the same shared model parameters.
Perceiver [21] and MBT [29] address the high computation cost of the
multimodal transformer by using a small set of fusion tokens as the attention
bottleneck to iteratively distill the uni-modal inputs. Despite our messengers
also serve as the attention bottleneck, it is used to suppress learning from
noisy labels. Moreover, the messengers are more effective in the small
multimodal models. The MBT and Perceiver initialize the fusion tokens randomly
and require multiple times updates with the uni-modal inputs for it to carry
meaningful cross-modal information, which is not applicable for the model with
small number of encoder layers. In contrast, our messenger is directly derived
from the global representation of each modality so that it is already
informative without multiple times of updates.

Figure 3: The architecture of the messenger-guided mid-fusion transformer.
Visual Encoder and Audio Encoder denote the transformer encoders for the
visual and audio, respectively. Visual Decoder and Audio Decoder denote the
transformer decoders for the visual and audio, respectively.

##  3 Method

Let us denote a video with $T$ non-overlapping segments as
$\\{\text{V}_{t},\text{A}_{t}\\}_{t=1}^{T}$, where $\text{V}_{t}$ and
$\text{A}_{t}$ are the visual and audio clip at the $t$-th segment,
respectively. The corresponding label for visual event, audio event and audio-
visual event at the $t$-th segment is denoted as
$y_{t}^{\text{v}}\in\\{0,1\\}^{C}$, $y_{t}^{\text{a}}\in\\{0,1\\}^{C}$ and
$y_{t}^{\text{av}}\in\\{0,1\\}^{C}$, respectively. $C$ is the total number of
classes in the dataset and $y_{t}^{\text{av}}=y_{t}^{\text{v}}\times
y_{t}^{\text{a}}$. An event is considered as an audio-visual event only if it
occurs in both modalities. Note that more than one event can happen in each
segment. Grouping all the segment-level labels together, we obtain the video-
level label $Y=\\{y_{t}^{\text{v}}\cup
y_{t}^{\text{a}}\\}_{t=1}^{T}\in\\{0,1\\}^{C}$. The goal of audio-visual video
parsing is to detect all the visual, audio and audio-visual events in the
video. The training of AVVP is conducted in weak supervision, where only
video-level labels $Y$ are provided.

In the following sections, we first introduce messenger-guided mid-fusion
transformer as the new multi-modal embedding for AVVP, and then the novel idea
of cross-audio prediction consistency to reduce the negative interference of
the unmatched audio context to the visual stream.

###  3.1 Messenger-guided Mid-Fusion Transformer

Fig. 3 shows our proposed messenger-guided mid-fusion transformer. We
instantiate the self-attention and cross-attention layers with transformers
[37] as it shows excellent performance in the uni-modal [9, 4] and multimodal
[11, 31, 16] tasks with just the attention mechanism. The pre-trained visual
and audio feature extractors extract segment-level visual features
$\\{f_{t}^{\text{v}}\\}_{t=1}^{T}$ and audio features
$\\{f_{t}^{\text{a}}\\}_{t=1}^{T}$, respectively. The
$\\{f_{t}^{\text{v}}\\}_{t=1}^{T}$ and $\\{f_{t}^{\text{a}}\\}_{t=1}^{T}$ are
the input to the multimodal embedding, where the uni-modal and cross-modal
context are modeled sequentially. Instead of directly feeding the full cross-
modal context to the fusion, we summarize it into compact messengers. Finally,
the outputs of the last layer of the decoders are sent into the classifiers to
detect segment-level events for each modality.

#### Uni-modal Context Refinement.

We model the uni-modal context with transformer encoders, where the self-
attention is capable of aggregating the temporal context for a better global
understanding of the raw input sequence. For brevity, we only illustrate the
working flow of the visual modality since the visual and audio branches work
symmetrically. The pre-extracted visual features are first converted into 1-D
tokens $S_{\text{v}}\in\mathbb{R}^{T\times d}$ with feature dimension $d$ as
follow:

$S_{\text{v}}=\left[f_{1}^{\text{v}}\mathbf{W}_{\text{v}}^{\text{enc}},f_{2}^{\text{v}}\mathbf{W}_{\text{v}}^{\text{enc}},\ldots,f_{\text{T}}^{\text{v}}\mathbf{W}_{\text{v}}^{\text{enc}}\right]+\mathbf{PE},$
(1)

where $\mathbf{W}_{\text{v}}^{\text{enc}}$ is the linear projection layer that
projects pre-extracted features to d dimension and $\mathbf{PE}$ is the
position embedding. Then, the tokens are sent into a $L$-layer transformer
encoder. We adopt the original architecture of transformer [37]. Each layer
consists of a multi-headed self-attention ($\operatorname{MSA}$) and a
position-wise fully connected feed-forward network ($\operatorname{FFN}$):

$\displaystyle\tilde{S}_{\text{v}}^{l}$
$\displaystyle=\operatorname{LN}\left(\operatorname{MSA}\left(S_{\text{v}}^{l}\right)+S_{\text{v}}^{l}\right),$
(2) $\displaystyle S_{\text{v}}^{l+1}$
$\displaystyle=\operatorname{LN}\left(\operatorname{FFN}\left(\tilde{S}_{\text{v}}^{l}\right)+\tilde{S}_{\text{v}}^{l}\right),$

where $\operatorname{LN}$ denotes layer normalization and $S_{\text{v}}^{l}$
is the input tokens at the $l$-th layer.

Figure 4: Comparison of different fusion methods. Each circle represents one
event. We only illustrate the cross-modal fusion at the visual branch for
brevity. (a) shows an oracle setting, where the visual label $Y_{\text{v}}$ is
available. In this case, the model only learns from the relevant audio
context. (b) shows the fusion with full audio context. (c) shows the fusion
with messengers.

#### Cross-modal Context Fusion with Messengers.

We model the cross-modal context using $M$-layer transformer decoders, where
each layer consists of the multi-headed self-attention ($\operatorname{MSA}$),
multi-headed cross-attention ($\operatorname{MCA}$), and the position-wise
feed-forward network ($\operatorname{FFN}$). The working flow inside the
$m$-th layer of the decoder is as follows:

$\displaystyle\tilde{R}_{\text{v}}^{m}$
$\displaystyle=\operatorname{LN}\left(\operatorname{MSA}\left(R_{\text{v}}^{m}\right)+R_{\text{v}}^{m}\right),$
(3) $\displaystyle\hat{R}_{\text{v}}^{m}$
$\displaystyle=\operatorname{LN}\left(\operatorname{MCA}\left(\tilde{R}_{\text{v}}^{m},S_{\text{a}}^{L}\right)+\tilde{R}_{\text{v}}^{m}\right),$
$\displaystyle R_{\text{v}}^{m+1}$
$\displaystyle=\operatorname{LN}\left(\operatorname{FFN}\left(\hat{R}_{\text{v}}^{m}\right)+\hat{R}_{\text{v}}^{m}\right),$

where $\operatorname{MCA}(\cdot)$ performs cross-modal fusion between the
visual feature $\tilde{R}_{\text{v}}^{m}$ (query) and the audio context
$S_{\text{a}}^{L}$ (key and value).

However, providing full cross-modal context is not ideal when the two
modalities are not fully correlated, and is even worse when the supervision is
noisy. The video-level label $Y$ is the union of the audio and visual events,
which can introduce noise when supervising each modality, _i.e_. an event that
is present only in one modality becomes a noisy label for the other modality.
As shown in Fig. 4(b), with connection with full audio context, the model is
assured that the audio-exclusive event truly happens in the visual stream
guided by the noisy supervision Y. Consequently, its generalization ability is
severely affected. To this end, we create a fusion bottleneck $M_{\text{a}}$
as shown in Fig. 4(c) so that they can suppress the irrelevant audio context,
and the model is less likely to overfit to the noisy label $Y$.

Specifically, we condense the full cross-modal context into the compact
representation $M_{\text{v}}$ as follows:

$M_{\text{v}}=\operatorname{Tanh}\left(\operatorname{Pool}\left(S_{\text{v}}^{L}\mathbf{W}_{\text{v}}^{\text{msg}};n_{\text{v}}\right)\right),$
(4)

where $\mathbf{W}_{\text{v}}^{\text{msg}}\in\mathbb{R}^{d\times d}$ is the
linear projection layer, $\operatorname{Pool}(\cdot)$ is the average pooling
along the temporal dimension with a target length of $n_{\text{v}}$ and
$\operatorname{Tanh}(\cdot)$ is the activation function.
$M_{\text{v}}\in\mathbb{R}^{n_{v}\times d}$ has limited capacity in storing
information compared to the full cross-modal context
$S_{v}^{L}\in\mathbb{R}^{T\times d}$, where T is much larger than $n_{v}$.
Consequently, it creates an attention bottleneck that gives priority to the
most relevant cross-modal context that fits the clean labels. The compact
audio context $M_{\text{a}}\in\mathbb{R}^{n_{a}\times d}$ at the audio stream
is obtained in a similar way. We name it messenger as it represents its source
modality as the direct input in the cross-modal fusion as follows:

$\hat{R}_{\text{v}}^{m}=\operatorname{MCA}\left(\operatorname{LN}\left(\tilde{R}_{\text{v}}^{m},M_{\text{a}}\right)\right)+\tilde{R}_{\text{v}}^{m},\\\
$ (5)

where the audio messengers $M_{\text{a}}$ replace the full cross-modal context
$S_{\text{a}}^{L}$.

#### Classification.

$R_{\text{v}}^{M}$ is considered as the final visual representation and is
sent into the modality-specific classifiers for segment-level event prediction
$P_{\text{v}}\in\mathbb{R}^{T\times C}$ as follow:

$P_{\text{v}}=\operatorname{Sigmoid}\left(R_{\text{v}}^{M}\mathbf{W}_{\text{v}}^{\text{cls}}\right),$
(6)

where $\mathbf{W}_{\text{v}}^{\text{cls}}\in\mathbb{R}^{d\times c}$ is the
classifier weights. During training, we aggregate $P_{\text{v}}$ into video-
level predictions $\tilde{P}_{\text{v}}\in\mathbb{R}^{C}$ via soft pooling
similar to [34] since only video-level label $Y\in\\{0,1\\}^{C}$ is provided.
The audio prediction $P_{\text{a}}$ and $\tilde{P}_{\text{a}}$ are obtained in
a similar way. We also combine $\tilde{P}_{\text{v}}$ and
$\tilde{P}_{\text{a}}$ into a modal-agnostic video-level prediction
$\tilde{P}_{\text{video}}\in\mathbb{R}^{C}$. In total, we have three
classification losses:

$\mathcal{L}_{\text{cls}}=\operatorname{CE}\left(\tilde{P}_{\text{v}},Y_{\text{v}}\right)+\operatorname{CE}\left(\tilde{P}_{\text{a}},Y_{\text{a}}\right)+\operatorname{CE}\left(\tilde{P}_{\text{video}},Y\right),$
(7)

where $\operatorname{CE}$ denotes the binary cross-entropy loss. We set
$Y_{\text{v}}=Y_{\text{a}}=Y$ when only the video-level label is available.

Method | Audio | Visual | Audio-Visual | Type@AV | Event@AV
---|---|---|---|---|---
|  Seg. |  Event |  Seg. |  Event |  Seg. |  Event |  Seg. |  Event |  Seg. |  Event
AVE [35] |  47.2 |  40.4 |  37.1 |  34.7 |  35.4 |  31.6 |  39.9 |  35.5 |  41.6 |  36.5
AVSDN [23] |  47.8 |  34.1 |  52.0 |  46.3 |  37.1 |  26.5 |  45.7 |  35.6 |  50.8 |  37.7
HAN [34] |  60.1 |  51.3 |  52.9 |  48.9 |  48.9 |  43.0 |  54.0 |  47.7 |  55.4 |  48.0
HAN ${\dagger}$ [34] |  59.8 |  52.1 |  57.5 |  54.4 |  52.6 |  45.8 |  56.6 |  50.8 |  56.6 |  49.4
MA [39] |  60.3 |  53.6 |  60.0 |  56.4 |  55.1 |  49.0 |  58.9 |  53.0 |  57.9 |  50.6
Lin _et al_. [24] |  60.8 |  53.8 |  63.5 |  58.9 |  57.0 |  49.5 |  60.5 |  54.0 |  59.5 |  52.1
JoMoLD [8] |  61.3 |  53.9 |  63.8 |  59.9 |  57.2 |  49.6 |  60.8 |  54.5 |  59.9 |  52.5
Ours |  61.9 |  53.9 |  64.8 |  61.6 |  57.6 |  50.2 |  61.4 |  55.2 |  60.9 |  53.1

Table 2: Comparison with the state-of-the-art methods of audio-visual video
parsing on the LLP test dataset. ‘Audio’, ‘Visual’ and ‘Audio-Visual’ denotes
audio event, visual event and audio-visual event detection, respectively.
Note that they are different from the event categories in Tab. 1 and we
illustrate the difference in the Supplementary Material. ‘Seg.’ denotes
segment-level evaluation and ‘Event’ denotes event-level evaluation.
‘HAN${\dagger}$’ is the variant of HAN that additionally uses label
refinement. The best result is marked in bold.

###  3.2 Cross-Audio Prediction Consistency

We further propose cross-audio prediction consistency (CAPC) to suppress the
inaccurate visual event prediction arising from unmatched audio information.
As analyzed in [39], audio-exclusive events are more common than visual-
exclusive events and thus the visual stream is more likely to be influenced by
the non-correlated cross-modal context. As shown in Fig. 6, the visual branch
confidently detects the audio-exclusive event ‘Chicken rooster’ when only
learnt with its paired audio. To alleviate this problem, we introduce a
consistency loss in the visual event prediction by pairing the same visual
sequence with different audios. Specifically, the visual modality V is paired
with not only its original audio counterpart $\text{A}_{\text{orig}}$, but is
also paired with audios that are randomly selected from other videos at each
training iteration. We denote the visual prediction from the original pair
$(\text{V},\text{A}_{\text{orig}})$ as
$\tilde{P}_{\text{v}}\in\mathbb{R}^{C}$, and the visual prediction from the
$i$-th random pair $(\text{V},\text{A}_{\text{rand}}^{i})$ as
$\tilde{P}_{\text{v}}^{i}\in\mathbb{R}^{C}$. CAPC requires
$\tilde{P}_{\text{v}}^{i}$ to be the same as $\tilde{P}_{\text{v}}$ as follow:

$\mathcal{L}_{\text{ccr}}=\frac{1}{N}\sum_{i=1}^{N}\left\|\tilde{P}_{\text{v}}^{i}-\tilde{P}_{\text{v}}\right\|_{2}^{2},$
(8)

where $N$ is the number of random pairs for each visual sequence. The cross-
attention at the visual stream will learn to only grab the useful audio
context (_i.e_. audio-visual event) from A and $\text{A}_{\text{rand}}^{i}$
and ignore the irrelevant information (_i.e_. audio-exclusive event) in order
to achieve this prediction consistency.

We notice that there may be a trivial solution to achieve this prediction
consistency. The cross-attention totally ignores all the audio context and
thus leads to complete independence of the visual prediction from the audio
information. However, we show in Tab. 4 that the model does not degenerate to
this trivial solution. Instead, CAPC improves the robustness of fusion under
non-fully correlated cross-modal context.

Finally, the total loss of our method is:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{cls}}+\mu\mathcal{L}_{\text{ccr}},$
(9)

where $\mu$ is the hyperparameter to balance the loss terms.

###  3.3 Discussions on CAPC

#### Comparison with Consistency Regularization.

Consistency regularization is widely adopted in semi-supervised learning [33,
6, 7], where the model is required to output the similar prediction when fed
perturbed versions of the same image. In contrast to augmenting the modality
itself, the CAPC keeps the modality itself intact and only augments its cross-
modal context, _i.e_. pairing the visual modality with different audios. By
learning consistency under cross-audio augmentation, the fusion robustness is
improved.

#### Comparison with Audio-Visual Correspondence.

Audio-visual pairing correspondence [3, 2] and audio-visual temporal
correspondence [22, 30] are the cross-modal self-supervision in the video.
They highlight the audio-visual alignment to learn good audio and visual
representation. In contrast, we highlight the audio-visual misalignment to
reduce the negative impact of the unmatched audio context to the visual
modality.

##  4 Experiments

###  4.1 Experiment Setup

#### Dataset.

We conduct experiments on the Look, Listen and Parse (LLP) Dataset [34]. It
contains 11,849 YouTube videos, each is 10-second long. It covers 25 real-life
event categories, including human activities, animal activities, music
performances, _etc_. 7,202 video clips are labeled with more than one event
category and per video has an average of 1.64 different event categories. We
follow the official data splits [34]. 10,000 video clips only have video-level
labels and are used as training sets. The remaining 1,849 videos that are
annotated with audio and visual events and second-wise temporal boundaries are
divided into 849 videos as the validation set and 1,000 as the test set.

#### Evaluation Metrics.

We use F-scores as the quantitative evaluation method. We parse the visual,
audio and audio-visual events, denoted as ‘Visual’, ‘Audio’ and ‘Audio-
Visual’, respectively. We use F-scores for segment-level and event-level
evaluations. The segment-level metrics evaluate the predictions for each
segment independently. The event-level evaluation first concatenates
consecutive positive segments as the event proposal and then compares the
alignment with the ground-truth event snippet under the mIoU=0.5 as the
threshold. Meanwhile, we use ‘Type@AV’ and ‘Event@AV’ for the overall access
of the model performance. The ‘Type@AV’ averages the evaluation scores of
‘Audio’, ‘Visual’ and ‘Audio-Visual. The ‘Event@AV’ considers all the audio
and visual events for each video rather than directly averaging results from
different event types.

#### Implementation Details.

Each video is downsampled at 8 fps and divided into 1-second segments. We use
both the ResNet-152 [18] model pre-trained on ImageNet and 18 layer deep
R(2+1)D [36] model pre-trained on Kinetics-400 to extract visual features. The
2D and 3D features are concatenated as the visual representation for the
visual input. For the audio signals, we use the VGGish network [19] pre-
trained on AudioSet [15] to extract 128-D features. The feature extractors are
fixed during training. For each modality, both the number of encoders L and
the number of decoders M are set to 1. The hidden size is set to 512 and the
number of attention head is set to 1. The number $n_{a}$ of audio messengers
and the number $n_{v}$ of visual messengers are set to 1, respectively.

We train our model in three stages. In the first stage, we optimize our
proposed audio-visual embedding with classification loss
$\mathcal{L}_{\text{cls}}$ on the video-level label Y. In the second stage, we
calculate the pseudo label [39] for each modality. In the third stage, we re-
train our embedding with Eqn. 9 using the pseudo label. $\mu$ is set to 0.5
and N is set to 1. We use Adam optimizer with learning rate $3\times 10^{-4}$
and batch size of 64. We train 40 epochs and decrease the learning rate by
$10^{-1}$ every 10 epochs in each training stage. All the experiments are
conducted using Pytorch on a NVIDIA GTX 1080 Ti GPU.

###  4.2 Comparison with State-of-the-art Results

We compare our method with state-of-the-art audio-visual event parsing methods
of AVE[35], AVSDN [23], HAN [34], MA [39], Lin _et al_. [24] and JoMoLD [8].
We report their results from their paper. All the methods are trained using
the same pre-extracted features as input. The recent methods, MA [39], Lin _et
al_. [24] and JoMoLD [8] all adopt the HAN [34] as the audio-visual embedding.

Tab. 2 shows the quantitative comparisons on the LLP dataset [34]. Our model
constantly outperforms other methods on all the evaluation metrics. Although
we remove the entanglement in the early layers, the performance of audio and
visual event detection are both improved than all the HAN-based methods. This
demonstrates that a compact fusion is better than the fully entangled fusion
approach when the audio and visual information are not always correlated.

$n_{a}$ | $n_{v}$ | Audio | Visual | Audio-Visual | Type@AV | Event@AV
---|---|---|---|---|---|---
No MSG | 61.5 | 63.2 | 55.7 | 60.1 | 59.9
MBT | 60.9 | 64.1 | 56.0 | 60.3 | 59.8
5 | 5 | 61.5 | 64.4 | 57.0 | 61.0 | 60.6
3 | 3 | 62.1 | 63.9 | 56.9 | 61.0 | 60.5
1 | 3 | 61.6 | 64.7 | 56.8 | 61.0 | 60.6
3 | 1 | 62.1 | 64.6 | 56.8 | 61.2 | 60.9
1 | 1 | 61.9 | 64.8 | 57.6 | 61.4 | 60.9

Table 3: Ablation study of the messengers. ‘No MSG’ denotes the model performs
cross-modal fusion without messengers. ‘MBT’ replaces the messengers with the
fusion bottleneck token [29]. $n_{\text{a}}$ and $n_{\text{v}}$ is the number
of audio and visual messengers, respectively, and
‘$n_{\text{a}}=1,n_{\text{v}}=1$’ is our final model. Segment-level results
are reported.  Figure 5: Qualitative comparison of the messengers. ‘Visual’
and ‘Audio’ represent the visual and audio event, respectively. ‘GT’ denotes
ground truth. ‘No MSG’ denotes model performs cross-modal fusion without
messengers. ‘Ours’ denotes the fusion with messengers.

###  4.3 Ablation Study

Effectiveness of the Messengers. Tab. 3 shows the effectiveness of the
messengers. ‘No Messenger’ abbreviated as ‘No MSG’ is the model which directly
performs cross-modal fusion using Eqn. 3. Compared with our final model
‘$n_{\text{a}}=1,n_{\text{v}}=1$’, both the audio and visual performance
decrease. We also provide qualitative analysis of the messenger in Fig. 5. ‘No
MSG’ wrongly detects the visible but not audible event ‘Car’ on the audio
stream due to the unconstrained visual context. By constraining the full
visual context into our compact messenger, the irrelevant visual information
‘Car’ is suppressed and only keeps the useful information of the ‘Motorcycle’
for the audio.

In addition, we compare our messenger with fusion bottleneck token [29],
denoted as ‘MBT’, by replacing the messenger with the same number of fusion
bottleneck tokens in the transformer model. Our messenger
‘$n_{\text{a}}=1,n_{\text{v}}=1$’ consistently outperforms ‘MBT’ in the
shallow transformer model, where only one layer of the encoder is used for
each modality.

Analysis of the number of messengers. Tab. 3 also shows the performance of our
model with different numbers of messengers. $n_{\text{a}}$ and $n_{\text{v}}$
is the number of audio and visual messengers, respectively. Using a large
number of messengers shows decreasing performance, suggesting the dilution of
the beneficial cross-modal context. Hence we adopt
‘$n_{\text{a}}=1,n_{\text{v}}=1$’ in our final model.

Figure 6: Qualitative comparison of CAPC. ‘GT’ denotes the ground truth event
labels. ‘No CAPC’ denotes the model without cross-audio prediction
consistency. ‘No FA’ denotes the model without fusion with audio at the visual
stream.

| Audio | Visual | Audio-Visual | Type@AV | Event@AV
---|---|---|---|---|---
No FA | 62.1 | 62.5 | 56.3 | 60.3 | 60.3
No CAPC | 61.9 | 64.2 | 56.4 | 60.8 | 60.7
Ours | 61.9 | 64.8 | 57.6 | 61.4 | 60.9

Table 4: Ablation study of the cross-audio prediction consistency. ‘No FA’
denotes the model without fusion with audio at the visual stream. ‘No CAPC’
denotes the model without using cross-audio prediction consistency. Segment-
level results are reported

Effectiveness of Cross-audio Prediction Consistency. Tab. 4 presents the
ablation study of cross-audio prediction consistency. ‘No CAPC’ is the model
trained without cross-audio prediction consistency. By forcing prediction
consistency on the visual stream, _i.e_. ‘Ours’, both the performance of
visual and audio-visual event detection show obvious improvement.

We also verify whether CAPC leads to the trivial solution, _i.e_. the visual
prediction does not need the audio information at all. We replace the input to
the cross-attention at the visual stream with the visual modality itself,
_i.e_. replacing $M_{\text{a}}$ with $S_{\text{v}}^{L}$ in Eqn. 5, denoted as
‘No FA’. The performance is worse than the ‘No CAPC’, and much worse than our
full model ‘Ours’. We also provide the qualitative comparison in Fig. 6. ‘No
FA’ shows the poor generalization ability on the visual event detection as it
wrongly detects a totally irrelevant event ‘Helicopter’. By learning with both
its paired audio and other non-paired audios, the model can correctly identify
that the ‘Chicken rooster’ is an audible but not a visible event. This shows
that our cross-audio prediction consistency does not de-activate the cross-
modal fusion. Instead, it improves the robustness of the fusion between two
non-fully correlated modalities.

$\mu$ | Audio | Visual | Audio-Visual | Type@AV | Event@AV
---|---|---|---|---|---
0.1 | 61.3 | 63.6 | 56.3 | 60.4 | 60.1
0.5 | 61.9 | 64.8 | 57.6 | 61.4 | 60.9
1 | 61.9 | 64.6 | 57.6 | 61.4 | 60.7

Table 5: Analysis of different value of $\mu$. Segment-level results are
reported.

Analysis of CAPC loss weight $\mu$. Tab. 5 presents the ablation study on
different value of $\mu$. Interestingly, using a small weight, _i.e_.
$\mu=0.1$ is worse than the model without CAPC (‘No CAPC’ in Tab. 4). The
possible reason is that the model trained with ‘$\mu=0.1$’ tends to focus on
the easy training samples ($\text{A}_{\text{orig}}$ and
$\text{A}_{\text{rand}}^{i}$ are both fully correlated with V). CAPC thus
encourages V to take in full audio context to achieve faster convergence in
this case, which provides the false signal. Only using larger weight can
effectively optimize hard samples ($\text{A}_{\text{orig}}$ and
$\text{A}_{\text{rand}}^{i}$ are not fully correlated with V), where CAPC
guides visual stream to only select useful audio information. It can also be
verified that CAPC loss of $\mu=0.1$ is much larger than $\mu=0.5$. Therefore,
we choose $\mu=0.5$ as our final setting.

N | Audio | Visual | Audio-Visual | Type@AV | Event@AV
---|---|---|---|---|---
1 | 61.9 | 64.8 | 57.6 | 61.4 | 60.9
2 | 61.7 | 64.5 | 57.0 | 61.1 | 60.5
3 | 61.7 | 64.0 | 57.0 | 60.9 | 60.6

Table 6: Analysis of different number of random pairs N. Segment-level results
are reported.

Analysis of the number of random pairs in CAPC. Tab. 6 presents the ablation
study of the number of random pairs in CAPC. Using a larger number of pairs,
_i.e_., $N=3$, not only increases computation cost but also exhibits a decline
in performance. We postulate the reason is that larger N provides a false
signal that the visual modality should ignore any audio context (including
correlated audio information). More analysis is provided in Supplementary
Material. Therefore, we choose $N=1$ in our final model.

##  5 Conclusion

We address the problem of fusion between two non-fully correlated modalities
in weakly supervised audio-visual video parsing. We propose the messenger-
guided mid-fusion transformer to reduce the unnecessary cross-modal
entanglement. The messengers act as fusion bottlenecks to mitigate the
detrimental effect of the noisy labels. Further, we propose cross-audio
prediction consistency to reduce the negative interference of unmatched audio
context to the visual stream. The effectiveness of our proposed method is
analyzed both quantitatively and qualitatively.

#### Acknowledgement.

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-024), and the
Tier 2 grant MOE-T2EP20120-0011 from the Singapore Ministry of Education.

## References

  * [1] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 208–224. Springer, 2020.
  * [2] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE international conference on computer vision, pages 609–617, 2017.
  * [3] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In Proceedings of the European conference on computer vision (ECCV), pages 435–451, 2018.
  * [4] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836–6846, 2021.
  * [5] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  Advances in neural information processing systems, 29:892–900, 2016.
  * [6] Philip Bachman, Ouais Alsharif, and Doina Precup.  Learning with pseudo-ensembles.  Advances in neural information processing systems, 27, 2014.
  * [7] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  Advances in neural information processing systems, 32, 2019.
  * [8] Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu, and Limin Wang.  Joint-modal label denoising for weakly-supervised audio-visual video parsing.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV, pages 431–448. Springer, 2022.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Rapantzikos, Georgios Skoumas, and Yannis Avrithis.  Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention.  IEEE Transactions on Multimedia, 15(7):1553–1568, 2013.
  * [11] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  In European Conference on Computer Vision, pages 214–229. Springer, 2020.
  * [12] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba.  Music gesture for visual sound separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478–10487, 2020.
  * [13] Ruohan Gao, Changan Chen, Ziad Al-Halah, Carl Schissler, and Kristen Grauman.  Visualechoes: Spatial image representation learning through echolocation.  In European Conference on Computer Vision, pages 658–676. Springer, 2020.
  * [14] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457–10467, 2020.
  * [15] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780. IEEE, 2017.
  * [16] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra.  Omnivore: A single model for many visual modalities.  arXiv preprint arXiv:2201.08377, 2022.
  * [17] Wangli Hao, Zhaoxiang Zhang, He Guan, and Guibo Zhu.  Integrating both visual and audio cues for enhanced video caption.  In AAAI Conference on Artificial Intelligence, 2018.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [19] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  Cnn architectures for large-scale audio classification.  In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131–135. IEEE, 2017.
  * [20] Di Hu, Feiping Nie, and Xuelong Li.  Deep multimodal clustering for unsupervised audiovisual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9248–9257, 2019.
  * [21] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In International Conference on Machine Learning, pages 4651–4664. PMLR, 2021.
  * [22] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  arXiv preprint arXiv:1807.00230, 2018.
  * [23] Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang.  Dual-modality seq2seq network for audio-visual event localization.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002–2006. IEEE, 2019.
  * [24] Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang.  Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing.  Advances in Neural Information Processing Systems, 34, 2021.
  * [25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [26] Pedro Morgado, Yi Li, and Nuno Nvasconcelos.  Learning representations from audio-visual spatial alignment.  Advances in Neural Information Processing Systems, 33:4733–4744, 2020.
  * [27] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos.  Robust audio-visual instance discrimination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934–12945, 2021.
  * [28] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra.  Audio-visual instance discrimination with cross-modal agreement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475–12486, 2021.
  * [29] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  Advances in Neural Information Processing Systems, 34, 2021.
  * [30] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018.
  * [31] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid.  Look before you speak: Visually contextualized utterances.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16877–16887, 2021.
  * [32] Xindi Shang, Zehuan Yuan, Anran Wang, and Changhu Wang.  Multimodal video summarization via time-aware transformers.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1756–1765, 2021.
  * [33] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  Advances in neural information processing systems, 33:596–608, 2020.
  * [34] Yapeng Tian, Dingzeyu Li, and Chenliang Xu.  Unified multisensory perception: Weakly-supervised audio-visual video parsing.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 436–454. Springer, 2020.
  * [35] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in unconstrained videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263, 2018.
  * [36] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [38] Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang.  Not only look, but also listen: Learning multimodal violence detection under weak supervision.  In European Conference on Computer Vision, pages 322–339. Springer, 2020.
  * [39] Yu Wu and Yi Yang.  Exploring heterogeneous clues for weakly-supervised audio-visual video parsing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1326–1335, 2021.
  * [40] Karren Yang, Bryan Russell, and Justin Salamon.  Telling left from right: Learning spatial correspondence of sight and sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9932–9941, 2020.
  * [41] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba.  The sound of motions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1735–1744, 2019.

## Supplementary Material

##  Appendix A Additional Analysis of CAPC.

Tab. A1 presents the results of detecting visual events by different models.
We split the visual events into visual-exclusive and multi-modality events.
Visual-exclusive events refers to events only happening in the visual
modality, while multi-modality events appear on audio and visual streams with
temporal overlap. The results are averaged F-scores per event. As shown in
Tab. A1, the results for the visual-exclusive events consistently improve with
an increase in the value of $N$, suggesting the CAPC can effectively reduce
the influence of unmatched audio context. However, the accuracy of multi-
modality events drops when $N=3$, suggesting that large $N$ can impede the
positive impact of audio on its correlated visual event. Only a small number
of $N$, _i.e_. $N=1$, improves fusion effectiveness for both events.

Model | Visual Event
---|---
Visual-exclusive | Multi-modality
No CAPC | 35.6 | 63.0
N=1 | 38.5 | 63.6
N=3 | 38.6 | 62.9

Table A1: Additional Analysis of cross-audio prediction consistency (CAPC).

##  Appendix B Qualitative Comparison with HAN

Fig. B1 shows the qualitative results. HAN [34] fails to identify the single-
modality event (’Speech’) in Fig. B1(a) or wrongly detect it on two modalities
in Fig. B1(b), suggesting the information on the audio and visual streams are
highly confounded. In contrast, our model can correctly detect the audio and
visual events.

Figure B1: Qualitative comparison with HAN.

##  Appendix C Illustration of Single-modality and Multi-modality Event

Single-modality event in Table 1 of main paper refers to events only happening
in one modality, while audio (visual) event in Table 2 includes event both
only happening in the audio (visual) and happening in the audio and visual
modality.

Multi-modality event in Table 1 of main paper refers to event appearing with
temporal overlap (either partial or full overlap) on audio and visual streams,
while audio-visual event in Table 2 refers to event with full overlap on audio
and visual streams.

The split of single-modality and multi-modality event in Table 1 is to
illustrate that the strong entanglement with another non-fully correlated
modality is harmful in detecting its own exclusive events. The evaluation of
audio, visual and audio-visual event in Table 2 is the standard benchmark of
audio-visual video parsing task [34].


#  Stability for backward problems in time for degenerate parabolic equations

1 Piermarco Cannarsa and 2 Masahiro Yamamoto

###### Abstract.

For solution $u(x,t)$ to degenearte parabolic equations in a bounded domain
$\Omega$ with homogenous boundary condition, we consider backward problems in
time: determine $u(\cdot,t_{0})$ in $\Omega$ by $u(\cdot,T)$, where $t$ is the
time variable and $0\leq t_{0}<T$. Our main results are conditional stability
under boundedness assumptions on $u(\cdot,0)$. The proof is based on a
weighted $L^{2}$-estimate of $u$ whose weight depends only on $t$, which is an
inequality of Carleman’s type. Moreover our method is applied to semilinear
degenerate parabolic equations.
Key words. degenerate parabolic equation, backward problem, Carleman estimate,
stability
AMS subject classifications. 35R30, 35K65, 35R25

1 Dipartimento di Matematica, Università di Roma Tor Vergata,
Via della Ricerca Scientifica, 00133, Roma, Italy
e-mail<EMAIL_ADDRESS>
2 Graduate School of Mathematical Sciences, The University of Tokyo
3-8-1 Komaba, Meguro-ku, Tokyo, 153-8914, Japan.
e-mail<EMAIL_ADDRESS>
##  1\. Introduction

The backward problem for parabolic equations is a typical ill-posed problem,
but it is practically meaningful for applications such as estimation of past
temperature by means of current temperature distribution, which is related to
thermo-archaelogy or adequate policies for the global warming by estimating
the past temperature.

Therefore the mathematical analysis is demanded and the uniqueness and the
stability are main theoretical topics. The backward problem in time for the
parabolic equation is severelly ill-posed, but it is known that we can recover
stability under a priori boundedness condition. This recovered stabiliy is
called conditional stability and a priori boundness recovering the stability,
can be introduced from physical viewpoints. For example, in the heat
conduction, such an a priori boundedness can be a fusion point of the material
under consideration. Moreover, the parabolic equations of degenerate type are
physically important, and we can refer to enormous references, but here only
to the monograph by Cannarsa, Martinez and Vanconstanoble [3], which describes
physical backgrounds of the degenerate parabolic equations and also Carleman
estmates with the applications to inverse problems with comprehensive
references.

The main purpose of this article is to establish the conditional stability for
degenerate parabolic equations. To the best knowledge of the authors, there
are no works on the backward problems for degenerate parabolic equations which
aim at comprehensive and systematic researches.

Now we formulate our problem. Let $\Omega\subset\mathbb{R}^{d}$ be a bounded
domain with smooth boundary $\partial\Omega$. We consider a degenerate
parabolic equation with boundary condition:

$(Lu)(x,t):=\partial_{t}u(x,t)-\sum_{i,j=1}^{d}\partial_{i}(a_{ij}(x,t)\partial_{j}u(x,t))-\sum_{k=1}^{d}b_{k}(x,t)\partial_{k}u(x,t)-c(x,t)u(x,t)$
$None$ $=F(x,t)$

for $(x,t)\in Q:=\Omega\times(0,T)$, and

$None$ $u=0\quad\mbox{on $\partial\Omega\times(0,T)$}$

or

$None$ $\partial_{\nu_{A}}u+r(x)u=0\quad\mbox{on
$\partial\Omega\times(0,T)$}.$

Here $\nu(x)=(\nu_{1}(x),....,\nu_{d}(x))$ denotes the unit outward normal
vector to $\partial\Omega$ and we set

$\partial_{\nu_{A}}v(x)=\sum_{i,j=1}^{d}a_{ij}(x)\nu_{i}\partial_{j}v(x)\quad\mbox{for
$x\in\partial\Omega$},$

and we assume that $r\in C(\partial\Omega)$. We assume that

$None$ $a_{ij}\in C^{1}([0,T];L^{\infty}(\Omega)),\quad a_{ij}=a_{ji},\quad
1\leq i,j\leq d,$

and there exists a function $\sigma\in C(\overline{\Omega})$, $\geq 0$ on
$\overline{\Omega}$ such that

$None$
$\sum_{i,j=1}^{d}a_{ij}(x,t)\xi_{i}\xi_{j}\geq\sigma(x)\sum_{k=1}^{d}\xi_{k}^{2},\quad(x,t)\in
Q,\,\xi_{1},...,\xi_{d}\in\mathbb{R},$

and $b:=(b_{1},...,b_{d})\in L^{\infty}(Q)$, $c\in L^{\infty}(Q)$.

We set

$H^{2,1}(Q):=\left\\{u\in
L^{2}(Q);\,\partial_{t}u,\partial_{i}u,\partial_{i}\partial_{j}u\in
L^{2}(Q),\quad 1\leq i,j\leq d\right\\}.$

Let $u\in H^{2,1}(Q)$ satisfy (1.1). Then we consider
Backward problem.
Let $0\leq t_{0}<T$ be given. Determine $u(x,t_{0})$, $x\in\Omega$ by
$u(x,T)$, $x\in\Omega$.

The conditional stability has been studied well for the non-degenerate
parabolic equation where

$None$ $\sigma(x)>0\quad\mbox{on $\overline{\Omega}$}.$

For the case (1.6), as available methodologies, we refer to

  * •

Logarithmic convexity: Ames and Straughan [2], Payne [9], Chapter 3 in Isakov
[4], for example.

  * •

the time analyticity and the maximum principle for holomorphic functions:
Kreĭn and Prozorovskaya [5].

  * •

Weight energy methods: [2], Lees and Protter [6], Payne [9].

Our main purpose of this article is to establish the conditional stability for
the backward problem in time for the degenerate case

$None$ $\sigma(x)\geq 0,\quad x\in\overline{\Omega},$

which means that $\sigma(x)$ in (1.5) is admitted to have zeros.

We introduce the main assumptions on the degeneracy:

There exists a constant $\lambda_{1}>0$ such that
$None$
$\sum_{i,j=1}^{d}(\lambda_{1}a_{ij}(x,t)-\partial_{t}a_{ij}(x,t))\xi_{i}\xi_{j}\geq
0,\quad(x,t)\in\overline{Q},\,\xi_{1},...,\xi_{d}\in\mathbb{R}.$

and

$None$ $\mbox{there exists a constant $C>0$ such that}\quad|b(x,t)|\leq
C\sqrt{\sigma(x)},\quad(x,t)\in\overline{Q}.$

If $|b(x,t)|\equiv 0$ in $Q$, then (1.9) is automatically satisfied.

Examples of (1.8) - (1.9).
(a) If $a_{ij}(x)$, $1\leq i,j\leq d$ are $t$-independent, then (1.8) is
satisfied.
(b) Let $b\equiv 0$ in $Q$. We assume that we can choose $\mu_{k}\in
C(\overline{\Omega})$, $\mu_{k}\geq 0$ on $\overline{\Omega}$,
$\widehat{a_{ij}^{k}}\in C^{1}([0,T];L^{\infty}(\Omega))$, $1\leq i,j\leq d$,
such that there exists a constant $\sigma_{1}>0$ such that

$a_{ij}(x,t)=\sum_{k=1}^{N}\mu_{k}(x)\widehat{a_{ij}^{k}}(x,t),$ $None$
$\sum_{i,j=1}^{d}\widehat{a_{ij}^{k}}(x,t)\xi_{i}\xi_{j}\geq\sigma_{1}\sum_{j=1}^{d}\xi_{j}^{2}\quad(x,t)\in\overline{Q},\,\xi_{1},...,\xi_{d}\in\mathbb{R},\quad
1\leq k\leq N.$

Then (1.8) - (1.9) are satisfied. For example,
$\widehat{a_{ij}^{k}}(x,t)=\delta_{ij}:=\left\\{\begin{array}[]{rl}1&i=j,\\\
0&i\neq j\end{array}\right.$ for $1\leq k\leq N$ and
$\mu_{k}(x)=r_{k}|x-x_{0}^{k}|^{\rho_{k}}$ with $x_{0}^{k}\in\Omega$ and
$r_{k}>0$, $\rho_{k}>0$ are constants for $1\leq k\leq N$.
(c) Let $\mu\in C(\overline{\Omega})$, $\geq 0$ on $\overline{\Omega}$,
$\widehat{a_{ij}}\in C^{1}([0,T];L^{\infty}(\Omega))$, $b_{k}\in
L^{\infty}(Q)$, $1\leq i,j,k\leq d$, $a_{ij}(x,t)=\mu(x)\widehat{a_{ij}}(x,t)$
and $b_{k}(x,t)=\mu(x)\widehat{b_{k}}(x,t)$ for $1\leq i,j,k\leq d$. We
further assume that we can find a constant $\sigma_{1}>0$ such that

$\sum_{i,j=1}^{d}\widehat{a_{ij}}(x,t)\xi_{i}\xi_{j}\geq\sigma_{1}\sum_{k=1}^{d}\xi_{k}^{2}\quad(x,t)\in\overline{Q},\,\xi_{1},...,\xi_{d}\in\mathbb{R}.$

Then (1.8) is satisfied.

Now we are ready to state our main results.
Theorem 1 (case $0<t_{0}<T$).
We assume (1.8) and (1.9), and $u\in H^{2,1}(Q)$ satisfy (1.1) and

$None$ $\|u(\cdot,0)\|_{H^{1}(\Omega)}\leq M$

with arbitrarily chosen constant $M>0$.
Case (1.2).
Then for $0<t_{0}<T$, there exist constants $C>0$ and $\theta\in(0,1)$
dependent on $t_{0}$ and $M$ such that

$None$ $\|u(\cdot,t_{0})\|_{L^{2}(\Omega)}\leq
C(\|u(\cdot,T)\|^{\theta}_{H^{1}(\Omega)}+\|u(\cdot,T)\|_{H^{1}(\Omega)}).$

Case (1.3).
We further assume that $\sigma(x)$ in (1.5) satisfies

$None$ $\sigma(x)>0\quad\mbox{for $x\in\partial\Omega$}.$

Then we have (1.12).
Theorem 2 (case $t_{0}=0$).
We assume (1.8) and (1.9), and $u\in H^{2,1}(Q)$ satisfy
$\partial_{t}u,\partial_{t}^{2}u\in H^{2,1}(Q)$ and (1.1), and

$None$ $\sum_{k=0}^{2}\|\partial_{t}^{k}u(\cdot,0)\|_{H^{1}(\Omega)}\leq M$

with arbitrarily chosen constant $M>0$.
Case (1.2).
Then, for any $\alpha\in(0,1)$, there exists a constant $C>0$ such that

$None$ $\|u(\cdot,0)\|_{L^{2}(\Omega)}\leq
C\left(\log\frac{1}{D}\right)^{-\alpha},$

provided that

$D:=\sum_{k=0}^{2}\|\partial_{t}^{k}u(\cdot,T)\|_{H^{1}(\Omega)}$

is small.
Case (1.3).
Assume (1.13) additionally. Then estimate (1.15) holds.

Our method is quite feasible, and is applicable for example, to semilinear
equations. We can consider more comprehensive class of nonlinear equations,
but for discussing the essence, we are restricted to the following case with
$0<t_{0}<T$:

$None$
$\left\\{\begin{array}[]{rl}&\partial_{t}u(x,t)=\sum_{i,j=1}^{d}\partial_{i}(a_{ij}(x,t)\partial_{j}u(x,t))+c(x,t)u+f(x,t,u(x,t)),\quad(x,t)\in
Q,\\\ &\partial_{\nu_{A}}u+r(x)u=0\quad\mbox{or}\quad u=0\quad\mbox{on
$\partial\Omega\times(0,T)$},\end{array}\right.$

where $f(x,t,\eta)$, $x\in\overline{\Omega}$, $0\leq t\leq T$ and
$\eta\in\mathbb{R}$, satisfies

$None$ $f,\,\partial_{\eta}f\in
C(\overline{\Omega}\times[0,T]\times\mathbb{R}).$

Then we can prove
Theorem 3.
Let $0<t_{0}<T$. Let $u,v\in H^{2,1}(Q)$ satisfy (1.16) and

$\|u(\cdot,0)\|_{H^{1}(\Omega)},\,\|v(\cdot,0)\|_{H^{1}(\Omega)}\leq
M,\quad\|u\|_{L^{\infty}(Q)},\,\|v\|_{L^{\infty}(Q)}\leq M$

with arbitrarily chosen constant $M>0$. Then there exist constants $C>0$ and
$\theta\in(0,1)$, depending on $t_{0}$ and $M$, such that

$\|u(\cdot,t)-v(\cdot,t)\|_{L^{2}(\Omega)}\leq
C(\|u(\cdot,T)-v(\cdot,T)\|^{\theta}_{H^{1}(\Omega)}+\|u(\cdot,T)-v(\cdot,T)\|_{H^{1}(\Omega)}).$

This article is composed of four sections. In Section 2, we prove a key
estimate of a Carleman type. Sections 3 and 4 are devoted to the proofs of
Theorem 1 and 3, and Theorem 2 respectively.

##  2\. Key estimate of a Carleman estimate

We set

$\varphi(t)=e^{\lambda t},\quad t>0,$

where a constant $\lambda>0$ is chosen later. We state the key inequality
without assumptions (1.8) and (1.9) on the degeneracy.
Lemma 1.
Case (1.2).
There exists a constant $\lambda_{0}>0$ such that for any
$\lambda>\lambda_{0}$, we can choose a constant $s_{0}(\lambda)>0$ satisfying:
there exist constants $C=C(s_{0},\lambda_{0})>0$ and $C_{0}>0$ such that

$None$
$\int_{Q}\Biggl{\\{}\frac{1}{s\varphi}|\partial_{t}u|^{2}+\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}u)(\partial_{j}u)+s\lambda^{2}\varphi|u|^{2}\Biggr{\\}}e^{2s\varphi}dxdt$
$\displaystyle\leq$ $\displaystyle
C\int_{Q}|F|^{2}e^{2s\varphi}dxdt+C\int_{Q}\sum_{j=1}^{d}|b_{j}\partial_{j}u|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}$
$\displaystyle+$ $\displaystyle
C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}$

for all $s>s_{0}$ and all $u\in H^{2,1}(Q)$ satisfying $Lu=F$ in $Q$ and the
boundary condition (1.2).
Case (1.3).
There exists a constant $\lambda_{0}>0$ such that for any
$\lambda>\lambda_{0}$, we can choose a constant $s_{0}(\lambda)>0$ satisfying:
there exist constants $C=C(s_{0},\lambda_{0})>0$ and $C_{0}>0$ such that

$None$
$\int_{Q}\Biggl{\\{}\frac{1}{s\varphi}|\partial_{t}u|^{2}+\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}u)(\partial_{j}u)+s\lambda^{2}\varphi|u|^{2}\Biggr{\\}}e^{2s\varphi}dxdt$
$\displaystyle\leq$ $\displaystyle
C\int^{T}_{0}\int_{\partial\Omega}\lambda|u|^{2}e^{2s\varphi}dSdt$
$\displaystyle+$ $\displaystyle
C\int_{Q}|F|^{2}e^{2s\varphi}dxdt+C\int_{Q}\sum_{j=1}^{d}|b_{j}\partial_{j}u|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}+C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}$

for all $s>s_{0}$ and all $u\in H^{2,1}(Q)$ satisfying $Lu=F$ in $Q$ and the
boundary condition (1.3).

Here the constant $C_{0}$ is independent of $s_{0},\lambda_{0}$.

We emphasize that we do not assume (1.8) and (1.9), so that Lemma 1 holds true
if

$\sum_{i,j=1}^{d}a_{ij}(x,t)\xi_{i}\xi_{j}\geq
0,\quad(x,t)\in\overline{Q},\,\xi_{1},...,\xi_{d}\in\mathbb{R}.$

In particular,
Proposition 1.
We assume

$\left\\{\begin{array}[]{rl}&\mbox{$a_{ij}$, $1\leq i,j\leq d$ are time-
independent and $b_{1}=\cdots=b_{d}=0$ on $\overline{Q}$},\\\
&\sum_{i,j=1}^{d}a_{ij}(x)\xi_{i}\xi_{j}\geq 0,\quad
x\in\overline{\Omega},\,\xi_{1},...,\xi_{d}\in\mathbb{R}.\\\
\end{array}\right.$

Then exists a constant $\lambda_{0}>0$ such that for any
$\lambda>\lambda_{0}$, we can choose a constant $s_{0}(\lambda)>0$ satisfying:
there exists a constant $C=C(s_{0},\lambda_{0})>0$ such that

$\displaystyle\int_{Q}\Biggl{\\{}\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\Biggr{\\}}e^{2s\varphi}dxdt\leq
C\int_{Q}|F|^{2}e^{2s\varphi}dxdt$ $\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}+C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}$

for all $s>s_{0}$ and all $u\in H^{2,1}(Q)$ satisfying $Lu=F$ in $Q$ and the
boundary condition (1.2), and

$\displaystyle\int_{Q}\Biggl{\\{}\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\Biggr{\\}}e^{2s\varphi}dxdt\leq
C\int^{T}_{0}\int_{\partial\Omega}\lambda|u|^{2}e^{2s\varphi}dSdt$
$\displaystyle+$ $\displaystyle C\int_{Q}|F|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}+C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}$

for all $s>s_{0}$ and all $u\in H^{2,1}(Q)$ satisfying $Lu=F$ in $Q$ and the
boundary condition (1.3).
Remark 2.1.
As a method with a similar spirit, we can refer to the weight energy method.
Concerning the weight energy method, there are many papers and see monographs
Ames and Straughan [2], Lees and Protter [6], Payne [9], and the references
therein. Except for Murray and Protter [8] for equations of hyperbolic types,
all the papers use just $t$ as weight function, and do not use the second
large parameter $\lambda$. In Murray and Protter [8], the weight function
$e^{st^{\lambda}}$ is used to prove properties for the asymptotic behaviour.

The essential diffference from the existing papers is the introduction of the
second large parameter $\lambda>0$. Such a second large parameter is very
flexible and gains a lot of possibility for better estimates.

Proof of Lemma 1.
First Step.
Set

$\displaystyle
Lu:=\partial_{t}u-\sum_{i,j=1}^{d}\partial_{i}(a_{ij}(x)\partial_{j}u),\quad
G:=F(x,t)+\sum_{k=1}^{d}b_{k}(x,t)\partial_{k}u(x,t),$ $\displaystyle
w=e^{s\varphi}u,\quad Pw=e^{s\varphi}L(e^{-s\varphi}w)=e^{s\varphi}G.$

Then

$e^{s\varphi}\partial_{t}(e^{-s\varphi}w)=\partial_{t}v-s\lambda\varphi w,$
$e^{s\varphi}\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}(we^{-s\varphi}))=\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w),$

and

$Pw=e^{s\varphi}L(e^{-s\varphi}w)=\partial_{t}w-\left(s\lambda\varphi
w+\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)\right)=e^{s\varphi}G.$

We have

$\displaystyle\|e^{s\varphi}G\|^{2}_{L^{2}(Q)}$ $\displaystyle=$
$\displaystyle\int_{Q}|\partial_{t}w|^{2}dxdt+2\int_{Q}(\partial_{t}w)\left(-s\lambda\varphi
w-\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)\right)dxdt$
$\displaystyle+$ $\displaystyle\int_{Q}\left|s\lambda\varphi
w+\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)\right|^{2}dxdt$
$\displaystyle\geq$
$\displaystyle\int_{Q}|\partial_{t}w|^{2}dxdt+2\int_{Q}\partial_{t}w\left(-\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)\right)dxdt$
$\displaystyle+$ $\displaystyle
2\int_{Q}(\partial_{t}w)(-s\lambda\varphi)wdxdt$ $None$
$=:\int_{Q}|\partial_{t}w|^{2}dxdt+J_{1}+J_{2}.$

Thus

$None$ $\int_{Q}|G|^{2}e^{2s\varphi}dxdt\geq J_{1}+J_{2}$

and

$None$ $\int_{Q}|\partial_{t}w|^{2}dxdt\leq\int_{Q}|G|^{2}e^{2s\varphi}dxdt-
J_{1}-J_{2}.$

Henceforth $C_{j}>0$ denote generic constants which are independent of
$s,\lambda$. We assume that $s>1$ and $\lambda>1$.

First we consider the boundary condition (1.2). By noting $a_{ij}=a_{ji}$, the
boundary condition (1.2) and integration by parts yields

$None$
$J_{1}=-2\int_{Q}(\partial_{t}w)\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)dxdt=2\int_{Q}\sum_{i,j=1}^{d}(\partial_{i}\partial_{t}w)a_{ij}(\partial_{j}w)dxdt$
$\displaystyle=$ $\displaystyle
2\int_{Q}\sum_{i>j}a_{ij}((\partial_{j}w)\partial_{i}\partial_{t}w+(\partial_{i}w)\partial_{j}\partial_{t}w)\,dxdt+2\sum_{i=1}^{n}\int_{Q}a_{ii}(\partial_{i}w)(\partial_{i}\partial_{t}w)\,dxdt$
$\displaystyle=$ $\displaystyle
2\int_{Q}\sum_{i>j}a_{ij}\partial_{t}((\partial_{i}w)(\partial_{j}w))\,dxdt+\int_{Q}\sum_{i=1}^{n}a_{ii}\partial_{t}((\partial_{i}w)^{2})\,dxdt$
$\displaystyle=$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}a_{ij}\partial_{t}((\partial_{i}w)\partial_{j}w)dxdt$
$\displaystyle=$
$\displaystyle-\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)\partial_{j}w\,dxdt+\sum_{i,j=1}^{d}\left[a_{ij}(\partial_{i}w)(\partial_{j}w)\right]^{t=T}_{t=0}dx$
$\displaystyle=$
$\displaystyle-\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)\,dxdt$
$\displaystyle+$
$\displaystyle\int_{\Omega}\sum_{i,j=1}^{d}(a_{ij}(x,T)(\partial_{i}w)(x,T)(\partial_{j}w)(x,T)-a_{ij}(x,0)(\partial_{i}w)(x,0)(\partial_{j}w)(x,0))dx.$

On the other hand,

$\displaystyle J_{2}=-s\lambda\int_{Q}2(\partial_{t}w)w\varphi
dxdt=-s\lambda\int_{Q}\partial_{t}(w^{2})\varphi dxdt$ $\displaystyle=$
$\displaystyle
s\lambda\int_{Q}(\partial_{t}\varphi)w^{2}dxdt-s\lambda\int_{\Omega}\left[\varphi
w^{2}\right]^{t=T}_{t=0}dx$ $None$ $=s\lambda^{2}\int_{Q}\varphi
w^{2}dxdt-s\lambda\int_{\Omega}(\varphi(T)|w(x,T)|^{2}-|w(x,0)|^{2})dx.$

Hence

$\|e^{s\varphi}G\|^{2}_{L^{2}(Q)}\geq s\lambda^{2}\int_{Q}\varphi
w^{2}dxdt-\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
$None$ $-s\lambda\int_{\Omega}(\varphi(T)|w(x,T)|^{2}+|w(x,0)|^{2})dx-
C_{1}\int_{\Omega}(|\nabla w(x,T)|^{2}+|\nabla w(x,0)|^{2})dx.$

Second Step.
On the right-hand side of (2.8), the term
$-\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
appears, and so we have to estimate the integrals including
$(\partial_{i}w)(\partial_{j}w)$. For it, we consider $\int_{Q}(Pw)w\,dxdt$:

$\displaystyle\int_{Q}(Pw)wdxdt=\int_{Q}(\partial_{t}w)wdxdt-\int_{Q}s\lambda\varphi
w^{2}dxdt-\int_{Q}\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)w\,dxdt$
$\displaystyle=:$ $\displaystyle I_{1}+I_{2}+I_{3}.$

We have

$\displaystyle|I_{1}|=\left|\int_{Q}(\partial_{t}w)wdxdt\right|=\left|\frac{1}{2}\int_{Q}\partial_{t}(w^{2})dxdt\right|$
$\displaystyle=$
$\displaystyle\left|\frac{1}{2}\int_{\Omega}\left[|w(x,t)|^{2}\right]^{t=T}_{t=0}dx\right|\leq\frac{1}{2}\int_{\Omega}(|w(x,T)|^{2}+|w(x,0)|^{2})dx.$

Next

$|I_{2}|=\left|-\int_{Q}s\lambda\varphi w^{2}dxdt\right|\leq
C_{2}\int_{Q}s\lambda\varphi w^{2}dxdt$

and

$None$
$I_{3}=-\sum_{i,j=1}^{d}\int_{Q}\partial_{i}(a_{ij}\partial_{j}w)wdxdt=\sum_{i,j=1}^{d}\int_{Q}a_{ij}(\partial_{j}w)(\partial_{i}w)dxdt.$

Hence

$\int_{Q}\lambda(Pw)w\,dxdt\geq\lambda\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt-
C_{2}\int_{Q}s\lambda^{2}\varphi w^{2}dxdt$ $None$
$-\frac{1}{2}\lambda\int_{\Omega}(|w(x,T)|^{2}+|w(x,0)|^{2})dx.$

On the other hand,

$\left|\int_{Q}\lambda(Pw)w\,dxdt\right|\leq\|Pw\|_{L^{2}(Q)}(\lambda\|w\|_{L^{2}(Q)})\leq\frac{1}{2}\|Pw\|^{2}_{L^{2}(Q)}+\frac{\lambda^{2}}{2}\|w\|_{L^{2}(Q)}^{2}$
$None$
$=\frac{1}{2}\int|G|^{2}e^{2s\varphi}dxdt+\frac{\lambda^{2}}{2}\|w\|_{L^{2}(Q)}^{2}.$

Hence (2.10) yields

$\displaystyle\lambda\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt\leq
C_{2}\int_{Q}s\lambda^{2}\varphi w^{2}dxdt$ $\displaystyle+$
$\displaystyle\frac{1}{2}\int_{Q}|Ge^{s\varphi}|^{2}dxdt+\frac{\lambda^{2}}{2}\int_{Q}w^{2}dxdt+\frac{1}{2}\lambda\int_{\Omega}(|w(x,T)|^{2}+|w(x,0)|^{2})dx.$

Estimating the first term on the right-hand side by (2.8), we obtain

$\displaystyle\lambda\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle\leq$ $\displaystyle
C_{3}\int_{Q}|Ge^{s\varphi}|^{2}dxdt+C_{3}\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)\,dxdt+C_{3}\int_{Q}\lambda^{2}w^{2}dxdt$
$\displaystyle+$ $\displaystyle
C_{3}\lambda(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})+C_{3}(\|\nabla
w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle+$ $\displaystyle
C_{3}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle\leq$ $\displaystyle
C_{3}\int_{Q}|Ge^{s\varphi}|^{2}dxdt+C_{3}\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)\,dxdt+C_{3}\int_{Q}\lambda^{2}w^{2}dxdt$
$None$
$+C_{3}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})+C_{3}(\|\nabla
w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$

Third Step.
Adding (2.8) and (2.12), we have

$\displaystyle\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\lambda\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle\leq$ $\displaystyle
C_{4}\int_{Q}|Ge^{s\varphi}|^{2}dxdt+C_{3}\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+C_{4}\int_{Q}\lambda^{2}w^{2}dxdt$
$\displaystyle+$ $\displaystyle C_{4}(\|\nabla
w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,0)\|^{2}_{L^{2}(\Omega)})+C_{4}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}).$

By $\varphi=e^{\lambda t}\geq 1$, we take $s>0$ and $\lambda>0$ large to
absorb the third term on the right-hand side into the left-hand side. Hence,

$\displaystyle\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\int_{Q}\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{3}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle\leq$ $\displaystyle C_{5}\int_{Q}|Ge^{s\varphi}|^{2}dxdt$
$\displaystyle+$ $\displaystyle
C_{5}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$None$ $+C_{5}(\|\nabla w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,0)\|^{2}_{L^{2}(\Omega)}).$

Fourth Step.
Next we will estimate $|\partial_{t}w|^{2}$. Since $u=e^{-s\varphi}w$, we have
$\partial_{t}u=-s\lambda\varphi e^{-s\varphi}w+e^{-s\varphi}\partial_{t}w$,
and

$\frac{1}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}\leq 2s\lambda^{2}\varphi
w^{2}+\frac{2}{s\varphi}|\partial_{t}w|^{2}.$

Let $\varepsilon\in\left(0,\frac{1}{2}\right)$ be a constant which we choose
later. We note that $\frac{1}{s\varphi}=\frac{1}{se^{\lambda
t}}\leq\frac{1}{s}\leq 1$ for $s\geq 1$. Therefore, for all large $s>0$ and
$\lambda>0$, we have

$\displaystyle\int_{Q}\frac{\varepsilon}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}dxdt\leq\int_{Q}2\varepsilon
s\lambda^{2}\varphi
w^{2}dxdt+\int_{Q}\frac{2\varepsilon}{s\varphi}|\partial_{t}w|^{2}dxdt$
$\displaystyle\leq$ $\displaystyle 2\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\varepsilon\int_{Q}|\partial_{t}w|^{2}dxdt$ $None$ $\leq
2\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\varepsilon\int_{Q}G^{2}e^{2s\varphi}dxdt+\varepsilon(-J_{1}-J_{2})$

by (2.5).

By (2.6) and (2.7), we have

$\displaystyle\varepsilon(-J_{1}-J_{2})=\int_{Q}\varepsilon\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt-s\lambda^{2}\varepsilon\int_{Q}\varphi
w^{2}dxdt$ $\displaystyle-$
$\displaystyle\varepsilon\int_{\Omega}\sum_{i,j=1}^{d}(a_{ij}(x,T)(\partial_{i}w)(x,T)(\partial_{j}w)(x,T)-a_{ij}(x,0)(\partial_{i}w)(x,0)(\partial_{j}w)(x,0))dx$
$\displaystyle+$ $\displaystyle\varepsilon
s\lambda\int_{\Omega}(\varphi(T)|w(x,T)|^{2}-|w(x,0)|^{2})dx,$

and the substitution of this into (2.13) yields

$\displaystyle\varepsilon\int_{Q}\frac{1}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}dxdt\leq
2\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\varepsilon\int_{Q}|G|^{2}e^{2s\varphi}dxdt$ $\displaystyle+$
$\displaystyle\sum_{i,j=1}^{d}\int_{Q}\varepsilon(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt-\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt$ $\displaystyle+$ $\displaystyle
C_{6}\varepsilon\int_{\Omega}(|\nabla w(x,T)|^{2}+|\nabla w(x,0)|^{2})dx$
$None$ $+\varepsilon
s\lambda\int_{\Omega}(\varphi(T)|w(x,T)|^{2}+|w(x,0)|^{2})dx.$

Adding (2.13) and (2.15), we obtain

$\displaystyle\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\int_{Q}\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{3}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+\varepsilon\int_{Q}\frac{1}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}dxdt$
$\displaystyle\leq$ $\displaystyle
C_{7}\int_{Q}|G|^{2}e^{2s\varphi}dxdt+\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt$ $\displaystyle+$
$\displaystyle\int_{Q}\varepsilon\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle+$ $\displaystyle
C_{7}(s\lambda\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,T)\|_{L^{2}(\Omega)}^{2})$ $\displaystyle+$ $\displaystyle
C_{7}(s\lambda\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,0)\|_{L^{2}(\Omega)}^{2}).$

Choosing $0<\varepsilon<\frac{1}{2}$, we can absorb the second term on the
right-hand side into the left-hand side, we obtain

$\displaystyle\varepsilon\int_{Q}\frac{1}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle\int_{Q}\sum_{i,j=1}^{d}(\lambda
a_{ij}-(C_{3}+\varepsilon)\partial_{t}a_{ij})(\partial_{i}u)(\partial_{j}u)e^{2s\varphi}dxdt+\frac{1}{2}\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt$ $\displaystyle\leq$ $\displaystyle
C_{7}\int_{Q}|G|^{2}e^{2s\varphi}dxdt$ $\displaystyle+$ $\displaystyle
C_{7}(s\lambda\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,T)\|_{L^{2}(\Omega)}^{2})$ $\displaystyle+$ $\displaystyle
C_{7}(s\lambda\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|\nabla
w(\cdot,0)\|_{L^{2}(\Omega)}^{2}).$

Thus the proof of Lemma 1 is complete in the case (1.2).
Fifth Step.
We will prove in the case (1.3). We recall the additional assumption (1.13).
In the above arguments, we need to modify only (2.6) and (2.9). That is,

$\displaystyle
J_{1}=-2\int_{Q}(\partial_{t}w)\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}w)dxdt$
$\displaystyle=$ $\displaystyle
2\int_{Q}\sum_{i,j=1}^{d}(\partial_{i}\partial_{t}w)a_{ij}(\partial_{j}w)dxdt-2\int_{\partial\Omega\times(0,T)}\sum_{i,j=1}^{d}a_{ij}(\partial_{j}w)\nu_{i}(\partial_{t}w)dSdt$
$\displaystyle=$ $\displaystyle
2\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}\partial_{t}w)(\partial_{j}w)dxdt-2\int_{\partial\Omega\times(0,T)}(\partial_{\nu_{A}}w)(\partial_{t}w)dSdt.$

By (1.3), we obtain

$\displaystyle-2\int_{\partial\Omega\times(0,T)}(\partial_{\nu_{A}}w)(\partial_{t}w)dSdt=2\int_{\partial\Omega\times(0,T)}rw(\partial_{t}w)dSdt$
$\displaystyle=$
$\displaystyle\int_{\partial\Omega\times(0,T)}r\partial_{t}(w^{2})dSdt=\int_{\partial\Omega}\left[rw^{2}\right]^{t=T}_{t=0}dS.$

Thus, similarly to (2.6), we have

$\displaystyle-
J_{1}=-2\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}\partial_{t}w)(\partial_{j}w)dxdt-\int_{\partial\Omega}\left[rw^{2}\right]^{t=T}_{t=0}dS$
$\displaystyle=$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle-$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(a_{ij}(x,T)(\partial_{i}w)(x,T)(\partial_{j}w)(x,T)-a_{ij}(x,0)(\partial_{i}w)(x,0)\partial_{j}w(x,0))dx$
$\displaystyle-$ $\displaystyle\int_{\Omega}r(w^{2}(x,T)-w^{2}(x,0))dS$

and by the trace theorem: $\|u\|_{L^{2}(\partial\Omega)}\leq
C\|u\|_{H^{1}(\Omega)}$, we can obtain the same estimate

$None$
$-J_{1}\leq\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+C_{8}(\|w(\cdot,T)\|_{H^{1}(\Omega)}^{2}+\|w(\cdot,0)\|_{H^{1}(\Omega)}^{2}).$

Therefore (2.4) and (2.7) yield

$\displaystyle\int_{Q}|G|^{2}e^{2s\varphi}dxdt\geq J_{1}+J_{2}$
$\displaystyle\geq$ $\displaystyle\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt-s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle-$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt-
C_{8}(\|w(\cdot,T)\|_{H^{1}(\Omega)}^{2}+\|w(\cdot,0)\|_{H^{1}(\Omega)}^{2}),$

that is,

$None$ $\int_{Q}s\lambda^{2}\varphi w^{2}dxdt$ $\displaystyle\leq$
$\displaystyle\int_{Q}|G|^{2}e^{2s\varphi}dxdt+s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle+$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+C_{8}(\|w(\cdot,T)\|_{H^{1}(\Omega)}^{2}+\|w(\cdot,0)\|_{H^{1}(\Omega)}^{2}).$

Moreover, for $I_{3}$ in (2.9), by (1.5) we have

$\displaystyle
I_{3}=\sum_{i,j=1}^{d}\int_{Q}a_{ij}(\partial_{i}w)(\partial_{j}w)dx-\int_{\partial\Omega\times(0,T)}w(\partial_{\nu_{A}}w)\,dSdt$
$\displaystyle=$
$\displaystyle\sum_{i,j=1}^{d}\int_{Q}a_{ij}(\partial_{i}w)(\partial_{j}w)dx+\int_{\partial\Omega\times(0,T)}rw^{2}dSdt$
$\displaystyle\geq$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt-
C_{9}\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}.$

Hence, since we can obtain the same estimates for $I_{1}$ and $I_{2}$, we see

$None$ $\int_{Q}\lambda(Pw)wdxdt=\lambda I_{1}+\lambda I_{2}+\lambda I_{3}$
$\displaystyle\geq$ $\displaystyle\int_{Q}\sum_{i,j=1}^{d}\lambda
a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt-
C_{10}\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}$ $\displaystyle-$
$\displaystyle\frac{1}{2}\lambda(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})-C_{10}\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt.$

Similarly to (2.12), using (2.11), by (2.18) we obtain

$\displaystyle\int_{Q}\sum_{i,j=1}^{d}\lambda
a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt$ $\displaystyle\leq$
$\displaystyle\int_{Q}\lambda(Pw)wdxdt+C_{10}\lambda\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}+\frac{1}{2}\lambda(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle+$ $\displaystyle C_{10}\int_{Q}s\lambda^{2}\varphi w^{2}dxdt$
$\displaystyle\leq$
$\displaystyle\frac{1}{2}\|Ge^{s\varphi}\|^{2}_{L^{2}(Q)}+\frac{\lambda^{2}}{2}\|w\|^{2}_{L^{2}(Q)}+C_{10}\lambda\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}$
$\displaystyle+$
$\displaystyle\frac{1}{2}\lambda(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}+C_{10}\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt.$

Hence (2.17) implies

$None$ $\int_{Q}\sum_{i,j=1}^{d}\lambda
a_{ij}(\partial_{i}w)(\partial_{j}w)dxdt$ $\displaystyle\leq$
$\displaystyle\frac{1}{2}\|Ge^{s\varphi}\|^{2}_{L^{2}(Q)}+C_{11}\lambda^{2}\|w\|^{2}_{L^{2}(Q)}$
$\displaystyle+$ $\displaystyle
C_{11}\lambda\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}+\frac{1}{2}\lambda(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle+$ $\displaystyle
C_{11}\int_{Q}|G|^{2}e^{2s\varphi}dxdr+C_{11}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$
$\displaystyle+$
$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(C_{11}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt$
$\displaystyle+$ $\displaystyle
C_{11}(\|w(\cdot,T)\|_{H^{1}(\Omega)}^{2}+\|w(\cdot,0)\|_{H^{1}(\Omega)}^{2}).$

Thus (2.17) and (2.19) yield

$\displaystyle\int_{Q}\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{11}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt$ $\displaystyle\leq$ $\displaystyle
C_{12}\int_{Q}|G|^{2}e^{2s\varphi}dxdt+C_{12}\lambda\|w\|^{2}_{L^{2}(0,T;L^{2}(\partial\Omega))}+C_{12}(\|w(\cdot,T)\|^{2}_{H^{1}(\Omega)}+\|w(\cdot,0)\|^{2}_{H^{1}(\Omega)})$
$None$
$+C_{12}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}).$

Here we used

$\frac{1}{2}(\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})\leq
C_{12}s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)})$

and we absorbed the term

$C_{11}\lambda^{2}\|w\|^{2}_{L^{2}(Q)}=C_{11}\int_{Q}\lambda^{2}w^{2}dxdt$

into $\int_{Q}s\lambda^{2}\varphi w^{2}dxdt$.

Next as for the estimate of $\partial_{t}w$, we can proceed similarly to the
argument starting (2.14) as follows. In terms of (2.7) and (2.16), we have

$None$ $\varepsilon(-J_{1}-J_{2})$ $\displaystyle\leq$ $\displaystyle
C_{13}\varepsilon\int_{Q}\sum_{i,j=1}^{d}(\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt+C_{13}\varepsilon(\|w(\cdot,T)\|^{2}_{H^{1}(\Omega)}+\|w(\cdot,0)\|^{2}_{H^{1}(\Omega)})$
$\displaystyle-$ $\displaystyle C_{13}\varepsilon\int_{Q}s\lambda^{2}\varphi
w^{2}dxdt+\varepsilon
s\lambda(\varphi(T)\|w(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|w(\cdot,0)\|^{2}_{L^{2}(\Omega)}).$

Thus, by (2.21) and (2.14), we can estimate
$\int_{Q}\frac{1}{s\varphi}|\partial_{t}u|^{2}e^{2s\varphi}dxdt$, and so
(2.21) completes the proof of Lemma 1 in the case (1.3). $\blacksquare$

##  3\. Proofs of Theorems 1 and 3

In Lemma 1, in order to estimate
$\left(\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\right)e^{2s\varphi}$,
we have to assume (1.8) and (1.9) because the term

$\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}u)(\partial_{j}u)e^{2s\varphi}$

is not necessarily non-negative. More precisely, we can state
Lemma 2.
We assume (1.8) and (1.9). In case (1.3), we further assume (1.13). Then there
exists a constant $\lambda_{0}>0$ such that for any $\lambda>\lambda_{0}$, we
can choose a constant $s_{0}(\lambda)>0$ satisfying: there exists a constant
$C=C(s_{0},\lambda_{0})>0$ such that

$None$
$\int_{Q}\Biggl{\\{}\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\Biggr{\\}}e^{2s\varphi}dxdt$
$\displaystyle\leq$ $\displaystyle
C\int_{Q}|F|^{2}e^{2s\varphi}dxdt+C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}$
$\displaystyle+$ $\displaystyle
C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}$

for all $s>s_{0}$ whenever a solution $u\in H^{2,1}(Q)$ satisfies (1.1) with
(1.2) or (1.3).
Proof of Lemma 2.
Case (1.2).
First let $b:=(b_{1},\cdots,b_{d})=0$ in $\Omega$. By (1.8), we can choose
$\lambda>0$ large such that

$None$ $\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})\xi_{i}\xi_{j}\geq 0\quad\mbox{for
$(x,t)\in\overline{Q}$ and $\xi_{1},...,\xi_{d}\in\mathbb{R}$.}$

Indeed, by (1.8) and (1.5) with $\sigma(x)\geq 0$, we verify that

$\displaystyle\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)$
$\displaystyle=$
$\displaystyle\sum_{i,j=1}^{d}(\lambda_{1}C_{0}a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)+(\lambda-\lambda_{1}C_{0})\sum_{i,j=1}^{d}a_{ij}(\partial_{i}w)(\partial_{j}w)$
$\displaystyle\geq$ $\displaystyle
C_{0}\sum_{i,j=1}^{d}(\lambda_{1}a_{ij}-\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)\geq
0\quad\mbox{on $\overline{Q}$},$

provided that $\lambda>0$ is so large that $\lambda-\lambda_{1}C_{0}\geq 0$,
which verifies (3.2).

We set $\varphi(t)=e^{\lambda t}$. Since $b=0$ in $\Omega$, by Lemma 1, we can
readily obtain (3.1).

Second let $b\not\equiv 0$ in $\Omega$. Then (1.8) and (1.5) yield

$\displaystyle\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)$
$\displaystyle=$
$\displaystyle\sum_{i,j=1}^{d}(\lambda-\lambda_{1}C_{0})a_{ij}(\partial_{i}w)(\partial_{j}w)+C_{0}\sum_{i,j=1}^{d}(\lambda_{1}a_{ij}-\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)$
$\displaystyle\geq$
$\displaystyle\sum_{i,j=1}^{d}(\lambda-\lambda_{1}C_{0})a_{ij}(\partial_{i}w)(\partial_{j}w)\geq(\lambda-\lambda_{1}C_{0})\sigma(x)|\nabla
w(x,t)|^{2},\quad(x,t)\in\overline{Q}.$

Therefore, (1.9) implies

$None$ $\int_{Q}\sum_{i,j=1}^{d}(\lambda
a_{ij}-C_{0}\partial_{t}a_{ij})(\partial_{i}w)(\partial_{j}w)dxdt-C\int_{Q}\sum_{j=1}^{d}|b_{j}\partial_{j}w|^{2}dxdt$
$\displaystyle\geq$
$\displaystyle\int_{Q}(\lambda-\lambda_{1}C_{0})\sigma(x)|\nabla
w(x,t)|^{2}dxdt-C\int_{Q}\sigma(x)|\nabla w(x,t)|^{2}dxdt$ $\displaystyle\geq$
$\displaystyle(\lambda-\lambda_{1}C_{0}-C)\int_{Q}\sigma(x)|\nabla
w(x,t)|^{2}dxdt.$

Hence, choosing $\lambda>0$ sufficiently large, we can absorb the second term
on the right-hand side of (2.1) into the left-hand side.

Fixing such sufficiently large $\lambda>0$, we obtain (3.1).
Case (1.3).
We assume that as a neighborhood of $\partial\Omega$, by means of (1.13), we
can find a subdomain $\Omega^{\prime}$ satisfying
$\overline{\Omega^{\prime}}\subset\Omega$ and a constant $\varepsilon_{1}>0$
such that

$None$
$\partial\Omega^{\prime}\supset\partial\Omega,\quad\sigma(x)\geq\varepsilon_{1}\quad\mbox{for
$x\in\Omega^{\prime}$}.$

We fix $\delta>0$ sufficiently small. Then, the interpolation inequality and
the trace theorem (e.g., Adams [1]) imply that for any $\delta>0$ there exists
a constant $C_{2}(\delta)>0$ such that

$None$ $\|u(\cdot,t)\|_{L^{2}(\partial\Omega)}\leq
C_{1}\|u(\cdot,t)\|_{H^{\frac{1}{2}+\delta_{0}}(\Omega^{\prime})}\leq\delta\|\nabla
u(\cdot,t)\|_{L^{2}(\Omega^{\prime})}+C_{2}(\delta)\|u(\cdot,t)\|_{L^{2}(\Omega^{\prime})}.$

Since (3.3) holds true also in the case (1.3), from (2.2) in Lemma 1, we can
derive

$None$
$\int_{Q}\left(\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\right))e^{2s\varphi}dxdt+(\lambda-\lambda_{1}C_{0}-C)\int_{Q}\sigma(x)|\nabla
u|^{2}e^{2s\varphi}dxdt$ $\displaystyle\leq$ $\displaystyle
C\int^{T}_{0}\int_{\partial\Omega}\lambda|u|^{2}e^{2s\varphi}dSdt+C\int_{Q}|F|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}$
$\displaystyle+$ $\displaystyle
C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}.$

Choosing $\lambda>0$ further large such that $\lambda-\lambda_{1}C_{0}-C>0$,
we see by (3.3) that

$None$ $(\lambda-\lambda_{1}C_{0}-C)\int_{Q}\sigma(x)|\nabla
u|^{2}e^{2s\varphi}dxdt\geq(\lambda-\lambda_{1}C_{0}-C)\int_{\Omega^{\prime}\times(0,T)}\sigma(x)|\nabla
u|^{2}e^{2s\varphi}dxdt$ $\geq
C_{3}\varepsilon_{1}\int_{\Omega^{\prime}\times(0,T)}|\nabla
u|^{2}e^{2s\varphi}dxdt.$

Therefore, in terms of (3.5), we obtain

$\displaystyle\lambda\|u(\cdot,t)e^{s\varphi(t)}\|^{2}_{L^{2}(\partial\Omega)}$
$\displaystyle\leq$ $\displaystyle C\lambda\delta\|\nabla
u(\cdot,t)e^{s\varphi(t)}\|^{2}_{L^{2}(\Omega^{\prime})}+C\lambda
C_{2}(\delta)\|u(\cdot,t)e^{s\varphi(t)}\|^{2}_{L^{2}(\Omega^{\prime})}.$

We substitute these inequalities into the second term on the left-hand side
and the first term on the right-hand side of (3.6) to reach

$\displaystyle\int_{Q}\left(\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\right))e^{2s\varphi}dxdt+C_{3}\varepsilon_{1}\int_{\Omega^{\prime}\times(0,T)}|\nabla
u|^{2}e^{2s\varphi}$ $\displaystyle\leq$ $\displaystyle
C\lambda\delta\int_{\Omega^{\prime}\times(0,T)}|\nabla
u|^{2}e^{2s\varphi}dxdt+C\lambda
C_{2}(\delta)\int_{\Omega^{\prime}\times(0,T)}|u|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle C\int_{Q}|F|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
C(s\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)})e^{2s\varphi(T)}$
$\displaystyle+$ $\displaystyle
C(s\lambda\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}+\|u(\cdot,0)\|^{2}_{H^{1}(\Omega)})e^{2s}.$

Choosing $\delta>0$ sufficiently small and $s>0$ sufficiently large, we can
absorb the first and the second terms on the right-hand side into the second
and the first terms on the left-hand side. Thus the proof of Lemma 2 is
complete. $\blacksquare$

Now we proceed to
Proof of Theorem 1.
Using (1.11) and $\varphi(t_{0})\leq\varphi(t)$ for $t_{0}\leq t\leq T$, we
shrink the integral region $Q$ on the right-hand side of (3.1), we obtain

$\displaystyle
e^{2s\varphi(t_{0})}\int_{\Omega\times(t_{0},T)}\left(\frac{1}{s\varphi}|\partial_{t}u|^{2}+s\lambda^{2}\varphi|u|^{2}\right)dxdt$
$\displaystyle\leq$ $\displaystyle
Cs\lambda\varphi(T)\|u(\cdot,T)\|^{2}_{H^{1}(\Omega)}e^{2s\varphi(T)}+Cs\lambda
M^{2}e^{2s}.$

for $s\geq s_{0}$. Fixing $\lambda>0$ sufficiently large, we do not need
specify the $\lambda$-dependency, and so

$\int_{\Omega\times(t_{0},T)}\left(\frac{1}{s}|\partial_{t}u|^{2}+s|u|^{2}\right)dxdt\leq
CsD_{0}^{2}e^{2s(\varphi(T)-\varphi(t_{0}))}+CsM^{2}e^{-2s\mu(t_{0})}.$

for $s\geq s_{0}$. Here the constant $C>0$ depends on $T$ and $\lambda$. We
set

$\mu(t_{0}):=\varphi(t_{0})-1=e^{\lambda t_{0}}-1>0,\quad
D_{0}:=\|u(\cdot,T)\|_{H^{1}(\Omega)}.$

Thus

$None$ $\|\partial_{t}u\|^{2}_{L^{2}(t_{0},T;L^{2}(\Omega))}\leq
Cs^{2}D_{0}^{2}e^{2s(\varphi(T)-\varphi(t_{0}))}+Cs^{2}M^{2}e^{-2s\mu(t_{0})}$

for all large $s>0$.

We note that the generic contants $C>0$, $C_{j}>0$ are independent of also
$t_{0}\in[0,T]$, but dependent on $T$, $\Omega$, $\lambda$.

Since

$u(x,t_{0})=\int^{t_{0}}_{T}\partial_{t}u(x,t)dt+u(x,T),\quad x\in\Omega,$

we can choose a constant $C_{4}>0$ such that

$None$ $\|u(\cdot,t_{0})\|^{2}_{L^{2}(\Omega)}\leq
C_{4}\|\partial_{t}u\|_{L^{2}(\Omega\times(t_{0},T))}^{2}+C_{4}\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}$

for all $t_{0}\in[0,T]$. Substituting (3.8) into (3.9) and using
$\varphi(T)>1$, we obtain

$\displaystyle\|u(\cdot,t_{0})\|_{L^{2}(\Omega)}^{2}\leq
C_{4}(Cs^{2}D_{0}^{2}e^{2s\varphi(T)}+Cs^{2}M^{2}e^{-2s\mu(t_{0})})+C_{4}\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}$
$\displaystyle\leq$ $\displaystyle
C_{5}s^{2}D_{0}^{2}e^{2s\varphi(T)}+C_{5}s^{2}M^{2}e^{-2s\mu(t_{0})}\leq
C_{6}D_{0}^{2}e^{3s\varphi(T)}+C_{6}M^{2}e^{-s\mu(t_{0})}$

for all large $s>s_{0}$ and all $t_{0}\in[0,T]$.

Setting $C_{7}:=C_{6}e^{3s_{0}\varphi(T)}$, we can have

$None$ $\|u(\cdot,t_{0})\|_{L^{2}(\Omega)}^{2}\leq
C_{7}D_{0}^{2}e^{3s\varphi(T)}+C_{7}M^{2}e^{-s\mu(t_{0})}$

for all large $s>0$ and all $t_{0}\in[0,T]$.

Now we choose $s>0$ in order to minimize the right-hand side of (3.10).
Case 1: $M\leq D_{0}$.
By setting $s=0$, the inequality (3.10) immediately yields
$\|u(\cdot,t_{0})\|_{L^{2}(\Omega)}^{2}\leq 2C_{7}D^{2}_{0}$.
Case 2: $M>D_{0}$.
We choose $s>0$ such that

$D_{0}^{2}e^{3s\varphi(T)}=M^{2}e^{-s\mu(t_{0})},$

that is,

$s=\frac{2}{3\varphi(T)+\mu(t_{0})}\log\frac{M}{D_{0}}>0,$

so that

$\|u(\cdot,t_{0})\|^{2}_{L^{2}(\Omega)}\leq
2C_{7}M^{\frac{6\varphi(T)}{3\varphi(T)+\mu(t_{0})}}D_{0}^{\frac{2\mu(t_{0})}{3\varphi(T)+\mu(t_{0})}}.$

Setting $\theta:=\frac{\mu(t_{0})}{3\varphi(T)+\mu(t_{0})}\in(0,1)$, we can
obtain (1.12). Also in the case of (1.3), thanks to Lemma 2, the same argument
completes the proof of Theorem 1. $\blacksquare$

Nest we provide
Proof of Theorem 3.
We set $y:=u-v$. Then

$\left\\{\begin{array}[]{rl}&\partial_{t}y=\sum_{i,j=1}^{d}\partial_{i}(a_{ij}\partial_{j}y)+c(x,t)y+(f(x,t,u(x,t))-f(x,t,v(x,t)),\quad(x,t)\in
Q,\\\ &\partial_{\nu_{A}}y+ry=0\quad\mbox{or}\quad y=0\quad\mbox{on
$\partial\Omega\times(0,T)$}.\end{array}\right.$

We apply Lemma 2 to obtain

$None$
$\int_{Q}\left(\frac{1}{s\varphi}|\partial_{t}y|^{2}+s\lambda^{2}\varphi|y|^{2}\right)e^{2s\varphi}dxdt$
$\displaystyle\leq$
$\displaystyle\int_{Q}|f(x,t,u(x,t))-f(x,t,v(x,t))|^{2}e^{2s\varphi}dxdt$
$\displaystyle+$ $\displaystyle
Cs\lambda\varphi(T)\|y(\cdot,T)\|^{2}_{H^{1}(\Omega)}e^{2s\varphi(T)}+Cs\lambda\|y(\cdot,0)\|^{2}_{H^{1}(\Omega)}e^{2s}$

for all large $s>0$.

By (1.2) and $\|u\|_{L^{\infty}(Q)}$, $\|u\|_{L^{\infty}(Q)}\leq M$, in terms
of the mean value theorem, we have

$|f(x,t,u(x,t))-f(x,t,v(x,t))|=|f(x,t,\eta_{x,t})||u(x,t)-v(x,t)|\leq
C_{8}|u(x,t)-v(x,t)|,$

where $\eta_{x,t}$ is a constant depending on $x,t$ such that
$|\eta_{x,t}|\leq M$. Therefore, we can absorb the first term on the right-
hand side of (3.11) into the left-hand side by choosing $s>0$ sufficiently
large, and we obtain

$\displaystyle\int_{Q}\left(\frac{1}{s\varphi}|\partial_{t}y|^{2}+s\lambda^{2}\varphi|y|^{2}\right)e^{2s\varphi}dxdt$
$\displaystyle\leq$ $\displaystyle
Cs\lambda\varphi(T)\|y(\cdot,T)\|^{2}_{H^{1}(\Omega)}e^{2s\varphi(T)}+Cs\lambda\|y(\cdot,0)\|^{2}_{H^{1}(\Omega)}e^{2s}$

for all large $s>0$.

Now the same argument as in the proof of Theorem 1 can complete the proof of
Theorem 3. $\blacksquare$

##  4\. Proof of Theorem 2

The proof is based on the stability for $t_{0}>0$ and

$u(x,0)=\int^{0}_{T}\partial_{t}u(x,t)dt+u(x,T),\quad x\in\Omega.$

We apply the same arguments towards (3.8) for $\partial_{t}u$ and
$\partial_{t}^{2}u$. We set

$M_{1}:=\sum_{k=0}^{2}\|\partial_{t}^{k}u(\cdot,0)\|_{H^{1}(\Omega)},\quad
D:=\sum_{k=0}^{2}\|\partial_{t}^{k}u(\cdot,T)\|_{H^{1}(\Omega)}.$

Then, arguing similarly to the proof of Theorem 1 to $\partial_{t}^{2}u$, we
obtain (3.8) for $\partial_{t}^{2}u$ replacing $u$:

$\int_{\Omega\times(t_{0},T)}\left(\frac{1}{s}|\partial_{t}^{3}u|^{2}+s|\partial_{t}^{2}u|^{2}\right)dxdt\leq
CsD^{2}e^{2s\varphi(T)}+CsM_{1}^{2}e^{-2s\mu(t_{0})},$

that is,

$None$ $\|\partial_{t}^{2}u\|^{2}_{L^{2}(\Omega\times(t_{0},T))}\leq
CD^{2}e^{2s\varphi(T)}+CM_{1}^{2}e^{-2s\mu(t_{0})}$

for all $s\geq s_{0}$: some constant and all $0<t_{0}<T$.

Since

$\partial_{t}u(x,t_{0})=\int^{t_{0}}_{T}\partial_{t}^{2}u(x,t)dt+\partial_{t}u(x,T),\quad
x\in\Omega,$

we have

$None$ $\|\partial_{t}u(\cdot,t_{0})\|^{2}_{L^{2}(\Omega)}\leq
C\|\partial_{t}^{2}u\|^{2}_{L^{2}(\Omega\times(t_{0},T))}+C\|\partial_{t}u(\cdot,T)\|^{2}_{L^{2}(\Omega)}.$

We substitute (4.1) to (4.2), so that

$\|\partial_{t}u(\cdot,t_{0})\|^{2}_{L^{2}(\Omega)}\leq
CD^{2}e^{2s\varphi(T)}+CM_{1}^{2}e^{-2s\mu(t_{0})}+C\|\partial_{t}u(\cdot,T)\|^{2}_{L^{2}(\Omega)}$
$None$ $\leq C_{1}D^{2}e^{2s\varphi(T)}+C_{1}M_{1}^{2}e^{-2s\mu(t_{0})}$

for all $s\geq s_{0}$: some constant and all $0<t_{0}<T$. Here we used
$\|\partial_{t}u(\cdot,T)\|_{L^{2}(\Omega)}^{2}\leq D^{2}\leq
CD^{2}e^{2s\varphi(T)}$.

Using

$u(x,0)=\int^{0}_{T}\partial_{t}u(x,t_{0})dt_{0}+u(x,T),\quad x\in\Omega,$

we obtain

$\displaystyle\int_{\Omega}|u(x,0)|^{2}dx\leq
2\int_{\Omega}\left|\int^{T}_{0}\partial_{t}u(x,t_{0})dt_{0}\right|^{2}dx+2\int_{\Omega}|u(x,T)|^{2}dx$
$\displaystyle\leq$ $\displaystyle
C\int^{T}_{0}\|\partial_{t}u(\cdot,t_{0})\|^{2}_{L^{2}(\Omega)}dt_{0}+C\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)}.$

Substitution of (4.3) yields

$\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}\leq
C_{1}D^{2}\int^{T}_{0}e^{2s\varphi(T)}dt_{0}+C_{1}M_{1}^{2}\int^{T}_{0}e^{-2s\mu(t_{0})}dt_{0}+C\|u(\cdot,T)\|^{2}_{L^{2}(\Omega)},$

that is,

$None$ $\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}\leq
C_{1}D^{2}\,e^{C_{2}s}+C_{1}M_{1}^{2}\int^{T}_{0}e^{-2s\mu(t_{0})}dt_{0}.$

We calculate $\int^{T}_{0}e^{-2s\mu(t_{0})}dt_{0}$ as follows. Changing the
variables $t_{0}\mapsto\xi$ by $\xi:=\mu(t_{0})=e^{\lambda t_{0}}-1$ and
setting $\widehat{T}:=e^{\lambda T}-1$, we have

$\int^{T}_{0}e^{-2s\mu(t_{0})}dt_{0}=\frac{1}{\lambda}\int^{\widehat{T}}_{0}e^{-2s\xi}\frac{1}{1+\xi}d\xi\leq\frac{1}{\lambda}\left[\frac{e^{-2s\xi}}{2s}\right]^{\xi=0}_{\xi=\widehat{T}}\leq\frac{1}{2\lambda
s}.$

Hence, (4.4) yields

$None$ $\|u(\cdot,0)\|^{2}_{L^{2}(\Omega)}\leq
C_{3}D^{2}\,e^{C_{2}s}+\frac{C_{3}}{s}M_{1}^{2}$

for all $s\geq s_{0}$.

We choose $s>0$ suitably for making the right-hand side small. Taking into
consideration that the first term $e^{C_{2}s}$ increases and the second
decreases with $\frac{1}{s}$, we may choose $s>0$ like order
$\left(\log\frac{1}{D}\right)^{-1}$. Precisely, we argue as follows.

First setting $C_{4}:=C_{3}e^{C_{2}s_{0}}$ and replacing $s:=s+s_{0}$. we can
obtain (4.5) for all $s>0$. Without loss of generality, we can assume $D<1$.
Setting

$s=\left(\log\frac{1}{D}\right)^{\alpha}>0$

with $0<\alpha<1$, we have

$\displaystyle
e^{C_{2}s}D^{2}=\exp\left(C_{2}\left(\log\frac{1}{D}\right)^{\alpha}\right)D^{2}$
$\displaystyle=$
$\displaystyle\exp\left(C_{2}\left(\log\frac{1}{D}\right)^{\alpha}\right)e^{2\log
D}=\exp\left(-2\left(\log\frac{1}{D}\right)+C_{2}\left(\log\frac{1}{D}\right)^{\alpha}\right).$

Since we can find a constant $C_{5}>0$ such that

$e^{-2\eta+C_{2}\eta^{\alpha}}\leq\frac{C_{5}}{\eta^{\alpha}}\quad\mbox{for
all $\eta>0$}$

by $\alpha<1$, we see

$e^{C_{2}s}D^{2}\leq C_{5}\left(\log\frac{1}{D}\right)^{-\alpha},$

and

$C_{4}D^{2}e^{C_{2}s}+\frac{C_{4}}{s}M_{1}^{2}\leq
C_{4}C_{5}\left(\log\frac{1}{D}\right)^{-\alpha}+C_{3}M_{1}^{2}\left(\log\frac{1}{D}\right)^{-\alpha}.$

Thus the proof of Theorem 2 is complete. $\blacksquare$

Acknowledgements. The work was supported by Grant-in-Aid for Scientific
Research (A) 20H00117 of Japan Society for the Promotion of Science.

## References

  * [1] R.A. Adams, Sobolev Spaces, Academic Press, New York, 1975.
  * [2] K. A. Ames and B. Straughan, Non-standard and Improperly Posed Problems, Academic Press, San Diego, 1997.
  * [3] P. Cannarsa, P. Martinez and J. Vancostenoble, Global Carleman Estimates for Degenerate Parabolic Operators with Applications, American Mathematical Society, Providence, RI, 2015.
  * [4] V. Isakov, Inverse Problems for Partial Differential Equations, Springer-Verlag, Berlin, 2006.
  * [5] S.G. Kreĭn and O.I. Prozorovskaya, Analytic semigroups and incorrect problems for evolutionary equations, Sov. Math.-Dokl. 133 (1960) 35-38.
  * [6] M. Lees and M.H. Protter, Unique continuation for parabolic differential equations and inequalities, Duke Math. J. 28 (1961) 369–382.
  * [7] K. Miller, Nonunique continuation for uniformly parabolic and elliptic equations in self-adjoint divergence form with Hölder continuous coefficients, Arch. Ration. Mech. Anal. 54 (1974) 105-117.
  * [8] A. C. Murray and M.H. Protter, The asymptotic behavior of solutions of second order systems of partial differential equations, J. Diff. Eqns 13 (1973) 57-80.
  * [9] L.E. Payne, Improperly Posed Problems in Partial Differential Equations, SIAM, Philadelphia, 1975.
  * [10] M. Yamamoto, Carleman estimates for parabolic equations and applications, Inverse Problems 25 (2009) 123013.