# Benchmarking Knowledge-Enhanced Commonsense Question Answering
via Knowledge-to-Text Transformation

Ning Bian,1,3 Xianpei Han,1,2, Bo Chen,1,2 Le Sun1,2,11footnotemark: 1
Corresponding authors.

###### Abstract

A fundamental ability of humans is to utilize commonsense knowledge in
language understanding and question answering. In recent years, many
knowledge-enhanced Commonsense Question Answering (CQA) approaches have been
proposed. However, it remains unclear: (1) How far can we get by exploiting
external knowledge for CQA? (2) How much potential of knowledge has been
exploited in current CQA models? (3) Which are the most promising directions
for future CQA? To answer these questions, we benchmark knowledge-enhanced CQA
by conducting extensive experiments on multiple standard CQA datasets using a
simple and effective knowledge-to-text transformation framework. Experiments
show that: (1) Our knowledge-to-text framework is effective and achieves
state-of-the-art performance on CommonsenseQA dataset, providing a simple and
strong knowledge-enhanced baseline for CQA; (2) The potential of knowledge is
still far from being fully exploited in CQA — there is a significant
performance gap from current models to our models with golden knowledge; and
(3) Context-sensitive knowledge selection, heterogeneous knowledge
exploitation, and commonsense-rich language models are promising CQA
directions.

## Introduction

Using a variety of knowledge to help in understanding the meaning of language
is one of the key abilities of humans (Minsky 2000). Commonsense question
answering (CQA) evaluates whether machines can understand language like humans
do by asking questions whose answers rely on commonsense knowledge. For
example, Figure 1 shows a question, and the answer to this question needs
commonsense knowledge “puzzle is used for intellectual challenge”.

Witnessed the importance of commonsense knowledge for CQA, many studies have
been conducted to incorporate external knowledge bases (KBs) in CQA models.
These approaches usually leverage knowledge to enhance a specific CQA
component: 1) enhancing representations (Weissenborn, Kočiskỳ, and Dyer 2017;
Bauer, Wang, and Bansal 2018; Mihaylov and Frank 2018; Ma et al. 2019); 2)
enhancing attention mechanism (Chen et al. 2018; Wang and Jiang 2019); and 3)
enhancing reasoning mechanism (Lin et al. 2019; Lv et al. 2020).

Although many knowledge-enhanced CQA approaches have been proposed, we found
it is still unclear: (1) How far can we get by exploiting external knowledge
for CQA? (2) How much potential of knowledge has been exploited in current
models? For example, can GNN-based models (Lin et al. 2019; Lv et al. 2020)
encode and exploit all useful evidence provided by external knowledge? (3)
Which are the most promising directions for knowledge-enhanced CQA? We believe
answering these questions can provide valuable insights for future CQA studies
and shed light on other knowledge-dependent tasks like reading comprehension
(Rajpurkar et al. 2016) and conversation generation (Zhou et al. 2018).

Figure 1: Our knowledge-to-text framework for benchmarking knowledge-enhanced
CQA with an example from CommonsenseQA (Talmor et al. 2019).

To answer the above questions, we benchmark knowledge-enhanced CQA by
conducting extensive experiments on multiple standard datasets via a simple
and effective knowledge-to-text transformation framework. Intuitively, to
benchmark knowledge-enhanced CQA, external knowledge should be incorporated in
a simple way that is not specialized to specific models/components. This is
challenging, due to 1) the heterogeneity between structured knowledge and
unstructured textual questions/answers, i.e., knowledge facts are usually
triples such as $<$person, Desires, Intellectual_challenge$>$, but questions
and answers are text; and 2) the context-sensitivity of knowledge, i.e., a KB
may contain thousands of facts about a concept, but only several of them are
relevant to the given question. For example, among the thousands of facts
about “person”, only $<$person, Desires, Intellectual_challenge$>$ is useful
for answering the question in Figure 1.

Specifically, our knowledge-to-text framework consists of three stages, which
are shown in Figure 1. Firstly, we retrieve facts from a commonsense knowledge
graph (CKG). Then we transform the knowledge facts to textual descriptions via
three transformation algorithms (template-based, paraphrasing-based, and
retrieval-based). Finally, we utilize machine reading comprehension (MRC)
models to predict answers by exploiting both the original questions and the
textural knowledge descriptions. This framework is simple and general for
benchmarking knowledge-enhanced CQA: 1) By transforming structured knowledge
into textual descriptions, our method resolves the heterogeneity problem
between knowledge and text. 2) By adopting MRC models, our method can learn to
select question-relevant knowledge automatically. 3) Our simple knowledge-
enhancing strategy allows us to easily compare the effects of different
commonsense knowledge.

We conduct thorough experiments on multiple standard CQA datasets (Talmor et
al. 2019; Levesque, Davis, and Morgenstern 2012; Zellers et al. 2019; Sap et
al. 2019b).

The contributions of our paper are:

1\. Through benchmarking experiments we found that the potential of external
knowledge is still far from exploited in knowledge-enhanced CQA, i.e., current
methods can only exploit knowledge to a limited extent. In our experiments,
there is a big performance gap from current models to our models using golden
knowledge.

2\. We propose a simple and effective knowledge-to-text framework for
knowledge-enhanced CQA which achieves state-of-the-art performance on the
CommonsenseQA dataset, providing a simple and strong knowledge-enhanced
baseline for CQA.

3\. Our experimental results shed light on three important future directions
for knowledge-enhanced CQA: context-sensitive knowledge selection,
heterogeneous knowledge exploitation, and commonsense-rich language models.

## Knowledge-enhanced CQA via Knowledge-to-Text Transformation

Following CommonsenseQA (Talmor et al. 2019), the CQA task in this paper is a
multiple-choice problem with five answer candidates. Given question
$Q=[q_{1},q_{2},…,q_{n}]$ and answer candidates $A=\\{A_{1},A_{2},…,A_{m}\\}$
with each answer candidate $A_{k}=[a_{1}^{k},a_{2}^{k},…,a_{l}^{k}]$,
$a_{j}^{k}$ and $q_{i}$ are words, $i$ and $j$ are indexes of words and $k$ is
the index of answer candidate, a CQA model needs to choose the correct answer
from $A$.

We propose a simple and effective knowledge-to-text framework for benchmarking
knowledge-enhanced CQA. Our framework includes three steps: 1) retrieving
facts from CKG; 2) transforming knowledge to text; and 3) adopting an MRC
model to select the answer.

Notice that the purpose of our paper is to benchmark knowledge-enhanced CQA
rather than to propose new techniques. So, it is critical to select classical,
robust, and well-known models, rather than new models which may lead to biased
conclusions. Our framework is not specialized to a specific CQA setting,
therefore it can also be used in other MRC or QA tasks.

In the following, we describe the three stages of our framework.

### Knowledge Retrieval

To answer a question $Q$, our method first retrieves relevant knowledge from a
given CKG. For example, to answer the question in Figure 1, we want to
retrieve facts like $<$person, Desires, Intellectual_challenge$>$ and
$<$puzzle, UsedFor, challenge$>$. Following a previous study (Lin et al.
2019), we retrieve paths on CKG connecting question concepts and answer
concepts as relevant facts, which provides a good precision/recall trade-off
for question-relevant facts.

Concretely, given a question $Q$ and an answer candidate $A_{k}$, we first
identify concepts in them by exactly matching n-grams with the concepts in
CKG. (we use ConceptNet (Speer, Chin, and Havasi 2017) in this paper). Then,
for each pair of $<$question concept, answer candidate concept$>$, we find all
paths between them on CKG (within $K$ hops) as facts for $A_{k}$ ($K$ is a
hyper-parameter here). For the example in Figure 1,
“puzzle$\to$IsA$\to$problem$\to$Synonym$\to$challenge” is a 2-hop knowledge
path for answer candidate “intellectual challenge”.

### Knowledge-to-Text Transformation

This section describes how to resolve the heterogeneity problem between
knowledge and text via knowledge-to-text transformation. Specifically, we
propose three transformation algorithms: template-based, paraphrasing-based,
and retrieval-based, which are described as follows.

Knowledge path |  Template-based |  Paraphrasing-based |  Retrieval-based
---|---|---|---
Silk$\to$AtLocation$\to$China |  Silk is located in China. |  Silk is in China. |  China is the world’s largest silk producer.
Puzzle$\to$IsA$\to$Problem$\to$Synonym $\to$Challenge |  Puzzle is a problem. Problem is the same as challenge. |  Puzzles are problems. The problem is the same as the challenge. |  Puzzle problem is a challenge game for children.
Walk$\leftarrow$MotivatedByGoal$\leftarrow$Hike$\to$Have- Subevent$\to$See beautiful views |  Hike in order to walk. Hike have subevent see beautiful views. |  You go hiking in order to go for a walk. You can see the beautiful scenery on hiking. |  Burghclere has some beautiful rural scenery, so you can walk along the railway or go for a hike.
Table 1:  Examples of knowledge descriptions generated by different knowledge-
to-text algorithms.

Template-based transformation. This algorithm transforms knowledge to text
using a description template for each relation in a CKG. For example, we can
use a template “X is a Y” to generate the description of $<$puzzle, IsA,
problem$>$ as “puzzle is a problem”. Because the number of relations in a CKG
is limited, we manually design a template for each relation type. For a
knowledge path $\\{k_{1},k_{2},…,k_{p},…\\}$ where $k_{p}$ is a knowledge
triple and $p$ is its index, we sequentially generate a sentence for each
tuple, i.e., $\\{s_{1},s_{2},…,s_{p},…\\}$ where sentence $s_{p}$ describes
triple $k_{p}$.

Paraphrasing-based transformation. The main drawback of the template-based
algorithm is the diversity issue, i.e., it always generates the same
description for one relation. To address this issue, we employ a paraphrasing
model to generate more diverse and fluent knowledge descriptions.
Specifically, given the template-based description of a knowledge path, we
generate its top-$M$ paraphrases using beam-search decoding and concatenate
them as the knowledge description. We adopt the encoder-decoder paraphrasing
model trained on PPDB (Pavlick et al. 2015) and WikiAnswers (Fader,
Zettlemoyer, and Etzioni 2013).

Retrieval-based transformation. The above two algorithms can only generate
pseudo textual descriptions, which are different from real-world knowledge
descriptions. Therefore, we propose a retrieval-based knowledge-to-text
algorithm, which retrieves texts from a real-world corpus (we use Wikipedia in
this paper) as knowledge descriptions. Specifically, we adopt the distant
supervision assumption (Mintz et al. 2009) that “if a sentence contains the
entities on a knowledge path, it will express the meaning of the knowledge
path”. We split all Wikipedia documents into separate sentences and build a
Wikipedia sentence retrieval system using Elastic Search. We use the knowledge
descriptions from template-based transformation as queries to retrieve
corresponding Wikipedia sentences containing the concepts on knowledge paths
via the BM25 algorithm (Robertson and Walker 1994). Finally, the rank 1
sentence is used as the description.

To compare different knowledge-to-text transformation algorithms, Table 1
shows some examples of generated knowledge descriptions. We can see that: (1)
The template-based algorithm can produce reasonable textual descriptions,
although they may contain grammar errors (like “Hike in order to walk” in the
3 example). (2) The paraphrasing-based algorithm can produce diverse and more
fluent sentences (“You go hiking in order to go for a walk”), but may change
some important words (e.g., “beautiful view” is changed to “beautiful scenery”
in the 3 example). (3) The retrieval-based algorithm can produce real-world
sentences (“China is the world’s largest silk producer”) but may contain extra
irrelevant content (like “Burghclere” in the 3 example).

### MRC-based Answer Prediction

Given a question and the generated knowledge descriptions, we predict its
answer using MRC models. We adopt MRC models because: 1) MRC models can
automatically learn to identify relevant information in a document (Seo et al.
2016). In our settings this ability can be used to automatically select
question-relevant knowledge, as all knowledge facts have been transformed into
a textual document; 2) MRC is a well-studied technique. Therefore, our method
can directly leverage the strong ability of existing state-of-the-art MRC
models, so that our benchmarking is effective, robust, and easy-to-implement.

Specifically, we model CQA as an MRC problem by treating knowledge
descriptions as a document. In this way, current MRC models can be directly
used, including BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), XLNet
(Yang et al. 2019b), and ALBERT (Lan et al. 2019) based MRC models. Figure 2
shows our MRC framework. For each question, we construct a sequence
$S_{k}=\\{K_{k},[SEP],Q,[SEP],A_{k}\\}$ for each answer candidate $A_{k}$,
where $K_{k}$ is the generated knowledge descriptions, $Q$ is the question,
and $[SEP]$ is the separation token in pretrained language models (PLMs).
Following Devlin et al. (2019), we use a feed-forward classifier as the output
layer which predicts the answer score $\rm{Score}$$(A_{k}|S_{k})$. Finally,
the highest-scored answer candidate is chosen as the answer.

Figure 2: The MRC model for predicting answers in our knowledge-enhanced CQA
method.

## Benchmarking Knowledge-Enhanced Commonsense Question Answering

This section benchmarks knowledge-enhanced CQA by conducting thorough
experiments. We first verify the effectiveness and robustness of our
knowledge-to-text-based CQA method, then we answer the three important
questions: (1) How far can we get by exploiting external knowledge for CQA?
(2) How much potential of knowledge has been exploited in current models? (3)
Which are the most promising directions for future knowledge-enhanced CQA?

Model | Knowledge Source | BERT | XLNet | RoBERTa | ALBERT
---|---|---|---|---|---
Human | – | 88.9 | 88.9 | 88.9 | 88.9
Golden Knowledge | Human Explanations | 81.1 | 85.1 | 84.7 | 83.7
Knowledge-to-Text |  |  |  |  |
Template-based | ConceptNet | 67.9 | 77.5 | 78.1 | 81.1
Paraphrasing-based | ConceptNet | 67.2 | 74.9 | 77.8 | 79.3
Retrieval-based | ConceptNet | 65.0 | 75.0 | 77.1 | 79.4
Full | ConceptNet | 70.4 | 80.3 | 80.8 | 83.3
Best Knowledge-enhanced System with Different PLMs | ConceptNet |  69.0 (Ma et al. 2019) |  79.3 (Lv et al. 2020) |  80.8 (KEDGN) |  (No available model so far)
Base Model | No knowledge | 63.6 | 68.9 | 76.2 | 78.6
Table 2:  Accuracies on CommonsenseQA.

### Experimental Settings

Datasets. We use CommonsenseQA dataset v1.11 (Talmor et al. 2019) as the
primary dataset, and adopt the Winograd Schema Challenge (WSC, Levesque,
Davis, and Morgenstern 2012), HellaSWAG (Zellers et al. 2019), and SOCIAL IQa
(Sap et al. 2019b) as secondary datasets.

(1) CommonsenseQA (Talmor et al. 2019) contains 12,102 human-generated
questions with 5 answer candidates for each question. All questions are
elaborately designed to make sure commonsense knowledge is needed for
correctly answering them. Furthermore, CoS-E (Rajani et al. 2019) provides
each question with a human-annotated golden knowledge explanation. Due to the
above advantages, We use CommonsenseQA as the primary benchmarking dataset.

(2) WSC (Levesque, Davis, and Morgenstern 2012) is a pronoun resolution
dataset that requires commonsense knowledge, which is recognized as one of the
most difficult CQA datasets (Zhou et al. 2020). Because WSC does not contain
training data, we use WSCR (Rahman and Ng 2012) for training.

(3) HellaSWAG (Zellers et al. 2019) is an update of the commonsense reasoning
dataset SWAG: given an event description like “A woman sits at a piano”, a
machine needs to select the most likely follow-up: “She sets her fingers on
the keys”. The “Overall accuracy” on the dev set is used in our evaluation.

(4) SOCIAL IQa (Sap et al. 2019b) is a QA dataset for commonsense reasoning
about social situations, which requires emotional and social commonsense in a
variety of every-day situations.

Knowledge base. We use ConceptNet 5 (Speer, Chin, and Havasi 2017) as the KB
for benchmarking, because: (i) ConceptNet is general and can provide a large
commonsense coverage for our CQA experiments. Other CKGs like ATOMIC (Sap et
al. 2019a, if-then relations of events) and ASER (Zhang et al. 2020, relations
of events, states, and actions) only contain partial knowledge for our
experiments. (ii) The primary CommonsenseQA dataset is constructed upon
ConceptNet and other datasets don’t accompany a given KB. ConceptNet concepts
can be easily and directly identified in questions and answers for
CommonsenseQA, so that we can better benchmark knowledge-enhanced CQA by
focusing on the ability of knowledge exploitation. We use the same 22
relations in ConceptNet as Talmor et al. (2019).

Baselines. We benchmark knowledge-enhanced CQA by assessing the performances
of different MRC models with/without external knowledge, including BERT-based
(Devlin et al. 2019), RoBERTa-based (Liu et al. 2019), XLNet-based (Yang et
al. 2019b), and ALBERT-based (Lan et al. 2019) MRC models.

To verify the effectiveness of knowledge-to-text transformation, we also
report the performances of current knowledge-enhanced systems with
corresponding pretrained language models as base encoders:

(1) Ma et al. (2019) (BERT + OCN + ConceptNet) is the best BERT-based
knowledge-enhanced CQA system on CommonsenseQA, which uses an attention
mechanism for knowledge incorporation and an Option Comparison Network (OCN)
model for answer prediction.

(2) Lv et al. (2020) (XLNet + Graph Reasoning) is the best XLNet-based system
on CommonsenseQA, which uses GNN to exploit knowledge from both ConceptNet and
Wikipedia.

(3) KEDGN (RoBERTa + Knowledge) is the unpublished best RoBERTa-based
knowledge-enhanced system on the leaderboard of CommonsenseQA, which exploits
knowledge via a dual graph network. For a fair comparison, in Table 2 we
report the accuracy of the best single model as described in its report.

Hyperparameters. For knowledge retrieval, we use knowledge paths within 2 hops
($K$ = 2). In paraphrasing-based transformation, we use the top 1 paraphrasing
result ($M$ = 1). For MRC models, we initialize them with the official
pretrained language models (BERT-Large, RoBERTa-Large, XLNet-Large, and
ALBERT-XXLarge) and fine-tune them using CQA training data. The output layers
have a 1024-dimensional hidden layer with a $tanh$ activation function. All
models are trained using Adam with a learning rate of 5e-6.

### Effect of Knowledge-to-Text Transformation

Table 2 and Table 3 show the experimental results on CommonsenseQA and other
datasets. For our method, we use four settings: template-based, paraphrasing-
based, retrieval-based, and a full model that uses a concatenation of all the
three generated descriptions as a document. We found that:

1) Knowledge-to-text transformation is effective for knowledge-enhanced CQA.
Our full model achieves state-of-the-art performance on CommonsenseQA. And all
template-based, paraphrasing-based, and retrieval-based models achieve
improvements over non-knowledge base models.

2) Knowledge-to-text transformation can robustly exploit knowledge for CQA.
Table 3 shows that our method can consistently improve the performances on
three extra CQA datasets by exploiting external commonsense knowledge. Notice
that although ConceptNet is not specially designed for WSC, HellaSWAG, and
SOCIAL IQA datasets, our method can still achieve improvements, which further
verifies the robustness of our method, and we believe the results on these
datasets can be further improved if more relevant commonsense knowledge
sources are available. In Table 2 our method achieves accuracy improvements on
all base models (BERT, RoBERTa, XLNet, and ALBERT) and all settings (template-
based, paraphrasing-based, and retrieval-based). Table 4 shows that our method
is also robust on different lengths of knowledge paths, and the 2-hop
knowledge path setting achieves the best performance.

3) The three knowledge-to-text transformation algorithms are complements of
each other. In Table 2, the full model can achieve the best performance by
combining all three knowledge-to-text algorithms, which verifies that these
algorithms can complement each other. Among the three single algorithms, the
template-based algorithm obtains the best performance. This may be because it
is easier for MRC models to capture regularities in simple and formal
sentences.

Overall, the above results verify that our simple knowledge-to-text
transformation is a good strategy for benchmarking the effectiveness and
robustness of knowledge-enhanced CQA.

In the following, we conduct benchmarking experiments on the primary
CommonsenseQA dataset using the full model and 2-hop knowledge path setting.

Models |  WSC | HellaSWAG | SOCIAL IQa
---|---|---|---
BERT | 66.0 | 42.3 | 66.2
\+ Knowledge | 68.1 | 44.2 | 68.8
RoBERTa | 81.4 | 82.5 | 74.3
\+ Knowledge | 82.5 | 83.0 | 75.0
ALBERT | 84.9 | 86.1 | 77.2
\+ Knowledge | 87.0 | 86.9 | 77.8
Human | 92.1 | 94.5 | 86.9
Table 3:  Accuracies on other CQA datasets. “+ Knowledge” means using our knowledge-to-text transformation method (template-based) with 2-hop knowledge paths on ConceptNet. Human accuracy of WSC is reported by Bender (2015). Path Length | BERT | XLNet | RoBERTa | ALBERT
---|---|---|---|---
1-hop | 67.1 | 74.7 | 77.9 | 80.0
2-hop | 67.9 | 77.5 | 78.1 | 81.1
3-hop | 65.0 | 68.6 | 77.2 | 79.2
Table 4:  Accuracies on different lengths of knowledge paths (template-based method). Missing Important Evidence |  Question |  What could people do that involves talking?
---|---|---
Answer candidates |  confession $|$ state park $|$ sing $|$ opera $|$ carnival
Golden knowledge |  confession involves talking.
Knowledge description |  people is located in confession. people is used for talk.
Complicated Descriptions |  Question |  They were getting ready for a really long hike, he put the food can in his what?
Answer candidates |  backpack $|$ make person sick $|$ cabinet $|$ house $|$ recycling center
Golden knowledge |  backpacks are used on hicks.
Knowledge description |  food can is located in backpack. backpack is in the context of sport. hike is in the context of sport……
Noisy Knowledge |  Question |  Most people who are family like to greet each other with a what?
Answer candidates |  listen to music $|$ have friends $|$ know what ophiolites $|$ hug $|$ apartments
Golden knowledge |  people who are family like to hug.
Knowledge description |  person desire hug. person is located in family. kissing have subevent hug. kissing cause like. meeting friend have subevent hug. hug in order to love. love is located in family. most people desire hug.
Table 5:  Bad examples of generated knowledge descriptions (template-based)
and golden knowledge, where: 1) In the 1 example, the relational knowledge
between “talking” and “confession” is missing in the generated knowledge
description because it is not covered by ConceptNet. 2) In the 2 example,
knowledge description provides the knowledge about “backpack” and “hike” using
two separate sentences, which is more complicated than golden knowledge and
thus puts an extra burden on MRC models. 3) In the 3 example, there are many
irrelevant/noisy sentences in knowledge description about unimportant question
words (like “people” and “like”).

### Effect of Knowledge for CQA

This section studies “how far can we get by exploiting external knowledge for
CQA?”. To answer this question, Table 2 further shows the performances of MRC
models using manually-annotated golden knowledge for each question (Rajani et
al. 2019) as the knowledge description. We can see that:

By incorporating golden external knowledge, CQA can be significantly improved
and can achieve close-to-human performance. On all BERT, XLNet, RoBERTa, and
ALBERT-based MRC models, incorporating golden knowledge can significantly
achieve 27%, 14%, 11%, and 7% accuracy improvements, correspondingly. The best
golden-knowledge enhanced system (XLNet + Golden) can achieve 85.1% accuracy,
which is not far from the human accuracy of 88.9%.

These results show that knowledge can get us quite far, and it is promising to
study more effective knowledge-enhanced CQA models.

### Effect of Knowledge in Current Models

This section investigates “how much potential of knowledge has been exploited
in current models?”. From Table 2, we can see that:

1) Current knowledge-enhanced CQA methods only exploit knowledge to a limited
extent. In Table 2, we can see that: (i) compared with models using golden
knowledge, all knowledge-enhanced CQA models have a big performance gap; and
(ii) our simple knowledge-to-text strategy can achieve competitive performance
with the complicated GNN-based strategies (KEDGN and XLNet + Graph Reasoning)
and Option Comparison Network.

2) Despite the effectiveness of our method, there is still great potential in
generating accurate question-relevant knowledge descriptions. To show this,
Table 5 shows several bad cases of knowledge descriptions. We can see that,
the golden knowledge descriptions are typically simple, relevant, and
accurate, while the automatically generated descriptions may miss important
evidence (1 example), be too complicated (2 example), or contain noisy
knowledge (3 example). Based on these observations, we believe seeking and
identifying more accurate question-relevant knowledge can further improve the
knowledge exploitation ability of CQA methods.

3) The commonsense knowledge embedded in current pretrained language models is
still not enough for CQA. In Table 2, we can see that there is a significant
performance gap between base models without using knowledge and knowledge-
enhanced models, although they have been trained using very large text corpus.
To further study this, we also experiment using ERNIE (Zhang et al. 2019b), a
knowledge-enhanced pretrained language model based on BERT, but the
performance is lower than BERT-based models (60.0% accuracy on CommonsenseQA).
We believe this is because ERNIE focuses on entity-centric facts, instead of
commonsense. This shows that, although trained on very large text corpus,
state-of-the-art pretrained language models still can not encode enough
commonsense knowledge.

The above results show that the potential of knowledge is still far from being
fully exploited by current knowledge-enhanced CQA methods. This is because of
1) the limited ability of current CQA models to exploit knowledge; 2) the lack
of ability to identify accurate question-relevant knowledge; 3) the limited
commonsense captured in pretrained language models.

### Detailed Analysis

This section analyzes our method in detail.

Figure 3: Performances of different commonsense skills using XLNet-based
model, with/without knowledge descriptions (template-based).

Performances on Different Commonsense Skills. CQA questions require different
types of commonsense skills (LoBue and Yates 2011). To analyze the effects of
knowledge on different commonsense skills, we randomly sample 200 questions
from CommonsenseQA and annotate their required skills using the commonsense
skill categories from Talmor et al. (2019).

Figure 3 shows the performances of our CQA method with/without knowledge on
different skills. From Figure 3, we can see that: (1) Knowledge can
significantly improve skills including “Spatial” (+12.3%), “Cause & Effect”
(+10.0%), “Activity” (+8.3%) and “Purpose” (+6.5%). (2) For “Definition”,
“Social”, and “Has parts” skills, the knowledge-enhanced model achieves
similar performances with the base model. We believe this may be because
ConceptNet has a low coverage for these types of knowledge.

Error type |  Example
---|---
Indistinguishable Knowledge (21/50) |  Question |  What do airplanes do as they are arriving at the gate?
Answer candidates |  $\surd$ slow down $|$ $\times$ land $|$ $\times$ crash $|$ $\times$ speed up $|$ $\times$ carry people
Knowledge for correct answer |  airplanes can slow down.
Knowledge for predicted answer |  airplanes can speed up.
Noisy Knowledge (15/50) |  Question |  I took my seat, the curtains drew back and I enjoyed the what?
Answer candidates |  $\times$ auditorium $|$ $\times$ theatre $|$ $\times$ movie $|$ $\surd$ show $|$ $\times$ airplane
Knowledge for correct answer |  curtain is located in show. cover is opposite to back. seat is located in show. show is located in opera. curtain is located in opera. person desire enjoy. curtain is located in theater……
Knowledge for predicted answer |  movie is located in theater. curtain is located in theater.
No Knowledge (13/50) |  Question |  Some animals can fly thanks to their lightweight hollow what?
Answer candidates |  $\times$ heads $|$ $\times$ tails $|$ $\surd$ bodies $|$ $\times$ bones $|$ $\times$ eyes
Knowledge for correct answer |  bones is located in person. person desire fly.
Knowledge for predicted answer |  [NO KNOWLEDGE FACT IS RETRIEVED]
Table 6:  Several error cases of XLNet-based model with template-based
knowledge descriptions.

Error Analysis. To understand why our model fails in some cases, we randomly
select 50 error cases and group them into several categories. Table 6 shows
the main error types with their examples:

1) Indistinguishable knowledge, i.e., retrieved knowledge cannot provide
enough information for distinguishing answer candidates. For example, the 1
error case provides strong support for both correct and incorrect answers
(“airplanes can slow down/speed up”). This is the main error type of our
method (21 out of 50).

2) Noisy knowledge. Noisy knowledge misleads MRC models to give wrong answers,
which often appears when knowledge descriptions are too long. In the 2 error
case, we can see that the important fact “curtain is located in show” is
obscured by noisy facts about irrelevant concepts like “seat”.

3) No Knowledge. Knowledge retrieval may not be able to retrieve question-
relevant facts and thus provides no useful information for MRC models. From
the 3 case, we can see that the knowledge facts are all irrelevant to the
answers.

The above three types of errors show that it is important to select accurate,
complete, and context-sensitive knowledge for more effective knowledge-
enhanced models.

## Related Work

Knowledge-enhanced CQA. Many studies have been proposed to exploit commonsense
knowledge for CQA. Rajani et al. (2019) propose to train a GPT-based
explanation generation model using manually labeled corpus, but it relies on
extra human effort. KagNet (Lin et al. 2019) represents external knowledge as
a graph and reasons via graph convolution and LSTM. Ma et al. (2019)
incorporate knowledge with text-to-knowledge attention and adopt a BERT-based
Option Comparison Network for answer prediction. Lv et al. (2020) propose a
GNN-based reasoning model over A heterogeneous knowledge graph of both
ConceptNet and Wikipedia sentences. Compared with these methods, our
knowledge-to-text method exploits knowledge in a simple way and knowledge can
be effectively used by the whole model.

Knowledge Exploitation in Neural Models. There are many studies which leverage
external knowledge to enhance models on a variety of NLP tasks (Lin, Sun, and
Han 2017; Yang and Mitchell 2017; An et al. 2018; Yang et al. 2019a; Logan et
al. 2019; Chen, Sun, and Han 2018). Chen et al. (2018) leverage semantic
relations in WordNet to enhance attention and inference abilities in the NLI
task. Mihaylov and Frank (2018) apply key-value memory to represent
commonsense facts and use word-to-knowledge attention for cloze-style MRC.
Bauer, Wang, and Bansal (2018) propose a mutual information-based knowledge
selection method and fuse knowledge using gated attention for multi-hop
reasoning. Zhang et al. (2019a) propose an attention-based knowledge selection
method for coreference resolution. ERNIE (Zhang et al. 2019b) and K-BERT (Liu
et al. 2020) incorporate knowledge in pretrained language models, but mainly
focus on entity-centric facts in KBs instead of commonsense.

Machine Reading Comprehension. In recent years, many effective end-to-end MRC
models have been proposed, including BERT (Devlin et al. 2019), RoBERTa (Liu
et al. 2019), XLNet (Yang et al. 2019b) and ALBERT (Lan et al. 2019) based
models. It has been proven that MRC models can effectively encode information
in a document and find the most relevant information for answer prediction. In
this paper, these abilities are utilized to select and exploit relevant
knowledge for knowledge-enhanced CQA.

## Conclusions and Future Work

We benchmark knowledge-enhanced CQA using a simple and effective knowledge-to-
text transformation framework and provides a strong knowledge-enhanced
baseline for CQA. By conducting thorough experiments, we found that: (1) Our
knowledge-to-text framework is effective and robust for knowledge-enhanced
CQA; (2) It is promising to incorporate knowledge in neural models for CQA;
(3) The potential of knowledge is still far from being fully exploited — there
is a large performance gap from current models to our models using golden
knowledge.

The above results also shed light on the promising directions for knowledge-
enhanced CQA:

1) Context-sensitive knowledge selection is critical for knowledge-enhanced
CQA. According to the error analysis, more than 70% of errors are caused by
noisy knowledge and indistinguishable knowledge.

2) The knowledge-text heterogeneity is a critical bottleneck for exploiting
the information from both knowledge and text. We address this heterogeneity
problem via simple knowledge-to-text transformation, and even such a simple
strategy can outperform many knowledge-enhanced models like GNN-based and
attention-based models. Therefore, we believe more advanced solutions for the
heterogeneity problem will further improve CQA, e.g., uniform representation
learning and joint graph representations.

3) It is valuable to incorporate more commonsense in pretrained language
models. From our experiments, we can see that current state-of-the-art
pretrained language models like BERT and XLNet still only encode limited
commonsense knowledge. So, we believe commonsense-rich language models will
provide valuable techniques and resources for CQA.

##  Acknowledgments

This research work is supported by National Key R&D Program of China under
Grant 2018YFB1005100, the National Natural Science Foundation of China under
Grants no. U1936207 and 61772505, Beijing Academy of Artificial Intelligence
(BAAI2019QN0502), and in part by the Youth Innovation Promotion Association
CAS (2018141).

## References

  * An et al. (2018) An, B.; Chen, B.; Han, X.; and Sun, L. 2018.  Accurate Text-Enhanced Knowledge Graph Representation Learning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 745–755. New Orleans, Louisiana: Association for Computational Linguistics.
  * Bauer, Wang, and Bansal (2018) Bauer, L.; Wang, Y.; and Bansal, M. 2018.  Commonsense for Generative Multi-Hop Question Answering Tasks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 4220–4230. Brussels, Belgium: Association for Computational Linguistics.
  * Bender (2015) Bender, D. 2015.  Establishing a Human Baseline for the Winograd Schema Challenge.  In _MAICS_ , 39–45.
  * Chen, Sun, and Han (2018) Chen, B.; Sun, L.; and Han, X. 2018.  Sequence-to-Action: End-to-End Semantic Graph Generation for Semantic Parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 766–777. Melbourne, Australia: Association for Computational Linguistics.
  * Chen et al. (2018) Chen, Q.; Zhu, X.; Ling, Z.-H.; Inkpen, D.; and Wei, S. 2018.  Neural Natural Language Inference Models Enhanced with External Knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , 2406–2417. Melbourne, Australia: Association for Computational Linguistics.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Fader, Zettlemoyer, and Etzioni (2013) Fader, A.; Zettlemoyer, L.; and Etzioni, O. 2013.  Paraphrase-Driven Learning for Open Question Answering.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics_ , 1608–1618. Sofia, Bulgaria: Association for Computational Linguistics.
  * Lan et al. (2019) Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ .
  * Levesque, Davis, and Morgenstern (2012) Levesque, H.; Davis, E.; and Morgenstern, L. 2012.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_. Citeseer.
  * Lin et al. (2019) Lin, B. Y.; Chen, X.; Chen, J.; and Ren, X. 2019.  KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2829–2839. Hong Kong, China: Association for Computational Linguistics.
  * Lin, Sun, and Han (2017) Lin, H.; Sun, L.; and Han, X. 2017.  Reasoning with Heterogeneous Knowledge for Commonsense Machine Comprehension.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2032–2043. Copenhagen, Denmark: Association for Computational Linguistics.
  * Liu et al. (2020) Liu, W.; Zhou, P.; Zhao, Z.; Wang, Z.; Ju, Q.; Deng, H.; and Wang, P. 2020.  K-BERT: Enabling Language Representation with Knowledge Graph.  In _Proceedings of the Thirty-Forth AAAI Conference on Artificial Intelligence_ , 2901–2908.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ .
  * LoBue and Yates (2011) LoBue, P.; and Yates, A. 2011.  Types of Common-Sense Knowledge Needed for Recognizing Textual Entailment.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 329–334. Portland, Oregon, USA: Association for Computational Linguistics.
  * Logan et al. (2019) Logan, R.; Liu, N. F.; Peters, M. E.; Gardner, M.; and Singh, S. 2019.  Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 5962–5971. Florence, Italy: Association for Computational Linguistics.
  * Lv et al. (2020) Lv, S.; Guo, D.; Xu, J.; Tang, D.; Duan, N.; Gong, M.; Shou, L.; Jiang, D.; Cao, G.; and Hu, S. 2020.  Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering.  In _Proceedings of the Thirty-Forth AAAI Conference on Artificial Intelligence_ , 8449–8456.
  * Ma et al. (2019) Ma, K.; Francis, J.; Lu, Q.; Nyberg, E.; and Oltramari, A. 2019.  Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering.  In _Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing_ , 22–32.
  * Mihaylov and Frank (2018) Mihaylov, T.; and Frank, A. 2018.  Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , 821–832. Melbourne, Australia: Association for Computational Linguistics.
  * Minsky (2000) Minsky, M. 2000.  Commonsense-based interfaces.  _Communications of the ACM_ 43(8): 66–73.
  * Mintz et al. (2009) Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , 1003–1011. Suntec, Singapore: Association for Computational Linguistics.
  * Pavlick et al. (2015) Pavlick, E.; Rastogi, P.; Ganitkevitch, J.; Van Durme, B.; and Callison-Burch, C. 2015.  PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_ , 425–430. Beijing, China: Association for Computational Linguistics.
  * Rahman and Ng (2012) Rahman, A.; and Ng, V. 2012.  Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , 777–789. Jeju Island, Korea: Association for Computational Linguistics.
  * Rajani et al. (2019) Rajani, N. F.; McCann, B.; Xiong, C.; and Socher, R. 2019.  Explain Yourself! Leveraging Language Models for Commonsense Reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4932–4942. Florence, Italy: Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2383–2392. Austin, Texas: Association for Computational Linguistics.
  * Robertson and Walker (1994) Robertson, S. E.; and Walker, S. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval.  In _SIGIR’94_ , 232–241. Springer.
  * Sap et al. (2019a) Sap, M.; Le Bras, R.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019a.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_ , volume 33, 3027–3035.
  * Sap et al. (2019b) Sap, M.; Rashkin, H.; Chen, D.; Le Bras, R.; and Choi, Y. 2019b.  Social IQa: Commonsense Reasoning about Social Interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 4463–4473. Hong Kong, China: Association for Computational Linguistics.
  * Seo et al. (2016) Seo, M.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H. 2016.  Bidirectional attention flow for machine comprehension.  _arXiv preprint arXiv:1611.01603_ .
  * Speer, Chin, and Havasi (2017) Speer, R.; Chin, J.; and Havasi, C. 2017.  ConceptNet 5.5: an open multilingual graph of general knowledge.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , 4444–4451.
  * Talmor et al. (2019) Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4149–4158.
  * Wang and Jiang (2019) Wang, C.; and Jiang, H. 2019.  Explicit Utilization of General Knowledge in Machine Reading Comprehension.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2263–2272. Florence, Italy: Association for Computational Linguistics.
  * Weissenborn, Kočiskỳ, and Dyer (2017) Weissenborn, D.; Kočiskỳ, T.; and Dyer, C. 2017.  Dynamic integration of background knowledge in neural NLU systems.  _arXiv preprint arXiv:1706.02596_ .
  * Yang et al. (2019a) Yang, A.; Wang, Q.; Liu, J.; Liu, K.; Lyu, Y.; Wu, H.; She, Q.; and Li, S. 2019a.  Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2346–2357. Florence, Italy: Association for Computational Linguistics.
  * Yang and Mitchell (2017) Yang, B.; and Mitchell, T. 2017.  Leveraging Knowledge Bases in LSTMs for Improving Machine Reading.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , 1436–1446. Vancouver, Canada: Association for Computational Linguistics.
  * Yang et al. (2019b) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019b.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , 5753–5763.
  * Zellers et al. (2019) Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019.  HellaSwag: Can a Machine Really Finish Your Sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4791–4800. Florence, Italy: Association for Computational Linguistics.
  * Zhang et al. (2020) Zhang, H.; Liu, X.; Pan, H.; Song, Y.; and Leung, C. W.-K. 2020.  ASER: A large-scale eventuality knowledge graph.  In _Proceedings of The Web Conference 2020_ , 201–211.
  * Zhang et al. (2019a) Zhang, H.; Song, Y.; Song, Y.; and Yu, D. 2019a.  Knowledge-aware Pronoun Coreference Resolution.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 867–876. Florence, Italy: Association for Computational Linguistics.
  * Zhang et al. (2019b) Zhang, Z.; Han, X.; Liu, Z.; Jiang, X.; Sun, M.; and Liu, Q. 2019b.  ERNIE: Enhanced Language Representation with Informative Entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 1441–1451. Florence, Italy: Association for Computational Linguistics.
  * Zhou et al. (2018) Zhou, H.; Young, T.; Huang, M.; Zhao, H.; Xu, J.; and Zhu, X. 2018.  Commonsense Knowledge Aware Conversation Generation with Graph Attention.  In _IJCAI_ , 4623–4629.
  * Zhou et al. (2020) Zhou, X.; Zhang, Y.; Cui, L.; and Huang, D. 2020.  Evaluating Commonsense in Pre-Trained Language Models.  In _Proceedings of the Thirty-Forth AAAI Conference on Artificial Intelligence_ , 9733–9740.


11institutetext: The Pennsylvania State University, State College, PA, 16802,
USA
11email<EMAIL_ADDRESS>
# What am I allowed to do here?: Online Learning of Context-Specific Norms by
Pepper

Ali Ayub  Alan R. Wagner

###### Abstract

Social norms support coordination and cooperation in society. With social
robots becoming increasingly involved in our society, they also need to follow
the social norms of the society. This paper presents a computational framework
for learning contexts and the social norms present in a context in an online
manner on a robot. The paper utilizes a recent state-of-the-art approach for
incremental learning and adapts it for online learning of scenes (contexts).
The paper further utilizes Dempster-Schafer theory to model context-specific
norms. After learning the scenes (contexts), we use active learning to learn
related norms. We test our approach on the Pepper robot by taking it through
different scene locations. Our results show that Pepper can learn different
scenes and related norms simply by communicating with a human partner in an
online manner111The final authenticated publication is available online at
https://doi.org/10.1007/978-3-030-62056-1_19.

Keywords: Online learning, indoor scene classification, norm learning, active
learning, human-robot interaction

##  1 Introduction

Norms are an implicit part of any society which guide the actions taken by
humans in that society. Norms support the actions taken by humans across time
and generations and bring coordination and cooperation in a society [1]. Some
norms are defined explicitly in the form of laws. Social norms, however, are
more implicit and are learned through the actions of the community. For
example, when in a library, people can implicitly learn that talking is
impermissible, while at a party talking is not only permissible but
obligatory.

With robots increasingly becoming an integral part of the society in different
roles, such as household robots or socially assistive robots, it is necessary
that they follow the implicit social norms of the society. Since social norms
can differ based on different communities, it is impossible to pre-program a
universal set of norms. Thus, similar to humans, robots must learn norms in a
lifelong manner using different sources, such as by asking questions or
observing other humans. The ultimate goal of this paper is to develop a method
allowing a robot to learn different scenes and the norms associated with the
scene while it wanders around different environments.

Despite its importance for social robotics applications, work in context-
specific norm learning on robots has been limited. Sarathay et al. [2, 3]
present a method for formal modeling of context-specific norms, however they
do not apply their approach on a real robot. For online learning of norms on a
robot, it is first necessary to develop perceptual systems that allow the
robot to learn different scene categories. The term scene is used here to
describe the perception-level representation of a context. To the best of our
knowledge, we are among the first to continually learn both the context a
robot is operating in and its associated acceptable behaviors using only
streaming video data.

In this paper, we present a computational framework to learn different scene
categories and associated norms from streaming video data in an online manner.
Online learning is a field of machine learning in which a single system is
required to both incrementally learn and perform open-set recognition. Where,
incremental learning is a field of machine learning focused on creating
systems that can incrementally learn over time without requiring all the
training data to be available in a single batch [4, 5, 6, 7, 8]. Open-set
recognition, on the other hand, uses a trained model to predict if a new
instance of data belongs to an already learned class or is contains entirely
new items. Thus making the online learning problem extremely challenging. We
adapt a recent state-of-the-art incremental learning approach termed Centroid-
Based Concept Learning and Pseudo-Rehearsal (CBCL-PR) [9] for online learning
of scenes in this paper. For norm modeling we use the mathematical framework
presented in [2]. Norm learning related to different scene categories is
accomplished by employing active learning to ask questions from a human
partner. We test our approach on the Pepper robot by taking it through
different locations on Penn State University campus for online learning of
scenes and their associated norms through question/answer sessions with human
partner.

The remainder of the paper is organized as follows: Section 2 reviews the
related work regarding scene classification, incremental learning of scenes
and norm learning. Section 3 describes our complete architecture for online
scene and norm learning on a robot. Section 4 presents empirical evaluations
of the system, demonstrating that our proposed system is capable of learning
scenes and norms in an online manner. Finally, Section 5 offers conclusions
and directions for future research.

##  2 Related Work

Scene classification has been strongly influenced by deep learning [10, 11].
Most approaches, however, have only been tested on indoor scene datasets that
do not include natural environments, variations in lighting conditions or the
differences stemming from the use of different robots. Most methods also
require the complete dataset, including all of the categories of scenes to be
learned, available in a single batch.

For robotics applications, data is available to the robot in the form of a
continuous stream. Learning from such streaming video data is known as online
learning. Only a few researchers have investigated online learning of scenes
for robots. Kawewong et al. [12] presents one of the earliest approaches for
incremental learning of indoor scenes. They utilize a pre-trained CNN for
feature extraction and utilize n-value self-organizing and incremental neural
networks for incremental learning of scenes. They, however, only test their
approach on the MIT 67 scene dataset [13] and not on a robot. Paul et al. [14]
presented an active learning approach for online learning of scenes. However,
they require the old data to be available when learning new labeled data.
Furthermore, they also do not test their approach on a robot or robot-centric
indoor scene dataset.

There has been limited work done on norm learning in the context of robotics
applications. For example, [15] presents a simple simulated autonomous robot
setup in which autonomous cars follow traffic signal norms. Tan et al. [16]
present another simple robotics application of learning ownership norms.
Neither of these approaches are applicable to the type of context-specific
norm learning considered in this paper. Sarathy et al. [2, 3] presents a
context-specific norm modeling strategy that is based upon Dempster-Shafer
theory [17]. They present a formal mathematical framework in which norms are
modeled using a deontic logic with context-specificity and uncertainty.
Unfortunately they do not test their approach on a real robot or with real
scene video/image data. In this work, we use the mathematical framework
presented in [2] to model context-specific norms on a Pepper robot in which
scene categories (contexts) and related norms are learned in an online manner
through active learning.

##  3 Online Scene and Norm Learning

Figure 1: Overall architecture of our approach. The robot captures new data
using its camera and then predicts the label for the scene category using
CBCL-PR. The Text-to-Speech module is used to communicate the predicted label.
The human provides the true label for the new data to the robot as a text
input. The robot further asks the human about the permitted and forbidden
norms specific to the scene category.

Figure 1 shows the complete architecture of our proposed system. The system
can learn new scene categories in an online manner while matching new data to
previously learned scene categories and unknown categories. After learning a
scene category, the system can learn the norms related to the corresponding
scene. The architecture uses Centroid-Based Concept Learning-Pseudorehearsal
(CBCL-PR) approach proposed in [9] to train a robot on new scenes in an online
manner with only a single example of a scene available at each increment. For
norm modeling and learning, we use the mathematical model proposed in [2, 3].
The major components of the architecture are described below.

###  3.1 Centroid-Based Concept Learning and Pseudo-Rehearsal(CBCL-PR)

CBCL-PR [9] is a revised version of CBCL [4] which is a recently developed
state-of-the-art method for incremental learning [4, 18] and RGB-D indoor
scene classification [19]. CBCL-PR has also been shown to learn objects from
only a few examples whereas prior state-of-the-art methods required a large
amount of training data, its memory footprint does not grow dramatically as it
learns new classes, and its learning time is faster than the other, mostly
deep learning, methods. One of the main advantages of CBCL-PR is that it can
be adapted to learn from data coming in an online manner such that the new
incoming data can belong to already learned classes or completely new classes
[20].

Figure 2: For each new class of images in a dataset, the feature extractor
generates feature vectors for all of the training images in the class. A set
of centroids and covariance matrices are created for the feature vectors using
the Agg-Var clustering algorithm and concatenated with the centroids and
covariance matrices of previously learned classes. Pseudo-exemplars for old
classes and training examples of new classes are then used to train a shallow
network for classification.

Figure 2 depicts the architecture for CBCL-PR. It is composed of three
modules: 1) a feature extractor, 2) Agg-Var clustering, and 3) a pseudo-
rehearsal and classification module. In the learning phase, once the human
provides the robot with the training examples for a new class, the first step
in CBCL-PR is the generation of feature vectors from the images of the new
class using a fixed feature extractor. In this paper, we use VGG-16 [21] pre-
trained on Places365 dataset [10] as the feature extractor.

####  3.1.1 Agg-Var Clustering:

Next, in the case of incremental learning, for each new image class $y$, CBCL-
PR clusters all of the training images in the class. Agg-Var clustering, a
cognitively-inspired approach, begins by creating one centroid from the first
image in the training set of class $y$. Next, for each image in the training
set of the class, feature vector $x_{i}^{y}$ (for the $i$the image) is
generated and compared using the Euclidean distance to all the centroids for
the class $y$. If the distance of $x_{i}^{y}$ to the closest centroid is below
a pre-defined distance threshold $D$, the closest centroid is updated by
calculating a weighted mean of the centroid and the feature vector
$x_{i}^{y}$. If the distance between the $i$th image and the closest centroid
is greater than the distance threshold $D$, a new centroid is created for
class $y$ and equated to the feature vector $x_{i}^{y}$ of the $i$th image.

CBCL also finds the covariance matrices related to each centroid using the
feature vectors of all the images clustered in the centroid. The result of
this process is a collection containing a set of centroids,
$C^{y}=\\{c_{1}^{y},...,c_{N^{*}_{y}}^{y}\\}$, and covariance matrices,
$\sum^{y}=\\{\sigma_{1}^{y},...,\sigma_{N^{*}_{y}}^{y}\\}$, for the class $y$
where $N^{*}_{y}$ is the number of centroids for class $y$. This process is
applied to the sample set $X^{y}$ of each class incrementally once they become
available to get a collection of centroids $C=C^{1},C^{2},...,C^{N}$ and
covariance matrices $\sum=\sum^{1},\sum^{2},...,\sum^{N}$ for all $N$ classes
in a dataset. Note that Agg-Var clustering calculates the centroids for each
class separately. Thus, its performance is not strongly impacted by when the
classes are presented incrementally.

####  3.1.2 Pseudorehearsal:

For classification of images from all the classes seen so far, CBCL-PR trains
a shallow neural network composed of a linear layer trained with a softmax
loss on examples from the current increment and pseudo-exemplars of the old
classes (pseudorehearsal). Pseudo-exemplars are not real exemplars of a class,
rather they are generated based on the class statistics to best resemble the
actual exemplars.

CBCL-PR uses each of the centroids $c_{i}^{y}$ and covariance matrices
$\sigma_{i}^{y}$ of class $y$ to generate a set of pseudo-exemplars. First a
multi-variate Gaussian distribution is created using the centroid as the mean
and the corresponding covariance matrix. The Gaussian distribution is then
sampled to generate the same number of pseudo-exemplars as the total number of
exemplars represented by the centroid/covariance matrix pair
$c_{i}^{y}$,$\sigma_{i}^{y}$. This process leads to a total of $N_{y}$ number
of pseudo-exemplars for class $y$. In this way the network is trained on
$N_{y}$ number of examples for each class $y$ in each increment. For
classification of a test image, the feature vector $x$ of the image is first
generated using the feature extractor. Next, this feature vector is passed
through the shallow network to produce a predicted label $y^{*}$. A more
detailed description of CBCL-PR can be found in [4, 19, 9]

###  3.2 Predicting Unknown Scene Categories

CBCL-PR is designed for incremental learning where the test images always
belong to one of the learned categories. However, for the online learning
setting considered in this paper the robot also has to make a prediction about
an unknown scene category. Further, in [9] CBCL-PR is provided with the
complete training set of a class at one time, however for the online learning
scenario considered here, data belonging to a previously known scene category
can also become available in an increment.

For this work we have adapted CBCL-PR to operate in an online learning manner.
First, to make a prediction about an unknown scene category, we calculate the
Euclidean distance of the feature vector of the new scene image to all of the
learned centroids. If the distance is below the distance threshold $D$ that
was used to get the centroids, we predict that the new image belongs to a new
scene category. Second, if the ground truth provided by the human for a new
scene image belongs to one of the already learned categories, we simply apply
Agg-Var clustering on the new scene images to either update the previously
learned centroids and covariance matrices of the scene category or create new
centroids and covariance matrices based upon the distance threshold $D$. A
more detailed explanation for adapting CBCL-PR for online learning is provided
in [20].

###  3.3 Norm Modeling and Learning

We follow the norm modeling framework presented in [2, 3]. In particular, we
use the idea of a context-specific belief-theoretic norm which is of the form:

$\mathcal{N}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\mathbb{D}_{C}^{[\alpha,\beta]}A$ (1)

where $\mathcal{N}$ is a context-specific belief theoretic norm for a formal
language $\mathcal{L}$, $\mathbb{D}_{C}^{[\alpha,\beta]}$ is an uncertain
context-specific deontic operator with $[\alpha,\beta]$ representing a
Dempster-Shafer uncertainty interval [17] for the operator with
$0<\alpha<\beta<1$. $\mathbb{D}$ is a collection of three deontic model
operators: obligatory $\mathbb{O}$, forbidden $\mathbb{F}$ and permissible
$\mathbb{P}$. $C\in\mathcal{L}$ represents the context and $A\in\mathcal{L}$
represents a state or an action. The above mentioned norm expression states
that in a context $C$, an action or state $A$ is either obligatory, forbidden
or permissible or not obligatory, forbidden or permissible.

We use the mathematical model of equation (1) to model and learn norms for
different scene categories (contexts) in an online manner. The initial
knowledge of the robot consists of a set of possible actions that the robot
can perform. For example, talk is an action available to the robot where it
can use its speech module to communicate. However, the robot does not know
which of its known actions are permissible, obligatory or forbidden in a
particular context. Hence, for each new context the robot learns about
context-specific norms through active learning i.e. by having a short
question/answer session with a human partner. We only use yes/no question
types in this paper. In each new context/scene, the robot asks its human
partner which of its actions are obligatory, forbidden or permissible. If a
norm does not exist in the robot’s knowledge base, it simply creates a new
norm for a particular context. The new norm is initialized with an uncertainty
interval of either $[0,0]$ or $[1,1]$ based upon the answer given by the human
(yes or no). However, if a norm already exists in the robot’s knowledge base
and it gets an answer from its human partner, it updates the uncertainty
interval of the norm based on the new answer. In section 4 we present results
of learning different context-specific norms by the Pepper robot. The learned
norms and the associated uncertainties can be used by the robot to reason
about before performing an action in a given context. Experiments regarding
norm reasoning for performing actions are left for future work.

##  4 Experiments

Figure 3: Images from two different bathrooms collected by the Pepper robot.
Both images are drastically different from each other.

We evaluate our method for online learning of scenes and related norms on the
Pepper robot. The robot was manually driven through 10 different locations
belonging to 5 different categories (bathroom, classroom, library, Office,
kitchen) on the Penn State University campus. The locations differed from one
another even when belonging to the same category, making the learning and
recognition task difficult. For example, Figure 3 shows images from two
different bathrooms captured by the Pepper robot. Note how different the
images are. Images were captured using the front head camera of the Pepper
with a resolution of 320$\times$240\. Pepper’s text-to-speech module was used
to communicate with the human while the labels and norm related answers were
provided by the human using a keyboard.

###  4.1 Online Scene Learning Using the Pepper Robot

Our first experiment demonstrates our method for online scene learning on a
Pepper robot. For each new location, the robot took the input of a scene
(either belonging to earlier learned categories or a new category) using its
camera. Next, it used the method described in Figure 1 to either predict the
label of the scene or predict that the scene category was unknown. The robot
further asked the human to confirm the label of the new scene. If the human
provided the label, the text module passed it to CBCL-PR to update the
previous centroids, otherwise if the new scene belonged to a known category
the robot created new centroids for the new scene category.

Figure 4: An example of the Pepper robot trying to predict the category
kitchen.

It took approximately 30 seconds to capture a video, while feature extraction
took about 11 seconds. The time required to make a prediction about the new
video was only 1 second, while confirmation of the label from the human took
approximately 5 seconds. Agg-Var clustering took only 1 second to find the
centroids and pseudorehearsal and classifier training took 10 seconds. Hence,
the total time to learn about a new location was approximately 58 seconds. It
should be noted that after receiving the label from the human, Pepper continue
to process the data while moving to other locations, so it only needs 47
seconds at one location to make a prediction.

At each of the locations, Pepper captured the video data by rotating its head
from center to the left and then left to right for 20 seconds. After capturing
the video, the feature extractor was used to extract the features for all the
images. Our method (Figure 1) used the shallow classifier in CBCL-PR to make a
prediction about the category of the location, while it used the distance to
the closest centroid to make a prediction if the location was known. Pepper’s
text-to-speech module was then used to communicate the prediction. After
making the prediction, Pepper confirmed its prediction in the case of a known
category or asked for a new label in the case of an unknown category. The
experimenter then typed the ground truth of the location in a terminal. Pepper
repeated the typed label to confirm with the experimenter. After the
confirmation, Agg-Var clustering in CBCL-PR was used to find the centroids for
the new data in the case of an unknown category or update the previously
learned centroids in the case of a known category. After learning the
centroids that represent the new location, the image data was discarded to
satisfy the conditions of online learning. Figure 4 shows the Pepper robot in
a kitchen trying to predict the location’s label.

Of the 10 locations tested, Pepper was presented with 5 locations that
belonged to new categories and 5 locations from a previously known category.
Pepper correctly predicted 80% of locations when the location belonged to a
new category and 80% of locations when predicting known categories. These
results clearly show the effectiveness of our approach to mitigate
catastrophic forgetting when learning scene categories using streaming video
data on a robot. Where catastrophic forgetting is a problem in continual
learning in which the model forgets the previously learned classes when
learning new classes and the overall accuracy decreases drastically. Detailed
experiments on incremental learning on benchmark datasets for CBCL-PR are
available in [4, 9].

Robot Actions |  Description
---|---
talkLoundly |  Speak in a higher volume
talkQuietly |  Lower volume when speaking
beQuiet |  Do not speak
listen |  Process the incoming audio input
watch |  Capture images through camera for processing
walk |  Move around

Table 1: Description of the six action types available to the Pepper robot.

###  4.2 Norm Learning by the Pepper Robot

The setup for this experiment was the same as the scene learning experiment.
For each new scene, after the robot predicted the scene and got the ground
truth label from the human partner, it initiated a question/answer session
regarding norm learning. For each location, Pepper was allowed to ask at most
three questions. In the three questions, Pepper randomly chose which norms to
ask about. The limit of three questions was set arbitrarily, keeping in mind
that the human would get annoyed if he/she has to answer a large number of
questions at a location. All of the questions were related to the permission
norms i.e. if one of the robot actions was permissible at a location. The
initial set of actions available to Pepper in its knowledge base were:
talkLoudly, talkQuietly, beQuiet, listen, watch and walk. Table 1 provides a
description of the six actions. Pepper used its speech to text module to ask
questions about the norms. The experimenter answered the questions using a
keyboard. The experimenter’s answers were based on the results shown by
previous studies in norm learning [2, 3]. Using the answers provided by the
experimenter, Pepper updated the uncertainties related to the previously
learned context-specific norms and also created new context-specific norms
with 100% certainty.

Table 2 shows all the permission norms learned by Pepper at the 10 locations
(5 scene categories). Most of the action types have either 100% or 0%
uncertainties. This is because of the limited number of actions available to
the robot and the limited amount of locations visited per scene category (2
per scene category). In cases when the experimenter gave different answers in
different locations of a scene category about an action, the uncertainties
were updated. For example, for the action type talkQuitely in the library
context the agent said that it was not permissible in the first library.
Hence, Pepper’s initial uncertainty interval for this action was [0,0].
However, in the second library, the experimenter said talkQuietly was
permissible. Therefore, Pepper updated the uncertainty interval to [0,0.5].
The robot was not able to ask questions about some actions at all because of
the limited number of actions allowed to be asked. The empty entries in Table
2 show that the robot did not ask about those actions at the corresponding
location. For example, the robot never asked if action watch was permissible
in the library.

The time required to communicate with the human partner for three questions
was about 10 seconds. These results depict the ability of the robot to learn
context-specific norms in an online manner without forgetting any previously
learned information. Social robots can use these context-specific norms while
reasoning about which actions to perform in a particular context. The norm
reasoning module is left for future work.

Permitted Action |  Bathroom |  Classroom |  Library |  Office |  Kitchen
---|---|---|---|---|---
talkLoudly |  [0,0] |  [0,0] |  [0,0] |  [0,0] |  [1,1]
talkQuietly |  [1,1] |  [0,0.5] |  [0,0.5] |  [1,1] |  [0.5,1]
beQuiet |  [1,1] |  [0,0.5] |  [0.5,1] |  [0,0.5] |  [1,1]
listen |  [1,1] |  [1,1] |  - |  [1,1] |  [1,1]
watch |  [0,0] |  - |  [1,1] |  - |  [1,1]
walk |  [1,1] |  - |  - |  [1,1] |  -

Table 2: Permission norms with uncertainty intervals learned by Pepper for
five different scene categories.

##  5 Conclusion

This paper presents and evaluates a method for a novel problem: online
learning of scenes and context-specific norms on a robot in unconstrained
environments. We evaluate our approach on the Pepper robot to learn various
scene categories and related norms in an online manner. The work presented
here makes only a few assumptions. We assume that the human provides correct
labels and correct answers related to norms when asked by the robot. Incorrect
information would certainly negatively impact the performance of our approach.

Nevertheless, we believe that this paper makes several important
contributions. Most importantly, we offer a novel and realistic approach to
online scene and context-specific norm learning. This method allows the robots
to not only learn different categories and appropriate behaviors associated
with these categories but to also use the recognition of these different
environments to moderate their behavior and decision-making. For example,
allowing a robot to recognize that its current location is in a library and
that it should therefore reduce the volume of its speech. Our future work will
focus on the reasoning module to choose appropriate actions based on learned
context-specific norms. Our overarching hope is that this work will evolve
into new applications and competencies for a variety of robot applications.

## Acknowledgements

This work was partially supported by the Air Force Office of Sponsored
Research contract FA9550-17-1-0017 and National Science Foundation grant
CNS-1830390. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.

## References

  * [1] E. Ullmann-Margalit.  The emergence of norms.  Oxford: Clarendon Press, 1976.
  * [2] V. Sarathy, M. Scheutz, and B. F. Malle.  Learning behavioral norms in uncertain and changing contexts.  In 2017 8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), pages 000301–000306, 2017.
  * [3] Vasanth Sarathy, Matthias Scheutz, Yoed Kenett, Mowafak Allaham, Joseph Austerweil, and Bertram Malle.  Mental representations and computational modeling of context-specific human norm systems.  In 39th annual meeting of the cognitive science society (CogSci), 2017.
  * [4] Ali Ayub and Alan R. Wagner.  Cognitively-inspired model for incremental learning using a few examples.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [5] Ali Ayub and Alan R. Wagner.  Storing encoded episodes as concepts for continual learning.  arXiv:2007.06637, 2020.
  * [6] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  iCaRL: Incremental classifier and representation learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [7] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [8] Ali Ayub and Alan R. Wagner.  Online learning of objects through curiosity-driven active learning.  IEEE RoMan (Workshop on Lifelong Learning for Long-term Human-Robot Interaction), 2020.
  * [9] Ali Ayub and Alan R. Wagner.  Cognitively-inspired model for incremental category learning using only a few examples, 2020.  In Review, Pattern Recognition.
  * [10] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017\.
  * [11] Z. Wang, L. Wang, Y. Wang, B. Zhang, and Y. Qiao.  Weakly supervised patchnets: Describing and aggregating local patches for scene recognition.  IEEE Transactions on Image Processing, 26(4):2028–2041, April 2017\.
  * [12] Aram Kawewong, Rapeeporn Pimup, and Osamu Hasegawa.  Incremental learning framework for indoor scene recognition.  In AAAI, 2013.
  * [13] A. Quattoni and A. Torralba.  Recognizing indoor scenes.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 413–420, June 2009.
  * [14] Sujoy Paul, Jawadul H. Bappy, and Amit K. Roy-Chowdhury.  Non-uniform subset selection for active learning in structured data.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [15] V. Krishnamoorthy, W. Luo, M. Lewis, and K. Sycara.  A computational framework for integrating task planning and norm aware reasoning for social robots.  In 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 282–287, 2018.
  * [16] Zong Xuan Tan, Jake Brawer, and Brian Scassellati.  That’s mine! learning ownership relations and norms for robots.  In AAAI, 2019.
  * [17] Glenn Shafer.  A mathematical theory of evidence.  Princeton University Press, 1976.
  * [18] Ali Ayub and Alan R. Wagner.  Tell me what this is: Few-shot incremental object learning by a robot.  arXiv:2008.00819, 2020.
  * [19] Ali Ayub and Alan R. Wagner.  Centroid based concept learning for rgb-d indoor scene classification.  arXiv:1911.00155, 2020.
  * [20] Ali Ayub, Carter Fendley, and Alan R. Wagner.  Boundaryless online learning of indoor scenes by a robot.  In Review, CoRL, 2020.
  * [21] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv 1409.1556, 09 2014.


# Kalman Filter from the Mutual Information Perspective

Yarong Luo<EMAIL_ADDRESS>
Jianlang Hu<EMAIL_ADDRESS>
Chi Guo<EMAIL_ADDRESS>
GNSS Research Center, Wuhan University

Abstract
Kalman filter is a best linear unbiased state estimator. It is also
comprehensible from the point view of the Bayesian estimation. However, this
note gives a detailed derivation of Kalman filter from the mutual information
perspective for the first time. Then we extend this result to the Rényi mutual
information. Finally we draw the conclusion that the measurement update of the
Kalman filter is the key step to minimize the uncertainty of the state of the
dynamical system.

Key Words
Kalman filter, mutual information, Rényi mutual information, uncertainty,
measurement update

##  Introduction

Kalman filter has been widely in various fields as an effective state
estimator for integrated navigation [1], robotics [2], etc. The classical
Kalman filter can be derived as a best linear unbiased estimate [1] and it is
easy understand it from the probabilistic perspective [2]. Recently, Kalman
filter has also been presented using the methods of maximum relative entropy
[3] and the temporal derivative of the Rényi entropy [4], which go beyond the
general Bayesian filter. More and more evidences show that Kalman filter can
be regarded as a direct extension of information theory. This note gives a new
perspective of the Kalman filter from the mutual information, which further
bridges the gap between the optimal state estimation and the information
theory. The main contribution of this note is to derive the Kalman filter from
the perspective of mutual information and extend it to the Rényi mutual
information case.

##  Kalman Filter from the Mutual Information

For following discrete-time state-space model

$X_{k}={\Phi_{k|k-1}}X_{k-1}+{\Gamma_{k|k-1}}W_{k-1}$ (1)
$Z_{k}=H_{k}X_{k}+V_{k}$ (2)

where $X_{k}$ is n-dimensional state vector; $Z_{k}$ is m-dimensional
measurement vector; ${\Phi_{k|k-1}}$, ${\Gamma_{k|k-1}}$ and $H_{k}$ are the
known system structure parameters, which are called the $n\times n$
dimensional one-step state update matrix, the $n\times l$ dimensional system
noise distribution matrix, and the $m\times n$ dimensional measurement matrix,
respectively; $W_{k-1}$ is the $l$-dimensional system noise vector, and
$V_{k}$ is the m-dimensional measurement noise vectors. Both of them are
Gaussian noise vector sequences with zero mean value, and independent to each
other:

$\mathbb{E}[W_{k}]=0,\mathbb{E}[W_{k}W_{j}^{T}]=Q_{k}\delta_{kj}$ (3)
$\mathbb{E}[V_{k}]=0,\mathbb{E}[V_{k}V_{j}^{T}]=R_{k}\delta_{kj}$ (4)
$\mathbb{E}[W_{k}V_{j}^{T}]=0$ (5)

The one-step prediction covariance matrix is denoted as $\Sigma_{k|k-1}$. The
state estimation at $t_{k}$ is denoted as
$\mathcal{N}(\hat{X}_{k},\Sigma_{k})$, where $\hat{X}_{k}$ is the mean of
estimated state and $\Sigma_{k}$ is the covariance matrix of the estimated
covariance matrix. Assuming the optimal estimation of the state can be
calculated as follows:

$\hat{X}_{k}={X_{k|k-1}^{-}}+K_{k}{\tilde{Z}_{k|k-1}}$ (6)

where $K_{k}$ is the undetermined correction factor matrix,
$X_{k|k-1}^{-}=X_{k|k-1}-\hat{X}_{k-1}$ is the state estimation error,
$\tilde{Z}_{k|k-1}=Z_{k}-H_{k}X_{k|k-1}^{-}$ is the measurement one-step
prediction error.

Then, the mean square error matrix of state estimation $\hat{X}_{k}$ is given
by [1]

${\Sigma}_{k}=({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}+K_{k}R_{k}K_{k}^{T}$
(7)

the mean square error matrix $\Sigma_{k}$ is positive definite as
$({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}$ is positive definite
and $K_{k}R_{k}K_{k}^{T}$ is positive definite.

The joint Gaussian distribution can be expressed

$p(X,Y)\sim\left(\begin{bmatrix}\hat{X}\\\
\hat{Y}\end{bmatrix},\begin{bmatrix}\Sigma_{xx}&\Sigma_{xy}\\\
\Sigma_{yx}&\Sigma_{yy}\end{bmatrix}\right)$ (8)

where
$X\sim\mathcal{N}(\hat{X},\Sigma_{xx})$,$Y\sim\mathcal{N}(\hat{Y},\Sigma_{yy})$.

The mutual information for a joint Gaussian PDF can be represented by

$\displaystyle I(X,Y)$ $\displaystyle=H(X)+H(Y)-H(X,Y)=H(X)-H(X|Y)$ (9)
$\displaystyle=\frac{1}{2}\ln((2\pi
e)^{N}\det\Sigma_{xx})+\frac{1}{2}\ln((2\pi
e)^{M}\det\Sigma_{yy})-\frac{1}{2}\ln((2\pi e)^{M+N}\det\Sigma)$
$\displaystyle=-\frac{1}{2}\ln\left(\frac{\det\Sigma}{\det\Sigma_{xx}\det\Sigma_{yy}}\right)$

where $H(X)=\frac{1}{2}\ln((2\pi e)^{N}\det\Sigma_{xx})$ is the entropy of a
Gaussian random variable, and

$\det\Sigma=\det\Sigma_{xx}\det\left(\Sigma_{yy}-\Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}\right)=\det\Sigma_{yy}\det\left(\Sigma_{xx}-\Sigma_{xy}\Sigma_{yy}^{-1}\Sigma_{yx}\right)$
(10)

Therefore, the mutual information describes the reduction of the uncertainty
in variable X due to gaining knowledge of variable Y.

Similarly, the mutual information at time step $t_{k+1}$ can be easily
computed by the a priori, a posteriori PDF and the Kalman gain $K_{k}$ as

$I(\hat{X}_{k|k-1},Z_{k})=\frac{1}{2}\ln\left(\frac{\det\Sigma_{k|k-1}}{\det\Sigma_{k}}\right)=\frac{1}{2}\ln\left(\frac{\det\Sigma_{k|k-1}}{\det\left(({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}+K_{k}R_{k}K_{k}^{T}\right)}\right)$
(11)

It describes the reduction of the uncertainty of the state due to gaining
knowledge from the measurement $Z_{k}$. Consequently, we want to maximize the
mutual information $I(\hat{X}_{k|k-1},Z_{k})$. It is obvious that the maximum
of the mutual information is equivalent to the minimum of $\ln\det\Sigma_{k}$.
We can note that equation (11) is the function of the unknown factor matrix
and thereby the maximization of it can be calculated by taking derivative of
it with respect to $K_{k}$ and setting it equal to zero,

$\frac{dI(\hat{X}_{k|k-1},Z_{k})}{dK_{k}}=-\Sigma_{k}^{-T}\frac{d\Sigma_{k}}{dK_{k}}=-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)=0$
(12)

where $\frac{\partial\ln\det X}{\partial X}=X^{-T}$ [5] has been used and then
solving for $K_{k}$ gives

$K_{k}=\Sigma_{k|k-1}H_{k}^{T}(H_{k}\Sigma_{k|k-1}H_{k}^{T}+R_{k})^{-1}$ (13)

A subsequent derivative of equation (12) must be performed to check for a
maximum, that is

$\frac{d}{dK_{k}}\left(-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)\right)$
(14)

After substituting the equation (12) into above equation results in

$\frac{d}{dK_{k}}\left(-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)\right)=-\Sigma_{k}^{-T}(H_{k}\Sigma_{k|k-1}H_{k}^{T}+R_{k})$
(15)

which is always negative definite by the definition of the covariance matrices
$R_{k}$ and $H_{k}\Sigma_{k|k-1}H_{k}^{T}$, ensuring the solution for $K_{k}$
is a maximum.

##  Kalman Filter from the Rényi Mutual Information

Moreover, the Rényi mutual information of a joint Gaussian PDF can be
calculated similar to equation (9):

$\displaystyle I_{R}^{\alpha}(X,Y)$
$\displaystyle=H_{R}^{\alpha}(X)+H_{R}^{\alpha}(Y)-H_{R}^{\alpha}(X,Y)$ (16)
$\displaystyle=\frac{1}{2}\ln|(2\pi)^{N}\alpha^{\frac{N}{\alpha-1}}\det\Sigma_{xx}|+\frac{1}{2}\ln|(2\pi)^{M}\alpha^{\frac{M}{\alpha-1}}\det\Sigma_{yy}|-\frac{1}{2}\ln|(2\pi)^{N+M}\alpha^{\frac{N+M}{\alpha-1}}\det\Sigma|$
$\displaystyle=\frac{1}{2}\ln\left(\frac{\det\Sigma_{xx}\det\Sigma_{yy}}{\det\Sigma}\right)$
$\displaystyle=I(X,Y)$

where
$H_{R}^{\alpha}(X)=\frac{1}{2}\ln|(2\pi)^{N}\alpha^{\frac{N}{\alpha-1}}\det\Sigma_{xx}|$
is the Rényi entropy of order $\alpha$ for a continuous Gaussian random
variable with a multivariate Gaussian PDF. Consequently, we can know that the
mutual information is the same as the Rényi mutual information for the joint
Gaussian PDF. Similarly, we can get the same result as equation (13) from the
Rényi mutual information.

##  Conclusions

In this paper, Kalman filter is derived from the perspective of mutual
information and extended to the Rényi mutual information case. We show that
the measurement update of the Kalman filter can minimize the uncertainty of
the state by formulating it as the mutual information between the evolving
state and the measurement and maximizing the mutual information. Furthermore,
we can think of Kalman filter a little more radically as an extension of the
information theory.

Acknowledgement
This research was supported by a grant from the National Key Research and
Development Program of China (2018YFB1305001). We express thanks to GNSS
Center, Wuhan University.

## References

  * [1] Y. Gongmin and W. Jun, _Lectures on Strapdown Inertial Navigation Algorithm and Integrated Navigation Principles_. Northwestern Polytechnical University Press: Xi’an, China, 2019.
  * [2] S. Thrun, W. Burgard, and D. Fox, _Probabilistic Robotics_. MIT Press, 2005.
  * [3] A. Giffin and R. Urniezius, “The kalman filter revisited using maximum relative entropy,” _Entropy_ , vol. 16, no. 2, pp. 1047–1069, 2014.
  * [4] Y. Luo, C. Guo, S. You, and J. Liu, “A novel perspective of the kalman filter from the rényi entropy,” _Entropy_ , vol. 22, no. 9, p. 982, 2020.
  * [5] X.-D. Zhang, _Matrix analysis and applications_. Cambridge University Press, 2017.