# MIML library: a Modular and Flexible Library for Multi-instance Multi-label
Learning

Álvaro Belmonte11footnotemark: 1<EMAIL_ADDRESS>Amelia Zafra22footnotemark:
233footnotemark: 3<EMAIL_ADDRESS>Eva Gibaja44footnotemark: 455footnotemark: 5
<EMAIL_ADDRESS>Department of Computer Science and Numerical Analysis.
University of Córdoba, Spain  Andalusian Research Institute in Data Science
and Computational Intelligence (DaSCI). University of Córdoba, Spain

###### Abstract

MIML library is a Java software tool to develop, test, and compare
classification algorithms for multi-instance multi-label (MIML) learning. The
library includes 15+15+13 algorithms and provides a specific format and
facilities for data managing and partitioning, holdout and cross-validation
methods, standard metrics for performance evaluation, and generation of
reports. In addition, algorithms can be executed through $xml$ configuration
files without needing to program. It is platform-independent, extensible,
free, open-source, and available on GitHub under the GNU General Public
License.

###### keywords:

Multi-instance Learning , Multi-label Learning , Weka , Mulan , Classification

††journal: Neurocomputing

basicstyle=, morestring=[b]", morestring=[s]><, morecomment=[s]<??>,
stringstyle=, identifierstyle=, keywordstyle=, showtabs=false, tabsize=2,
morekeywords=header, configuration, classifier, multiInstanceClassifier,
evaluator, data, trainFile,testFile,xmlFile,reportFileName,
multiLabelClassifier, parameters, parameter, value, class, transformMethod,
standardDeviation, macroMeasuresLabels, threshold, seed,
sampleWithReplacement, useConfidences, samplePercentage, numClassifiers,
baseLearner, measures, measure, numFolds, nReferences, nCiters, metric,
listOptions, report, fileName, @attribute, @relation, @data, label, @end,
labels, file basicstyle=, morestring=[b]", morestring=[s]><,
morecomment=[s]<??>, stringstyle=, identifierstyle=, keywordstyle=,
showtabs=false, tabsize=2, morekeywords=header, configuration, classifier,
multiInstanceClassifier, evaluator, data,
trainFile,testFile,xmlFile,reportFileName, multiLabelClassifier, parameters,
parameter, value, class, transformMethod, standardDeviation,
macroMeasuresLabels, threshold, seed, sampleWithReplacement, useConfidences,
samplePercentage, numClassifiers, baseLearner, measures, measure, numFolds,
nReferences, nCiters, metric, listOptions, report, fileName, @attribute,
@relation, @data, label, @end, labels, file numbers=left, stepnumber=1,
numberstyle=, numbersep=10pt language=Java, commentstyle=, keywordstyle=,
stringstyle=, xleftmargin=9pt, xrightmargin=2pt,

##  1 Introduction

Multi-instance multi-label (MIML) learning has emerged as a promising
supervised learning paradigm by combining multi-instance (MI) and multi-label
(ML) learning. On the one hand, MI allows a more flexible representation of
input space by associating a pattern (_bag_) with multiple instances, all of
them with the same number of attributes [1]. On the other hand, ML introduces
greater flexibility representation in the output space by associating a
pattern with not one, but a set of binary classes (_labels_) [2]. For example,
an image could be represented by multiple instances being each one a region in
the image and each image could have several binary labels (e.g. cloud, lion,
landscape). MIML carries out a natural formulation of complex objects in real
problems such as text and image categorization, audio and video annotation, or
bioinformatics [3].

Currently, there are available several well-known Java frameworks such as Weka
[4] to work with MI learning and Mulan [5] or Meka [6] to work with ML
learning. Nevertheless, none of these libraries can work with MIML data. To
the best of our knowledge, there are only publicly available some Matlab
implementations of particular MIML algorithms [7]. However, they are not
integrated into a library. Each one has its specific configuration and data
format. The motivation of this work is to develop a modular and extensible
Java library, based on the Weka and Mulan libraries, to facilitate the running
and development of MIML classification algorithms.

The rest of the document is organized as follows: an overview of the
architecture and functionalities of the library is specified in Section 2. An
illustrative example is presented in Section 3. Finally, conclusions are
pointed out in Section 4.

##  2 MIML Framework

###  2.1 Software Architecture

MIML library is available in GitHub666https://github.com/kdis-lab/MIML under
the GNU license. It is organized in packages, each one containing the
interfaces and the classes required to extend its functionality (see Figure
1). The core and run packages contain classes related to the configuration and
running of algorithms by means of xml files. The evaluation package contains
cross-validation and holdout evaluators. The data package includes classes to
load and save MIML data files and two subpackages: miml.data.statistics with
classes to obtain statistics about the dataset and miml.data.partitioning with
classes for holdout and cross-validation data partitioning. The transformation
package comprises methods to transform a MIML dataset into either ML
(transformation.mimlTOml) or MI dataset (transformation.mimlTOmi). Interfaces
and abstract classes required to develop MIML classification algorithms are
included in classifers.miml which comprises six subpackages: mimlTOmi and
mimlTOml with algorithms which transform the MIML problem to a MI or ML
problem respectively. The meta package is for ensemble methods, lazy for
instance-based algorithms, neural for neural networks algorithms, and
optimization for proposals based on optimization. The report package contains
classes to generate reports for the experiments carried out, and the tutorial
package includes practical usage examples (e.g. transforming and managing a
dataset, running holdout and cross-validation experiments, data partitioning,
etc.). In addition, the distribution provides an extensive user manual with a
complete description of the library777https://github.com/kdis-
lab/MIML/tree/master/documentation. It includes topics such as an introduction
to MIML learning, getting and running the library, managing MIML data, running
a classification algorithm included in the library, developing a new MIML
classification algorithm and practical examples for all algorithms. The API
Javadoc in both html and pdf is also provided.

Figure 1: Architecture of MIML library.

###  2.2 Software Functionalities

The library uses a data format specifically designed for MIML learning which
is based on Weka and Mulan data formats. It additionally provides
functionality for loading, manipulating, saving, and obtaining representative
metrics of MIML data (e.g. number of bags, number of labels, cardinality,
density, label co-occurrence, imbalance ratio, etc.). Besides, 3 strategies
for holdout and cross-validation MIML data partitioning are available: random,
powerset [8] and iterative [8]. The library includes 3 transformations of MIML
data to ML representation (arithmetic, geometric, and min-max) and 2
transformations to MI representation (BR and LP) as well as 3 different
distance functions between bags (average, minimal, and maximal Hausdorff). It
can run both cross-validation and holdout experiments.

The library includes 15+15+13 MIML classification algorithms (see Table 2.2).
On the one hand, $15$ MI Weka algorithms can be used by transforming the
problem to MI. On the other hand, $15$ Mulan algorithms can be executed by
transforming the problem to ML. Besides, the library includes $13$ algorithms
that cope with a MIML problem without transformation, some of them developed
as wrappers. Concretely, MIMLBagging that follows a bagging scheme [9],
algorithms based on neural networks such as MIMLNN [10], MIMLRBF [11] and
EnMIMLNNmetric [12]; algorithms based on optimization such as MIMLFast [13],
KiSar [14], MIMLSVM [15] and MIMLWel [16] and algorithms based on nearest
neighbors such as MIMLkNN algorithm [17]. Finally, it includes MIMLBRkNN,
MIMLMAPkNN, DMIMLkNN and MIMLIBLR that are adaptations to the MIML framework
of BRkNN_ML [18], MLkNN [19], DMLkNN [20] and IBLR_ML [21] ML algorithms.

Algorithms can be easily executed through configuration files in xml format to
set holdout and cross-validation methods and a wide set of performance metrics
that will include the output report. In addition to this easy way to run the
algorithms, its modular structure eases the development of new proposals
simply extending the class miml.classifiers.miml.MIMLClassifier.

Table 1: Algorithms included in the MIML library.

gray MIMLtoMI |  MIMLtoML |  MIML
---|---|---
lightgray Label | MI Algorithm | Bag | ML Algorithm | MIML
lightgray transf. | (Weka) | transf. | (Mulan) | Algorithm
|  CitationKNN [22] |  |  BR [23] |  |
|  MDD [24] |  |  LP [23] | Bagging | MIMLBagging
|  MIDD [24] |  |  RPC [25] |  |  MIMLNN [10]
|  MIBoost [26] |  |  CLR [27] | ANN |  MIMLRBF [11]
|  MILR [28] |  |  BRkNN [18] |  |  EnMIMLNNmetric [12]
BR [23] |  MIOptimalBall [29] |  |  DMLkNN [20] |  |  MIMLkNN [17]
|  MIRI [30] | Arithmetic |  IBLR_ML [21] |  | DMIMLkNN
|  MISMO [31] | Geometric |  MLkNN [19] | Lazy | MIMLIBLR
|  MISVM [32] | Min-Max |  HOMER [33] |  | MIMLMAPkNN
|  MITI [34] |  [35] |  RAkEL [36] |  | MIMLBRkNN
|  MIWrapper [37] |  |  PS [38] |  |  MIMLWel [16]
|  SimpleMI [35] |  |  EPS [38] | Other |  MIMLFast [13]
|  CitationKNN [22] |  |  CC [39] |  |  KiSar [14]
LP [23] |  MIWrapper [37] |  |  ECC [39] |  |  MIMLSVM [15]
|  SimpleMI [35] |  |  MLStacking [40] |  |

##  3 An illustrative example

Classification algorithms included in the library are executed by the
RunAlgorithm class by using a xml configuration file. The configuration file
path is specified through the command line with the option -c. An example of
an execution command would be:

⬇

$ java -cp dist/miml-1.0.jar miml.run.RunAlgorithm -c
dist/configurations/MIMLClassifier/MIMLkNN.config

Configuration files start at root element <configuration> and contain three
branches: <classifier>, <evaluator> and <report>. The <classifier> element
specifies the classification algorithm by means of its name attribute. In the
example, the specification of the parameters <nReferences>, <nCiters> and
<metric> of MIMLkNN algorithm is shown. The <evaluator> element sets the
validation method by means of its name attribute. The dataset used is
described in element <data>. In the example, cross-validation evaluator is
used to set the seed and the number of folds in <seed> and <numFolds>
elements. Finally, the <report> element specifies the class of the output
report by means of name attribute. This class can be easily extended to obtain
the most convenient output format. The <fileName> element contains the path
where the report will be stored. Optionally, the <measures> element can be
defined to include a specific set of measures in the output report. If no
measures are configured, all measures allowed by the classifier are shown. In
the example, the hamming loss, macro-averaged precision and micro-averaged
recall are specified. For macro-averaged measures, the perLabel attribute sets
if separated measures for each label are included in the report. Examples of
configuration files for all algorithms in the library are included in
configurations folder as well as in the user’s manual. The user manual details
the rest of the functionalities (e.g. transforming a dataset, developing new
proposals, etc.).

⬇

<classifier name=”miml.classifiers.miml.lazy.MIMLkNN”>

<nReferences>4</nReferences>

<nCiters>6</nCiters>

<metric name=”miml.core.distance.AverageHausdorff”></metric>

</classifier>

<evaluator name=”miml.evaluation.EvaluatorCV”>

<seed>712637</seed>

<numFolds>5</numFolds>

<data>

<file>data/miml_birds.arff</file>

<xmlFile>data/miml_birds.xml</xmlFile>

</data>

</evaluator>

<report name=”miml.report.BaseMIMLReport”>

<fileName>results/mimlknnn.csv</fileName>

<measures perLabel=”true”>

<measure>Hamming Loss</measure>

<measure>Macro-averaged Precision</measure>

<measure>Micro-averaged Recall</measure>

</measures>

</report>

##  4 Conclusion

This work presents a free, open-source Java library that eases the
development, testing, and comparison of MIML classification algorithms. As it
is based on Weka and Mulan, researchers on MIML who have used any of these
libraries will be familiar with it. Besides, it is extensible, platform-
independent, and easy to install, configure and use. It is accompanied by a
user manual with usage examples, a step-by-step tutorial on preparing and
running experiments, a detailed overview of the architecture and instructions
to extend the software for customization purposes. We consider this library
can facilitate the development of experimental studies and the design of new
proposals, contributing to future advances in the field.

## Acknowledgments

This research was supported by the Spanish Ministry of Science and Innovation
and the European Regional Development Fund, project PID2020-115832GB-I00.

## References

  * [1] T. G. Dietterich, R. H. Lathrop, T. Lozano-Pérez, Solving the multiple instance problem with axis-parallel rectangles, Artificial intelligence 89 (1-2) (1997) 31–71.
  * [2] E. Gibaja, S. Ventura, A tutorial on multilabel learning, ACM Computing Surveys (CSUR) 47 (3) (2015) 52.
  * [3] F. Herrera, S. Ventura., R. Bello, C. Cornelis, A. Zafra, D. Sánchez-Tarragó, S. Vluymans, Multiple Instance Learning. Foundations and Algorithms, Springer, 2016.
  * [4] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten, The weka data mining software: an update, ACM SIGKDD explorations newsletter 11 (1) (2009) 10–18.
  * [5] G. Tsoumakas, E. Spyromitros-Xioufis, J. Vilcek, I. Vlahavas, Mulan: A java library for multi-label learning, Journal Machine Learning Research 12 (2011) 2411–2414.
  * [6] J. Read, P. Reutemann, B. Pfahringer, G. Holmes, Meka: a multi-label/multi-target extension to weka, Journal of Machine Learning Research 17 (1) (2016) 667–671.
  * [7] LAMDA, Lamda learning and mining from data, http://www.lamda.nju.edu.cn/Data.ashx, accessed: 2020-07-19 (2019).
  * [8] K. Sechidis, G. Tsoumakas, I. Vlahavas, On the stratification of multi-label data, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2011, pp. 145–158.
  * [9] L. Breiman, Bagging predictors, Machine learning 24 (2) (1996) 123–140.
  * [10] Z. Zhou, M. Zhang, S. Huang, Y. Li, Multi-instance multi-label learning, Artificial Intelligence 176 (1) (2012) 2291–2320.
  * [11] M. Zhang, Z. Wang, Mimlrbf: Rbf neural networks for multi-instance multi-label learning, Neurocomputing 72 (16-18) (2009) 3951–3956.
  * [12] J. Wu, S. Huang, Z. Zhou, Genome-wide protein function prediction through multi-instance multi-label learning, Transactions on Computational Biology and Bioinformatics 11 (5) (2014) 891–902.
  * [13] S.-J. Huang, W. Gao, Z.-H. Zhou, Fast multi-instance multi-label learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (11) (2019) 2614–2627.  doi:10.1109/TPAMI.2018.2861732.
  * [14] Y.-F. Li, J.-H. Hu, Y. Jiang, Z.-H. Zhou, Towards discovering what patterns trigger what labels, in: 26th AAAI Conference on Artificial Intelligence, Toronto, Canada, 2012.
  * [15] Z. H. Zhou, M. L. Zhang, Multi-instance multi-label learning with application to scene classification, in: Advances in Neural Information Processing Systems 19 (NIPS’06), Vancouver, Canada, 2007, pp. 1609–1616.
  * [16] S.-J. Yang, Y. Jiang, Z.-H. Zhou, Multi-instance multi-label learning with weak label, in: Proceedings of the Twenty-Third international joint conference on Artificial Intelligence (IJCAI’13), Beijing, China, 2013, pp. 1862–1868.
  * [17] M. Zhang, A k-nearest neighbor based multi-instance multi-label learning algorithm, in: Proceedings of the 22nd International Conference on Tools with Artificial Intelligence, Vol. 2, 2010, pp. 207–212.
  * [18] E. Spyromitros, G. Tsoumakas, I.Vlahavas, An empirical study of lazy multilabel classification algorithms, in: Proc. 5th Hellenic Conference on Artificial Intelligence (SETN 2008), 2008.
  * [19] M.-L. Zhang, Z.-H. Zhou, Ml-knn: A lazy learning approach to multi-label learning, Pattern recognition 40 (7) (2007) 2038–2048.
  * [20] Z. Younes, F. Abdallah, T. Denoeux, Multi-label classification algorithm derived from k-nearest neighbor rule with label dependencies, in: 2008 16th European Signal Processing Conference, 2008, pp. 1–5.
  * [21] W. Cheng, E. Hüllermeier, Combining instance-based learning and logistic regression for multilabel classification, Machine Learning 76 (2-3) (2009) 211–225.
  * [22] J. Wang, J.-D. Zucker, Solving the multiple-instance problem: A lazy learning approach, in: P. Langley (Ed.), Proceedings of the 17th International Conference on Machine Learning, ICML ’00, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2000, pp. 1119–1126.
  * [23] G. Tsoumakas, I. Katakis, I. Vlahavas, Mining multi-label data, in: Data mining and knowledge discovery handbook, Springer, 2009, pp. 667–685.
  * [24] O. Maron, T. Lozano-Pérez, A framework for multiple-instance learning, in: Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS ’97, MIT Press, Cambridge, MA, USA, 1998, pp. 570–576.
  * [25] E. Hüllermeier, J. Fürnkranz, W. Cheng, K. Brinker, Label ranking by learning pairwise preferences, Artificial Intelligence 172 (2008) 1897–1916.
  * [26] X. Xu, E. Frank, Logistic regression and boosting for labeled bags of instances, in: Advances in knowledge discovery and data mining, Springer, 2004, pp. 272–281.
  * [27] J. Fürnkranz, E. Hüllermeier, E. Loza mencía, K. Brinker, Multilabel classification via calibrated label ranking, Machine Learning 73(2) (2008) 133 – 153.
  * [28] S. Ray, M. Craven, Supervised versus multiple instance learning: An empirical comparison, in: Proceedings of the 22nd international conference on Machine learning, 2005, pp. 697–704.
  * [29] P. Auer, R. Ortner, A boosting approach to multiple instance learning, in: 15th European Conference on Machine Learning, Springer, 2004, pp. 63–74, lNAI 3201.
  * [30] L. Bjerring, E. Frank, Beyond trees: Adopting miti to learn rules and ensemble classifiers for multi-instance data, in: Proceedings of the Australasian Joint Conference on Artificial Intelligence, Springer, 2011.
  * [31] T. Gärtner, P. A. Flach, A. Kowalczyk, A. J. Smola, Multi-instance kernels., in: ICML, Vol. 2, 2002, pp. 179–186.
  * [32] S. Andrews, I. Tsochantaridis, T. Hofmann, Support vector machines for multiple-instance learning, in: Advances in Neural Information Processing Systems 15, MIT Press, 2003, pp. 561–568.
  * [33] G. Tsoumakas, I. Katakis, I. Vlahavas, Effective and efficient multilabel classification in domains with large number of labels, in: Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD’08), 2008.
  * [34] H. Blockeel, D. Page, A. Srinivasan, Multi-instance tree learning, in: Proceedings of the International Conference on Machine Learning, ACM, 2005, pp. 57–64.
  * [35] L. Dong, A comparison of multi-instance learning algorithms, Master’s thesis, The University of Waikato (2006).
  * [36] G. Tsoumakas, I. Katakis, I. Vlahavas, Random k-labelsets for multilabel classification, IEEE Transactions on Knowledge and Data Engineering 23 (7) (2011) 1079–1089.
  * [37] E. T. Frank, X. Xu, Applying propositional learning algorithms to multi-instance data, Tech. rep., University of Waikato, Department of Computer Science, University of Waikato, Hamilton, NZ (06 2003).
  * [38] J. Read, B. Pfahringer, G. Holmes, Multi-label classification using ensembles of pruned sets, in: Data Mining, 2008. ICDM’08. Eighth IEEE International Conference on, IEEE, 2008, pp. 995–1000.
  * [39] J. Read, B. Pfahringer, G. Holmes, E. Frank, Classifier chains for multi-label classification, Machine learning 85 (3) (2011) 333.
  * [40] G. Tsoumakas, A. Dimou, E. Spyromitros-Xioufis, V. Mezaris, I. Kompatsiaris, I. Vlahavas, Correlation-based pruning of stacked binary relevance models for multi-label learning, in: Proceedings of the 1st International Workshop on Learning from Multi-Label Data, 2009, pp. 101–116.

## Required Metadata

## Current executable software version

Table 2: Software metadata Nr. |  (executable) Software metadata description |  Please fill in this column
---|---|---
S1 |  Current software version |  1.0
S2 |  Permanent link to executables of this version |  https://github.com/kdis-lab/miml
S3 |  Legal Software License |  GPLv3
S4 |  Computing platform/Operating System |  GNU Linux, Microsoft Windows, OS X
S5 |  Installation requirements & dependencies |  Java version 1.8
S6 |  If available, link to user manual - if formally published include a reference to the publication in the reference list |  https://github.com/kdis-lab/MIML/blob/master/ documentation/MIML-UserManual.pdf
S7 |  Support email for questions |<EMAIL_ADDRESS>

## Current code version

Table 3: Software metadata Nr. |  (executable) Software metadata description |  Please fill in this column
---|---|---
C1 |  Current code version |  1.0
C2 |  Permanent link to code/repository used of this code version |  https://github.com/kdis-lab/miml
C3 |  Legal Code License |  GPLv3
C4 |  Code versioning system used |  git
C5 |  Software code languages, tools, and services used |  Java version 1.8
C6 |  Compilation requirements, operating environments & dependencies& dependencies |  junit 4.12
C7 |  If available Link to developer documentation/manual |  https://github.com/kdis-lab/MIML/blob/master/ documentation/MIML-UserManual.pdf
C8 |  Support email for questions |<EMAIL_ADDRESS>


# Few-shot Subgoal Planning with Language Models

Lajanugen Logeswaran , Yao Fu‡, Moontae Lee∗†, Honglak Lee∗‡
LG AI Research∗, University of Illinois at Chicago†, University of Michigan‡
Correspondence to<EMAIL_ADDRESS>
###### Abstract

Pre-trained large language models have shown successful progress in many
language understanding benchmarks. This work explores the capability of these
models to predict actionable plans in real-world environments. Given a text
instruction, we show that language priors encoded in pre-trained language
models allow us to infer fine-grained subgoal sequences. In contrast to recent
methods which make strong assumptions about subgoal supervision, our
experiments show that language models can infer detailed subgoal sequences
from few training sequences without any fine-tuning. We further propose a
simple strategy to re-rank language model predictions based on interaction and
feedback from the environment. Combined with pre-trained navigation and visual
reasoning components, our approach demonstrates competitive performance on
subgoal prediction and task completion in the ALFRED benchmark compared to
prior methods that assume more subgoal supervision.

##  1 Introduction

Developing autonomous agents that can complete specific tasks given goal
descriptions embodies human-level intelligence. Successful agents in this
setting require multiple reasoning capabilities including natural language
understanding, visual reasoning, and acting over long temporal horizons.
Training black-box models that map instructions and observations to suitable
actions has proven to be difficult due to challenges in interpreting and
reasoning with multimodal information, especially in the absence of strong
supervision. Thus, generalization in this setting demands effective strategies
for planning, exploration, and incorporating feedback from the environment.

Figure 1:  Overview of subgoal prediction approach. (a) A pre-trained language
model prompted with a sequence of training instances, i.e., (instruction,
subgoal sequence) pairs, and a query instruction predicts top-k hypotheses
using beam search. (b) These predictions are then re-ranked by incorporating
information about the environment.

Generalization in human agents, on the other hand, stems from our ability to
naturally brainstorm abstract subgoals, better calibrating executable actions
and their sequences. Planning at the level of subgoals instead of low-level
actions allows us to better adapt to unfamiliar settings. We posit that
language supervision can help realize such planning capabilities effectively
in artificial agents. First, text is a natural API for interacting with
intelligent agents that act in the real world to complete tasks. Knowledge
available in the form of text corpora, descriptions and instructions can be
exploited to build better agents (Branavan et al., 2012; Zhong et al., 2020).
Second, strong language priors are useful to reason about causal sequences of
events (Li et al., 2021). Language priors can further inform about object
affordances (e.g. an apple is sliceable, whereas a table is not) and other
contextual knowledge (e.g. a slicing task is more likely to be performed in a
kitchen than a bathroom) (Chen et al., 2020). Recent advances have
demonstrated that large language models are able to capture such priors, as
evidenced by their strong capabilities in language understanding and beyond
(Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Bommasani et
al., 2021). This leads to the natural question of whether priors learned by
language models can help reason about subgoals.

We study the ability of language models to reason about plans composed of a
sequence of intermediate goals for completing basic object manipulation tasks
in a household environment specified using text instructions. In particular,
we use the in-context learning ability (Brown et al., 2020) of large pre-
trained language models to reason about subgoals. In contrast to prior methods
that fine-tune language models to predict subgoals/actions (Jansen, 2020; Yao
et al., 2020), we show that they can predict subgoal sequences effectively
without any fine-tuning. We teach the model how instructions translate into
subgoal sequences by constructing a prompt using few examples. Given the
prompt and a query instruction the language model predicts likely subgoal
sequences (see Figure 1 for an illustration).

While language models are capable of generating strong hypotheses, we observe
that these predictions may not be directly usable by agents acting in real
environments. First, they suffer from calibration issues: Language models have
a tendency to repeat content from the prompt (Zhao et al., 2021). We show that
mutual-information inspired metrics help mitigate calibration issues and lead
to better ranking of model generated hypotheses.

Second, real-world agents have to update their beliefs and predictions based
on interaction and feedback from the environment. Without such feedback we
cannot expect the predicted plan to be executable in the environment. We
execute plans proposed by the language model in the environment using a pre-
trained low-level policy and collect feedback about task success/failure. We
use this feedback as a learning signal to train a ranking model that re-ranks
language model predictions. In contrast to prior methods that rely on strong
subgoal supervision and task level expert trajectories, we show that combining
subgoal predictions with a pre-trained subgoal execution policy leads to a
strong embodied agent baseline.

We make the following contributions in this work. We show that

  * •

Large language models can predict subgoals from text instructions with very
little supervision using in-context learning.

  * •

Incorporating a small amount of feedback from interaction with the environment
such as agent state and task success/failure outcome improves language model
predictions.

  * •

Combining predicted subgoals with a pre-trained low-level policy for
navigation and visual reasoning leads to a simple modular agent policy that
performs well on an embodied learning setting.

##  2 Related work

#### Language models for planning and interaction

The use of language models for planning and action prediction has been
explored in prior work. Jansen (2020) fine-tuned a language model to predict
subgoal sequences for text instructions from the ALFRED benchmark. Micheli and
Fleuret (2021) take a similar approach, but show that imitation learning with
few instances combined with reinforcement learning produces models that work
well on the ALFWorld benchmark (Shridhar et al., 2021). Yao et al. (2020)
demonstrate a similar approach for interactive fiction games (Hausknecht et
al., 2020). In contrast to these prior methods, our approach does not assume
strong supervision and we demonstrate generalization with limited training
examples. Furthermore, in order to exploit the generalization capabilities of
large language models, we do not fine-tune these models and instead use their
in-context learning ability. Finally, our approach allows us to build policies
that inherit the strong generalization capabilities of these large pre-trained
models such as compositional generalization.

#### Large language models and few-shot learning

Brown et al. (2020) showed that pre-trained large language models have few-
shot learning capabilities. Given a few examples
$\\{(x_{i},y_{i}=f(x_{i}))\\}$ that define a task $f$ such as classification
or translation and a query instance $x^{q}$, prompting a language model with a
string such as "$x_{1}=y_{1};x_{2}=y_{2};...;x_{n}=y_{n};x^{q}=$" leads to
meaningful completions by the language model $y^{q}\approx f(x^{q})$. This
few-shot learning capability of language models has since then been studied
and improved upon with approaches like prefix engineering Schick and Schütze
(2021), prompt tuning (Li and Liang, 2021), model calibration Zhao et al.
(2021) and other methods Min et al. (2021a). We adopt a similar approach for
few-shot subgoal inference. We assume that subgoal supervision is available
for a small number of training tasks and use the language model to infer
subgoals for unseen tasks.

#### Instruction following

There is rich literature on agents that follow language instructions (Branavan
et al. (2009); Mei et al. (2016); Fried et al. (2018); Suhr et al. (2019)
inter alia111See Luketina et al. (2019) for a comprehensive survey). Recent
developments in simulated environments and benchmarks with human annotated
instructions have driven progress in embodied agents that learn from text
instructions (Shridhar et al., 2020; Kolve et al., 2017). Successful agents in
these settings require multiple reasoning capabilities including language
understanding, visual reasoning and learning to act over long time-horizons.
Recent embodied learning literature exploit subgoal supervision, pre-trained
visual reasoning components and pre-trained transformer models to do well on
the task (Singh et al., 2020; Suglia et al., 2021; Zhang and Chai, 2021;
Corona et al., 2021; Blukis et al., 2022). Unlike these methods, we do not
assume access to strong subgoal supervision or task level expert supervision.
We combine language model predictions with pre-trained low-level navigation
and interaction policies to obtain a competitive agent policy.

#### Few-shot semantic parsing

Subgoal inference from text instructions can be considered a semantic parsing
problem where the subgoal sequences serves as a formal representation of text.
Shin et al. (2021) show that few-shot semantic parsers can be derived from
language models and demonstrate their applicability on text-to-SQL (Finegan-
Dollak et al., 2018) and SCAN (Lake and Baroni, 2018) benchmarks. Furrer et
al. (2020) and Herzig et al. (2021) further study the compositional
generalization ability of such semantic parsers. In our work we make use of
ideas introduced in these works such as dynamic prompt creation, constrained
decoding and intermediate representations.

##  3 Approach

We first consider subgoal inference as a semantic parsing problem where a text
instruction needs to be translated to a sequence of subgoals and propose an
approach to few-shot subgoal inference based on pre-trained language models in
Section 3.1. We extend this setting to an agent acting in a simulated
environment which can execute these subgoals, observe feedback, and improve
upon language model predictions for more accurate subgoal inference in Section
3.2.

###  3.1 Few-shot subgoal inference

#### Subgoals

We are interested in a particular sub-class of instruction following problems
which involve performing a sequence of object interactions in an embodied
environment. Each object interaction requires navigating to a particular
object and performing an action on it. A task is considered successfully
completed if the state of objects in the end satisfy a set of task-specific
constraints (for instance, objects that need to be
sliced/warmed/cooled/cleaned have the appropriate state change). It is thus
natural to define a subgoal as one or more sequence of object interactions. A
subgoal $g$ is specified as $g=(b,o)\in\mathcal{B}\times\mathcal{O}$ where
$b\in\mathcal{B}=$ {Pickup, Clean, Heat, ..} is one of a pre-defined set of
abstract actions and $o\in\mathcal{O}=${Apple, Microwave, DeskLamp, Ottoman,
..} is an object category.

#### Subgoal inference problem

Given a text instruction $\tau$, subgoal inference seeks to predict a sequence
of subgoals $\tau\mapsto g=(g^{(1)},...,g^{(n)})$. To perform in-context
learning with a language model, we consider a representation $v(g)$ of $g$
that looks like natural text, where $v$ is a pre-defined invertible mapping.
Such representations have been referred to in the literature as verbalizers
(Min et al., 2021a), intermediate representations (Herzig et al., 2021) and
canonical representations (Shin et al., 2021), where the purpose is to
represent the output in a format the language model understands. In a slight
abuse of notation, we will use $g$ to refer to either a subgoal sequence or
it’s textual representation $v(g)$ depending on the context.

Figure 2:  Re-ranking language model predictions with interaction and feedback
from the environment. Given the task, language model predictions, completed
subgoals and the agent state we come up with a ranked list of subgoal
sequences. The agent then executes the next subgoal from the highest ranked
plan. The completed subgoal is added to the partial plan and the process
continues until stop subgoal is encountered. During training the agent
receives a positive reward if the task is successfully completed, which we use
as supervision to train the ranking model.

#### Generating subgoals

We assume that a small amount of training data
$\\{(\tau_{1},g_{1}),\cdots,(\tau_{n},g_{n})\\}$ is given. The language model
is prompted with a comma separated concatenation of the training examples,
each in the format "$\tau_{i}=g_{i}$", followed by a query $\tau$, formatted
as "$\tau=$". We assume that the probability of a hypothesis $h$ (i.e., text
representaton of a subgoal sequence) can be modeled as in Equation 1, where
$h_{i}$ is the $i^{\text{th}}$ token of $h$ and the token probabilities are
derived from the language model.

$p(h|\tau)=\prod_{i}p_{\text{LM}}(h_{i}|h_{<i},\tau,\\{\tau_{j},g_{j}\\}_{j=1}^{n})$
(1)

We use beam search to identify the top-k hypotheses according to $p(h|\tau)$.
Generated hypotheses are constrained to be valid representations of subgoal
sequences by considering only tokens which lead to a valid partial prefix of a
subgoal sequence at each step of beam search.

#### Re-ranking predictions

Recent studies have found that language models have popularity and recency
biases: the tendency to repeat content mentioned in the prompt, especially
content appearing later in the prompt (Zhao et al., 2021). They considered a
simple approach to mitigate such biases in model predictions for
classification tasks by comparing the likelihood of an output label with and
without the query. In contrast to this ‘direct model’ which models the
probability of a label given the input $p(y|x)$, Min et al. (2021a) showed
that a ‘channel model’ which models $p(x|y)$ leads to better, more stable
models.

Inspired by these observations, we propose to use $p(\tau|h)$ to score
hypotheses in addition to $p(h|\tau)$. Mutual Information based ranking
metrics are a natural candidate and they have been explored in the text
generation literature (Li et al., 2016; Li and Jurafsky, 2016). We generate
multiple hypotheses from the model using $p(h|\tau)$ and the generated
hypotheses are re-scored using the weighted mutual information metric
$(1-\lambda)\text{log }p(h|\tau)+\lambda\text{log }p(\tau|h)$ where $\lambda$
is a hyperparameter222Appendix A details the connection to Mutual
Information.. To compute $p(\tau|h)$, we again use the language model prompted
with "$g_{1}=\tau_{1},...,g_{n}=\tau_{n},h=$" as the query and compute the
conditional probability of $\tau$. We expect this paradigm of generating a set
of strong hypotheses, followed by accurate re-ranking is more generally
applicable to other few-shot language understanding problems.

###  3.2 Agent policy and incorporating environment feedback

We next consider building an agent that acts in a visual environment to
complete tasks given text instructions. While Section 3.1 treated the language
model as a knowledge extraction system, in the real world plans need to be
updated based on interaction and feedback from the environment. We thus
propose a method to improve language model predictions based on environment
interaction. Since our goal is to learn the planning component of the agent,
we assume a pre-trained low-level policy is provided and optimize over the
space of plans. Jointly learning both components is beyond the scope of this
work and left as future work.

We assume that a representation of the agent state $s$ is available. The state
representation captures information about the environment (e.g. objects
present and their locations) estimated based on the agent’s visual
observations. As the agent explores the environment and collects new
observations the state representation is updated. Assuming that a low-level
policy $\pi_{L}$ pre-trained to execute a given subgoal is provided, our goal
is to train a high-level policy $\pi_{H}$ which proposes the subgoals to be
executed by the low-level policy. More formally, the high-level policy models
$\pi_{H}(g^{(t)}|\tau,s_{t},g^{(<t)})$ where $\tau$ is a text instruction,
$s_{t}$ is the state representation and $g^{(<t)}$ is the sequence of subgoals
completed so far at high-level time-step $t$333Alternatively, this can be
framed as a POMDP in a hierarchical reinforcement learning setting..

While the language model can generate compelling subgoal hypotheses, it
doesn’t take into account information about the environment. For instance,
knowledge about the type of room the agent is in (kitchen, bathroom, etc.) and
the objects present in it are useful to infer the kind of tasks and subgoals
that can be performed. We propose to re-rank hypotheses generated by the
language model based on information from the environment to construct
$\pi_{H}$. The plans generated by the language model are executed in the
environment using $\pi_{L}$. The success/failure outcomes of these plan
executions are used to construct a labeled dataset of instructions $\tau$,
plans $g$ and agent state $s$. A supervised ranking model $f(g,\tau,s;\theta)$
is trained using this data to re-rank the language model predictions. We
represent the ranking model as
$f(g,\tau,s;\theta)=\theta^{T}\text{concat}(f^{\text{state}}(s),f^{\text{text}}(\tau,g))$
where $f^{\text{state}}(s)$ is a state embedding, $f^{\text{text}}(\tau,g)$ is
a joint encoding of $\tau$ and $g$ produced by a text encoder and $\theta$ is
a parameter vector. Although a text embedding can be derived from the language
model, we use a BERT encoder in favor of obtaining a smaller dimensional
representation ($f^{\text{text}}=\text{BERT}_{\text{CLS}}$). See Appendix C
for more details.

During inference, an instruction $\tau$ is given, and we use the procedure in
Section 3.1 to generate top-k hypotheses. At each step, hypotheses
inconsistent with the sequence of subgoals executed so far are pruned and the
remaining hypotheses are re-ranked based on the current agent state using $f$.
The agent attempts the next subgoal proposed by the top hypothesis. The
process ends when the stop subgoal is predicted. See Figure 2 for an
illustration and Appendix D for more details about the training and inference
algorithms.

##  4 Experiments

#### Data

We use data from the ALFRED benchmark proposed by Shridhar et al. (2020) in
our experiments. The ALFRED task requires an agent to execute instructions
specified in text to accomplish basic tasks in an embodied environment. A
given task is described using a high-level language directive as well as low-
level step-by-step instructions (We only use the high-level description). The
dataset consists of 7 task types (and 11 fine-grained types), and has more
than 20k natural language task descriptions collected from human annotators.
In addition, expert demonstrations computed by a planner are also made
available. Tasks require acting over many time-steps, with an average of 50
actions, and the longest tasks require 100+ steps.

The ground truth subgoal sequences in the dataset consist of both navigation
subgoals and object interaction subgoals. We discard the navigation subgoals
and only retain the interaction subgoals for the following reasons. First, the
interaction subgoals are sufficient for an agent to successfully complete the
task. Second, predicting navigation subgoals from the text instruction alone
may not always be possible as they often depend on the scene layout.

#### Subgoal representation

A subgoal $g^{s}$ is specified as $g^{s}=(b,o)\in\mathcal{B}\times\mathcal{O}$
where $|\mathcal{B}|=7$ and $|\mathcal{O}|=80$. We define a textual
representation $v(b)$ of each action type (e.g. $v$(Pickup) $=$ ‘pick up’,
$v$(Heat) $=$ ‘heat in’). The object types $o$ are identified by a text string
$v(o)$ in the dataset and we directly use them as the text representation with
minimal pre-processing (e.g. $v$(apple) $=$ ‘apple’, $v$(desklamp) $=$ ‘desk
lamp’). The subgoal is represented as $v(g^{s})=$‘$v(b)$ $v(o)$’ (e.g.
$v$((Pickup, apple)) $=$ ‘pick up apple’). A subgoal sequence
$g=(g^{(1)},...,g^{(n)})$ is represented as $v(g)=$‘$v(g^{(1)})$, …,
$v(g^{(n)})$.’. Text representations of all subgoals are given in Appendix B.
Note that there are many plausible choices for the representation $v$ and a
different set of choices can lead to different results.

#### Metrics

We use top-k recall to evaluate the ability of language models to generate
plans from instructions by comparing against ground truth plans. In addition,
we also evaluate the performance of an agent acting in the AI2-Thor (Kolve et
al., 2017) simulator to complete tasks using task success rate (the percentage
of tasks successfully completed).

Top-$k$ recall | Ranking criteria | GPT2-XL | GPT-J
---|---|---|---
$N=11$ | $N=22$ | $N=33$ | $N=11$ | $N=22$ | $N=33$
[ column count=8, no head, table head= |  |  |  |  |  |
, late after line= |  |  |  |  |  |  |
|  |  |  |  |  |  |

Table 1:  Re-ranking model generated hypotheses using different criteria. The
first section shows top-10 recall of generated hypotheses (using $p(h|\tau)$).
The second section shows top-1 recall after re-ranking these hypotheses using
different criteria. Results are shown for GPT2-XL and GPT-J models when the
number of training instances $N$ is varied.

Task | GPT2-XL | GPT-J
---|---|---
top-1 | top-10 | top-1 | top-10
[ column count=5, no head, table head= |  |  |  |
, late after line= |  |  |  |
|  |  |  |

Table 2: Top-k recall for subgoal sequences predicted by GPT2-XL and GPT-J
models categorized by task type.

###  4.1 Few-shot subgoal inference

We construct a training set of $N=22$ instances by randomly choosing two
instances per fine-grained task type. The language model is prompted with a
concatenation of these training examples and the query instance. We perform
constrained beam search decoding with a beam size of 10 to generate subgoal
sequences. At each step of beam search, only tokens which lead to a valid
partial prefix of a subgoal sequence are considered. All model generated
hypotheses thus correspond to valid subgoal sequences. We evaluate models on
the valid-seen split of the dataset which has 800 instances.

Section 4 shows subgoal inference results categorized by task type. We use
publicly available pre-trained transformer language models GPT2-XL (Radford et
al., 2019) and GPT-J (Wang and Komatsuzaki, 2021) via the HuggingFace library
(Wolf et al., 2020), which respectively have 1.5B and 6B parameters, in our
experiments. The first six of the seven task types have two object arguments
each. The pick place movable task type has three object arguments and hence a
lower recall than the other task types. The top-10 recall of GPT2-XL and GPT-J
are respectively 59% and 71%, which shows that large language models have
strong ability to reason about plans from few training examples.

#### Re-ranking hypotheses

The top-k recall performance reported in Section 4 is based on log
$p(h|\tau)$. We confirmed that the biases reported in the literature such as
predicting content from the prompt are present in model predictions (Zhao et
al., 2021). Consider the query example Place a martini glass with a fork on it
on the table. The following two are among the top generated hypotheses:

a) pick up fork, put in cup, pick up cup, put in sink.

b) pick up fork, put in cup, pick up cup, put in table.

When the prompt contains training examples that mention ‘sink’, the model
assigns the following log $p(h|\tau)$ to these hypotheses: a) -2.4 and b)
-4.3. However, when all training instances in the prompt involving ‘sink’ are
removed, the log probabilities now become, a) -13.7 and b) -9.1 The incorrect
hypotheses involving ‘sink’ is now ranked below the correct hypothesis
involving table. While language models can retrieve strong hypotheses as
indicated by the high top-10 recall, this observation shows that the ranking
of these hypotheses, as determined by $p(h|\tau)$, may not be accurate. We
thus consider mutual information based ranking approaches. Section 4 shows
top-1 recall when model generated hypotheses are ranked according to different
criteria. We also vary the number of training examples $N$ by randomly
choosing respectively 1, 2 and 3 instances per fine-grained task type.

We first observe that $p(\tau|h)$ ranks hypotheses better than $p(h|\tau)$
with very limited supervision ($N=11$). However, it is often worse when more
supervision is available. In contrast, combining the two log probabilities
with $\lambda=\frac{1}{2}$ yields consistently better performance across
models and number of training examples. This shows that generating a large
number of hypotheses with a language model, followed by more accurate re-
ranking using Mutual Information inspired metrics can be an effective paradigm
for few-shot generation tasks with in-context learning.

#### Comparison with prior work

We compare our prediction performance against prior work in Section 4.1.
Jansen (2020) fine-tunes a GPT2-Medium model (325M parameters) to predict
subgoals from instructions and report prediction
results444https://github.com/cognitiveailab/alfred-gpt2 when the model is
trained on varying amounts of training data: 1%, 10%, 25%, 100% of the
training set, which has 7793 instances. We ignore the navigation subgoals in
this evaluation and only compare the sequence of object interactions. We
report prediction performance of GPT-J using our approach on the same test
set. The results show that large language models encode useful knowledge that
can help plan from instructions effectively when supervision is limited.
However, fine-tuning can be effective when more supervision is available due
to the fixed context length limitation of in-context learning. See Section 5
for ablations and more discussion about fine-tuning.

Model | Top-1 | Training
---|---|---
| recall | instances
Jansen (2020) (Fine-tuned GPT2-Medium) |  05.07 |  0077
41.92 |  0779
53.80 | 1948
61.00 | 7793
1-3  Ours (In-context GPT-J) | 44.02 |  0011
47.80 |  0022
49.63 |  0033

Table 3:  Comparison against subgoal prediction performance of Jansen (2020).

#### Prediction errors

We examine prediction errors in identifying task type and object type. Key
sources of model errors include annotation issues and ambiguity in object
types. Table 4 shows the object types that have the least prediction accuracy,
along with the object categories the model is confused about. Annotations can
fail to correctly identify the target object - identifying a _butter knife_ as
a _knife_ or a _pepper shaker_ as _salt shaker_. Ambiguity can also arise from
identifying an object with different names, depending on the context. For
instance, depending on the scene layout, the argument for a look at object in
light task can be a floor lamp or a desk lamp. Unless the type of lamp is
identified precisely in the instruction, it is not possible to correctly
predict the type of lamp. Correctly identifying these objects requires
feedback from interaction with the environment.

The experiments so far evaluate the ability of a language model to retrieve
ground truth subgoal sequences. Next we examine embodied agents that make use
of these predictions and collect more supervision in order to improve subgoal
predictions.

###  4.2 Agent policy and incorporating environment feedback

We now use subgoal predictions to construct an agent policy that acts in a
simulated environment to complete tasks. The agent state representation and
pre-trained low-level subgoal policy are borrowed from the HLSM model proposed
in Blukis et al. (2022). HLSM represents the agent state as a spatial
persistent voxel representation of the room which models the location and
category of objects present. The representation is constructed using modules
that estimate segmentation and depth maps and other visual reasoning
components and is updated as the agent gathers new observations. We use pre-
trained models made available by the authors555https://hlsm-alfred.github.io
for state estimation and the low-level policy in our experiments.

We combine subgoal predictions with the pre-trained HLSM low-level policy and
evaluate the overall agent policy on the ALFRED task in Table 5. Unlike the
results reported in Section 4.1 which were based on the static dataset, these
results are based on subgoals executed against the AI2-Thor simulator. In
addition to task success rate, we also report the percentage of goal
conditions satisfied, which rewards the model for partial task completions.

Obj category | Confusion categories
---|---
wateringcan | pencil, kettle
glassbottle | vase
cart | shelf, sidetable, microwave, cart, fridge
butterknife | knife, butterknife
floorlamp | desklamp, floorlamp
vase | pencil, vase, bowl, winebottle, pot
ladle | spoon, ladle
pot | pot, pan
soapbottle | soapbottle, winebottle
Table 4:  Object categories the model makes most errors on and the top object
categories it confuses with.

We compare against the following baselines on the ALFRED task. Seq2seq
(Shridhar et al., 2020) is a simple sequence-to-sequence baseline trained to
map text instructions to low-level actions. MOCA (Singh et al., 2020) improves
on Seq2seq using subgoal supervision and pre-trained visual reasoning
components. Recent work such as HLSM (Blukis et al., 2022) and FiLM (Min et
al., 2021b) build and use spatial semantic state representations and achieve
stronger performance on the task. Note that, unlike these prior methods (MOCA,
HLSM, FiLM) that rely on full subgoal supervision (20k instances), our
approach is based on a small amount of subgoal supervision and additional
supervision collected using active interaction with the environment. In
addition, our approach does not require task-level expert trajectories and
only assumes that a subgoal execution policy is provided.

Using the top language model prediction as is without using any information
from the environment leads to 20% success rate. Next, we collect plan
execution feedback for 1000 text instructions to train the ranking model
described in Section 3.2. Re-ranking language model predictions using the
trained ranking model improves the performance to 24%, which shows the
importance of incorporating feedback from environment interaction. In
comparison, the HLSM model with full subgoal supervision has success rate 30%.
Although our predictions fall short of HLSM, they are competitive with the
other baselines with subgoal supervision. The performance upper bound
estimated using oracle subgoals is 37%, which shows the room for improvement
over our predictions. These results show that accurate subgoal inference
coupled with pre-trained low-level components leads to agents that perform
well in embodied environments.

Figure 1 shows an example where the ranking model uses environment information
to identify better plans. In this example, the instruction ambiguously
specifies the receptacle as ‘white table with shelving’. The language model’s
top two predictions for the target receptacle are ‘side table’ and ‘shelf’,
neither of which are present in the environment. The agent state captures this
information and helps identify ‘dining table’ as the correct receptacle.

Model | Success rate
---|---
Task | Goal-Cond
Seq2seq (Shridhar et al., 2020) | 3.7 | 10.0
MOCA (Singh et al., 2020) | 19.2 | 28.5
FiLM (Min et al., 2021b) | 24.6 | 37.2
HLSM (Blukis et al., 2022) | 29.6 | 38.8
HLSM [-1pt] low-level [-1pt] policy | Predicted subgoals | 19.8 | 31.4
Re-ranked subgoals | 23.9 | 35.0
Oracle subgoals | 37.2 | 48.2
Table 5:  Task completion and goal condition success rates of models on the
ALFRED validation seen split (results are based on task executions in the
AI2-Thor simulator). The performance of our subgoal predictions combined with
the HLSM low-level policy are shown at the bottom. We show the performance
before and after re-ranking language model predictions based on agent state.
Oracle subgoals shows the performance upper bound.

##  5 Ablations

We perform a series of ablations to identify the robustness of model
predictions. We compare the performance of in-context learning to a sequence-
to-sequence model fine-tuned to translate instructions to subgoal sequences.
In addition, we observe the effect of varying the number of training examples
and choice of training examples.

$11$$22$$33$$44$$0$$20$$40$$60$$80$Number of training examplesRecallGPT-J in-
context (top-1)T5-large finetune (top-1)GPT-J in-context (top-10)T5-large
finetune (top-10)

Figure 3:  Comparison between GPT-J with in-context learning and a fine-tuned
T5-large model for varying number of training examples. See text for details.

#### Number of training examples

Figure 3 shows model recall for varying number of training examples. For zero
training examples, the prompt consists of just the query instruction and the
model decodes subgoals. Beyond 44 examples (4 examples per task type), the
model prompt no longer fits the sequence length restriction (1024 tokens) of
GPT models. A steady increase in performance can be initially observed when
increasing the number of training examples and the performance saturates
towards the end. In-context learning further has the limitation of not being
able to accommodate a larger number of training examples due to the length
restriction. It would be interesting to explore how to make effective use of
large number of training examples in future work.

#### Choice of training examples

We also estimate performance variance by varying the random seed for choosing
examples randomly from the training set and compute standard deviation based
on five random seeds for each setting. The plot shows that top-1 predictions
from in-context learning have lower variance compared to fine-tuning.

#### Comparison with fine-tuning

In order to understand how well the in-context learning approach compares to
fine-tuned models, we fine-tune a T5-large model (Raffel et al., 2019) with
770M parameters on varying amounts of training data (this was the largest
model we could fine-tune on our compute infrastructure). Note that this is not
a head-to-head comparison between in-context learning and fine-tuning due to
the difference in model size. Furthermore, there are other fine-tuning
mechanisms such as prompt tuning and head tuning (Min et al., 2021a) which are
not considered here. However, the result suggests that in-context learning
with large pre-trained models can be favorable when computational constraints
do not allow full fine-tuning of large models.

These ablations show that the in-context learning ability of large language
models leads to predictions that are accurate, robust and stable in the
presence of a small amount of training data.

##  6 Conclusion

This work explores the use of pre-trained language models for planning in
real-world tasks. We showed that language models have strong capability to
reason about subgoal sequences given a small number of training examples. We
further demonstrated some simple mechanisms to incorporate feedback from
interaction with the environment and show that this leads to more usable
predictions. Finally, we show that combining subgoal predictions with a pre-
trained low-level policy yields a strong baseline for embodied agent learning.

Our ablations demonstrate that in-context learning with a small amount of
subgoal demonstrations has robust generalization properties. However, we also
point out that in-context learning has the limitation of not being able to
incorporate a large number of training examples due to the fixed context
length restriction. It would further be beneficial to perform end-to-end
learning with language model based subgoal prediction and a low-level policy,
which would be interesting to explore in future work.

## References

  * Blukis et al. (2022) Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022.  A persistent spatial semantic representation for high-level natural language instruction execution.  In _Conference on Robot Learning_ , pages 706–717. PMLR.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Branavan et al. (2009) S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009.  Reinforcement learning for mapping instructions to actions.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 82–90, Suntec, Singapore. Association for Computational Linguistics.
  * Branavan et al. (2012) SRK Branavan, David Silver, and Regina Barzilay. 2012.  Learning to win by reading manuals in a monte-carlo framework.  _Journal of Artificial Intelligence Research_ , 43:661–704.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901.
  * Chen et al. (2020) Haonan Chen, Hao Tan, Alan Kuntz, Mohit Bansal, and Ron Alterovitz. 2020.  Enabling robots to understand incomplete natural language instructions using commonsense reasoning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1963–1969. IEEE.
  * Corona et al. (2021) Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. 2021\.  Modular networks for compositional instruction following.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1033–1040, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Finegan-Dollak et al. (2018) Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018.  Improving text-to-SQL evaluation methodology.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 351–360, Melbourne, Australia. Association for Computational Linguistics.
  * Fried et al. (2018) Daniel Fried, Jacob Andreas, and Dan Klein. 2018.  Unified pragmatic models for generating and following instructions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics.
  * Furrer et al. (2020) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020.  Compositional generalization in semantic parsing: Pre-training vs. specialized architectures.  _arXiv preprint arXiv:2007.08970_.
  * Hausknecht et al. (2020) Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2020.  Interactive fiction games: A colossal adventure.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7903–7910.
  * Herzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021.  Unlocking compositional generalization in pre-trained models using intermediate representations.  _arXiv preprint arXiv:2104.07478_.
  * Jansen (2020) Peter Jansen. 2020.  Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4412–4417, Online. Association for Computational Linguistics.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International conference on machine learning_ , pages 2873–2882. PMLR.
  * Li et al. (2021) Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021.  Implicit representations of meaning in neural language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1813–1827, Online. Association for Computational Linguistics.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016.  A diversity-promoting objective function for neural conversation models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California. Association for Computational Linguistics.
  * Li and Jurafsky (2016) Jiwei Li and Dan Jurafsky. 2016.  Mutual information and diverse decoding improve neural machine translation.  _arXiv preprint arXiv:1601.00372_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Luketina et al. (2019) Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. 2019.  A survey of reinforcement learning informed by natural language.  _arXiv preprint arXiv:1906.03926_.
  * Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016.  Listen, attend, and walk: Neural mapping of navigational instructions to action sequences.  In _Thirtieth AAAI Conference on Artificial Intelligence_.
  * Micheli and Fleuret (2021) Vincent Micheli and Francois Fleuret. 2021.  Language models are few-shot butlers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9312–9318, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Min et al. (2021a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021a.  Noisy channel language model prompting for few-shot text classification.  _arXiv preprint arXiv:2108.04106_.
  * Min et al. (2021b) So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021b.  Film: Following instructions in language with modular methods.  _arXiv preprint arXiv:2110.07342_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Shin et al. (2021) Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7699–7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020.  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10740–10749.
  * Shridhar et al. (2021) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021.  ALFWorld: Aligning Text and Embodied Environments for Interactive Learning.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Singh et al. (2020) Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. 2020.  Moca: A modular object-centric approach for interactive instruction following.  _arXiv preprint arXiv:2012.03208_.
  * Suglia et al. (2021) Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. 2021.  Embodied bert: A transformer model for embodied, language-guided visual task completion.  _arXiv preprint arXiv:2108.04927_.
  * Suhr et al. (2019) Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019.  Executing instructions in situated collaborative interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2119–2130, Hong Kong, China. Association for Computational Linguistics.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yao et al. (2020) Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020.  Keep CALM and explore: Language models for action generation in text-based games.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8736–8754, Online. Association for Computational Linguistics.
  * Zhang and Chai (2021) Yichi Zhang and Joyce Chai. 2021.  Hierarchical task learning from language instructions with unified transformers and self-monitoring.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4202–4213, Online. Association for Computational Linguistics.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR.
  * Zhong et al. (2020) Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. 2020.  Rtfm: Generalising to new environment dynamics via reading.  In _International Conference on Learning Representations_.

##  Appendix A Mutual Information based scoring

Mutual Information between random variables $X,Y$ is defined as in Equation 2.
We consider a weighted Mutual Information metric as defined as in Equation 3
similar to Li and Jurafsky (2016) and introduce the hyperparameter $\lambda$.
Identifying $Y$ that maximizes the weighted Mutual Information is equivalent
to maximizing the expression in Equation 4. We use this metric to rank
hypotheses generated by the language model.

$\displaystyle\text{MI}(X,Y)$ $\displaystyle=\text{log
}\frac{p(x,y)}{p(x)p(y)}$ (2) $\displaystyle\text{wMI}(X,Y)$
$\displaystyle=\text{log }\frac{p(x,y)}{p(x)p(y)^{\lambda}}$ (3)

$\displaystyle\operatorname*{argmax}_{y}\text{wMI}(X,Y)$
$\displaystyle=\operatorname*{argmax}_{y}\text{log
}\frac{p(x,y)}{p(x)p(y)^{\lambda}}$
$\displaystyle=\operatorname*{argmax}_{y}\text{log
}\left(\frac{p(x,y)}{p(x)}\right)^{1-\lambda}\left(\frac{p(x,y)}{p(y)}\right)^{\lambda}\frac{1}{p(x)^{\lambda}}$
$\displaystyle=\operatorname*{argmax}_{y}(1-\lambda)\text{log
}p(y|x)+\lambda\text{log }p(x|y)$ $\displaystyle\hskip
150.00023pt-\lambda\text{log }p(x)$
$\displaystyle=\operatorname*{argmax}_{y}(1-\lambda)\text{log
}p(y|x)+\lambda\text{log }p(x|y)$ (4)

##  Appendix B Subgoal representation

Table 6 shows the subgoal representation we use in this work.

Subgoal | Representation
---|---
(Pickup, X) | pick up X
(Put, X) | put in X
(Heat, X) | heat in X
(Cool, X) | cool in X
(Clean, X) | clean in X
(Slice, X) | slice X
(ToggleOn, X) | turn on X
Table 6:  Subgoals and corresponding text representation. X represents an
object argument.

##  Appendix C Ranking model: Architecture

#### State embedding

HLSM represents the agent state as a semantic voxel representation
$s\in[0,1]^{X\times Y\times Z\times C}$ where the value $s(x,y,z,c)$
represents if there is an object of type $c$ at position $(x,y,z)$ of the room
layout. We pool across the spatial dimensions of the representation and
project it using a linear mapping to obtain $f^{\text{state}}(s)$. We use this
encoding as the state embedding.

#### Instruction and subgoal sequence encoding

The instruction $\tau$ and a candidate subgoal sequence $g$ are jointly
processed using a BERT encoder and the CLS representation is used as a
representation vector $\text{BERT}_{\text{CLS}}(\tau,g)$. The ranking model is
represented as
$f(g,\tau,s;\theta)=\theta^{T}\text{concat}(f^{\text{state}}(s),\text{BERT}_{\text{CLS}}(\tau,g))$
where $\theta$ is a parameter vector.

##  Appendix D Ranking Model: Training and Inference

Formally, the learning problem is a MDP
$(\mathcal{S},\mathcal{G},\mathcal{L},\mathcal{R},\mathcal{T})$, where
$s_{t}\in\mathcal{S}$ is the agent state, $g^{(t)}\in\mathcal{G}$ is a
subgoal, $\tau\in\mathcal{L}$ is a text instruction, $\mathcal{R}(\tau,s_{t})$
is a reward function that provides success/failure feedback for completing a
given instruction, $\mathcal{T}:(s_{t},g^{(t)})\rightarrow s_{t+1}$ is a state
transition function where $s_{t+1}$ is computed by a low-level policy
$\pi_{L}$ pre-trained to execute a given subgoal $g$. Our goal is to train a
high-level policy $\pi_{H}(g^{(t)}|\tau,s_{t},g^{(<t)})$ where $s_{t}$ is the
agent state and $g^{(<t)}$ is the sequence of subgoals completed so far at
high-level time-step $t$.

Algorithm 1 describes how we collect training data to train the ranking model.
Algorithm 2 shows how the ranking model is used during inference.

Algorithm 1 Training

Given: $\text{epochs}=100$, $\mathcal{D}^{\text{ins}}$ (set of instructions)

Collect training data

$\mathcal{D}\leftarrow\\{\\}$ (Initialize training set)

for $\tau$ in $\mathcal{D}^{\text{ins}}$ do

Generate plans and re-rank using mutual information metric
$g_{1},\ldots,g_{k}\sim p_{\text{LM}}(\cdot|\tau)$

for $i=1\ldots k$ do

Initialize agent state $s$

$S\leftarrow\\{s\\}$ (Record agent states)

for $j=1\ldots|g_{i}|$ do

$s\leftarrow\mathcal{T}(s,g_{i}^{(j)})$ (Execute $g_{i}^{(j)}$ using
$\pi_{L}$)

$S\leftarrow S\cup\\{s\\}$

end for

if $\mathcal{R}(\tau,s)>0$ then (Task succeeded)

$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(g_{i},\tau,s)|s\in S\\}$

break

end if

end for

end for

Train model

for $i=1\ldots$ epochs do

loss $\leftarrow 0$

for $(g,\tau,s)\in\mathcal{D}$ do

Generate plans $g_{1},..,g_{k}\sim p_{\text{LM}}(\cdot|\tau)$

loss $\leftarrow$ loss $-$ log $\frac{\text{exp
}f(g,\tau,s;\theta)}{\sum_{i=1}^{k}\text{exp }f(g_{i},\tau,s;\theta)}$

end for

$\theta\leftarrow\text{Optimizer Update}(\theta,\nabla_{\theta}f)$

end for

return $f$

Algorithm 2 Inference

Given: Instruction $\tau$, $i^{\text{thresh}}=10$

Generate plans $g_{1},\ldots,g_{k}\sim p_{\text{LM}}(\cdot|\tau)$

$G\leftarrow\\{g_{1},\ldots,g_{k}\\}$

Initialize agent state $s$

$i\leftarrow 0$ (subgoal index)

$g\leftarrow\operatorname*{argmax}_{g\in G}f(g,\tau,s;\theta)$

while $|G|\neq 0$ and $g^{(i)}\neq$ <stop> and $i<i^{\text{thresh}}$ do

$s\leftarrow\mathcal{T}(s,g^{(i)})$ (Execute $g^{(i)}$ using $\pi_{L}$)

$G\leftarrow\\{h|h\in G\text{ and }h^{(i)}=g^{(i)}\\}$ (Retain plans
consistent with subgoals completed so far)

$g\leftarrow\operatorname*{argmax}_{g\in G}f(g,\tau,s;\theta)$

$i\leftarrow i+1$

end while

return $g,s$

##  Appendix E Analysis of model errors

Figure 4 shows the confusion matrix for object type prediction. Predictions
are from top-1 subgoal sequences predicted by GPT-J.

Figure 4: Object type prediction confusion matrix.


6 |  Whenever an ATTACH REJECT message with the EMM cause #14 "EPS services not allowed in this PLMN" is received by the UE, the chosen PLMN identity shall be stored in the "forbidden PLMNs for GPRS service" and if the UE is configured to use timer T3245 (see 3GPP S 24.368 [15A] or 3GPP S 31.102 [17]) then the UE shall start timer T3245 and proceed as described in clause 5.3.7a. |  Additionally, the UE shall delete the list of equivalent PLMNs and reset the attach attempt counter In S1 mode, the UE shall store the PLMN identity in the "forbidden PLMN list" and enter state EMM-DEREGISTERED.PLMN-SEARCH and if the UE is configured to use timer T3245 (see 3GPP S 24.368 [15] or 3GPP S 31.102 [17]) then the UE shall start timer T3245 and proceed as described in clause 5.3.7a. |  5.5.1.1 |  5.5.1.3.5
7 |  In addition, the UE shall include Old GUTI type IE with GUTI type set to "native GUTI". If there is no valid GUTI available, the UE shall include the IMSI in the ATTACH REQUEST message; or 2) If the UE supports A/Gb mode or Iu mode or both and i) if the TIN indicates "P-TMSI" and the UE holds a valid native P-TMSI and RAI, the UE shall map the P-TMSI and RAI into the EPS mobile identity IE, and include Old GUTI type IE with GUTI type set to "mapped GUTI". |  In addition, the UE shall include Old GUTI type IE with GUTI type set to "native GUTI"; or 2) if the UE supports A/Gb mode or Iu mode or both, the UE shall handle the Old GUTI IE as follows If the TIN indicates "P-TMSI" and the UE holds a valid native P-TMSI and RAI, the UE shall map the P-TMSI and RAI into the Old GUTI IE, and include Old GUTI type IE with GUTI type set to "mapped GUTI". |  5.5.1.2.2 |  5.5.3.2.2
Table 12: Subset of inconsistent pairs discovered from 4G (Continued) : security specification and : NAS specification Pair No. |  $\text{Text}_{1}$ |  $\text{Text}_{2}$ |  $\text{Loc}_{1}$ |  $\text{Loc}_{2}$
---|---|---|---|---
8 |  The UE shall consider the USIM as invalid for EPS services and non-EPS services until switching off or the UICC containing the USIM is removed or the timer T3245 expires as described in clause 5.3.7a. Additionally, the UE shall delete the list of equivalent PLMNs and enter state EMM-DEREGISTERED.NO-IMSI. If the message has been successfully integrity checked by the NAS and the UE maintains a counter for "SIM/USIM considered invalid for GPRS services", then the UE shall set this counter to UE implementation-specific maximum value. |  The UE shall consider the USIM as invalid for EPS services until switching off or the UICC containing the USIM is removed or the timer T3245 expires as described in clause 5.3.7a. The UE shall delete the list of equivalent PLMNs and shall enter the state EMM-DEREGISTERED.NO-IMSI. If the UE maintains a counter for "SIM/USIM considered invalid for GPRS services", then the UE shall set this counter to UE implementation-specific maximum value. |  5.5.1.2.5 |  5.5.2.3.2
9 |  Additionally, the UE shall delete the list of equivalent PLMNs and reset the attach attempt counter In S1 mode, the UE shall store the PLMN identity in the "forbidden PLMN list" and enter state EMM-DEREGISTERED.PLMN-SEARCH and if the UE is configured to use timer T3245 (see 3GPP S 24.368 [15] or 3GPP S 31.102 [17]) then the UE shall start timer T3245 and proceed as described in clause 5.3.7a. |  Furthermore, the UE shall delete any GUTI, last visited registered TAI, TAI  list and KSI. The UE shall reset the attach attempt counter and shall enter the state EMM-DEREGISTERED.PLMN-SEARCH The UE shall store the PLMN identity in the "forbidden PLMNs for GPRS service" list and if the UE is configured to use timer T3245 (see 3GPP S 24.368 [15A] or 3GPP S 31.102 [17]) then the UE shall start timer T3245 and proceed as described in clause 5.3.7a. |  5.5.1.2.5 |  5.5.2.3.2
10 |  Additionally, the UE shall reset the registration attempt counter #12 (Tracking area not allowed) The UE shall set the EPS update status to EU3 ROAMING NOT ALLOWED (and shall store it according to clause 5.1.3.3) and shall delete any GUTI, last visited registered TAI, TAI list and eKSI. Additionally, the UE shall reset the attach attempt counter In S1 mode, the UE shall store the current TAI in the list of "forbidden tracking areas for regional provision of service" and enter the state EMM-DEREGISTERED.LIMITED-SERVICE. |  In addition, the UE shall reset the registration attempt counter #15 (No suitable cells in tracking area) The UE shall set the EPS update status to EU3 ROAMING NOT ALLOWED (and shall store it according to clause 5.1.3.3). The UE shall reset the tracking area updating attempt counter and shall enter the state EMM-REGISTERED.LIMITED-SERVICE The UE shall store the current TAI in the list of "forbidden tracking areas for roaming". |  5.5.1.2.5 |  5.5.3.3.5
11 |  Additionally, the UE shall reset the registration attempt counter #15 (No suitable cells in tracking area) The UE shall set the EPS update status to EU3 ROAMING NOT ALLOWED (and shall store it according to clause 5.1.3.3) and shall delete any GUTI, last visited registered TAI, TAI list and eKSI. Additionally, the UE shall reset the attach attempt counter The UE shall store the current TAI in the list of "forbidden tracking areas for roaming". |  Additionally, the UE shall reset the registration attempt counter #15 (No suitable cells in tracking area) The UE shall set the EPS update status to EU3 ROAMING NOT ALLOWED (and shall store it according to clause 5.1.3.3) and shall delete any GUTI, last visited registered TAI, TAI list and eKSI. Additionally the UE shall reset the attach attempt counter and enter the state EMM-DEREGISTERED.LIMITED-SERVICE The UE shall store the current TAI in the list of "forbidden tracking areas for roaming". |  5.5.1.2.5 |  5.5.3.3.5
12 |  It is FFS how to prevent the UE from making repeated attempts at selecting the same satellite access PLMN if there are no other available PLMNs at UE ’s location Other values are considered as abnormal cases. The behaviour of the UE in those cases is specified in clause 5.5.1.2.6 . |  Additionally, the UE shall reset the registration attempt counter Editor ’s note: [IoT_SAT_ARCH_EPS, CR#3620]. It is FFS how to prevent the UE from making repeated attempts at selecting the same satellite access PLMN if there are no other available PLMNs at UE ’s location Other EMM cause values or if no EMM cause IE is included is considered as abnormal cases. |  5.5.1.2.5 |  5.5.2.3.2
13 |  Additionally, the UE shall delete the list of equivalent PLMNs and shall enter the state EMM-DEREGISTERED.NO-IMSI. If the message has been successfully integrity checked by the NAS and the UE maintains a counter for "SIM/USIM considered invalid for GPRS services", then the UE shall set this counter to UE implementation-specific maximum value. |  The UE shall consider the USIM as invalid for EPS services until switching off or the UICC containing the USIM is removed or the timer T3245 expires as described in clause 5.3.7a. The UE shall delete the list of equivalent PLMNs and shall enter the state EMM-DEREGISTERED.NO-IMSI. If the UE maintains a counter for "SIM/USIM considered invalid for GPRS services", then the UE shall set this counter to UE implementation-specific maximum value. |  5.6.1.5 |  5.5.2.3.2
14 |  If the UE is operating in single-registration mode, the UE shall in addition set the 5GMM state to 5GMM-DEREGISTERED #42 (Severe network failure) The UE shall set the EPS update status to EU2 NOT UPDATED, and shall delete any GUTI, last visited registered TAI, TAI list, eKSI, and list of equivalent PLMNs, and set the tracking area update counter to 5. |  If the UE is operating in single-registration mode, the UE shall in addition set the 5GMM state to 5GMM-DEREGISTERED #42 (Severe network failure) The UE shall set the EPS update status to EU2 NOT UPDATED, and shall delete any GUTI, last visited registered TAI, TAI list, eKSI, and list of equivalent PLMNs. |  5.5.3.2.5 |  5.5.1.2.5
Table 13: Subset of inconsistent pairs discovered from 4G (Continued) : security specification and : NAS specification Pair No. |  $\text{Text}_{1}$ |  $\text{Text}_{2}$ |  $\text{Loc}_{1}$ |  $\text{Loc}_{2}$
---|---|---|---|---
15 |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures and the EMM sublayer shall not indicate the abort of the service request procedure to the MM sublayer. Otherwise the EMM sublayer shall indicate the abort of the service request procedure to the MM sublayer NOTE 8: If the UE disables the E-UTRA capability, then subsequent mobile terminating calls could fail If the service request was initiated for CS fallback for emergency call and a CS fallback cancellation request was not received, the UE may attempt to select GERAN or UTRAN radio access technology. |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures and the EMM sublayer shall not indicate the abort of the service request procedure to the MM sublayer. Otherwise the EMM sublayer shall indicate the abort of the service request procedure to the MM sublayer, and the UE shall also set the EPS update status to EU2 NOT UPDATED and enter the state EMM-REGISTERED.ATTEMPTING-TO-UPDATE |  5.6.1.5 |  5.6.1.5
16 |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures and the EMM sublayer shall not indicate the abort of the service request procedure to the MM sublayer. Otherwise the EMM sublayer shall indicate the abort of the service request procedure to the MM sublayer NOTE 8: If the UE disables the E-UTRA capability, then subsequent mobile terminating calls could fail If the service request was initiated for CS fallback for emergency call and a CS fallback cancellation request was not received, the UE may attempt to select GERAN or UTRAN radio access technology. |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures and the EMM sublayer shall not indicate the abort of the service request procedure to the MM sublayer. Otherwise the EMM sublayer shall indicate the abort of the service request procedure to the MM sublayer, and the UE shall also set the EPS update status to EU2 NOT UPDATED and enter the state EMM-REGISTERED.ATTEMPTING-TO-UPDATE |  5.6.1.5 |  5.6.1.5
17 |  The following abnormal cases can be identified a) Expiry of timer T3485 On the first expiry of the timer T3485, the MME shall resend the ACTIVATE DEFAULT EPS BEARER CONTEXT REQUEST and shall reset and restart timer T3485. This retransmission is repeated four times, i.e. on the fifth expiry of timer T3485, the MME shall release possibly allocated resources for this activation and shall abort the procedure |  The following abnormal cases can be identified a) Expiry of timer T3485 On the first expiry of the timer T3485, the MME shall resend the ACTIVATE DEDICATED EPS BEARER CONTEXT REQUEST and shall reset and restart timer T3485. This retransmission is repeated four times, i.e. on the fifth expiry of timer T3485, the MME shall abort the procedure, release any resources allocated for this activation and enter the state BEARER CONTEXT INACTIVE |  6.4.1.6 |  6.4.2.6
18 |  When the UE is switched off, when the USIM is removed, or when there is a change in the value indicated by the network in the 15 bearers bit of the EPS network feature support IE, the UE shall clear all previous determinations representing PLMNs maximum number of EPS bearer contexts in S1 mode. |  When the UE is switched off or when the USIM is removed, the UE shall clear all previous determinations representing any PLMN ’s maximum number of EPS bearer contexts in S1 mode. |  6.5.1.4.3 |  6.5.3.4.2
19 |  Tracking area updating and detach procedure collision EPS detach containing detach type "re-attach required" or "re-attach not required": If the UE receives a DETACH REQUEST message before the tracking area updating procedure has been completed, the tracking area updating procedure shall be aborted and the detach procedure shall be progressed. |  Change of cell into a new tracking area If a cell change into a new tracking area that is not in the stored TAI list occurs before the UE initiated detach procedure is completed, the UE proceeds as follows: 1) If the detach procedure was initiated for reasons other than removal of the USIM or the UE is to be switched off, the detach procedure shall be aborted and re-initiated after successfully performing a tracking area updating procedure |  5.5.3.2.6 |  5.5.2.2.4
20 |  The UE shall store the current TAI in the list of "forbidden tracking areas for roaming". If the ATTACH REJECT message is not integrity protected, the UE shall memorize the current TAI was stored in the list of "forbidden tracking areas for roaming" for non-integrity protected NAS reject message. Additionally, the UE shall enter the state EMM-DEREGISTERED.LIMITED-SERVICE |  The UE shall store the current TAI in the list of "forbidden tracking areas for roaming". If the ATTACH REJECT message is not integrity protected, the UE shall memorize the current TAI was stored in the list of "forbidden tracking areas for roaming" for non-integrity protected NAS reject message. |  5.5.1.2.5 |  5.5.1.3.5
21 |  From this time onward the UE shall cipher and integrity protect all NAS signalling messages with the selected NAS ciphering and NAS integrity algorithms. |  From this time onward, all NAS messages exchanged between the UE and the MME are sent integrity protected and except for the messages specified in clause 4.4.5, all NAS messages exchanged between the UE and the MME are sent ciphered |  5.4.3.3 |  4.4.2.3
22 |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures; otherwise the EMM sublayer shall indicate the abort of the EMM procedure to the MM sublayer . |  If the UE finds a suitable GERAN or UTRAN cell, it then proceeds with the appropriate MM and CC specific procedures; otherwise if there is a CS fallback emergency call or CS fallback call pending, the EMM sublayer shall indicate the abort of the EMM procedure to the MM sublayer . |  5.5.1.3.6 |  5.5.3.3.4.3