§ INTRODUCTION

Modern data collection and analysis pipelines increasingly involve collections
of decoupled components in order to better manage software complexity
through reusability, modularity, and fault isolation [Wegiel and Krintz, 2010].
These pipelines are frequently built using different programming
languages for the different phases of data analysis — collection,
cleaning, modeling, analysis, post-processing, and
presentation — in order to take advantage of the unique combination of
performance, speed of development, and library support offered by
different environments and languages.  Each stage of such a data
analysis pipeline may produce intermediate results that need to be
stored in a file, or sent over the network for further processing.

Given these requirements, how do we safely and efficiently share intermediate results
between different applications, possibly written in different
languages, and possibly running on different computer systems?
In computer programming, serialization is the process of
translating data structures, variables, and session state into a
format that can be stored or transmitted and then reconstructed in the
original form later [Cline, 2013].
languages such as R, Julia, Java, and Python include built-in
support for serialization, but the default formats
are usually language-specific and thereby lock the user into a single

Data analysts and researchers often use character-separated text formats such
as  [Shafranovich, 2005] to export and import
data. However, anyone who has ever used  files will have noticed
that this method has many limitations: it is restricted to tabular data,
lacks type-safety, and has limited precision for numeric values.  Moreover,
ambiguities in the format itself frequently cause problems.  For example,
conventions on which characters is used as separator or decimal point vary by
country.  Extensible Markup Language () is another
well-established and widely-supported format with the ability to define just
about any arbitrarily complex schema [Nolan and Temple Lang, 2013]. However, it pays
for this complexity with comparatively large and verbose messages, and added
complexity at the parsing side (which are somewhat mitigated by the
availability of mature libraries and parsers). Because  is
text-based and has no native notion of numeric types or arrays, it usually not a
very practical format to store numeric data sets as they appear in statistical

A more modern format is JavaScript ObjectNotation
(), which is derived from the object literals of
JavaScript, and already widely-used on the world wide web.
Several R packages implement functions to parse and generate
data from R objects [Couture-Beil, 2012, Temple Lang, 2011, Ooms, 2014].
natively supports arrays and four primitive types: numbers, strings,
booleans, and null. However, as it too is a text-based format, numbers are
stored as human-readable decimal notation which is inefficient and
leads to loss of type (double versus integer) and precision.
A number of binary formats based on  have been proposed
that reduce the parsing cost and improve efficiency, but these formats
are not widely supported.  Furthermore, such formats lack a separate
schema for the serialized data and thus still duplicate field names
with each message sent over the network or stored in a file.

Once the data serialization needs of an application become complex
enough, developers typically benefit from the use of an
interface description language, or IDL.  IDLs like
Protocol Buffers [Google, 2012], Apache Thrift, and Apache Avro
provide a compact well-documented schema for cross-language data
structures and efficient binary interchange formats.  Since the schema
is provided separately from the data, the data can be
efficiently encoded to minimize storage costs when
compared with simple “schema-less” binary interchange formats.
Many sources compare data serialization formats
and show Protocol Buffers perform favorably to the alternatives; see
Sumaray and Makki, 2012 for one such comparison.

This paper describes an R interface to Protocol Buffers,
and is organized as follows. Section <ref>
provides a general high-level overview of Protocol Buffers as well as a basic
motivation for their use.
Section <ref> describes the interactive R interface
provided by the RProtoBuf package, and introduces the two main abstractions:
Messages and Descriptors.  Section <ref>
details the implementation details of the main S4 classes and methods.
Section <ref> describes the challenges of type coercion
between R and other languages.  Section <ref> introduces a
general R language schema for serializing arbitrary R objects and evaluates
it against the serialization capabilities built directly into R.  Sections <ref>
and <ref> provide real-world use cases of RProtoBuf
in MapReduce and web service environments, respectively, before
Section <ref> concludes.

§ PROTOCOL BUFFERS

Protocol Buffers are a modern, language-neutral, platform-neutral,
extensible mechanism for sharing and storing structured data.  Some of
the key features provided by Protocol Buffers for data analysis include:

* Portable:  Enable users to send and receive data between
applications as well as different computers or operating systems.

* Efficient:  Data is serialized into a compact binary
representation for transmission or storage.

* Extensible:  New fields can be added to Protocol Buffer schemas
in a forward-compatible way that does not break older applications.

* Stable:  Protocol Buffers have been in wide use for over a

Example protobuf usage

Figure <ref> illustrates an example
communication work flow with Protocol Buffers and an interactive R session.
Common use cases include populating a request remote-procedure call (RPC)
Protocol Buffer in R that is then serialized and sent over the network to a
remote server.  The server would then deserialize the message, act on the
request, and respond with a new Protocol Buffer over the network.
The key difference to, say, a request to an Rserve instance is that
the remote server may be implemented in any language, with no
dependence on R.

While traditional IDLs have at times been criticized for code bloat and
complexity, Protocol Buffers are based on a simple list and records
model that is flexible and easy to use.  The schema for structured
Protocol Buffer data is defined in  files, which may
contain one or more message types.  Each message type has one or more
fields.  A field is specified with a unique number (called a tag number), a name, a value
type, and a field rule specifying whether the field is optional,
required, or repeated.  The supported value types are numbers,
enumerations, booleans, strings, raw bytes, or other nested message
types.  The  file syntax for defining the structure of Protocol
Buffer data is described comprehensively on Google Code[See
Table <ref> shows an example  file that
defines the  type[The compound name
in R is derived from the name of the
message (Person) and the name of the package defined at the top of the
file in which it is defined (tutorial).].  The R code in the right
column shows an example of creating a new message of this type and
populating its fields.

Schema :      Example R Session

package tutorial;
message Person
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType
MOBILE = 0; HOME = 1;
WORK = 2;

message PhoneNumber
required string number = 1;
optional PhoneType type = 2;

repeated PhoneNumber phone = 4;

R> library(RProtoBuf)
R> p <- new(tutorial.Person,id=1,name="Dirk")
R> p$name
\end{Sinput}
\begin{Soutput}
[1] "Dirk"
\end{Soutput}
\begin{Sinput}
R> p$name <- "Murray"
R> cat(as.character(p))
\end{Sinput}
\begin{Soutput}
name: "Murray"
id: 1
\end{Soutput}
\begin{Sinput}
R> serialize(p, NULL)
\end{Sinput}
\begin{Soutput}
[1] 0a 06 4d 75 72 72 61 79 10 01
\end{Soutput}
\begin{Sinput}
R> class(p)
\end{Sinput}
\begin{Soutput}
[1] "Message"
[1] "RProtoBuf"
\end{Soutput}
\end{Schunk}
\end{minipage} \\
\bottomrule
\end{tabular}
\caption{The schema representation from a \texttt{.proto} file for the
\texttt{tutorial.Person} class (left) and simple \proglang{R} code for creating
an object of this class and accessing its fields (right).}
\label{tab:proto}
\end{table}

For added speed and efficiency, the \proglang{C++}, \proglang{Java},
and \proglang{Python} bindings to
Protocol Buffers are used with a compiler that translates a Protocol
Buffer schema description file (ending in \texttt{.proto}) into
language-specific classes that can be used to create, read, write, and
manipulate Protocol Buffer messages.  The \proglang{R} interface, in contrast,
uses a reflection-based API that makes some operations slightly
slower but which is much more convenient for interactive data analysis.
All messages in \proglang{R} have a single class
structure, but different accessor methods are created at runtime based
on the named fields of the specified message type, as described in the
next section.

\section{Basic Usage: Messages and descriptors}
\label{sec:rprotobuf-basic}

This section describes how to use the \proglang{R} API to create and manipulate
Protocol Buffer messages in \proglang{R}, and how to read and write the
binary representation of the message (often called the \emph{payload}) to files and arbitrary binary
\proglang{R} connections.
The two fundamental building blocks of Protocol Buffers are \emph{Messages}
and \emph{Descriptors}.  Messages provide a common abstract encapsulation of
structured data fields of the type specified in a Message Descriptor.
Message Descriptors are defined in \texttt{.proto} files and define a
schema for a particular named class of messages.

\subsection[Importing message descriptors from .proto files]{Importing message descriptors from \texttt{.proto} files}

To create or parse a Protocol Buffer Message, one must first read in
the message type specification from a \texttt{.proto} file. The
\texttt{.proto} files are imported using the \code{readProtoFiles}
function, which can either import a single file, all files in a directory,
or every \texttt{.proto} file provided by a particular \proglang{R} package.

After importing proto files, the corresponding message descriptors are
available by name from the \code{RProtoBuf:DescriptorPool} environment in
the \proglang{R} search path.  This environment is implemented with the
user-defined tables framework from the \pkg{RObjectTables} package
available from the OmegaHat project [Temple Lang}, 2012].  Instead of
being associated with a static hash table, this environment
dynamically queries the in-memory database of loaded descriptors
during normal variable lookup.

\begin{Schunk}
\begin{Sinput}
R> ls("RProtoBuf:DescriptorPool")
\end{Sinput}
\begin{Soutput}
[1] "rexp.CMPLX"                   "rexp.REXP"
[3] "rexp.STRING"                  "rprotobuf.HelloWorldRequest"
[5] "rprotobuf.HelloWorldResponse" "tutorial.AddressBook"
[7] "tutorial.Person"
\end{Soutput}
\end{Schunk}

\subsection{Creating a message}

New messages are created with the \texttt{new} function which accepts
a Message Descriptor and optionally a list of ``name = value'' pairs
to set in the message.
%The objects contained in the special environment are
%descriptors for their associated message types. Descriptors will be
%discussed in detail in another part of this document, but for the
%purpose of this section, descriptors are just used with the \texttt{new}
%function to create messages.

\begin{Schunk}
\begin{Sinput}
R> p1 <- new(tutorial.Person)
R> p <- new(tutorial.Person, name = "Murray", id = 1)
\end{Sinput}
\end{Schunk}

\subsection{Access and modify fields of a message}

Once the message is created, its fields can be queried
and modified using the dollar operator of \proglang{R}, making Protocol
Buffer messages seem like lists.

\begin{Schunk}
\begin{Sinput}
R> p$name
\end{Sinput}
\begin{Soutput}
[1] "Murray"
\end{Soutput}
\begin{Sinput}
R> p$id
\end{Sinput}
\begin{Soutput}
[1] 1
\end{Soutput}
\begin{Sinput}
R> p$email <-<EMAIL_ADDRESS>\end{Sinput}
\end{Schunk}

As opposed to \proglang{R} lists, no partial matching is performed
and the name must be given entirely.
The \verb|[[| operator can also be used to query and set fields
of a messages, supplying either their name or their tag number:

\begin{Schunk}
\begin{Sinput}
R> p[["name"]] <- "Murray Stokely"
R> p[[ 2 ]] <- 3
R> p[["email"]]
\end{Sinput}
\begin{Soutput}
[1]<EMAIL_ADDRESS>\end{Soutput}
\end{Schunk}

Protocol Buffers include a 64-bit integer type, but \proglang{R} lacks native
64-bit integer support.  A workaround is available and described in
Section~\ref{sec:int64} for working with large integer values.

\subsection{Display messages}

Protocol Buffer messages and descriptors implement \texttt{show}
methods that provide basic information about the message:

\begin{Schunk}
\begin{Sinput}
R> p
\end{Sinput}
\begin{Soutput}
[1] "message of type 'tutorial.Person' with 3 fields set"
\end{Soutput}
\end{Schunk}

For additional information, such as for debugging purposes,
the \texttt{as.character} method provides a more complete ASCII
representation of the contents of a message.

\begin{Schunk}
\begin{Sinput}
R> writeLines(as.character(p))
\end{Sinput}
\begin{Soutput}
name: "Murray Stokely"
id: 3
email<EMAIL_ADDRESS>\end{Soutput}
\end{Schunk}

\subsection{Serializing messages}

One of the primary benefits of Protocol Buffers is the efficient
binary wire-format representation.
The \texttt{serialize} method is implemented for
Protocol Buffer messages to serialize a message into a sequence of
bytes (raw vector) that represents the message.
The raw bytes can then be parsed back into the original message safely
as long as the message type is known and its descriptor is available.

\begin{Schunk}
\begin{Sinput}
R> serialize(p, NULL)
\end{Sinput}
\begin{Soutput}
[1] 0a 0e 4d 75 72 72 61 79 20 53 74 6f 6b 65 6c 79 10 03 1a 12 6d 75
[23] 72 72 61 79 40 73 74 6f 6b 65 6c 79 2e 6f 72 67
\end{Soutput}
\end{Schunk}

The same method can be used to serialize messages to files:

\begin{Schunk}
\begin{Sinput}
R> tf1 <- tempfile()
R> serialize(p, tf1)
R> readBin(tf1, raw(0), 500)
\end{Sinput}
\begin{Soutput}
[1] 0a 0e 4d 75 72 72 61 79 20 53 74 6f 6b 65 6c 79 10 03 1a 12 6d 75
[23] 72 72 61 79 40 73 74 6f 6b 65 6c 79 2e 6f 72 67
\end{Soutput}
\end{Schunk}

Or to arbitrary binary connections:

\begin{Schunk}
\begin{Sinput}
R> tf2 <- tempfile()
R> con <- file(tf2, open = "wb")
R> serialize(p, con)
R> close(con)
R> readBin(tf2, raw(0), 500)
\end{Sinput}
\begin{Soutput}
[1] 0a 0e 4d 75 72 72 61 79 20 53 74 6f 6b 65 6c 79 10 03 1a 12 6d 75
[23] 72 72 61 79 40 73 74 6f 6b 65 6c 79 2e 6f 72 67
\end{Soutput}
\end{Schunk}

\texttt{serialize} can also be called in a more traditional
object oriented fashion using the dollar operator:

\begin{Schunk}
\begin{Sinput}
R> # serialize to a file
R> p$serialize(tf1)
R> # serialize to a binary connection
R> con <- file(tf2, open = "wb")
R> p$serialize(con)
R> close(con)
\end{Sinput}
\end{Schunk}

\subsection{Parsing messages}

The \CRANpkg{RProtoBuf} package defines the \code{read} and
\code{readASCII} functions to read messages from files, raw vectors,
or arbitrary connections.  \code{read} expects to read the message
payload from binary files or connections and \code{readASCII} parses
the human-readable ASCII output that is created with
\code{as.character}.

The binary representation of the message
does not contain information that can be used to dynamically
infer the message type, so we have to provide this information
to the \code{read} function in the form of a descriptor:

\begin{Schunk}
\begin{Sinput}
R> msg <- read(tutorial.Person, tf1)
R> writeLines(as.character(msg))
\end{Sinput}
\begin{Soutput}
name: "Murray Stokely"
id: 3
email<EMAIL_ADDRESS>\end{Soutput}
\end{Schunk}

The \texttt{input} argument of \texttt{read} can also be a binary
readable \proglang{R} connection, such as a binary file connection:

\begin{Schunk}
\begin{Sinput}
R> con <- file(tf2, open = "rb")
R> message <- read(tutorial.Person, con)
R> close(con)
R> writeLines(as.character(message))
\end{Sinput}
\begin{Soutput}
name: "Murray Stokely"
id: 3
email<EMAIL_ADDRESS>\end{Soutput}
\end{Schunk}

Finally, the payload of the message can be used:

\begin{Schunk}
\begin{Sinput}
R> # reading the raw vector payload of the message
R> payload <- readBin(tf1, raw(0), 5000)
R> message <- read(tutorial.Person, payload)
\end{Sinput}
\end{Schunk}

\texttt{read} can also be used as a pseudo-method of the descriptor

\begin{Schunk}
\begin{Sinput}
R> # reading from a file
R> message <- tutorial.Person$read(tf1)
R> # reading from a binary connection
R> con <- file(tf2, open = "rb")
R> message <- tutorial.Person$read(con)
R> close(con)
R> # read from the payload
R> message <- tutorial.Person$read(payload)
\end{Sinput}
\end{Schunk}

\section{Under the hood: S4 classes, methods, and pseudo methods}
\label{sec:rprotobuf-classes}

The \CRANpkg{RProtoBuf} package uses the S4 system to store
information about descriptors and messages.  Using the S4 system
allows the package to dispatch methods that are not
generic in the S3 sense, such as \texttt{new} and
\texttt{serialize}.
Table~\ref{class-summary-table} lists the six
primary Message and Descriptor classes in \CRANpkg{RProtoBuf}.  Each \proglang{R} object
contains an external pointer to an object managed by the
\texttt{protobuf} \proglang{C++} library, and the \proglang{R} objects make calls into more
than 100 \proglang{C++} functions that provide the
glue code between the \proglang{R} language classes and the underlying \proglang{C++}

\begin{table}[bp]
\centering
\begin{tabular}{lccl}
\toprule
\textbf{Class}      &
\textbf{Slots} &
\textbf{Methods} &
\textbf{Dynamic Dispatch}\\
\cmidrule{1-4}
Message             & 2 & 20 & yes (field names)\\
Descriptor          & 2 & 16 & yes (field names, enum types, nested types)\\
FieldDescriptor     & 4 & 18 & no\\
EnumDescriptor      & 4 & 11 & yes (enum constant names)\\
EnumValueDescriptor & 3 & \phantom{1}6 & no\\
FileDescriptor      & 3 & \phantom{1}6 & yes (message/field definitions)\\
\bottomrule
\end{tabular}
\caption{\label{class-summary-table}Overview of class, slot, method and
dispatch relationships}
\end{table}

The \CRANpkg{Rcpp} package
[Eddelbuettel and Fran{\c{c}}ois, 2011, Eddelbuettel, 2013] is used to
facilitate this integration of the \proglang{R} and \proglang{C++} code for these objects.
Each method is wrapped individually which allows us to add
user-friendly custom error handling, type coercion, and performance
improvements at the cost of a more verbose implementation.
The \CRANpkg{RProtoBuf} package in many ways motivated
the development of \CRANpkg{Rcpp} Modules [Eddelbuettel and Fran{\c{c}}ois, 2013],
which provide a more concise way of wrapping \proglang{C++} functions and classes
in a single entity.

The \CRANpkg{RProtoBuf} package supports two forms for calling
functions with these S4 classes:
\begin{itemize}
\item The functional dispatch mechanism of the the form
\verb|method(object, arguments)| (common to \proglang{R}), and
\item The traditional object oriented notation
\verb|object$method(arguments)|.
\end{itemize}

Additionally, \CRANpkg{RProtoBuf} supports tab completion for all
classes.  Completion possibilities include pseudo-method names for all
classes, plus \emph{dynamic dispatch} on names or types specific to a given
object.  This functionality is implemented with the
\texttt{.DollarNames} S3 generic function defined in the \pkg{utils}

\subsection{Messages}

The \texttt{Message} S4 class represents Protocol Buffer Messages and
is the core abstraction of \CRANpkg{RProtoBuf}. Each \texttt{Message}
contains a pointer to a \texttt{Descriptor} which defines the schema
of the data defined in the Message, as well as a number of
\texttt{FieldDescriptors} for the individual fields of the message.  A
complete list of the slots and methods for \texttt{Messages}
is available in Table~\ref{Message-methods-table}.

\begin{Schunk}
\begin{Sinput}
R> new(tutorial.Person)
\end{Sinput}
\begin{Soutput}
[1] "message of type 'tutorial.Person' with 0 fields set"
\end{Soutput}
\end{Schunk}

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & External pointer to the \texttt{Message} object of the \proglang{C++} protobuf library. Documentation for the
\texttt{Message} class is available from the Protocol Buffer project page. \\
%(\url{http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.message.html#Message}) \\
\texttt{type} & Fully qualified name of the message. For example a \texttt{Person} message
has its \texttt{type} slot set to \texttt{tutorial.Person} \\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{has} & Indicates if a message has a given field.   \\
\texttt{clone} & Creates a clone of the message \\
\texttt{isInitialized} & Indicates if a message has all its required fields set\\
\texttt{serialize} & serialize a message to a file, binary connection, or raw vector\\
\texttt{clear} & Clear one or several fields of a message, or the entire message\\
\texttt{size} & The number of elements in a message field\\
\texttt{bytesize} & The number of bytes the message would take once serialized\\[3mm]
\texttt{swap} & swap elements of a repeated field of a message\\
\texttt{set} & set elements of a repeated field\\
\texttt{fetch} & fetch elements of a repeated field\\
\texttt{setExtension} & set an extension of a message\\
\texttt{getExtension} & get the value of an extension of a message\\
\texttt{add} & add elements to a repeated field \\[3mm]
\texttt{str} & the \proglang{R} structure of the message\\
\texttt{as.character} & character representation of a message\\
\texttt{toString} & character representation of a message (same as \texttt{as.character}) \\
\texttt{as.list} & converts message to a named \proglang{R} list\\
\texttt{update} & updates several fields of a message at once\\
\texttt{descriptor} & get the descriptor of the message type of this message\\
\texttt{fileDescriptor} & get the file descriptor of this message's descriptor\\
\hline
\end{tabular}
\end{small}
\caption{\label{Message-methods-table}Description of slots and methods for the \texttt{Message} S4 class}
\end{table}

\subsection{Descriptors}

Descriptors describe the type of a Message.  This includes what fields
a message contains and what the types of those fields are.  Message
descriptors are represented in \proglang{R} by the \emph{Descriptor} S4
class. The class contains the slots \texttt{pointer} and
\texttt{type}.  Similarly to messages, the \verb|$| operator can be
used to retrieve descriptors that are contained in the descriptor, or
invoke pseudo-methods.

When \CRANpkg{RProtoBuf} is first loaded it calls
\texttt{readProtoFiles} to read in the example \texttt{addressbook.proto} file
included with the package.  The \texttt{tutorial.Person} descriptor
and all other descriptors defined in the loaded \texttt{.proto} files are
then available on the search path\footnote{This explains why the example in
Table~\ref{tab:proto} lacked an explicit call to
\texttt{readProtoFiles}.}.

\begin{Schunk}
\begin{Sinput}
R> tutorial.Person$email # field descriptor
\end{Sinput}
\begin{Soutput}
[1] "descriptor for field 'email' of type 'tutorial.Person' "
\end{Soutput}
\begin{Sinput}
R> tutorial.Person$PhoneType # enum descriptor
\end{Sinput}
\begin{Soutput}
[1] "descriptor for enum 'PhoneType' of type 'tutorial.Person' with 3 values"
\end{Soutput}
\begin{Sinput}
R> tutorial.Person$PhoneNumber # nested type descriptor
\end{Sinput}
\begin{Soutput}
[1] "descriptor for type 'tutorial.Person.PhoneNumber' "
\end{Soutput}
\begin{Sinput}
R> # same as
R> tutorial.Person.PhoneNumber
\end{Sinput}
\begin{Soutput}
[1] "descriptor for type 'tutorial.Person.PhoneNumber' "
\end{Soutput}
\end{Schunk}

Table~\ref{Descriptor-methods-table} provides a complete list of the
slots and available methods for Descriptors.

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & External pointer to the \texttt{Descriptor} object of the \proglang{C++} proto library. Documentation for the
\texttt{Descriptor} class is available from the Protocol Buffer project page.\\
%\url{http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.descriptor.html#Descriptor} \\
\texttt{type} & Fully qualified path of the message type. \\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{new} & Creates a prototype of a message described by this descriptor.\\
\texttt{read} & Reads a message from a file or binary connection.\\
\texttt{readASCII} & Read a message in ASCII format from a file or
text connection.\\
\texttt{name} & Retrieve the name of the message type associated with
this descriptor.\\
\texttt{as.character} & character representation of a descriptor\\
\texttt{toString} & character representation of a descriptor (same as \texttt{as.character}) \\
\texttt{as.list} & return a named
list of the field, enum, and nested descriptors included in this descriptor.\\
\texttt{asMessage} & return DescriptorProto message. \\
\texttt{fileDescriptor} & Retrieve the file descriptor of this
\texttt{containing\_type} & Retrieve the descriptor describing the message type containing this descriptor.\\
\texttt{field\_count} & Return the number of fields in this descriptor.\\
\texttt{field} & Return the descriptor for the specified field in this descriptor.\\
\texttt{nested\_type\_count} & The number of nested types in this descriptor.\\
\texttt{nested\_type} & Return the descriptor for the specified nested
type in this descriptor.\\
\texttt{enum\_type\_count} & The number of enum types in this descriptor.\\
\texttt{enum\_type} & Return the descriptor for the specified enum
type in this descriptor.\\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{Descriptor-methods-table}Description of slots and methods for the \texttt{Descriptor} S4 class}
\end{table}

\subsection{Field descriptors}
\label{subsec-field-descriptor}

The class \emph{FieldDescriptor} represents field
descriptors in \proglang{R}. This is a wrapper S4 class around the
\texttt{google::protobuf::FieldDescriptor} \proglang{C++} class.
Table~\ref{fielddescriptor-methods-table} describes the methods
defined for the \texttt{FieldDescriptor} class.

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & External pointer to the \texttt{FieldDescriptor} \proglang{C++} variable \\
\texttt{name} & Simple name of the field \\
\texttt{full\_name} & Fully qualified name of the field \\
\texttt{type} & Name of the message type where the field is declared \\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{as.character} & Character representation of a descriptor\\
\texttt{toString} & Character representation of a descriptor (same as \texttt{as.character}) \\
\texttt{asMessage} & Return FieldDescriptorProto message. \\
\texttt{name} & Return the name of the field descriptor.\\
\texttt{fileDescriptor} & Return the fileDescriptor where this field is defined.\\
\texttt{containing\_type} & Return the containing descriptor of this field.\\
\texttt{is\_extension} & Return TRUE if this field is an extension.\\
\texttt{number} & Gets the declared tag number of the field.\\
\texttt{type} & Gets the type of the field.\\
\texttt{cpp\_type} & Gets the \proglang{C++} type of the field.\\
\texttt{label} & Gets the label of a field (optional, required, or repeated).\\
\texttt{is\_repeated} & Return TRUE if this field is repeated.\\
\texttt{is\_required} & Return TRUE if this field is required.\\
\texttt{is\_optional} & Return TRUE if this field is optional.\\
\texttt{has\_default\_value} & Return TRUE if this field has a default value.\\
\texttt{default\_value} & Return the default value.\\
\texttt{message\_type} & Return the message type if this is a message type field.\\
\texttt{enum\_type} & Return the enum type if this is an enum type field.\\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{fielddescriptor-methods-table}Description of slots and
methods for the \texttt{FieldDescriptor} S4 class}
\end{table}

\subsection{Enum descriptors}
\label{subsec-enum-descriptor}

The class \emph{EnumDescriptor} represents enum descriptors in \proglang{R}.
This is a wrapper S4 class around the
\texttt{google::protobuf::EnumDescriptor} \proglang{C++} class.
Table~\ref{enumdescriptor-methods-table} describes the methods
defined for the \texttt{EnumDescriptor} class.

The \verb|$| operator can be used to retrieve the value of enum
constants contained in the EnumDescriptor, or to invoke

The \texttt{EnumDescriptor} contains information about what values this type
defines, while the \texttt{EnumValueDescriptor} describes an
individual enum constant of a particular type.

\begin{Schunk}
\begin{Sinput}
R> tutorial.Person$PhoneType
\end{Sinput}
\begin{Soutput}
[1] "descriptor for enum 'PhoneType' of type 'tutorial.Person' with 3 values"
\end{Soutput}
\begin{Sinput}
R> tutorial.Person$PhoneType$WORK
\end{Sinput}
\begin{Soutput}
[1] 2
\end{Soutput}
\end{Schunk}

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & External pointer to the \texttt{EnumDescriptor} \proglang{C++} variable \\
\texttt{name} & Simple name of the enum \\
\texttt{full\_name} & Fully qualified name of the enum \\
\texttt{type} & Name of the message type where the enum is declared \\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{as.list} & return a named
integer vector with the values of the enum and their names.\\
\texttt{as.character} & character representation of a descriptor\\
\texttt{toString} & character
representation of a descriptor (same as \texttt{as.character}) \\
\texttt{asMessage} & return EnumDescriptorProto message. \\
\texttt{name} & Return the name of the enum descriptor.\\
\texttt{fileDescriptor} & Return the fileDescriptor where this field is defined.\\
\texttt{containing\_type} & Return the containing descriptor of this field.\\
\texttt{length} & Return the number of constants in this enum.\\
\texttt{has} & Return TRUE if this enum contains the specified named constant string.\\
\texttt{value\_count} & Return the number of constants in this enum (same as \texttt{length}).\\
\texttt{value} & Return the EnumValueDescriptor of an enum value of specified index, name, or number.\\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{enumdescriptor-methods-table}Description of slots and methods for the \texttt{EnumDescriptor} S4 class}
\end{table}

\subsection{Enum value descriptors}
\label{subsec-enumvalue-descriptor}

The class \emph{EnumValueDescriptor} represents enumeration value
descriptors in \proglang{R}.  This is a wrapper S4 class around the
\texttt{google::protobuf::EnumValueDescriptor} \proglang{C++} class.
Table~\ref{EnumValueDescriptor-methods-table} describes the methods
defined for the \texttt{EnumValueDescriptor} class.

The \verb|$| operator can be used to invoke pseudo-methods.

\begin{Schunk}
\begin{Sinput}
R> tutorial.Person$PhoneType$value(1)
\end{Sinput}
\begin{Soutput}
[1] "enum value descriptor tutorial.Person.MOBILE"
\end{Soutput}
\begin{Sinput}
R> tutorial.Person$PhoneType$value(name="HOME")
\end{Sinput}
\begin{Soutput}
[1] "enum value descriptor tutorial.Person.HOME"
\end{Soutput}
\begin{Sinput}
R> tutorial.Person$PhoneType$value(number=1)
\end{Sinput}
\begin{Soutput}
[1] "enum value descriptor tutorial.Person.HOME"
\end{Soutput}
\end{Schunk}

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & External pointer to the \texttt{EnumValueDescriptor} \proglang{C++} variable \\
\texttt{name} & simple name of the enum value \\
\texttt{full\_name} & fully qualified name of the enum value \\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{number} & return the number of this EnumValueDescriptor. \\
\texttt{name} & Return the name of the enum value descriptor.\\
\texttt{enum\_type} & return the EnumDescriptor type of this EnumValueDescriptor. \\
\texttt{as.character} & character representation of a descriptor. \\
\texttt{toString} & character representation of a descriptor (same as \texttt{as.character}). \\
\texttt{asMessage} & return EnumValueDescriptorProto message. \\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{EnumValueDescriptor-methods-table}Description of slots
and methods for the \texttt{EnumValueDescriptor} S4 class}
\end{table}

\subsection{File descriptors}
\label{subsec-file-descriptor}

\begin{table}[tbp]
\centering
\begin{small}
\begin{tabular}{lp{10cm}}
\toprule
\textbf{Slot} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{pointer} & external pointer to the \texttt{FileDescriptor} object of the \proglang{C++} proto library. Documentation for the
\texttt{FileDescriptor} class is available from the Protocol Buffer project page:
\url{http://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor.html#FileDescriptor} \\
\texttt{filename} & fully qualified pathname of the \texttt{.proto} file.\\
\texttt{package} & package name defined in this \texttt{.proto} file.\\[.3cm]
\textbf{Method} & \textbf{Description} \\
\cmidrule(r){2-2}
\texttt{name} & Return the filename for this FileDescriptorProto.\\
\texttt{package} & Return the file-level package name specified in this FileDescriptorProto.\\
\texttt{as.character} & character representation of a descriptor. \\
\texttt{toString} & character representation of a descriptor (same as \texttt{as.character}). \\
\texttt{asMessage} & return FileDescriptorProto message. \\
\texttt{as.list} & return named list of descriptors defined in this file descriptor.\\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{filedescriptor-methods-table}Description of slots and methods for the \texttt{FileDescriptor} S4 class}
\end{table}

The class \emph{FileDescriptor} represents file descriptors in \proglang{R}.
This is a wrapper S4 class around the
\texttt{google::protobuf::FileDescriptor} \proglang{C++} class.
Table~\ref{filedescriptor-methods-table} describes the methods
defined for the \texttt{FileDescriptor} class.

The \verb|$| operator can be used to retrieve named fields defined in
the FileDescriptor, or to invoke pseudo-methods.

\begin{Schunk}
\begin{Sinput}
R> f <- tutorial.Person$fileDescriptor()
R> f
\end{Sinput}
\begin{Soutput}
[1] "file descriptor for package tutorial (/Library/Frameworks/R.framework/Versions/3.0/Resources/library/RProtoBuf/proto/addressbook.proto)"
\end{Soutput}
\begin{Sinput}
R> f$Person
\end{Sinput}
\begin{Soutput}
[1] "descriptor for type 'tutorial.Person' "
\end{Soutput}
\end{Schunk}

\section{Type coercion}
\label{sec:types}

One of the benefits of using an Interface Definition Language (IDL)
like Protocol Buffers is that it provides a highly portable basic type
system. This permits different language and hardware implementations to map to
the most appropriate type in different environments.

Table~\ref{table-get-types} details the correspondence between the
field type and the type of data that is retrieved by \verb|$| and \verb|[[|
extractors.  Three types in particular need further attention due to
specific differences in the \proglang{R} language: booleans, unsigned
integers, and 64-bit integers.

\begin{table}[h]
\centering
\begin{small}
\begin{tabular}{lp{5cm}p{5cm}}
\toprule
Field type & \proglang{R} type (non repeated) & \proglang{R} type (repeated) \\
\cmidrule(r){2-3}
double	& \texttt{double} vector & \texttt{double} vector \\
float	& \texttt{double} vector & \texttt{double} vector \\[3mm]
uint32	  & \texttt{double} vector & \texttt{double} vector \\
fixed32	  & \texttt{double} vector & \texttt{double} vector \\[3mm]
int32	  & \texttt{integer} vector & \texttt{integer} vector \\
sint32	  & \texttt{integer} vector & \texttt{integer} vector \\
sfixed32  & \texttt{integer} vector & \texttt{integer} vector \\[3mm]
int64	  & \texttt{integer} or \texttt{character}
vector    & \texttt{integer} or \texttt{character} vector \\
uint64	  & \texttt{integer} or \texttt{character} vector & \texttt{integer} or \texttt{character} vector \\
sint64	  & \texttt{integer} or \texttt{character} vector & \texttt{integer} or \texttt{character} vector \\
fixed64	  & \texttt{integer} or \texttt{character} vector & \texttt{integer} or \texttt{character} vector \\
sfixed64  & \texttt{integer} or \texttt{character} vector & \texttt{integer} or \texttt{character} vector \\[3mm]
bool	& \texttt{logical} vector & \texttt{logical} vector \\[3mm]
string	& \texttt{character} vector & \texttt{character} vector \\
bytes	& \texttt{character} vector & \texttt{character} vector \\[3mm]
enum & \texttt{integer} vector & \texttt{integer} vector \\[3mm]
message & \texttt{S4} object of class \texttt{Message} & \texttt{list} of \texttt{S4} objects of class \texttt{Message} \\
\bottomrule
\end{tabular}
\end{small}
\caption{\label{table-get-types}Correspondence between field type and
\proglang{R} type retrieved by the extractors. Note that \proglang{R} lacks native
64-bit integers, so the \code{RProtoBuf.int64AsString} option is
available to return large integers as characters to avoid losing
precision.  This option is described in Section~\ref{sec:int64}.}
\end{table}

\subsection{Booleans}

\proglang{R} booleans can accept three values: \texttt{TRUE}, \texttt{FALSE}, and
\texttt{NA}.  However, most other languages, including the Protocol
Buffer schema, only accept \texttt{TRUE} or \texttt{FALSE}.  This means
that we simply can not store \proglang{R} logical vectors that include all three
possible values as booleans.  The library will refuse to store
\texttt{NA}s in Protocol Buffer boolean fields, and users must instead
choose another type (such as enum or integer) capable of storing three
distinct values.

% We want a cleaner error message here.
\begin{CodeChunk}
\begin{CodeInput}
R> a <- new(protobuf_unittest.TestAllTypes)
R> a$optional_bool <- TRUE
R> a$optional_bool <- FALSE
R> a$optional_bool <- NA
\end{CodeInput}
\begin{CodeOutput}
Error: NA boolean values can not be stored in bool Protocol Buffer fields
\end{CodeOutput}
\end{CodeChunk}

\subsection{Unsigned integers}

\proglang{R} lacks a native unsigned integer type.  Values between $2^{31}$ and
$2^{32} - 1$ read from unsigned integer Protocol Buffer fields must be
stored as doubles in \proglang{R}.

\begin{Schunk}
\begin{Sinput}
R> as.integer(2^31-1)
\end{Sinput}
\begin{Soutput}
[1] 2147483647
\end{Soutput}
\begin{Sinput}
R> as.integer(2^31 - 1) + as.integer(1)
\end{Sinput}
\begin{Soutput}
[1] NA
\end{Soutput}
\begin{Sinput}
R> 2^31
\end{Sinput}
\begin{Soutput}
[1] 2.147e+09
\end{Soutput}
\begin{Sinput}
R> class(2^31)
\end{Sinput}
\begin{Soutput}
[1] "numeric"
\end{Soutput}
\end{Schunk}

\subsection{64-bit integers}
\label{sec:int64}

\proglang{R} also does not support the native 64-bit integer type. Numeric vectors
with values $\geq 2^{31}$ can only be stored as doubles, which have
limited precision. Thereby \proglang{R} loses the ability to distinguish some
distinct integers:

\begin{Schunk}
\begin{Sinput}
R> 2^53 == (2^53 + 1)
\end{Sinput}
\begin{Soutput}
[1] TRUE
\end{Soutput}
\end{Schunk}

However, most modern languages do have support for 64-bit integers,
which becomes problematic when \CRANpkg{RProtoBuf} is used to exchange data
with a system that requires this integer type. To work around this,
\CRANpkg{RProtoBuf} allows users to get and set 64-bit integer values by specifying
them as character strings.

If we try to set an int64 field in \proglang{R} to double values, we lose

\begin{Schunk}
\begin{Sinput}
R> test <- new(protobuf_unittest.TestAllTypes)
R> test$repeated_int64 <- c(2^53, 2^53+1)
R> length(unique(test$repeated_int64))
\end{Sinput}
\begin{Soutput}
[1] 1
\end{Soutput}
\end{Schunk}

But when the values are specified as character strings, \CRANpkg{RProtoBuf}
will automatically coerce them into a true 64-bit integer types
before storing them in the Protocol Buffer message:

\begin{Schunk}
\begin{Sinput}
R> test$repeated_int64 <- c("9007199254740992", "9007199254740993")
\end{Sinput}
\end{Schunk}

When reading the value back into \proglang{R}, numeric types are returned by
default, but when the full precision is required a character value
will be returned if the \code{RProtoBuf.int64AsString} option is set
to \texttt{TRUE}.  The character values are useful because they can
accurately be used as unique identifiers and can easily be passed to \proglang{R}
packages such as \CRANpkg{int64} [Fran{\c{c}}ois, 2011] or \CRANpkg{bit64}
[Oehlschl\"{a}gel, 2012] which represent 64-bit integers in \proglang{R}.

\begin{Schunk}
\begin{Sinput}
R> options("RProtoBuf.int64AsString" = FALSE)
R> test$repeated_int64
\end{Sinput}
\begin{Soutput}
[1] 9.007e+15 9.007e+15
\end{Soutput}
\begin{Sinput}
R> length(unique(test$repeated_int64))
\end{Sinput}
\begin{Soutput}
[1] 1
\end{Soutput}
\begin{Sinput}
R> options("RProtoBuf.int64AsString" = TRUE)
R> test$repeated_int64
\end{Sinput}
\begin{Soutput}
[1] "9007199254740992" "9007199254740993"
\end{Soutput}
\begin{Sinput}
R> length(unique(test$repeated_int64))
\end{Sinput}
\begin{Soutput}
[1] 2
\end{Soutput}
\end{Schunk}

\section[Converting R data structures into Protocol Buffers]{Converting \proglang{R} data structures into Protocol Buffers}
\label{sec:evaluation}

The previous sections discussed functionality in the \CRANpkg{RProtoBuf} package
for creating, manipulating, parsing, and serializing Protocol Buffer
messages of a defined schema.  This is useful when there are
pre-existing systems with defined schemas or significant software
components written in other languages that need to be accessed from
within \proglang{R}.
The package also provides methods for converting arbitrary \proglang{R} data structures into Protocol
Buffers and vice versa with a universal \proglang{R} object schema. The \code{serialize\_pb} and \code{unserialize\_pb}
functions serialize arbitrary \proglang{R} objects into a universal Protocol Buffer

\begin{Schunk}
\begin{Sinput}
R> msg <- serialize_pb(iris, NULL)
R> identical(iris, unserialize_pb(msg))
\end{Sinput}
\begin{Soutput}
[1] TRUE
\end{Soutput}
\end{Schunk}

In order to accomplish this, \CRANpkg{RProtoBuf} uses the same catch-all \texttt{proto}
schema used by \pkg{RHIPE} for exchanging \proglang{R} data with Hadoop [Guha, 2010]. This
schema, which we will refer to as \texttt{rexp.proto}, is printed in
%appendix \ref{rexp.proto}.
the appendix.
The Protocol Buffer messages generated by \CRANpkg{RProtoBuf} and
\pkg{RHIPE} are naturally compatible between the two systems because they use the
same schema. This shows the power of using a schema-based cross-platform format such
as Protocol Buffers: interoperability is achieved without effort or close coordination.

The \texttt{rexp.proto} schema supports all main \proglang{R} storage types holding \emph{data}.
These include \texttt{NULL}, \texttt{list} and vectors of type \texttt{logical},
\texttt{character}, \texttt{double}, \texttt{integer}, and \texttt{complex}. In addition,
every type can contain a named set of attributes, as is the case in \proglang{R}. The \texttt{rexp.proto}
schema does not support some of the special \proglang{R} specific storage types, such as \texttt{function},
\texttt{language} or \texttt{environment}. Such objects have no native equivalent
type in Protocol Buffers, and have little meaning outside the context of \proglang{R}.
When serializing \proglang{R} objects using \texttt{serialize\_pb}, values or attributes of
unsupported types are skipped with a warning. If the user really wishes to serialize these
objects, they need to be converted into a supported type. For example, the  can use
\texttt{deparse} to convert functions or language objects into strings, or \texttt{as.list}
for environments.

\subsection[Evaluation: Converting R data sets]{Evaluation: Converting \proglang{R} data sets}

To illustrate how this method works, we attempt to convert all of the built-in
data sets from \proglang{R} into this serialized Protocol Buffer representation.

\begin{Schunk}
\begin{Sinput}
R> datasets <- as.data.frame(data(package="datasets")$results)
R> datasets$name <- sub("\\s+.*$", "", datasets$Item)
R> n <- nrow(datasets)
\end{Sinput}
\end{Schunk}

There are 206 standard data sets included in the \pkg{datasets}
package included with \proglang{R}. These data sets include data frames, matrices, time series, tables lists,
and some more exotic data classes. The \texttt{can\_serialize\_pb} method is
used to determine which of those can fully be converted to the \texttt{rexp.proto}
Protocol Buffer representation. This method simply checks if any of the values or
attributes in an object is of an unsupported type:

\begin{Schunk}
\begin{Sinput}
R> m <- sum(sapply(datasets$name, function(x) can_serialize_pb(get(x))))
\end{Sinput}
\end{Schunk}

192 data sets can be converted to Protocol Buffers
without loss of information (93\%). Upon closer
inspection, all other data sets are objects of class \texttt{nfnGroupedData}.
This class represents a special type of data frame that has some additional
attributes (such as a \emph{formula} object) used by the \pkg{nlme} package [Pinheiro \emph{et~al.}, 2013].
Because formulas are \proglang{R} \emph{language} objects, they have little meaning to
other systems, and are not supported by the \texttt{rexp.proto} descriptor.
When \texttt{serialize\_pb} is used on objects of this class, it will serialize
the data frame and all attributes, except for the formula.

\begin{Schunk}
\begin{Sinput}
R> attr(CO2, "formula")
\end{Sinput}
\begin{Soutput}
uptake ~ conc | Plant
<environment: R_EmptyEnv>
\end{Soutput}
\begin{Sinput}
R> msg <- serialize_pb(CO2, NULL)
R> object <- unserialize_pb(msg)
R> identical(CO2, object)
\end{Sinput}
\begin{Soutput}
[1] FALSE
\end{Soutput}
\begin{Sinput}
R> identical(class(CO2), class(object))
\end{Sinput}
\begin{Soutput}
[1] TRUE
\end{Soutput}
\begin{Sinput}
R> identical(dim(CO2), dim(object))
\end{Sinput}
\begin{Soutput}
[1] TRUE
\end{Soutput}
\begin{Sinput}
R> attr(object, "formula")
\end{Sinput}
\begin{Soutput}
\end{Soutput}
\end{Schunk}

\subsection{Compression performance}
\label{sec:compression}

This section compares how many bytes are used to store data sets
using four different methods:

\begin{itemize}
\item normal \proglang{R} serialization [Tierney, 2003],
\item \proglang{R} serialization followed by gzip,
\item normal Protocol Buffer serialization, and
\item Protocol Buffer serialization followed by gzip.
\end{itemize}

Table~\ref{tab:compression} shows the sizes of 50 sample \proglang{R} data sets as
returned by object.size() compared to the serialized sizes.
%The summary compression sizes are listed below, and a full table for a
%sample of 50 data sets is included on the next page.
Note that Protocol Buffer serialization results in slightly
smaller byte streams compared to native \proglang{R} serialization in most cases,
but this difference disappears if the results are compressed with gzip.
%Sizes are comparable but Protocol Buffers provide simple getters and setters
%in multiple languages instead of requiring other programs to parse the \proglang{R}
%serialization format. % [Tierney, 2003].
One takeaway from this table is that the universal \proglang{R} object schema
included in \CRANpkg{RProtoBuf} does not in general provide
any significant saving in file size compared to the normal serialization
mechanism in \proglang{R}.
% redundant: which is seen as equally compact.
The benefits of \CRANpkg{RProtoBuf} accrue more naturally in applications where
multiple programming languages are involved, or when a more concise
application-specific schema has been defined.  The example in the next
section satisfies both of these conditions.

% latex table generated in \proglang{R} 3.0.2 by xtable 1.7-0 package
% Fri Dec 27 17:00:03 2013
\begin{table}[h!]
\begin{center}
\small
\scalebox{0.9}{
\begin{tabular}{lrrrrr}
\toprule
Data Set & object.size & \multicolumn{2}{c}{\proglang{R} Serialization} &
\multicolumn{2}{c}{RProtoBuf Serial.} \\
& & default & gzipped & default & gzipped \\
\cmidrule(r){2-6}
uspop & 584 & 268 & 172 & 211 & 148 \\
Titanic & 1960 & 633 & 257 & 481 & 249 \\
volcano & 42656 & 42517 & 5226 & 42476 & 4232 \\
euro.cross & 2728 & 1319 & 910 & 1207 & 891 \\
attenu & 14568 & 8234 & 2165 & 7771 & 2336 \\
ToothGrowth & 2568 & 1486 & 349 & 1239 & 391 \\
lynx & 1344 & 1028 & 429 & 971 & 404 \\
nottem & 2352 & 2036 & 627 & 1979 & 641 \\
sleep & 2752 & 746 & 282 & 483 & 260 \\
co2 & 4176 & 3860 & 1473 & 3803 & 1453 \\
austres & 1144 & 828 & 439 & 771 & 410 \\
ability.cov & 1944 & 716 & 357 & 589 & 341 \\
EuStockMarkets & 60664 & 59785 & 21232 & 59674 & 19882 \\
treering & 64272 & 63956 & 17647 & 63900 & 17758 \\
freeny.x & 1944 & 1445 & 1311 & 1372 & 1289 \\
Puromycin & 2088 & 813 & 306 & 620 & 320 \\
warpbreaks & 2768 & 1231 & 310 & 811 & 343 \\
BOD & 1088 & 334 & 182 & 226 & 168 \\
sunspots & 22992 & 22676 & 6482 & 22620 & 6742 \\
beaver2 & 4184 & 3423 & 751 & 3468 & 840 \\
anscombe & 2424 & 991 & 375 & 884 & 352 \\
esoph & 5624 & 3111 & 548 & 2240 & 665 \\
PlantGrowth & 1680 & 646 & 303 & 459 & 314 \\
infert & 15848 & 14328 & 1172 & 13197 & 1404 \\
BJsales & 1632 & 1316 & 496 & 1259 & 465 \\
stackloss & 1688 & 917 & 293 & 844 & 283 \\
crimtab & 7936 & 4641 & 713 & 1655 & 576 \\
LifeCycleSavings & 6048 & 3014 & 1420 & 2825 & 1407 \\
Harman74.cor & 9144 & 6056 & 2045 & 5861 & 2070 \\
nhtemp & 912 & 596 & 240 & 539 & 223 \\
faithful & 5136 & 4543 & 1339 & 4936 & 1776 \\
freeny & 5296 & 2465 & 1518 & 2271 & 1507 \\
discoveries & 1232 & 916 & 199 & 859 & 180 \\
state.x77 & 7168 & 4251 & 1754 & 4068 & 1756 \\
pressure & 1096 & 498 & 277 & 427 & 273 \\
fdeaths & 1008 & 692 & 291 & 635 & 272 \\
euro & 976 & 264 & 186 & 202 & 161 \\
LakeHuron & 1216 & 900 & 420 & 843 & 404 \\
mtcars & 6736 & 3798 & 1204 & 3633 & 1206 \\
precip & 4992 & 1793 & 813 & 1615 & 815 \\
state.area & 440 & 422 & 246 & 405 & 235 \\
attitude & 3024 & 1990 & 544 & 1920 & 561 \\
randu & 10496 & 9794 & 8859 & 10441 & 9558 \\
state.name & 3088 & 844 & 408 & 724 & 415 \\
airquality & 5496 & 4551 & 1241 & 2874 & 1294 \\
airmiles & 624 & 308 & 170 & 251 & 148 \\
quakes & 33112 & 32246 & 9898 & 29063 & 11595 \\
islands & 3496 & 1232 & 563 & 1098 & 561 \\
OrchardSprays & 3600 & 2164 & 445 & 1897 & 483 \\
WWWusage & 1232 & 916 & 274 & 859 & 251 \\
\bottomrule
%  Total & 391176 & 327537 & 99161 & 313456 & 100308 \\
Relative Size & 100\% & 83.7\% & 25.3\% & 80.1\% & 25.6\%\\
\bottomrule
\end{tabular}
\caption{Serialization sizes for default serialization in \proglang{R} and
\CRANpkg{RProtoBuf} for 50 \proglang{R} data sets.}
\label{tab:compression}
\end{center}
\end{table}

\section{Application: Distributed data collection with MapReduce}
\label{sec:mapreduce}

Many large data sets in fields such as particle physics and information
processing are stored in binned or histogram form in order to reduce
the data storage requirements [Scott, 2009].  In the
last decade, the MapReduce programming model [Dean and Ghemawat, 2008]
has emerged as a popular design pattern that enables the processing of
very large data sets on large compute clusters.

Many types of data analysis over large data sets may involve very rare
phenomenon or deal with highly skewed data sets or inflexible
raw data storage systems from which unbiased sampling is not feasible.
In such situations, MapReduce and binning may be combined as a
pre-processing step for a wide range of statistical and scientific
analyses [Blocker and Meng, 2013].

There are two common patterns for generating histograms of large data
sets in a single pass with MapReduce.  In the first method, each
mapper task generates a histogram over a subset of the data that it
has been assigned, serializes this histogram and sends it to one or
more reducer tasks which merge the intermediate histograms from the

In the second method, illustrated in
Figure~\ref{fig:mr-histogram-pattern1}, each mapper rounds a data
point to a bucket width and outputs that bucket as a key and '1' as a
value.  Reducers then sum up all of the values with the same key and
output to a data store.

\begin{figure}[h!]
\begin{center}
\includegraphics[width=\textwidth]{figures/histogram-mapreduce-diag1.pdf}
\end{center}
\caption{Diagram of MapReduce histogram generation pattern}
\label{fig:mr-histogram-pattern1}
\end{figure}

In both methods, the mapper tasks must choose identical bucket
boundaries in advance if we are to construct the histogram in a single
pass, even though they are analyzing disjoint parts of the input set
that may cover different ranges.  All distributed tasks involved in
the pre-processing as well as any downstream data analysis tasks must
share a schema of the histogram representation to coordinate

The \CRANpkg{HistogramTools} package [Stokely, 2013] enhances
\CRANpkg{RProtoBuf} by providing a concise schema for \proglang{R} histogram objects:

\begin{example}
package HistogramTools;

message HistogramState {
repeated double breaks = 1;
repeated int32 counts = 2;
optional string name = 3;
\end{example}

This HistogramState message type is designed to be helpful if some of
the Map or Reduce tasks are written in \proglang{R}, or if those components are
written in other languages and only the resulting output histograms
need to be manipulated in \proglang{R}.  For example, to create HistogramState
messages in Python for later consumption by \proglang{R}, we first compile the
\texttt{histogram.proto} descriptor into a python module using the
\texttt{protoc} compiler:

\begin{verbatim}
protoc histogram.proto --python_out=.
\end{verbatim}
This generates a Python module called \texttt{histogram\_pb2.py}, containing both the
descriptor information as well as methods to read and manipulate the histogram
message data.  The following simple Python script uses this generated
module to create a histogram and write out the Protocol Buffer
representation to a file:

\begin{Code}
from histogram_pb2 import HistogramState;

# Create empty Histogram message
hist = HistogramState()

# Add breakpoints and binned data set.
hist.counts.extend([2, 6, 2, 4, 6])
hist.name="Example Histogram Created in Python"

# Output the histogram
outfile = open("/tmp/hist.pb", "wb")
\end{Code}

The Protocol Buffer can then be read into \proglang{R} and converted to a native
\proglang{R} histogram object for plotting:

\begin{Code}

# Read the Histogram schema

# Read the serialized histogram file.
hist <- HistogramTools.HistogramState$read("/tmp/hist.pb")
[1] "message of type 'HistogramTools.HistogramState' with 3 fields set"

# Convert to native R histogram object and plot
\end{Code}

\begin{center}
\includegraphics{figures/fig-034}
\end{center}

One of the authors has used this design pattern for several
large-scale studies of distributed storage systems
[Stokely \emph{et~al.}, 2012, Albrecht \emph{et~al.}, 2013].

\section{Application: Data Interchange in web Services}
\label{sec:opencpu}

As described earlier, the primary application of Protocol Buffers is data
interchange in the context of inter-system communications.  Network protocols
such as HTTP provide mechanisms for client-server communication, i.e., how to
initiate requests, authenticate, send messages, etc.  However, network
protocols generally do not regulate the \emph{content} of messages: they
allow transfer of any media type, such as web pages, static files or
multimedia content.  When designing systems where various components require
exchange of specific data structures, we need something on top of the network
protocol that prescribes how these structures are to be represented in
messages (buffers) on the network. Protocol Buffers solve exactly this
problem by providing a cross-platform method for serializing arbitrary
structures into well defined messages, which can then be exchanged using any
protocol. The descriptors (\texttt{.proto} files) are used to formally define
the interface of a remote API or network application. Libraries to parse and
generate protobuf messages are available for many programming languages,
making it relatively straightforward to implement clients and servers.

\subsection[Interacting with R through HTTPS and Protocol Buffers]{Interacting with \proglang{R} through HTTPS and Protocol Buffers}

One example of a system that supports Protocol Buffers to interact
with \proglang{R} is OpenCPU [Ooms, 2013]. OpenCPU is a framework for embedded statistical
computation and reproducible research based on \proglang{R} and \LaTeX. It exposes a
HTTP(S) API to access and manipulate \proglang{R} objects and allows for performing
remote \proglang{R} function calls. Clients do not need to understand
or generate any \proglang{R} code: HTTP requests are automatically mapped to
function calls, and arguments/return values can be posted/retrieved
using several data interchange formats, such as Protocol Buffers.
OpenCPU uses the \texttt{serialize\_pb} and \texttt{unserialize\_pb} functions
from the \CRANpkg{RProtoBuf} package to convert between \proglang{R} objects and protobuf
messages. Therefore, clients need the \texttt{rexp.proto} descriptor mentioned
earlier to parse and generate protobuf messages when interacting with OpenCPU.

\subsection[HTTP GET: Retrieving an R object]{HTTP GET: Retrieving an \proglang{R} object}

The \texttt{HTTP GET} method is used to read a resource from OpenCPU. For example,
to access the data set \texttt{Animals} from the package \texttt{MASS}, a
client performs the following HTTP request:

\begin{verbatim}
GET https://public.opencpu.org/ocpu/library/MASS/data/Animals/pb
\end{verbatim}
The postfix \texttt{/pb} in the URL tells the server to send this
object in the form of a protobuf message. Alternative formats include
\texttt{/json}, \texttt{/csv}, \texttt{/rds} and others. If the request
is successful, OpenCPU returns the serialized object with HTTP status
code 200 and HTTP response header \texttt{Content-Type: application/x-protobuf}.
The latter is the conventional MIME type that formally notifies the client to
interpret the response as a protobuf message.

Because both HTTP and Protocol Buffers have libraries available for many
languages, clients can be implemented in just a few lines of code. Below
is example code for both \proglang{R} and Python that retrieves a data set from \proglang{R} with
OpenCPU using a protobuf message. In \proglang{R}, we use the HTTP client from
the \texttt{httr} package [Wickham, 2012]. In this example we
download a data set which is part of the base \proglang{R} distribution, so we can
verify that the object was transferred without loss of information.

\begin{Schunk}
\begin{Sinput}
R> # Load packages
R> library(RProtoBuf)
R> library(httr)
R> # Retrieve and parse message
R> req <- GET('https://public.opencpu.org/ocpu/library/MASS/data/Animals/pb')
R> output <- unserialize_pb(req$content)
R> # Check that no information was lost
R> identical(output, MASS::Animals)
\end{Sinput}
\end{Schunk}

This code suggests a method for exchanging objects between \proglang{R} servers, however this might as
well be done without Protocol Buffers. The main advantage of using an inter-operable format
is that we can actually access \proglang{R} objects from within another
programming language. For example, in a very similar fashion we can retrieve the same
data set in a Python client. To parse messages in Python, we first compile the
\texttt{rexp.proto} descriptor into a python module using the \texttt{protoc} compiler:

\begin{verbatim}
protoc rexp.proto --python_out=.
\end{verbatim}
This generates Python module called \texttt{rexp\_pb2.py}, containing both the
descriptor information as well as methods to read and manipulate the \proglang{R} object
message. In the example below we use the HTTP client from the \texttt{urllib2}

\begin{verbatim}
# Import modules
import urllib2
from rexp_pb2 import REXP

# Retrieve message
req = urllib2.Request('https://public.opencpu.org/ocpu/library/MASS/data/Animals/pb')
res = urllib2.urlopen(req)

# Parse rexp.proto message
msg = REXP()
\end{verbatim}
The \texttt{msg} object contains all data from the Animals data set. From here we
can easily extract the desired fields for further use in Python.

\subsection[HTTP POST: Calling an R function]{HTTP POST: Calling an \proglang{R} function}

The example above shows how the \texttt{HTTP GET} method retrieves a
resource from OpenCPU, for example an \proglang{R} object. The \texttt{HTTP POST}
method on the other hand is used for calling functions and running scripts,
which is the primary purpose of the framework. As before, the \texttt{/pb}
postfix requests to retrieve the output as a protobuf message, in this
case the function return value. However, OpenCPU allows us to supply the
arguments of the function call in the form of protobuf messages as well.
This is a bit more work, because clients needs to both generate messages
containing \proglang{R} objects to post to the server, as well as retrieve and parse
protobuf messages returned by the server. Using Protocol Buffers to post
function arguments is not required, and for simple (scalar) arguments
the standard \texttt{application/x-www-form-urlencoded} format might be sufficient.
However, with Protocol Buffers the client can perform function calls with
more complex arguments such as \proglang{R} vectors or lists. The result is a complete
RPC system to do arbitrary \proglang{R} function calls from within
any programming language.

The following example \proglang{R} client code performs the remote function call
\texttt{stats::rnorm(n=42, mean=100)}. The function arguments (in this
case \texttt{n} and \texttt{mean}) as well as the return value (a vector
with 42 random numbers) are transferred using a protobuf message. RPC in
OpenCPU works like the \texttt{do.call} function in \proglang{R}, hence all arguments
are contained within a list.

\begin{Schunk}
\begin{Sinput}
R> #requires httr >= 0.2.99
R> library(httr)
R> library(RProtoBuf)
R> args <- list(n=42, mean=100)
R> payload <- serialize_pb(args, NULL)
R> req <- POST (
+    url = "https://public.opencpu.org/ocpu/library/stats/R/rnorm/pb",
+    body = payload,
+    add_headers (
+      "Content-Type" = "application/x-protobuf"
+    )
+  )
R> #This is the output of stats::rnorm(n=42, mean=100)
R> output <- unserialize_pb(req$content)
R> print(output)
\end{Sinput}
\end{Schunk}
The OpenCPU server basically performs the following steps to process the above RPC request:

\begin{Schunk}
\begin{Sinput}
R> fnargs <- unserialize_pb(inputmsg)
R> val <- do.call(stats::rnorm, fnargs)
R> outputmsg <- serialize_pb(val)
\end{Sinput}
\end{Schunk}

\section{Summary}
\label{sec:summary}
Over the past decade, many formats for interoperable
data exchange have become available, each with their unique features,
strengths and weaknesses.
Text based formats such as \texttt{CSV} and \texttt{JSON} are easy to use, and will likely
remain popular among statisticians for many years to come. However, in the
context of increasingly complex analysis stacks and applications involving
distributed computing as well as mixed language analysis pipelines, choosing a more
sophisticated data interchange format may reap considerable benefits.
%Protocol Buffers is itself not a protocol.
%Forward-compatibility is one of the features. No need to re-iterate those
The Protocol Buffers standard and library offer a unique combination of features,
performance, and maturity, that seems particularly well suited for data-driven
applications and numerical computing.

The \CRANpkg{RProtoBuf} package builds on the Protocol Buffers \proglang{C++} library,
and extends the \proglang{R} system with the ability to create, read,
write, parse, and manipulate Protocol
Buffer messages. \CRANpkg{RProtoBuf} has been used extensively inside Google
for the past three years by statisticians, analysts, and software engineers.
At the time of this writing there are over 300 active
users of \CRANpkg{RProtoBuf} using it to read data from and otherwise interact
with distributed systems written in \proglang{C++}, \proglang{Java}, \proglang{Python}, and
other languages. We hope that making Protocol Buffers available to the
\proglang{R} community will contribute towards better software integration
and allow for building even more advanced applications and analysis pipelines
with \proglang{R}.

\section*{Acknowledgments}

The first versions of \CRANpkg{RProtoBuf} were written during 2009-2010.
Very significant contributions, both in code and design, were made by
Romain Fran\c{c}ois whose continued influence on design and code is
greatly appreciated. Several features of the package reflect
the design of the \CRANpkg{rJava} package by Simon Urbanek.
The user-defined table mechanism, implemented by Duncan Temple Lang for the
purpose of the \pkg{RObjectTables} package, allows for the dynamic symbol lookup.
Kenton Varda was generous with his time in reviewing code and explaining
obscure Protocol Buffer semantics.  Karl Millar was very
helpful in reviewing code and offering suggestions.  Saptarshi Guha's
work on RHIPE and implementation of a universal message type for \proglang{R}
language objects allowed us to add the \code{serialize_pb} and \code{unserialize_pb}
methods for turning arbitrary R objects into Protocol Buffers without
a specialized pre-defined schema.

\newpage
\appendix
\setcounter{secnumdepth}{0}

\section*{Appendix: The rexp.proto schema descriptor}
\label{rexp.proto}

Below a print of the \texttt{rexp.proto} schema (originally designed by [Guha, 2010])
that is included with the \CRANpkg{RProtoBuf} package and used by \texttt{serialize\_pb} and
\texttt{unserialize\_pb}.

\begin{verbatim}
package rexp;

message REXP {
enum RClass {
STRING = 0;
RAW = 1;
REAL = 2;
COMPLEX = 3;
INTEGER = 4;
LIST = 5;
LOGICAL = 6;
NULLTYPE = 7;
enum RBOOLEAN {

required RClass rclass = 1 ;
repeated double realValue = 2 [packed=true];
repeated sint32 intValue = 3 [packed=true];
repeated RBOOLEAN booleanValue = 4;
repeated STRING stringValue = 5;
optional bytes rawValue = 6;
repeated CMPLX complexValue = 7;
repeated REXP rexpValue = 8;
repeated string attrName = 11;
repeated REXP attrValue = 12;
message STRING {
optional string strval = 1;
optional bool isNA = 2 [default=false];
message CMPLX {
optional double real = 1 [default=0];
required double imag = 2;
\end{verbatim}
% \end{appendices}
\newpage
\begin{thebibliography}{26}
\newcommand{\enquote}[1]{``#1''}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi:\discretionary{}{}{}#1}\else
\providecommand{\doi}{doi:\discretionary{}{}{}\begingroup
\urlstyle{rm}\Url}\fi
\providecommand{\eprint}[2][]{\url{#2}}

[Albrecht \emph{et~al.}, 2013]
Albrecht C, Merchant A, Stokely M, Waliji M, Labelle F, Coehlo N, Shi X,
Schrock E (2013).
\newblock \enquote{Janus: Optimal Flash Provisioning for Cloud Storage
\newblock In \emph{Proceedings of the USENIX Annual Technical Conference}, pp.
91--102. 2560 Ninth Street, Suite 215, Berkeley, CA 94710, USA.
\newblock
\urlprefix\url{https://www.usenix.org/system/files/conference/atc13/atc13-albrecht.pdf}.

[Blocker and Meng, 2013]
Blocker AW, Meng XL (2013).
\newblock \enquote{The potential and perils of preprocessing: Building new
\newblock \emph{Bernoulli}, \textbf{19}(4), 1176--1211.
\newblock \doi{10.3150/13-BEJSP16}.
\newblock \urlprefix\url{http://dx.doi.org/10.3150/13-BEJSP16}.

[Cline, 2013]
Cline M (2013).
\newblock \enquote{C++ faq.}
\newblock \emph{Also available as http://www. parashift.
com/c++-faq-lite/index. html}.

[Couture-Beil, 2012]
Couture-Beil A (2012).
\newblock \emph{rjson: JSON for R}.
\newblock R package version 0.2.10,
\urlprefix\url{http://CRAN.R-project.org/package=rjson}.

[Dean and Ghemawat, 2008]
Dean J, Ghemawat S (2008).
\newblock \enquote{MapReduce: simplified data processing on large clusters.}
\newblock \emph{Communications of the ACM}, \textbf{51}(1), 107--113.

[Eddelbuettel, 2013]
Eddelbuettel D (2013).
\newblock \emph{Seamless R and C++ Integration with Rcpp}.
\newblock Springer.

[Eddelbuettel and Fran{\c{c}}ois, 2011]
Eddelbuettel D, Fran{\c{c}}ois R (2011).
\newblock \enquote{Rcpp: Seamless R and C++ integration.}
\newblock \emph{Journal of Statistical Software}, \textbf{40}(8), 1--18.

[Eddelbuettel and Fran{\c{c}}ois, 2013]
Eddelbuettel D, Fran{\c{c}}ois R (2013).
\newblock \emph{Exposing C++ functions and classes with Rcpp modules}.
\newblock Vignette included in R package Rcpp,
\urlprefix\url{http://CRAN.R-project.org/package=Rcpp}.

[Fran{\c{c}}ois, 2011]
Fran{\c{c}}ois R (2011).
\newblock \emph{int64: 64 bit integer types}.
\newblock R package version 1.1.2,
\urlprefix\url{http://CRAN.R-project.org/package=int64}.

[Google, 2012]
Google (2012).
\newblock \emph{Protocol Buffers: Developer Guide}.
\newblock
\urlprefix\url{http://code.google.com/apis/protocolbuffers/docs/overview.html}.

[Guha, 2010]
Guha S (2010).
\newblock \emph{RHIPE: A Distributed Environment for the Analysis of Large and
Complex Datasets}.
\newblock \urlprefix\url{http://www.stat.purdue.edu/~sguha/rhipe/}.

[Nolan and {Temple Lang}, 2013]
Nolan D, {Temple Lang} D (2013).
\newblock \emph{XML and Web Technologies for Data Sciences with R}.
\newblock Springer.

[Oehlschl\"{a}gel, 2012]
Oehlschl\"{a}gel J (2012).
\newblock \emph{bit64: A S3 class for vectors of 64bit integers}.
\newblock R package version 0.9-3,
\urlprefix\url{http://CRAN.R-project.org/package=bit64}.

[Ooms, 2013]
Ooms J (2013).
\newblock \emph{OpenCPU system for embedded statistical computation and
reproducible research}.
\newblock R package version 1.2.2, \urlprefix\url{http://www.opencpu.org}.

[Ooms, 2014]
Ooms J (2014).
\newblock \emph{jsonlite: A smarter JSON encoder/decoder for R}.
\newblock R package version 0.9.4,
\urlprefix\url{http://github.com/jeroenooms/jsonlite#readme}.

[Pinheiro \emph{et~al.}, 2013]
Pinheiro J, Bates D, DebRoy S, Sarkar D, {EISPACK authors}, {R Core} (2013).
\newblock \emph{nlme: Linear and Nonlinear Mixed Effects Models}.
\newblock R package version 3.1-113,
\urlprefix\url{http://CRAN.R-project.org/package=nlme}.

[Scott, 2009]
Scott DW (2009).
\newblock \emph{Multivariate density estimation: theory, practice, and
visualization}, volume 383.
\newblock Wiley. com.

[Shafranovich, 2005]
Shafranovich Y (2005).
\newblock \enquote{Common format and mime type for comma-separated values (csv)
\newblock \urlprefix\url{http://tools.ietf.org/html/rfc4180}.

[Stokely, 2013]
Stokely M (2013).
\newblock \emph{HistogramTools: Utility Functions for R Histograms}.
\newblock R package version 0.3,
\urlprefix\url{https://r-forge.r-project.org/projects/histogramtools/}.

[Stokely \emph{et~al.}, 2012]
Stokely M, Mehrabian A, Albrecht C, Labelle F, Merchant A (2012).
\newblock \enquote{Projecting Disk Usage Based on Historical Trends in a Cloud
\newblock In \emph{ScienceCloud 2012 Proceedings of the 3rd International
Workshop on Scientific Cloud Computing}, pp. 63--70.

[Sumaray and Makki, 2012]
Sumaray A, Makki SK (2012).
\newblock \enquote{A Comparison of Data Serialization Formats for Optimal
Efficiency on a Mobile Platform.}
\newblock In \emph{Proceedings of the 6th International Conference on
Ubiquitous Information Management and Communication}, ICUIMC '12, pp.
48:1--48:6. ACM, New York, NY, USA.
\newblock ISBN 978-1-4503-1172-4.
\newblock \doi{10.1145/2184751.2184810}.
\newblock \urlprefix\url{http://doi.acm.org/10.1145/2184751.2184810}.

[Temple Lang}, 2011]
{Temple Lang} D (2011).
\newblock \emph{RJSONIO: Serialize R objects to JSON, JavaScript Object
\newblock R package version 0.96-0,
\urlprefix\url{http://CRAN.R-project.org/package=RJSONIO}.

[Temple Lang}, 2012]
{Temple Lang} D (2012).
\newblock \emph{User-Defined Tables in the R Search Path}.
\newblock
\urlprefix\url{http://www.omegahat.org/RObjectTables/RObjectTables.pdf}.

[Tierney, 2003]
Tierney L (2003).
\newblock \enquote{A New Serialization Mechanism for R.}
\newblock
\urlprefix\url{http://www.cs.uiowa.edu/~luke/R/serialize/serialize.ps}.

[Wegiel and Krintz, 2010]
Wegiel M, Krintz C (2010).
\newblock \enquote{Cross-language, Type-safe, and Transparent Object Sharing
for Co-located Managed Runtimes.}
\newblock \emph{SIGPLAN Not.}, \textbf{45}(10), 223--240.
\newblock ISSN 0362-1340.
\newblock \doi{10.1145/1932682.1869479}.
\newblock \urlprefix\url{http://doi.acm.org/10.1145/1932682.1869479}.

[Wickham, 2012]
Wickham H (2012).
\newblock \emph{httr: Tools for working with URLs and HTTP}.
\newblock R package version 0.2,
\urlprefix\url{http://CRAN.R-project.org/package=httr}.

\end{thebibliography}

\end{document}


# SAT-NGP : Unleashing Neural Graphics Primitives for Fast Relightable
Transient-Free 3D reconstruction from Satellite Imagery

###### Abstract

Current stereo-vision pipelines produce high accuracy 3D reconstruction when
using multiple pairs or triplets of satellite images. However, these pipelines
are sensitive to the changes between images that can occur as a result of
multi-date acquisitions. Such variations are mainly due to variable shadows,
reflexions and transient objects (cars, vegetation). To take such changes into
account, Neural Radiance Fields (NeRF) have recently been applied to multi-
date satellite imagery. However, Neural methods are very compute-intensive,
taking dozens of hours to learn, compared with minutes for standard stereo-
vision pipelines. Following the ideas of Instant Neural Graphics Primitives we
propose to use an efficient sampling strategy and multi-resolution hash
encoding to accelerate the learning. Our model, Satellite Neural Graphics
Primitives (SAT-NGP) decreases the learning time to 15 minutes while
maintaining the quality of the 3D reconstruction.

Index Terms—  Neural Radiance Fields, Neural Graphics Primitives, 3D
Reconstruction, Satellite imagery, Transient objects, Relighting

Fig. 1: Compared to the ground truth LiDAR and unseen WorldView-3 image (top),
SAT-NGP (middle) yields more accurate surface extraction and similar novel
views in 14 minutes, as SAT-NeRF (bottom) in 12 hours.

##  1 Introduction

The way we perceive and understand our planet has been revolutionised by the
advent of satellite imagery. In fields as diverse as meteorology, biodiversity
conservation and climate change, satellite imagery has become an invaluable
source of information. Accurate knowledge of 3D surface elements is critical
in several domains, including flood risk mitigation, vegetation monitoring,
urban planning, etc. State-of-the-art stereo-vision techniques [1] provide
large-scale 2.5D Digital Surface Models (DSM) with sufficient accuracy for
many applications. When applied to a single pair or triplet of satellite
images, these methods suffer from occlusions, homogeneous/poorly structured
areas, shadow zones, and reflective surfaces (like water or metal structures).
Such errors can be mitigated by merging the 3D models from several
pairs/triplets acquired at different dates. Nonetheless, the accuracy of these
methods is negatively impacted by changes in the contents of the scene (mobile
objects, vegetation) and variations in lighting conditions. In the field of
computer vision, a new concept has emerged called Neural Radiance Field [2]
(NeRF), which has already shown remarkable results in the synthesis of novel
views and 3D reconstruction from multi-date satellite imagery [3, 4, 5, 6].

NeRF [2] represents the 3D scene as a continuous radiance field, encoded in a
feed-forward Multi-Layer Perceptron (MLP). Novel views are synthesised by
alpha-compositing radiance values (queried from the NeRF) at coordinates and
viewing directions along camera rays. Previous works [3, 4, 5, 6] applied to
remote sensing have focused on unique challenges linked to the complexity of
the scene, namely changes in lighting conditions, satellite sensor geometry,
and transient objects. Nonetheless, retraining an entire neural network on
each scene constrains the scalability to large-scale areas, Indeed, state of
the art stereo-vision pipelines such as CARS [1]) produce relatively accurate
DSMs with 40 000 $\times$ 40 000 pixel images in less than an hour, using only
few parallel CPUs. To address these challenges, we combine previous works on
the satellite image model of NeRF [3, 4] with the acceleration brought by
Instant Neural Graphics Primitives (I-NGP) [7]. Our method, Satellite Neural
Graphics Primitives (SAT-NGP) reduces the time needed to extract a 3D model of
a terrestrial scene from satellite images from 24 hours to less than 15
minutes, without compromising the quality of the reconstruction.

##  2 Related works

###  2.1 NeRF for Satellite imagery

An important concern when applying NeRF to multi-date satellite images is the
changes in lighting conditions. Shadow NeRF (S-NeRF) [3] uses the solar angles
to learn the amount of light reaching each point in the scene, and achieves
more reliable modelling of shadow areas than with a NeRF model alone. SAT-NeRF
[4] learns the transient objects (cars, etc.) present in each view with a
similar approach to NeRF in the Wild [8] which introduces an uncertainty
coefficient. This coefficient predicts for each point on the ray whether that
point corresponds to a transient object or not. Earth Observation NeRF (EO-
NeRF) [6] adds geometrically consistent shading, which provides realistic
shadows and relighting capabilities. The adaptations from [3, 4] generate DSMs
similar in terms of altimetric accuracy to the state-of-the-art stereo-vision
pipelines (CARS [1]), as is demonstrated in Table 1. However, training a NeRF
for each scene is slow due to the immense number of inferences that are
required for the network to converge. This issue is made worse by the
complexity of learning from multi-date satellite images. Taking into account
shading effects and transient objects increases the complexity of the
rendering, and the use of multiple competing losses tends to slow down
convergence.

###  2.2 NeRF Training and Inference acceleration

Fig. 2: Comparison between ground truth unseen view (left column) and NVS
(right column). We show transient-free (top two row) and relightable
capabilities on NVS.

The authors of [7] have accelerated NeRF using a multi-resolution hash-table
that stores features decoded by a smaller, faster neural network. In addition,
the samples along each ray are concentrated near the surface, using a voxel
occupancy grid updated and kept in cache during training. A concurrent work
RS-NeRF [9] also offers acceleration based on I-NGP. However, it does not
provide relighting capabilities and handles transient objects with an
inpainting method based on a pretrained network.

##  3 METHODOLOGY

Our work focuses on speeding up the previous versions of NeRF applied to
satellite imagery [3, 4] using the principles of I-NGP [7]. Our goal is to
achieve the same qualitative results in the Novel View Synthesis (NVS) and
DSMs generation. We consider the same experimental setting as SAT-NeRF. First
we use the solar angles as a network input in order to model shading effects.
Second, we learn an uncertainty image as a network output, based on a latent
time vector, to constrain the loss to focus on areas without transient
objects. Finally, the sensor models of the different acquisitions are refined
using bundle-adjustment to reduce inaccuracies between models. We use the
Universal Transverse Mercator (UTM) based representation of geographic 3D
point coordinates as in [6] instead of using an Earth-Centered Earth-Fixed
reference. This is particularly helpful for DSM generation since $z$ axis
corresponds to the altitude in UTM.

###  3.1 Encoding

Previous works employ either sinusoidal networks [3] or a frequency-based
positional encoding [4, 6]. Instead, we follow [7] and use a multi-resolution
hash encoding which allows us to work with smaller neural networks. The linear
interpolation of features learned and encoded in a hash table is more
computationally efficient than querying a large neural network.

###  3.2 Architecture

Our model is based on the architecture in [4] and consists of the same first
small MLP with 2 hidden layers of 64 neurons instead of 8 layers and 512
neurons. We evaluate MISH [10] activation function on hidden layers over SIREN
used in SAT-NeRF. MISH is a smooth, continuous, self regularized and non-
monotonic activation function which is, unlike ReLU, continuously
differentiable. We also explore the use of Spherical Harmonics (SH) to encode
solar directions, a technique not employed in previous studies [3, 4].

###  3.3 Loss

Modeling transient objects with a variable learned during training remains a
proven approach with [8] but comes at an additional computational cost. The
authors of [11] promote a loss function based on the principle of robust
estimation for training a NeRF in a setting with transient perturbations. This
is achieved by modeling the transient objects as outliers during the NeRF
model training. This method assumes no a priori knowledge and focuses on the
optimisation problem rather than on pre-processing or modeling transient
objects.

We define our loss 1 as :

$\displaystyle\mathcal{L}_{final}=\mathcal{L}_{robust}+\lambda\mathcal{L}_{solar}$
(1)

a linear sum of the robust loss 3 :

$\displaystyle\mathcal{L}_{rgb}=\sum_{r\in\mathcal{R}}\big{|}\big{|}\mathbf{C}(r)-\mathbf{C}_{\small
GT}(r)\big{|}\big{|}^{2}_{2}$ (2)

$\displaystyle\mathcal{L}_{robust}=\omega(\mathcal{L}_{rgb}^{t-1})\cdot\mathcal{L}_{rgb}^{t}$
(3)

with $\omega(\cdot)$ the weighted function detailed in equation (10) of [11].
And the solar correction 4 of [3] :

$\displaystyle\mathcal{L}_{solar}=\displaystyle\sum_{r\in\mathcal{R_{SC}}}\left(\displaystyle\sum\limits_{i=1}^{N_{SC}}\left(T_{i}-s_{i}\right)^{2}+1-\displaystyle\sum\limits_{i=1}^{N_{SC}}T_{i}\alpha_{i}s_{i}\right)$
(4)

weighted by a $\lambda$ factor of 0.05. With the first term, the illuminance
factor $s_{i}$, predicted at point $i$, must be equal to the transmittance
$T_{i}$. The second term encourages all direct light to be absorbed by the
scene.

###  3.4 Implementation details

We use orthogonal initialization as described in [12], which in our
experiments seems to enhance gradient stability during training. For each
test, we used the RAdam Optimizer with a learning rate of $0.01$ using the
LambdaLR scheduler. The batch size is 1024 rays with a number of samples
between 4 and 256 points per ray (see Section 2.2). The multi-resolution
encoding is driven by five main parameters as presented in [7]. After
hyperparameter tuning we have chosen : a hash table of size $2^{19}$, 8
levels, a grid size of 128 and coarsest resolution of 16. NeRF experiments
were done using a 12GB GPU and the stere-ovision pipeline CARS on 4 parallel
CPU.

##  4 EXPERIMENTS AND ANALYSIS

The main objective of the experiments is to measure whether the acceleration
comes at the cost of a loss in quality of novel view synthesis or the
geometric quality of the DSM. We want to know whether our model stands close
to the state-of-the-art stereo-vision pipelines, while retaining the high 3D
rendering quality and novel view synthesis of NeRF methods.

###  4.1 Datasets and ground truth details

The experiments are conducted using the Data Fusion Contest (DFC2019)
dataset111https://github.com/pubgeo/dfc2019 which is a 3d reconstruction
featuring satellite images of Jacksonville (JAX) taken by World-View 3 with
0.3m/pixel resolution over a year, used in [4, 6, 3]. An airborne LiDAR DSM
with a ground sampling distance of 0.5m is used as ground truth for the 3d
reconstruction.

###  4.2 Evaluation Metrics

In the following experiments, two metrics are used. The Peak Signal Noise
Ratio (PSNR) is measured on images generated from viewing angles that are
absent from the training dataset. It should be noted that in the case of EO-
NeRF, the PSNR may be higher due to the fact that their evaluation is based on
the training images. The values were taken from their publication as their
code is private. On the other hand, the Mean Absolute Error (MAE) quantifies
the average error of surface elevation prediction in relation to the ground
truth measured by each rasterized LiDAR point.

$0$$1$$2$$3$$4$$5$$0$$1$$2$$4$$6$$8$14 min28 min39 min46 min58 min3 min6 min8
min11 min14 minEpochsMAE in metersSAT-NeRFSAT-NGP (ours) Fig. 3: MAE evolution
during the first 5 epochs with the training time of each one of SAT-NeRF
compared to our method.

###  4.3 Results

Table 1 shows that SAT-NGP is significantly faster than compared to other NeRF
variants, while being slower than CARS. This is due to the higher number of
images (10-18 instead of 2-3).

In terms of 3D reconstruction, Figure 3 demonstrates that our method already
converges to a lower MAE score in the initial epochs. The quality of the DSM
is further shown in Figure 1, which compares the airborne LiDAR, with the DSM
of SAT-NGP and SAT-NeRF after convergence. The DSM produced by our method is
devoid of any holes or bumps on the car-park. We attribute this to the use of
robust loss which provides a smoother inductive bias when combined with MISH
and RAdam.

Finally we observe that the novel views generated at unseen viewing and solar
angles by SAT-NGP are less accurate than the slower NeRF variants. The large
difference with EO-NeRF can be due to the fact that they compute the test
scores on images have already been seen during the training unlike S-NeRF,
SAT-NeRF and ours.

Our PSNR values are similar or slightly worse than S-NeRF and SAT-NeRF. This
can be attributed to the darker shadows and a smoother rendering. Moreover,
our method not only recovers the 3D scene but also eliminates the transient
objects from the NVS as shown in Figure 2.

PSNR ↑ / MAE ↓ / TIME ↓ Area index 004 214 S-NeRF [3] 26.14 / 1.472 / 10h
24.93 / 2.406 / 20h SAT-NeRF [4] 26.67 / 1.288 $\bullet$ / 10h 25.50 / 2.009
$\bullet$ / 20h EO-NeRF [6] 28.56 $\bullet$ / 1.25 $\bullet$ / 10h 26.59
$\bullet$ / 1.52 $\bullet$ / 20h CARS [1] \- / - / - \- / 3.24 / 3min
$\bullet$ SAT-NGP (ours) 25.03 / 1.31 $\bullet$ / 8min $\bullet$ 23.43 / 2.17
$\bullet$ / 14min $\bullet$

PSNR ↑ / MAE ↓ / TIME ↓ Area index 260 068 S-NeRF [3] 21.24 / 2.299 / 20h
24.07 / 1.374 $\bullet$ / 20h SAT-NeRF [4] 21.78 / 1.864 $\bullet$ / 20h 25.07
/ 1.249 $\bullet$ / 20h EO-NeRF [6] 26.09 $\bullet$ / 1.43 $\bullet$ / 20h
27.25 $\bullet$ / 0.91 $\bullet$ / 20h CARS [1] \- / 13.81 / 2min $\bullet$ \-
/ 1.40 / 2min $\bullet$ SAT-NGP (ours) 23.02 / 1.68 $\bullet$ / 12min$\bullet$
22.58 / 2.03 / 13min $\bullet$

Table 1: Evaluation metrics and computation time of various methods on four
JAX areas. SAT-NGP (with Robust Loss and MISH) is the fastest among NeRF
methods and provides competitive PSNR and MAE values. Best values are in bold.
The values from S-NeRF and SAT-NeRF are taken from [4], while EO-NeRF is taken
from [6].

##  5 CONCLUSION

In this paper, we presented SAT-NGP, a fast relightable transient-free 3D
reconstruction method from satellite images. This model requires minimal GPU
capacity and surpasses the performance of the previously adapted NeRF variant
for multi-date satellite imagery. SAT-NGP can accurately reconstruct DSM in
less than 15 minutes, a significant improvement from the 20 hours required by
previous methods. We also highlight the benefits of using a Robust loss
function and MISH activation function. Future improvements to the quality of
DSM and NVS production, while maintaining rapid training and significantly
reducing the number of view inputs, would be advantageous for real satellite
reconstruction use cases. After making necessary adjustments, our code will be
made available at SAT-NGP.

##  6 ACKNOWLEDGEMENTS

This work was performed using HPC resources from CNES Computing Center (DOI
10.24400/263303/CNES_C3). The authors would like to thank the Johns Hopkins
University Applied Physics Laboratory and IARPA for providing the data used in
this study, and the IEEE GRSS Image Analysis and Data Fusion Technical
Committee for organizing the Data Fusion Contest.

## References

  * [1] Julien Michel, Emmanuelle Sarrazin, David Youssefi, Myriam Cournet, Fabrice Buffe, Jean-Marc Delvit, Aurélie Emilien, Julien Bosman, Olivier Melet, and Céline L’Helguen,  “A new satellite imagery stereo pipeline designed for scalability, robustness and performance,”  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 2, pp. 171–178, 2020.
  * [2] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng,  “Nerf: Representing scenes as neural radiance fields for view synthesis,”  Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
  * [3] Dawa Derksen and Dario Izzo,  “Shadow neural radiance fields for multi-view satellite photogrammetry,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1152–1161.
  * [4] Roger Marí, Gabriele Facciolo, and Thibaud Ehret,  “Sat-nerf: Learning multi-view satellite photogrammetry with transient objects and shadow modeling using rpc cameras,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1311–1321.
  * [5] Lulin Zhang and Ewelina Rupnik,  “Sparsesat-nerf: Dense depth supervised neural radiance fields for sparse satellite images,”  arXiv preprint arXiv:2309.00277, 2023.
  * [6] Roger Marí, Gabriele Facciolo, and Thibaud Ehret,  “Multi-date earth observation nerf: The detail is in the shadows,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2034–2044.
  * [7] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller,  “Instant neural graphics primitives with a multiresolution hash encoding,”  ACM transactions on graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022.
  * [8] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth,  “Nerf in the wild: Neural radiance fields for unconstrained photo collections,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7210–7219.
  * [9] Songlin Xie, Lei Zhang, Gwanggil Jeon, and Xiaomin Yang,  “Remote sensing neural radiance fields for multi-view satellite photogrammetry,”  Remote Sensing, vol. 15, no. 15, pp. 3808, 2023.
  * [10] Diganta Misra,  “Mish: A self regularized non-monotonic activation function,”  arXiv preprint arXiv:1908.08681, 2019.
  * [11] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi,  “Robustnerf: Ignoring distractors with robust losses,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20626–20636.
  * [12] Andrew M Saxe, James L McClelland, and Surya Ganguli,  “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,”  arXiv preprint arXiv:1312.6120, 2013.