Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

This paper has been submitted to PLOS ONE in January 2024 and was desk
rejected as it is more a policy/method paper. It has been resubmitted to
F1000Research

[orcid=0000-0003-2820-6564] [orcid=0000-0003-0839-8167]
<EMAIL_ADDRESS>[orcid=0000-0001-6347-9926] [orcid=0000-0002-0690-7151]
[orcid=0000-0003-1028-1670] [orcid=0000-0001-6159-4875]
[orcid=0000-0003-0617-3319] [orcid=0000-0001-8151-8722]
[orcid=0000-0002-4526-2472] [orcid=0000-0002-0727-8326]
[orcid=0000-0002-0964-4457] [orcid=0000-0001-9868-4671]
[orcid=0000-0002-5588-0290]

[1]Corresponding author.

# Towards a Quality Indicator for Research Data publications and Research
Software publications
– A vision from the Helmholtz Association

Wolfgang zu Castell  Doris Dransch  Guido Juckeland  Marcel Meistring
Bernadette Fritzsch  Ronny Gey  Britta Höpfner  Martin Köhler  Christian
Meeßen  Hela Mehrtens  Felix Mühlbauer  Sirko Schindler  Thomas Schnicke
Roland Bertelmann  Helmholtz Center Potsdam - German Research Center for
Geosciences, Potsdam, Germany  Helmholtz-Zentrum Dresden-Rossendorf (HZDR),
Dresden, Germany  Helmholtz Association, Helmholtz Open Science Office,
Berlin, Germany  Alfred Wegener Institute, Helmholtz Center for Polar and
Marine Research, Bremerhaven, Germany  Helmholtz Centre for Environmental
Research, Leipzig, Germany  Helmholtz-Zentrum Berlin für Materialien und
Energie (HZB), Berlin, Germany  Deutsches Elektronen-Synchrotron DESY,
Hamburg, Germany  GEOMAR Helmholtz Centre for Ocean Research, Kiel, Germany
Institute of Data Science, German Aerospace Center (DLR), Jena, Germany

(2024)

###### Abstract

Research data and software are widely accepted as an outcome of scientific
work. However, in comparison to text-based publications, there is not yet an
established process to assess and evaluate quality of research data and
research software publications. This paper presents an attempt to fill this
gap. Initiated by the Working Group Open Science of the Helmholtz Association
the Task Group Helmholtz Quality Indicators for Data and Software Publications
currently develops a quality indicator for research data and research software
publications to be used within the Association. This report summarizes the
vision of the group of what all contributes to such an indicator. The proposed
approach relies on generic well-established concepts for quality criteria,
such as the FAIR Principles and the COBIT Maturity Model. It does – on purpose
– not limit itself to technical implementation possibilities to avoid using an
existing metric for a new purpose. The intention of this paper is to share the
current state for further discussion with all stakeholders, particularly with
other groups also working on similar metrics but also with entities that use
the metrics.

##  1 Background and objective

Measuring the performance and progress of research based on means of output is
a common task of research assessment. While it is commonly accepted to use
quality-assured scientific publications as a proxy for quality assessment in
research evaluation, other artifacts of scientific work such as research data
and research software are often not yet considered. For text-based scientific
work, there are well established processes of quality assurance during the
process of publication. For publishing research data and research software no
commonly accepted standards exist yet, although research data and software
represent a significant output of the scientific endeavor especially given the
rise of data- and software-intensive research.

The Helmholtz Association has already decided to integrate research data
publications and research software publications in its internal key
performance indicator system and is currently developing and implementing a
quality indicator for those artifacts. It should become a driver within
Helmholtz in a threefold way: a) to create awareness and appreciation for the
diversity of research output, b) to align the assessment of research and
research practice to the conditions of digitalization and openness of science,
and c) as a stimulus and incentive for scientists to improve their data and
software publications related to specific quality criteria, and thus, to
improve the quality and trustworthiness of science as a whole. Helmholtz
Centers from all research fields of the Association contributed in an
iterative discussion process to cover a broad perspective originating from
their respective research environment. To meet all aspects relevant for the
quality indicator, a group of scientists, scientific software developers, data
curators, and colleagues reporting on research output were involved in its
development. This paper provides insight to the current status of discussion.

##  2 Terminology

The following definitions should clarify what type of research data and
research software we address in this paper.

Research data

In accordance with the EOSC-Glossary [1] we define research data as data that
has been collected, created and/or is being used in “scientific research
activities and used as evidence in the research process, or commonly accepted
in the research community as necessary to validate research findings and
results”. It ranges from highly complex and individual long-tail-data from
experiments and observations to automatically produced generic data
collections.

Research software

We define research software as software that has been developed in the
research process [2]. We consider software of all levels of maturity: from
personal use, reuse by others, long term support, and criticality as defined
by [3] in the form of software classes.

Published data and software

In both cases we concentrate on research output which is published. We define
Published data and software as digital objects that have been made as
permanently available as possible on the Internet [4]. The different levels of
“Published” are addressed in the framework of quality dimensions and
attributes (Section 4 and Section 5).

##  3 Methodical framework – quality criteria and quality quantification

Recalling the publishing process of scientific publications: A paper published
e.g. in a scientific journal has been quality checked with respect to several
criteria over the course of the publication process. When accepted, a
manuscript is marked as being approved by the underlying quality assurance
mechanisms. Aggregating the number of papers that went through this process of
quality assurance is therefore used as a quantitative measure of the output
performance of a scientific entity. Yet, this process of solely using simple
and one-dimensional metrics to assess research quality and performance is
broadly criticized. But the convenience of having one number easy at hand has
so far hindered all attempts to replace this practice. Taking this into
consideration a quality indicator for research data and software has to be
developed with the goal to have one easily comparable value that still does
more then only counting - by representing a diverse set of quality features in
itself.

Therefore, any analog of this process for scientific data publications or
research software publications has to technically map a multi-dimensional
suite of quality criteria onto one value, the quality indicator. Thus,
developing a quality indicator has to address the following questions: 1) What
quality criteria should be considered? 2) How to measure each of the quality
criteria? 3) How to condense individual quality measures into one number, the
quality indicator? In addition, for practicability, 4) how to collect the
quality measures for the most part in an automated fashion? Since the
Helmholtz quality indicator has to match all types of research data
publications and research software publications generated in the diverse
scientific disciplines within the Helmholtz Association, generic approaches
are required to provide answers to these questions. We decided to build on the
FAIR Principles [5] to specify relevant quality criteria. In order to provide
a generic way to estimate quality for a wide range of criteria, we are using
an approach based on the COBIT Maturity Model [6] to quantify quality in terms
of a maturity scale. Finally, a weighted summation and radar plot approach is
used to derive one single value from the various quantitative measures while
also providing a more detailed representation. The collection of quality
measures should rely on automated procedures and tools as much as possible.
However, in the end, any indicator must match the target set, technical issues
must not lead to deviation from that target. Therefore, our basic approach for
developing the quality indicator was to define the quality criteria that are
essential for the intended objectives in a first step, and to then condense
the list of criteria in a second step with respect to technical feasibility of
automated collection. This approach discusses quality criteria of research
data publications and software publications without immediately considering
the technical implementation of measuring them. Thus, we selected metrics not
simply for the ease of measurement. Rather, the measurement of some criteria
deemed highly relevant might be deferred until a corresponding, automated
approach of evaluating them becomes available.

The overall approach then consists of the following five steps.

  1. (i)

Define a suitable number of dimensions, according to which quality of research
data and software can be assessed.

  2. (ii)

For each of these dimensions, collect a set of specific attributes capturing
the respective dimension.

  3. (iii)

Using a generic maturity model, each attribute is mapped onto a numeric scale

  4. (iv)

Using weighted averages, an aggregated maturity level is obtained for each
dimension.

  5. (v)

The summarizing quality assessment can then be visualized using radar plots.

These steps lead to the current state of a quality summary of the assessed
research data and software. Using more than one dimension allows us to take
different perspectives into account. For example, an organization aiming at
building-up a large, integrated software library might have other targets for
quality assessment than a single software developer working on individual
projects. As well as an international campaign gathering huge amounts of
automatically generated sensor-data has other objectives, restrictions, and
conditions than long-tail data generated in experiments in a laboratory
setting or a medical scientist working with confidential patient data. In
analogy to the aggregation of quality-assured scientific publications, a
quantitative estimate of high quality data and software publications could
then be obtained by aggregating the number of data/software showing a maturity
above a given minimum in each of the dimensions. Afterwards, the maturity of
each dimension also has to be aggregated to form the final indicator for a
publication.

To determine an initial set of dimensions for quality assessment, we are
building on the FAIR principles [5]. The FAIR principles have been widely
adopted as a minimum set of quality criteria for research data and research
software thus, our approach generally also builds on the four principles
findability, accessibility, interoperability and re-usability. Considering
upcoming discussions within communities on the limitations of pure
implementation of FAIR to grasp a comprehensive view on the quality of
research data and research software publications, the partly interrelated
aspects and minor inaccuracies of wording within the four core principles we
adopted and extended the FAIR principles to best suit our needs.

For research software we amended the FAIR criteria with two additional
criteria: scientific quality, addressing the question of research software
being scientifically well grounded, and technical competence, capturing
aspects of the software being technically well grounded (FAIR-ST).

For evaluating research data publications, we regrouped the FAIR criteria into
four dimensions that are easier to grasp related to datasets but still cover
the same aspects. Those being 1. Publishing, 2. Openness, 3. Curation, and 4.
Metadata. To compensate for inadequacies between different disciplines when
applying a general framework like the proposed, the 5. dimension of the
External View has been added to the set (POCME).

In order to allow for a comparable and unbiased quality assessment for data
and software publications based on our modified FAIR-framework, we need a
broadly applicable scheme leading to a numerical value for each quality
criterion. In certification processes, maturity models are commonly used to
solve this task.

###  3.1 The COBIT Maturity Model

To achieve comparability among different attributes of quality, stages of
product evolution can be assessed using a generic model of maturity. Being
generic, the model can be applied to all categories of scientific output
leading to quantitative results which can be compared. Our approach utilizes
the COBIT Maturity Model [6]. The COBIT Maturity Model quantifies maturity of
a product in six levels by mainly evaluating the processes that lead to the
product. Looking at processes for achieving a certain quality is a common
approach in quality management, where quality management systems (QMS) are
defined as “formalized systems that document processes, procedures, and
responsibilities for achieving quality policies and objectives” [7].
Typically, lower levels of maturity focus on individual outcomes of a process
to be certified. The higher the maturity, the more the process is considered,
accounting for the fact that a well-developed process naturally leads to
products of a certain quality. COBIT starts with a default level 0 (non-
existent), two levels of maturity acknowledge the successive completion of a
set of quality criteria for the product to be targeted. Starting with level 3
maturity implies a process to be existent which guarantees the addressed
quality to be kept. Two further levels finally focus on the improvement of the
underlying process towards full optimization with maturity level 5. Clearly,
the explicit meaning of the maturity levels for a given quality criterion has
to be defined as part of the community process. Obviously, the maturation
levels need to be interpreted within the context of a given aspect of quality
assessment. The levels are cumulative in the sense that a higher level
includes the requirements of all lower levels, i.e. marking a higher level as
fulfilled means that the requirements of all lower levels are fulfilled plus
the requirements of the highest marked level. Thus, a higher level can either
refine/specialize characteristics that were already evaluated for compliance
with a lower level or introduce a new and additional characteristic to assess.
As a result, comparability across quality dimensions is given.

The specific application of maturity levels derived from the COBIT Maturity
Model with regards to data and software publications are introduced in the
respective subsection of this paper (Subsection 4.2 and Subsection 5.2).

###  3.2 Weighted summation and radar plot

Guidelines for reviewers for academic journals typically provide a set of
criteria being considered in preparing a review for a manuscript. Examples are
scientific originality, proper use of methods, organization of the
material/figures/tables, and comprehensibility. Such criteria can likewise be
grouped into different dimensions of assessment such as scientific novelty,
being scientifically grounded, quality of presentation etc. However, such
dimensions are commonly not transparently weighted against each other.
Nevertheless, there is a common understanding in each scientific community on
what has to be considered more important. To make such an understanding open
and transparent, we introduce radar plots to visualize the joint maturity of a
data and software publication with respect to all quality dimensions. The
relative importance of every single attribute within each dimension of
assessment can be fine-tuned by agreeing on weights to be used in a weighted
average. Doing so, minor criteria can also be included into the assessment
without the risk of leading to a biased assessment.

For the overall assessment of the data and software publication, we
deliberately do not suggest to use weighted averages as a default. The multi-
dimensionality of each quality assessment should rather be kept transparent.
This allows to optimize towards different goals in using the quality
indicator. Figure 1 exemplifies the result of the quality assessment of two
fictitious research software or data publications.

Figure 1:  Example of a fictitious assessment for two research software or
data publications showing how the radar plot can visualize the fulfillment of
some criteria at higher quality levels than others. Maturity levels are
ordered from center (0, non-existent) to outside (5, optimized). The minimum
requirements and maximum scores might also differ for each dimension.

##  4 The Quality Indicator for research data

###  4.1 Dimensions of quality for research data assessment

The quality of published research data and hence its potential value for
future reuse in other contexts can be assessed based on the quality of the
metadata describing the data formally, on metadata describing the data on the
content-level, on the quality of the data itself and the conditions of
Openness/Licensing. With regards to the diversity of research data over the
different scientific domains represented at Helmholtz, the heterogeneous ways
of collecting, creating, editing, and curating data, a single quality
criterion for research data assessment would not be constructive. To allow for
a multi-dimensional perspective on research data, the quality assessment being
presented in this document is spanning five dimensions of data quality being
partly based on the FAIR-principles and the FAIR Data Maturity Model of the
Research Data Alliance [8].

The derived quality dimensions of the POCME-framework for research data
publications, each with indication of the covered FAIR-principles, are:

Publishing

This dimension covers especially the areas of Findability and Accessibility.
It addresses the use of identifiers to enable unambiguous identification as
well as the level of storage and hence the possibilities and information on
the general accessibility to published data.

Openness

As publishing a dataset does not necessarily mean to make it openly available
(Intelligent Openness), this dimension takes this circumstance into account
and covers aspects of Accessibility, Interoperability and Reusability.

Curation

Curation can be crucial for the quality of published data itself as well as
for the provided formal and content related information. Hence, it can
significantly improve the value of a published dataset for Reusability as well
as for Findability, Accessibility, and Interoperability.

Metadata

Qualitative metadata can help in many aspects of Findability, Accessibility,
Interoperability, and Reusability of published data. By providing content
related aspects, helpful information to retrieve a dataset as well as context
around a dataset.

External View

This dimension is meant as a handle to compensate inadequacies between
different disciplines which is a sensible approach when applying a general
framework like the proposed.

These five dimensions of quality for research data are still too generic to be
evaluated using the maturity model. Therefore, each dimension is split into a
set of attributes. The latter will be assessed using the maturity model. To
derive a single value for each of the five dimensions, an aggregation step has
to be incorporated. This is done by computing a weighted average of the
maturation values for all attributes constituting a quality dimension. Using
weights provides the freedom not to be forced to work with attributes of equal
importance. Minor aspects are just given smaller weight.

###  4.2 Quality assessment of the dimensions

To assess the quality of research data publications, the developed POCME
assessment-framework measures/estimates the maturity of different quality
dimensions and attributes. To be able to do this, a generic five-level scale
of maturity derived from the COBIT Maturity Model is applied. The levels are
always to be seen in the context of the respective dimension and attribute
they are applied on.

The first level (0) usually implies that the respective attribute is non-
existent or not applied. The following levels built upon each other, always
following the assumption, that the previous levels are largely met, leading to
the highest possible level of maturity and hence quality of the data itself
and the accompanying meta-information maximizing the findability,
accessibility, interoperability, and re-usability.

  1. (0)

Non-existent: no information available or not applied

  2. (1)

Most necessary information provided or measure taken

  3. (2)

Basic information provided or measure taken (sensible level of
information/measures)

  4. (3)

Advanced information provided or measure taken, allowing to generally
understand and (re)use the published data

  5. (4)

Complete and accurate information provided or measure taken, to an extend that
allows maximal understanding and usage of data

The following sections list the attributes and their maturity levels of
research data publications that are proposed for each dimension. It must again
be noted that the COBIT model is cumulative, i.e. a higher level includes the
fulfillment of all requirements of a lower level.

####  4.2.1 Dimension Publishing

##### Published with Identifier

  1. (0)

No identifier (resource may only be found via personal communication)

  2. (1)

Basic Uniform Resource Identifier

  3. (2)

Dataset is identifiable via internal handle (does not resolve globally,
generally no metadata)

  4. (3)

Dataset is basically identifiable via formalized, standardized, persistent
identifier (resolves globally, general metadata provided)

  5. (4)

Dataset is identifiable via globally unique, formalized, standardized,
persistent identifier supported by general metadata (e.g. DOI).

##### Published via a Repository or Collection, that is indexed in a Meta-
Repository (e.g. re3data)

  1. (0)

No information available, the data is not published via a
repository/collection

  2. (1)

The data is published in a repository/ collection which is not listed in an
eligible meta-repository

  3. (2)

The repository/collection is listed in an eligible meta-repository, basic no.
of quality indicators assigned by the meta-repository are achieved

  4. (3)

The repository/collection is listed in an eligible meta-repository, medium no.
of quality indicators assigned by the meta-repository are achieved

  5. (4)

The repository/collection is listed in an eligible meta-repository, high no.
of quality indicators assigned by the meta-repository are achieved

Remark: Meta-repositories like re3data usually do not perform in depth-checks
on the quality of published research data in the indexed sources. Still, if
sources indexed in meta-repositories like re3data fulfill certain quality-
criteria respective indicators (e.g. icons) are assigned. The icons themselves
represent different quality-levels on the level of the source, assuming that
contained dataset likely meet those quality levels represented by the source.

##### Published with Information on Access to the Data

  1. (0)

No metadata available

  2. (1)

Metadata available, but no data access-information available in the metadata

  3. (2)

Metadata available, data access-information available only in human-readable
form

  4. (3)

Metadata available, data access-information available only in human readable
form, including general license information

  5. (4)

Metadata available, data access-information available in human-readable and
machine-readable form*, including license information

*The information is available in a structured form that allows, for example, that the respective information can be read out via script.

####  4.2.2 Dimension Openness

##### General Degree of Openness

  1. (0)

No information on open accessibility/availability of the data at all

  2. (1)

Information available: no open accessibility/availability of the data. No
justification, no information on possible contact or restrictions

  3. (2)

Like (1) + information on possible contact, restrictions or potential use
cases on request available

  4. (3)

Like (2) + with justification AND/OR date of moratorium

  5. (4)

Open accessibility with corresponding license (no login or contact needed or
otherwise with justification)

##### Primary Data Formats

  1. (0)

No primary data available in digital form

  2. (1)

Primary data generally available

  3. (2)

Primary data stored in common proprietary data formats

  4. (3)

Primary data stored in open formats

  5. (4)

Primary data makes use of common, domain specific terminologies (e.g.,
codelists)

####  4.2.3 Dimension Curation

##### Level of Curation

  1. (0)

Data is published in raw form without any curation or documentation (e.g. raw
long-tail data)

  2. (1)

Data is published in raw form without curation but according to standard with
basic documentation like readme (e.g. automatic generated sensor data, long-
tail data following a basic scheme)

  3. (2)

Data is published in cleaned form with some curation (e.g. brief checking,
documentation according to standard)

  4. (3)

Data is published in cleaned form with enhanced curation and/ or reprocessing
(e.g. conversion to new formats, enhancement of documentation)

  5. (4)

Data is published after undergoing extensive curation and/or reprocessing
according to discipline specific standards in order to enhance to max. quality
(like (3) + additional editing of deposited data for accuracy)

####  4.2.4 Dimension Metadata

##### Metadata to find/retrieve a Resource / Formal Metadata

  1. (0)

No metadata available

  2. (1)

Metadata available for/with the data publication that is not structured
according to a commonly accepted scheme (i.e. no scheme applied)

  3. (2)

Metadata provided with the data publication that is structured in a basic way
according to a commonly accepted scheme (e.g. completed DataCite mandatory-
properties/discovery ; Dublin Core, etc.)

  4. (3)

Metadata provided with the data publication that is structured in an advanced
way, according to a commonly accepted scheme (e.g., completed Datacite
mandatory- and recommended-properties for discovery + discovery-supporting
basic content metadata according to DataCite scheme)

  5. (4)

Full Metadata provided with the data publication (complete DataCite mandatory-
and recommended- and optional-properties for discovery + comprehensive
discovery-supporting content metadata according to DataCite scheme)

##### Content related Metadata

  1. (0)

No content related metadata available

  2. (1)

Some content related metadata available, following a (generic) scheme (e.g.
DataCite)

  3. (2)

Complete content related metadata available following a (generic) scheme (e.g.
DataCite)

  4. (3)

Some content related metadata available following standardized form or domain
specific scheme

  5. (4)

Complete and curated content related metadata available following a
standardized form and domain specific scheme (see 3)

####  4.2.5 Dimension External View

##### Score from Domain Specific Fair Assessment Tool

  1. (0)

0-20% Score reached

  2. (1)

21-40% Score reached

  3. (2)

41-60% Score reached

  4. (3)

61-80% Score reached

  5. (4)

81-100% Score reached

Remark: By integrating this dimension, it would be possible to take a domain
specific evaluation of a data publication in regards to community best
practices and domain specific requirements on metadata and ideally the data
itself into account. This is based on the assumption that in addition to more
generalistic FAIR assessment tools currently available, further tools with
domain-specific evaluation schemes will be developed. The rating scales of
these tools can then presumably always be converted into a percentage scale. A
rather low weighting of this dimension must be considered in order to minimize
the stimulus for gamification.

##  5 The Quality Indicator for research software

###  5.1 Dimensions of quality for research software assessment

Quality of research software has various aspects to be considered. Depending
on the type of software, the field of application and the scientific goals to
be followed in using research software, different aspects of research software
are being considered as important. Therefore, a single quality criterion for
research software assessment would not be constructive. To allow a multi-
dimensional perspective on research software, the quality assessment being
presented in this document is spanning six dimensions of software quality
being given by the proposed FAIR-ST framework.

Using the principles for quality assessment, the FAIR principles have to be
spelled-out explicitly with respect to research software. This is given by
FAIR4RS [9]. As mentioned above, the four FAIR principles have been augmented
with two further quality dimensions. Thus, the six dimensions of research
software quality being presented in the assessment scheme are given as
follows.

  1. (a)

Findable: Researchers need to be able to find the software using typical
search strategies. Having found a given software product, they have to be able
to identify a given version/release and be provided with enough information to
value the software for their specific research.

  2. (b)

Accessible: Software needs to be accessible in order to unfold its potential.
Hereby, accessibility has both a legal side as well as an operational side.

  3. (c)

Interoperable: For software to be interoperable, it must be capable of being
included into larger contexts or frameworks. Towards this aim, input/output
formats need to be considered as well as interfaces to use the software in
automated pipelines.

  4. (d)

Reusable: Using software for exactly the purpose it has been developed for is
considered as accessibility within the quality assessment. In contrast,
reusability considered the aspect of being able/allowed to use the code for
one’s own purposes, to adapt and extend the software.

  5. (e)

Scientific basis: Research software is an integral part of the research
process. It therefore has to also follow community/organization specified
common standards in performing research. While the contribution of software to
a certain scientific achievement must be evaluated by dedicated experts in the
field, some aspects of scientific quality can also be considered in wider
generality.

  6. (f)

Technical basis: Quality of software also reflects general aspects of software
engineering. At the end, software has to guarantee to actually produce what it
has been aimed to do. Professional software development aims at producing
software following state-of-the-art software engineering concepts. Thus,
aspects of professional software engineering must also be considered as part
of a quality assessment of research software development.

The six dimensions of quality for research software are still too generic to
be evaluated using the maturity model. Therefore similar to the approach for
research data, each dimension is split into a set of attributes. The same
principles as outlined previously for research data are applied.

###  5.2 Quality assessment of the dimensions

The following descriptions summarize the proposed five levels of maturity used
for evaluating the quality of software publications. Level 0 is the default
level, when no information for the evaluated attribute is provided.

  1. (0)

Non-existent: no information available

  2. (1)

Initial: initial information available being obtained in an ad-hoc,
unorganized manner

  3. (2)

Repeatable: the information is complete, being produced in a repeatable, yet
intuitive manner

  4. (3)

Defined: a process is established guaranteeing the complete compilation of the
required information

  5. (4)

Managed: the process being established is managed, i.e. monitoring/measuring
is included

  6. (5)

Optimized: practices are put in place optimizing the way the process is
operated, leading to improved quality over time

For each of the quality dimensions, the assessment is done by rating every
corresponding attribute according to its maturity. The explicit interpretation
of each maturity level for a given attribute is given in terms of asking of
compliance with statements. Since the maturity levels are building on each
other in a cumulative way, the number of the last achieved statement defines
the maturity of the attribute.

The following sections provide the assessment scheme for the six dimensions of
the FAIR-ST framework.

####  5.2.1 Dimension Findable

The following statements address the aspect of being able to find and uniquely
identify the software.

##### Open Publication Repository

  1. (0)

There is no information available on where to find the software.

  2. (1)

The software is contained in an online repository.

  3. (2)

Some kind of description is available giving further information on the
software in this repository (e.g. readme file).

  4. (3)

A structured meta data description (e.g. following DataCite) given for
software is in this repository.

  5. (4)

The repository is listed in some overarching meta-repository (e.g. Helmholtz
Research Software Directory (RSD)).

  6. (5)

The meta-repository is performing quality checks (e.g. re3data) for the used
publication repository.

##### Versioning

  1. (0)

No software versioning applied.

  2. (1)

There is some kind of versioning for the software.

  3. (2)

The software uses structured (e.g. semantic) versioning.

  4. (3)

A description of the versioning scheme is available.

  5. (4)

There is a documentation on release cycles for the software.

  6. (5)

The versioning scheme allows for automatic tagging by CI/CD processes.

##### Persistent Identifier (PID)

  1. (0)

No PIDs given.

  2. (1)

A handle/URL is provided to identify the software.

  3. (2)

The handle/URL is provided with a defined metadata scheme.

  4. (3)

A persistent identifier is provided.

  5. (4)

A PID allowing for automated harvesting of metadata information is provided.

  6. (5)

The PID is part of an established community standard.

##### Rich Metadata

  1. (0)

No metadata given.

  2. (1)

Some metadata information is provided with the software.

  3. (2)

The metadata information is following a given metadata scheme complete.

  4. (3)

A metadata curation process reflects changes/updates.

  5. (4)

All metadata information following the given metadata scheme can be
automatically harvested.

  6. (5)

An external quality assessment of the metadata exists.

####  5.2.2 Dimension Accessible

The following questions address the aspect of being able to access research
software. Accessing included the possibility to run the software, which might
also be in terms of a web service. However, accessibility does not include the
possibility to adjust the code which is rather being captured under the aspect
of reusability.

##### Access Conditions (organizational)

  1. (0)

Not specified.

  2. (1)

A contact is given which to inquire about the right to use the software.

  3. (2)

The software has a license describing rights of use.

  4. (3)

The license allows for open use of the software (e.g. OSI licenses).

  5. (4)

There is a way to also obtain some kind of support in using the software.

  6. (5)

There isa community, providing the opportunity of support and exchange
concerning aspects of using the software.

##### Access Options (process)

  1. (0)

There is only one specific form of accessing the software or no option at all.

  2. (1)

The software (source code or executable) is provided.

  3. (2)

The sources or executables being provided include some documentation on how to
install/use the software.

  4. (3)

Provided test cases allow to determine whether installation/execution worked
as being expected.

  5. (4)

Provided checks make sure the software works correctly.

  6. (5)

A software service is provided, i.e. are reported bugs taken into the
development cycle.

##### Technical Accessibility (run/start)

  1. (0)

No information given.

  2. (1)

“How to install” information is provided.

  3. (2)

Installation scripts are provided.

  4. (3)

The software allows for (semi-)automated installation, e.g. a Makefile or
manual package (like Python modules).

  5. (4)

Sources are provided such that a package manager or automated build tools ,
e.g. automake, can be used.

  6. (5)

A complete package that enables execution (e.g. container, app package) is
available.

####  5.2.3 Dimension Interoperable

The following questions address the aspect of being interoperable, i.e., the
possibilities of being able to integrate the software into one’s own software
framework or execution pipelines.

##### Input/Output Formats

  1. (0)

Not specified.

  2. (1)

Some description of input and output formats is provided.

  3. (2)

The software builds on standard formats for input and output.

  4. (3)

Additional options for varying input/output formats are provided.

  5. (4)

The software builds on accepted community standards for input/output data.

  6. (5)

The software provides in addition further tools for processing input/output
data.

##### Adaptability/Flexibility of Use

  1. (0)

No information given.

  2. (1)

There is a way to use the software with one defined set of input data.

  3. (2)

There are parameters to adjust the way the software is working.

  4. (3)

There is some way of logging what is done during execution.

  5. (4)

Documented API(s) are provided to integrate the software into one’s own
framework.

  6. (5)

There is documented way to integrate the software into open workflows, e.g.
via containers, web-services etc.

####  5.2.4 Dimension Reusable

The following questions address the aspect of being reusable. In addition of
being accessible, i.e. executable, reusability includes the possibility to
actually change/adapt the code.

##### Reusability Conditions

  1. (0)

Not clear.

  2. (1)

The software uses a custom license allowing reuse.

  3. (2)

The software uses a FOSS/OSI approved license including that license
dependencies are at least being checked manually.

  4. (3)

The software uses an appropriate license for different file types (code, text,
images etc.) following e.g. the REUSE specification.

  5. (4)

There is a process available for automatically checking e.g. the REUSE
specification.

  6. (5)

There is a process available such that all license dependencies are
automatically controlled.

####  5.2.5 Dimension Scientific basis

The following questions address the aspect of the software being
scientifically well grounded. While domain specific scientific requirements
have to be assessed as part of a scientific peer-review process, certain
generic aspect of good scientific practice can be assessed for all research
software.

##### Community Standards

  1. (0)

No information given.

  2. (1)

The connection to known (scientific) standards is drawn.

  3. (2)

The software follows standards of the relevant scientific community.

  4. (3)

The software complies with relevant scientific standards of the field.

  5. (4)

There is an indication on how further evolution of community standards will be
addressed.

  6. (5)

A closed feedback-loop is established, making sure that further evolutions of
community standards are being adopted.

##### Team Expertise

  1. (0)

No information given.

  2. (1)

Clear expertise from a single, relevant domain is part of the software
development team.

  3. (2)

The software development team has access to expertise in several relevant
domains.

  4. (3)

The software development team has access to expertise in all relevant domains.

  5. (4)

A fixed, established, interdisciplinary team works on the software.

  6. (5)

An established and coordinated community of software developers works on the
software.

##### Scientific Embedding

  1. (0)

No information given.

  2. (1)

At least one scientific use case is documented.

  3. (2)

A broader scientific context is documented including several examples.

  4. (3)

The software development is at least loosely connected to some scientific
initiative.

  5. (4)

The software development is part of a larger scientific initiative.

  6. (5)

The software development ispart of a larger scientific initiative with
dedicated processes for software development.

####  5.2.6 Dimension Technical basis

The following questions address aspects of professional software development
leading to sustainable, high quality research software.

##### Project Management

  1. (0)

No information on project management and code history being provided.

  2. (1)

Some kind of version control is used.

  3. (2)

A version control system is used.

  4. (3)

A version control system being part of a code project management platform
(e.g. GitHub, GitLab) and an associated ticket system is in place.

  5. (4)

A transparent process for ticket resolving, code review by other developer,
and merge requests is established.

  6. (5)

A release process with guaranteed changelog generation, testing, and product
provisioning is established.

##### Repository Structure

  1. (0)

No information given.

  2. (1)

All files are provided in some structured/unstructured way inside the
repository.

  3. (2)

The repository is structured albeit maybe in a manner such that every
contributor is free to follow own way of organizing files.

  4. (3)

A contribution mechanism is documented, e.g. CONTRIBUTORS.md file, as well as
a defined structure for the repository and a documented onboarding process.

  5. (4)

A common template for the repository structure is available, as well as some
kind of identification of deviation.

  6. (5)

A repository structure is enforced following community standards.

##### Code Structure

  1. (0)

No information given.

  2. (1)

Every developer is free to use his/her own style of coding.

  3. (2)

There are general recommendations for coding, albeit every developer being
able to follow his/her own style.

  4. (3)

There is some harmonization of code style being enforced following common
standards including meaningful naming of functions/variables etc.

  5. (4)

The code style is checked when accepting changes into the repository.

  6. (5)

The code style is enforced via a review process (e.g. failed pipelines or
auto-formatting).

##### Reproducibility (Code)

  1. (0)

No tests, or duplicated code.

  2. (1)

The code follows a modular structure allowing for component reusability.

  3. (2)

Clear system requirements are documented with min/max versions, albeit version
pinning, modularity etc. being enforced manually.

  4. (3)

A package manager is used for dependency pinning and testing enforced.

  5. (4)

Test coverage is measured, albeit tests may be written on a voluntary basis.

  6. (5)

Automated testing for different system environments, requirements for minimal
test coverage, and provisioning of containerized packages is done.

##### Code change process

  1. (0)

No information

  2. (1)

Internal 4-eye principle for accepting changes

  3. (2)

Code changes via transparent processes, e.g. merge/pull request

  4. (3)

Approval of code changes via transparent processes and with a 4-eye principle

  5. (4)

Integration of code changes into main development branch/releases only allowed
for specifically named/trained persons.

  6. (5)

Software releases involve an external review (by someone outside of the core
developer team)

##### Security

  1. (0)

No security concepts given.

  2. (1)

There are at least sporadic updates and dependency checks.

  3. (2)

There is a systematic assessment of dependencies and documentation of the
software stack.

  4. (3)

Deployment is provided within a CI/CD framework for different environments
including tools for check for security leaks.

  5. (4)

There is some process for monitoring dependency updates including reporting.

  6. (5)

There are regular and automated security monitoring and an automated update
process in place allowing merges only of security checks have been passed.

##  6 Conclusion, Next Steps

The procedure described so far represents the result of the conceptualization
phase of defining a quality indicator for data and software publications as
initiated within the Helmholtz Association. The following main principles were
guiding the developing process:

  * •

The indicator must match the quality criteria that are essential for the
intended objectives, technical issues must not lead to deviation from that
target. The objectives defined in the Helmholtz Association are: a) to create
awareness and appreciation to the diversity of research output, b) to align
the assessment of research and research practice to the conditions of
digitalization and openness of science, and c) as a stimulus and incentive for
scientists to improve their data and software publications related to specific
quality criteria, and thus, to improve the quality and trustworthiness of
science as a whole. Therefore, our basic approach for developing the quality
indicator was to define the quality criteria that are essential for the
intended objectives in a first step, and to then condense the list of criteria
in a second step with respect to technical feasibility of automated
collection.

  * •

The indicator should map a multi-dimensional suite of quality criteria onto
one value, the quality indicator. The approach of solely using simple and one-
dimensional metrics to assess research quality and performance is broadly
criticized. Taking this into consideration the quality indicator for research
data and research software publications should provide one easily comparable
value that still does more then only counting - by representing a diverse set
of quality features in itself.

  * •

The quality indicator has to match all types of research data publications and
research software publications generated in the diverse scientific disciplines
within the Helmholtz Association, generic approaches are required to provide
answers to these questions.

  * •

The indicator should rely on generic well-established concepts for quality
criteria and quality assessment to make use of already existing preliminary
work.

The result of the conceptual development process is a generic multi-
dimensional quality indicator based on a defined set of goal-oriented quality
dimensions and attributes as well as well-established, maturity-based concepts
for quality measurement.

Feasibility of the concept is currently evaluated in pilot implementations and
further refinements. Several aspects have to be clarified in the following
piloting phase of the implementation of the proposed procedure.

  1. (i)

While the attributes for each quality dimension have been discussed and
iterated in several rounds of community engagement, their actual application
to assess the quality of given data or software will show whether the
formulation of the attributes is clear enough to be applied in a wider
community, where documentation of the assessment needs to be improved, or
where even manual assessment is not easily possible at the moment.

  2. (ii)

In particular, the applicability of the quality indicators in various research
communities needs to be proven by running a couple of exemplifying
assessments.

  3. (iii)

The quality assessment should only lead to a minimal increase in the resources
required for the assessment itself. Therefore, the assessment of all the
quality dimensions and attributes should be automatized as much as possible.
Clearly, working with repositories for research data and research software is
a key requirement for automation on a higher level. Therefore, introducing
quality indicators for data and software publications hopefully further
fosters the level of professionalization in dealing with these outcomes of
scientific work.

  4. (iv)

The final assessment and visualization should be done in a digital manner.
Suitable tools are currently assessed or being developed.

It is obvious that the described approach to derive multi-dimensional quality
indicators can be used in an analog way to assess the research endeavor as a
whole. In the end, the adoption of a precise definition of these indicators,
the fine-tuned weighting scheme of quality dimensions and attributes against
each other, as well as the definition of a minimal level of maturity to be
expected for research data and research software, will make the process of
research assessment transparent and open. Moving towards multi-dimensional
assessments rather than single numbers will better reflect the complexity and
diversity of modern scientific work and, at the end, lead to a fairer
assessment of the contributions of the various parts of the scientific system
towards the common goal of contributing to increase collective human
knowledge.

With this paper, we share the current state of the process within the
Helmholtz Association. We invite all interested parties to further expand,
refine, or change the dimensions and attributes being proposed here. It is our
belief that any list of criteria has to be a living document to account for
the dynamic nature of research. Only through constant evolution, the proposed
framework is able to keep up with new thoughts, ideas, and findings on how to
improve the scientific process.

###### Acknowledgements.

This paper reflects the work of the Task Group Helmholtz Quality Indicators
for Data and Software Publications of the Helmholtz Association from the past
two years. While not all group members were able to participate in this paper,
the authors acknowledge and highly appreciate the constructive discussions of
the whole group, namely Gisbert Breitbach (Hereon), Torsten Bronger (FZJ),
Stefanie Castell (HZI), Claas Faber (GEOMAR), Tamara Husch (HZB), Ulrike
Kleeberg (Hereon), Matthias Kretz (GSI), Marco Nolden (DKFZ), Maria Nüchter
(KIT), Florian Ott (GFZ), David Pfaff (CISPA), Will Rayner (HMGU), Marco
Schaefer-Herte (DZNE), Dagmar Sitek (DKFZ), Christian Tacke (GSI), Anne Talk
(HZI), Regine Tobias (KIT), and Andreas Walker (AWI). An initial version of
the paper was also shared with research software and data experts within the
Helmholtz Association. The authors acknowledge and also incorporated comments
to these early versions from Stephan Druskat (DLR), Tobias Huste (HZDR), Sven
Kiele (HZDR), Uwe Konrad (HZDR), Tobias Schlauch (DLR), Bernard Silenou (HZI),
and Thomas White (DESY).

## CRediT author statement

Wolfgang zu Castell

Conceptualization, Methodology, Writing – Original Draft

Doris Dransch

Conceptualization, Investigation, Writing – Review and Editing, Supervision,
Project administration

Guido Juckeland

Conceptualization, Writing – Review and Editing, Visualization,
Supervision

Marcel Meistring

Conceptualization, Writing – Review and Editing, Supervision,
Project administration

Bernadette Fritzsch

Conceptualization, Writing – Review and Editing

Ronny Gey

Conceptualization, Writing – Review and Editing

Britta Höpfner

Conceptualization, Writing – Review and Editing

Martin Köhler

Conceptualization, Writing – Review and Editing

Christian Meeßen

Conceptualization, Investigation, Writing – Original Draft

Hela Mehrtens

Conceptualization, Writing – Review and Editing

Felix Mühlbauer

Conceptualization, Investigation, Writing – Original Draft

Sirko Schindler

Conceptualization, Writing – Review and Editing

Thomas Schnicke

Conceptualization, Writing – Review and Editing

Roland Bertelmann

Supervision, Project administration

## References

  * European Open Science Cloud (2023) [EOSC] European Open Science Cloud (EOSC), EOSC Glossary, 2023. URL: https://eosc-portal.eu/about/glossary.
  * Gruenpeter et al. [2021] M. Gruenpeter, D. S. Katz, A.-L. Lamprecht, T. Honeyman, D. Garijo, A. Struck, A. Niehues, P. A. Martinez, L. J. Castro, T. Rabemanantsoa, N. P. Chue Hong, C. Martinez-Ortiz, L. Sesink, M. Liffers, A. C. Fouilloux, C. Erdmann, S. Peroni, P. Martinez Lavanchy, I. Todorov, M. Sinha, Defining research software: a controversial discussion, 2021\. doi:10.5281/ZENODO.5504016.
  * Schlauch et al. [2018] T. Schlauch, C. Haupt, M. Meinel,  Software Engineering Guidelines - From Theory To Practice,  RSE Conference 2018 (2018). doi:10.5281/ZENODO.1411431.
  * Lawrence et al. [2011] B. Lawrence, C. Jones, B. Matthews, S. Pepler, S. Callaghan,  Citation and Peer Review of Data: Moving Towards Formal Data Publication,  International Journal of Digital Curation 6 (2011) 4–37. doi:10.2218/ijdc.v6i2.205.
  * Wilkinson et al. [2016] M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, A. Gonzalez-Beltran, A. J. Gray, P. Groth, C. Goble, J. S. Grethe, J. Heringa, P. A. ’t Hoen, R. Hooft, T. Kuhn, R. Kok, J. Kok, S. J. Lusher, M. E. Martone, A. Mons, A. L. Packer, B. Persson, P. Rocca-Serra, M. Roos, R. van Schaik, S.-A. Sansone, E. Schultes, T. Sengstag, T. Slater, G. Strawn, M. A. Swertz, M. Thompson, J. van der Lei, E. van Mulligen, J. Velterop, A. Waagmeester, P. Wittenburg, K. Wolstencroft, J. Zhao, B. Mons,  The FAIR Guiding Principles for scientific data management and stewardship,  Scientific Data 3 (2016). doi:10.1038/sdata.2016.18.
  * De Haes and Van Grembergen [2015] S. De Haes, W. Van Grembergen, COBIT as a Framework for Enterprise Governance of IT, Springer International Publishing, 2015, pp. 103–128. doi:10.1007/978-3-319-14547-1_5.
  * American Society for Quality (2023) [ASQ] American Society for Quality (ASQ), What Is a Quality Management System (QMS)?, 2023. URL: https://asq.org/quality-resources/quality-management-system.
  * Research Data Alliance FAIR Data Maturity Model Working Group [2020] Research Data Alliance FAIR Data Maturity Model Working Group,  FAIR Data Maturity Model: specification and guidelines,  RDA Endorsed Recommendation (2020). doi:10.15497/RDA00050.
  * Chue Hong et al. [2021] N. P. Chue Hong, D. S. Katz, M. Barker, A.-L. Lamprecht, C. Martinez, F. E. Psomopoulos, J. Harrow, L. J. Castro, M. Gruenpeter, P. A. Martinez, T. Honeyman,  FAIR Principles for Research Software (FAIR4RS Principles),  RDA Endorsed Recommendation (2021). doi:10.15497/RDA00065.


# Fields2Cover: An open-source coverage path planning library for unmanned
agricultural vehicles

Gonzalo Mier1 João Valente2 Sytze de Bruin3 1Gonzalo Mier with the Laboratory
of Geo-Information Science and Remote Sensing, Wageningen University and
Research, Wageningen, the Netherlands<EMAIL_ADDRESS>Valente
with the Information Technology Group, Wageningen University and Research,
Wageningen, The Netherlands3Sytze de Bruin with the Laboratory of Geo-
Information Science and Remote Sensing, Wageningen University and Research,
Wageningen, the Netherlands

###### Abstract

This paper describes
Fields2Cover111https://github.com/Fields2Cover/Fields2Cover, a novel open
source library for coverage path planning (CPP) for agricultural vehicles.
While there are several CPP solutions nowadays, there have been limited
efforts to unify them into an open source library and provide benchmarking
tools to compare their performance. Fields2Cover provides a framework for
planning coverage paths, developing novel techniques, and benchmarking state-
of-the-art algorithms. The library features a modular and extensible
architecture that supports various vehicles and can be used for a variety of
applications, including farms. Its core modules are: a headland generator, a
swath generator, a route planner and a path planner. An interface to the Robot
Operating System (ROS) is also supplied as an add-on. In this paper, the
functionalities of the library for planning a coverage path in agriculture are
demonstrated using 8 state-of-the-art methods and 7 objective functions in
simulation and field experiments.

Keywords – Agricultural Automation, Software Architecture for Robotic and
Automation, Field Robots

##  I Introduction

In developed countries, there is a shortage of skilled workers to operate
agricultural machinery [1]. This shortage can be alleviated with the
development of autonomous machinery. Unlike manually operated machinery,
autonomous vehicle operations need meticulous planning beforehand. The problem
of determining a path to cover a field is known as coverage path planning
(CPP). CPP is of high importance for cleaning [2], surveillance robots [3],
lawn mowers [4], and agricultural vehicles [5], where it has been addressed in
several works.

Whilst there have been many efforts, most of the (partial) CPP solutions have
not been released as open-source software thus hindering more rapid advances
in CPP by the scientific community. The packages shown in Table I are the only
open-source software to the best of our knowledge. Note that the software
packages listed in Table I solve the CPP problem partially, but require
several modifications in order to be customized to different unmanned vehicles
and applications.

This paper aims to fill the above mentioned gap by proposing and releasing to
the community an open-source CPP library for field coverage. The library was
designed focusing in four modules that are the core of CPP solutions: a
headland generator, a swath generator, a route planner, and a path planner.
Each module includes at least one state-of-the-art method and one objective
function. The library currently only supports convex fields. Regardless, there
is an urgent need for an open source software solution to fill the existing
gap in the CPP problem in agriculture. The ultimate goal of the library is to
ease the state of-the-art algorithm benchmark and to accelerate CPP research
and application.

TABLE I: Comparison between coverage path planning open-source software
solutions.

Package name |  Docs |  No grid used |  Non-holonomous |  Headlands support |  Customizable objective function |  Terrestrial agricultural vehicles
---|---|---|---|---|---|---
RJJxp/CoveragePlanning [6] |  ✗ |  ✔ |  ✗ |  ✗ |  ✗ |  ✗
Nobleo/full_coverage_path_planner [7] |  ✔ |  ✗ |  ✔ |  ✗ |  ✗ |  ✗
Ipa320/ipa_coverage_planning [8, 9, 2] |  ✔ |  ✔ |  ✔ |  ✗ |  ✗ |  ✗
Ethz-asl/polygon_coverage_planning [10, 11] |  ✔ |  ✔ |  ✗ |  ✗ |  ✗ |  ✗
Irvingvasquez/ocpp [12, 13] |  ✗ |  ✔ |  ✗ |  ✗ |  ✗ |  ✗
Greenzie/boustrophedon_planner [14] |  ✗ |  ✔ |  ✔ |  ✔ |  ✗ |  ✗
Ipiano/coverage-planning [15, 16] |  ✔ |  ✔ |  ✗ |  ✗ |  ✔ |  ✗
Fields2Cover |  ✔ |  ✔ |  ✔ |  ✔ |  ✔ |  ✔

###  I-A Related work

Owing to the non-holonomous nature of agricultural vehicles, a region of the
field known as headlands must be reserved for turning the vehicle. The most
basic approach is to allocate a constant width area around the field. This
strategy allocates a large amount of space to a poor yield area. Depending on
how the swaths are arranged, some headland areas are parallel to the swaths
and hence they are not needed for turning. By only constructing headlands
along the field edges where turns are made, the area reserved for them can be
minimized [5, 17].

Swaths are generated in the inner field, which is the remaining region after
subtracting the headlands. In two-dimensional planar fields, a reference line
can be applied as a guide for the generation of swaths, where each parallel
creates a swath [17, 5, 18]. This line can be chosen for convenience or by an
algorithm such as brute force or a meta-heuristic. Oksanen [5] describes a
driving angle search strategy that requires fewer iterations than brute force
search but it does not guarantee finding the global minimum. Objective
functions such as the number of turns or the sum of swath lengths are used to
determine optimality in swath generation [17].

The distance [17] and time [19] required to cover the field are affected by
the order of the swaths. A route is the sequence of the swaths to cover. The
Boustrophedon order, which travels the swaths sequentially from one side of
the field to the other, and the snake order, which skips one swath at each
turn and returns through the uncovered swaths, are popular preset routing
patterns [20]. Objective functions such as distance, number of rotations, or
time necessary to traverse the field [17, 19] are minimized by finding the
optimal route through meta-heuristics [21].

A path is composed of the swaths of a route connected by turns, forming a
continuous line along which the vehicle will drive. Dubins’ [22] or Reeds-
Shepp’s [23] curves are turns that minimize the path length of the turns.
These curves are made by either curve segments or straight lines. The main
problem is that there is an instantaneous change of curvature at the
transition point between two segments. Techniques such as numerical
integrators [24] or clothoids [25] are employed to smooth the turn to avoid
the curvature discontinuity. Furthermore, to navigate from a swath to the
headlands, turns such as non-uniform rational B-spline (NURBS) curves [26] can
be adopted.

CPP problems are composed of numerous sub-problems, several of which have
received special attention in literature. For example, Spekken [21] presents
an approach for calculating the coverage path in undulating terrain that
however does not consider turns between rows or headland creation. Nilsson
[27] and Nørremark [28] divide the CPP problem into two major modules: Field
Partitioning/Representation, where the distribution of headlands and swaths in
the field is set up, and Route Planning, which determines the optimal order of
travelling the swaths within sub-fields. In the latter framework, each module
has more than one function, increasing the complexity of comparing multiple
variations of the module.

###  I-B Existing open-source software

There have been web applications, such as GAOS [18], that allowed farmers to
design or adapt coverage paths with a user-friendly interface. Many of such
web applications, despite being a great help to the farming community, have
been developed in collaboration with companies, restricting the possibility to
release the code to the public domain.

The currently existing open source CPP repositories are listed in Table I.
Although seven other projects were found, none of them can be adopted for
farming purposes with ground robots. As mentioned above, ground robots in
agriculture are generally not-holonomous, so turning maneuvers must be planned
to move from one swath to another. Unfortunately, some packages [6, 10, 12,
15] only compute the route to cover a region. These packages are designed for
quadrotors [10, 12] or for indoor robots [6]. However, the code needs to be
modified to support path generation for non-holonomous robots. A special case
of CPP is the Nobleo package [7] which, although the vehicle used is non-
holonomous, uses a grid to define the nodes that should be covered at least
once. In agriculture it is important to reduce the damage caused by the wheels
of the vehicle, so it is not recommended to cover the same swath several times
[7] or to cross through the main field [6, 10, 8]. On the other hand, Greenzie
[14], which was developed for lawn mowers, is the only package that supports
headlands, along with Fields2Cover. Unlike arable farming, mowers are
constrained to avoid repeated tracks for field traffic, thus the coverage path
is created with random sweep angles. For this reason, Greenzie does neither
provide an optimizer nor an objective function for planning the swaths. In
contrast, Ipiano [15] provides an interface to change the objective function
used by its optimizer, but here no headland support is offered. Fields2Cover
is the only software solution that provides algorithms to create a coverage
path for terrestrial agricultural robots, including optimizers and objective
functions to generate the best path, headland support and turn planning.

###  I-C Contributions

This paper introduces the first open-source extensible CPP library for
agriculture. It provides the following contributions:

  1. 1.

A publicly-available library (Fields2cover) providing connectable modules to
address CPP problems with unmanned agricultural vehicles. Those modules can be
effortlessly customized for other CPP problems.

  2. 2.

Benchmark tools for quantitative comparison between the CPP algorithms and
approaches.

  3. 3.

A quantitative comparison using 38 convex fields between eight state-of-the-
art CPP approaches/methods and seven objective functions.

  4. 4.

Experiments with a commercial unmanned agricultural vehicle demonstrating
Fields2Cover’s capability to provide real-world solutions.

To the best of our knowledge, this is the first work proposing an open-source
library for CPP on agriculture.

##  II Fields2Cover

Figure 1: Diagram of the Fields2Cover library. The library contains four core
modules: 1) Headland Generator, 2) Swath Generator, 3) Route Planner and 4)
Path Planner. Each of the modules is represented as a box with 3 slots: the
name of the module, the methods implemented, and the objective functions
available. The output of a module is the input of the next module.

Fields2Cover is designed in four modules (Fig. 1): 1) Headland Generator, 2)
Swath Generator, 3) Route Planner and 4) Path Planner. The inputs of the CPP
problem are the shape of the field and the vehicle specifications, while the
output is the coverage path of the field. Methods from the same module can be
used interchangeably to compare their solutions independently from the rest of
the CPP problem.

###  II-A Headland Generator module

The Headland Generator module currently implements a single method that
buffers the border of the field in inward direction by a custom constant width
(see Module 1 in Fig. 1). The objective function of this module is the area of
the remaining field after removing the headlands.

$A_{r}=\frac{A_{\bar{hl}}}{A_{f}}$ (1)

where $A_{r}$ is the area remaining, $A_{\bar{hl}}$ the area of the field
without headlands, and $A_{f}$ the area of the original field.

###  II-B Swath Generator module

The inner field (i.e., excluding the headlands) is the input of the Swath
Generator module (see Module 2 in Fig. 1). This region is divided into
parallel swaths matching the operating width. In the current version, the
library only supports parallel non-overlapping swaths. Fields2Cover has a
brute force algorithm to find the optimal sweep angle by trying discretized
angles using a given step size. If the computer running the library supports
multiple threads, several sweep angles are tried in parallel [29].

This module currently implements 3 objective functions:

  * •

Minimize the Number of Swaths. This objective function depends on the shape
and the area of the field, and the width of the robot. The number of swaths is
limited by the equation:

$0\leq\\#S_{\alpha}\leq\frac{A_{\bar{hl}}}{R_{w}},$ (2)

where $\\#S_{\alpha}$ is the number of swaths for a given sweep angle
$\alpha$, $A_{\bar{hl}}$ is the area of the field without headland, and
$R_{w}$ is the operational width of the robot. The shape of the field that
maximizes the minimun number of swaths is the square field, which results in:

$min_{\alpha}\
\\#S_{\alpha}^{\tiny\diamondsuit}\simeq\frac{\sqrt{A_{\bar{hl}}}}{R_{w}},$ (3)

where $\\#S_{\alpha}^{\tiny\diamondsuit}$ is number of swaths in a square
field with a given sweep angle $\alpha$. Therefore, the optimal value of this
objective function is less than the square root of the area of the field.

  * •

Maximize the Field Coverage:

$A_{cov}=\frac{A_{\bar{hl}}\cap\\{\cup_{i}\ S^{i}\\})}{A_{\bar{hl}}},$ (4)

where $A_{cov}$ is the amount of area covered, $A_{\bar{hl}}$ is the field
without headlands, $S^{i}$ is the $i^{th}$ swath, $\cap$ is the intersection
operator, and $(\cup_{i}\ S^{i})$ is the union of all the swaths.

  * •

Minimize the Swaths Length:

$\sum_{i}^{N}length(S^{i})=\sum_{i}^{N}\sum_{j}^{S^{i}_{p}-1}||S^{i}_{p=j+1}-S^{i}_{p=j}||_{2},$
(5)

where $\sum_{i}length(S^{i})$ is the sum of the length of the swaths, $N$ is
the number of swaths, $S^{i}_{p}$ is the number of points that the $i^{th}$
swath has, ${}^{i}_{p=j}$ is the $j^{th}$ point of the $i^{th}$ swath, and
$||x||_{2}$ is the euclidian norm.

###  II-C Route Planner module

The Route Planner module uses the swaths created earlier to produce the route
(see Module 3 in Fig. 1). Fields2Cover contains several predefined route
patterns, which include the boustrophedon pattern, the snake pattern, the
spiral pattern and a custom pattern. The Boustrophedon pattern covers the
swaths sequentially, and the Snake pattern skips one swath each time to
traverse the field in one direction and returns through covering the uncovered
swaths. The Spiral pattern is a variation of the Snake pattern, that sort the
swaths in clusters of a fixed size with the snake pattern. The custom pattern
requires specification of the swath order by the user. To compare different
routes, the library provides as objective function the length of the path
generated by the Path Planning module. In addition, it provides the path
length when the minimum turning radius of the vehicle is zero.

###  II-D Path Planner module

The inputs of the Path Planner module (see Module 4 in Fig. 1) are the route
(sorted swaths) and the vehicle parameters. Once the route is known, the turns
to complete the path are computed. In the current version of the library, the
path planner applies the same type of curves for all the headland turns.
Fields2Cover currently supports straight curves, the Dubins’ curves[22] and
the Reeds-Shepp’s[23] curves, using the path length as the single objective
function.

###  II-E ROS wrapper

Although the Fields2Cover library does not depend on ROS, an interface with
ROS is provided as an add-on. The fields2cover-
ros222https://github.com/Fields2Cover/fields2cover-ros package provides
functions that convert Fields2Cover data types into ROS messages. Services are
created to execute modules directly from ROS topics. Launch files are used to
script examples of the package. RVIZ-support is also provided to visualize the
results of the modules. Methods, objective functions and parameters can be
modified on real time thanks to
rqt_reconfigure333http://wiki.ros.org/rqt_reconfigure.

###  II-F Design & Implementation

Fields2Cover is implemented using C++17, with a Python interface using Swig
[30], and released under BSD-3 license. The design of Fields2Cover aims to
serve both scientists and service providers and is intended to be easily used.

The reason for making Fields2Cover an open-source library is that doing so
encourages the development of additional functionality by providing the code
to the community. Likewise, Fields2Cover widely employs open-source libraries
from third parties to streamline the development process of state-of-the-art
algorithms. For scientists, priority is given to a flexible design, which
allows to extend or modify existing algorithms. Additionally, a benchmark
against which to compare new solutions is added. For service providers,
utility concerns the ability to plan the best coverage path for a given
objective function in a straightforward manner. The modularity of Fields2Cover
is key to ensure its usefulness for both cases. In addition, the library
provides tests, tutorials, and extended
documentation444https://fields2cover.github.io/ to reduce the learning curve.

##  III Results

Several experiments were conducted to demonstrate the functionalities of
Fields2Cover. Firstly, coverage paths were created for convex fields from the
Nilsson’s benchmark [27]. In these simulations, the experiments focus on the
optimization of the objective functions and the computation time of those
methods. Secondly, real field experiments were conducted in an agricultural
field with a commercial robot (Fig. 2) of the company AgXeed B.V (The
Netherlands). The aim of the experiment was to program the coverage trajectory
of the robot using the Fields2Cover library and assess whether a designed
coverage path is efficiently traversed by the robot. The planned path is
previously transferred to the robot with Protobuf[31]. The protobuf message
defines the path as timestamps, positions, velocities and orientations. It
also contains the geometry of the field boundary to prevent the vehicle from
leaving the field. The sensor data collected during the coverage path, such as
the GPS position and the velocity, is returned from the AgBot as a rosbag[32].

Experiments were done with a laptop MSI GF627RE with Intel(R) Core(TM)
i7-7700HQ CPU @ 2.80GHz (4 cores, 8 threads) with Ubuntu 20.04.5.

Figure 2: The AgBot 5.115T2, from the company Agreed B.V (The Netherlands), is
a differential robot with continuous treads. The weight of the robot is 7.8t,
the total width of the robot is 2.5 m, the minimum turning radius avoiding
excessive soil damage is 2.1 m. For the experiments the operational width of
the robot (width of the coupled tool) was assigned the same value as the width
of the robot. The AgBot 5.115T2 has 4-cylinder Deutz Diesel Engine, stage 5
with 156hp, and an electric drive train with a maximum speed of 13.5km/h. Some
onboard sensors are 2 cameras, a RTK-GNSS receiver and an IMU.

###  III-A Simulation results

Three simulation experiments were performed. Firstly, the optimal route was
computed for three different fields to visually inspect the effects of the
objective function. Secondly, the coverage path was computed for 38 convex
fields with every possible combination of the algorithms provided by the
library. The combination of algorithms for creating a coverage path were
compared using the path length as the objective function. Thirdly, the time
for computing coverage paths was recorded using several objective functions of
the Swath Generator module. The relationship between the area of the field and
the computation time was found.

The first decision for coverage path planning of a field is the objective
function to be optimised by the swath generator (Brute force algorithm). The
optimal pitch angle of the swaths may vary with the chosen objective function.
Therefore, the first experiment provides examples of optimal swaths for the
fields REC_A, CIR_B and SAL_B from the Nilsson’s benchmark [27], which are
shown in table II. The fields were re-scaled to an area of $100m^{2}$. If the
number of swaths is minimized, the number of turns is also reduced. For
instance, fields CIR_B and SAL_B are covered using a single turn. If maximum
field coverage is to be achieved, CIR_B needs seven turns while SAL_B needs
five. Field coverage is typically achieved when swaths are parallel or
perpendicular to one of the edges. In contrast, the swath-length objective
function may produce many short swaths (bottom-left of CIR_B with swath
length), that reduce the total length of the swaths.

TABLE II: Comparison of swaths generated using brute force optimizing one of the three objective functions: sum of swath lengths (minimization problem), number of swaths (minimization problem) and field coverage (maximization problem). The parallel lines inside the field are the centers of the generated swaths. Field Name |  Swath length |  Number Swaths |  Field Coverage
---|---|---|---
REC_A |  |  |
CIR_B |  |  |
SAL_B |  |  |
Figure 3: Coverage path length comparison. Plots in the same columns
correspond to the same objective function that was optimized using the brute
force algorithm, while plots in the same row refer to the same route planner
algorithm. Each dot represents a field from the Nilsson’s benchmark [27]. Red
dots correspond to paths created by a implementing Dubins’ curves, blue dots
are for Reeds-Shepp’s curves, and the black lines are for in-place turns.

The second experiment was conducted using 38 convex fields of the Nilsson’s
benchmark [27], re-scalated to an area of $1$ ha (Fig. 3). For each field, a
headland of $7.5m$ (three times the operational width of the robot) was
generated with constant width generator. Then, the brute force algorithm
generated the optimal swaths for each objective functions shown in Table II.
The route planners sorted the swaths with the boustrophedon, snake or spiral
(bulk of 6 swaths) pattern. Lastly, the path length was used to compare the
coverage path computed with Dubins’ or Reeds-Shepp’s curves. The resulting
path lengths were compared against the length of paths with in-place turning
(minimum turning radius equal to 0), which is the least possible path length
for a holonomic vehicle. As shown in Fig. 3, a percentage between $0.5\%$ and
$50\%$ of the coverage path was spent on turns. When the number of turns is
reduced, the distance traveled is reduced accordingly. The distance used for
turning increases when the boustrophedon pattern is applied since a shorter
width between swaths requires a larger turn to comply with the minimum turning
radius requirement. For instance, in the first column of the figure 3, the
difference between the path length using Dubins’ curves and in-place turns is
smaller than in the other columns. Field coverage and swath length behaved
equally in terms of coverage path length. With any of the objective functions
presented, the boustrophedon pattern produced the shortest pattern with in-
place turns, the snake pattern was the second and the spiral pattern the
longest. The length of the boustrophedon pattern increases when the minimum
turning radius is required.

Figure 4: Time required to compute a path according to the objective function
used. Algorithms used are constant width headland generator, parallel brute
force for swath generation, Boustrophedon route order and Dubins’ curves.

In the last simulated experiment, the computation time of planning a coverage
path was measured in relation to the area of the field and the objective
function of the swath generator (Fig. 4). The constant headland width was set
to three times the width of the robot. Next, the parallel brute force
algorithm optimized the pitch angle of the swaths, which were sorted using a
boustrophedon pattern. Finally, the path planner used Dubins’ Curves to create
the coverage path. Fields2Cover computed a coverage path for a field of $1$ ha
in less than 3.5 seconds using Field coverage as the objective function, while
only 0.5 seconds were needed using the number of swaths or the swath length as
the objective functions. Since the computation of those objective functions is
proportional to the number of swaths and the number of swaths is proportional
to the width of the field perpendicular to the direction of the driving angle,
the computational time grows proportional to the square root of the area of
the field. The computation time for all the objective functions can be
approximated by:

$T_{c}=C_{0}*\frac{\sqrt{A_{\bar{hl}}}}{R_{w}}+C_{1}$ (6)

where $T_{c}$ is the computation time, $C_{0}$ and $C_{1}$ are constants,
$A_{\bar{hl}}$ is the area of the field, and $R_{w}$ is the operational width
of the robot. This relationship is only true when the field is convex, so it
can be covered with the same pattern.

The field coverage is computationally the most demanding objective function
because it computes the difference between the field and the union of the
areas of each swath. Geometrical operations such as ’difference’ and ’union’
are more expensive than returning the number of swaths, which is the size of
the vector of swaths.

Computational time analysis is focused on the objective function of the brute
force algorithm because it consumes more than 80% of the total time of
coverage path planning.

###  III-B Field experiment

A field experiment was conducted using the AgBot shown in (Fig. 2). In the
extreme case shown in Figure 5, the AgBot covered an elongated narrow area.
Objective functions like the minimum swath length or the number of turns would
produce swaths parallel to the longest edge of the field. However, here we
show a coverage pattern given a custom angle that allows observing the turns
in the field. The produced swaths were sorted using the Snake pattern and
connected by Dubins’ curves.

The difference between the planned path and the recorded track in Figure 5 can
be explained by an offset between the controlled point (front part) and the
GPS antenna. Owing to this offset, the recorded GPS data has wider turns than
the planned path. Turns made with the snake pattern always skip one swath,
except for the turn at the rightmost part of the field where coverage
direction changes. This turn is sharper, causing wider tracks on the ground,
greater soil slippage, and thus more soil damage [33]. In spite of soil
slippage, the AgBot was capable of covering the field with the path designed
by the library routines.

Figure 5: AgBot covering a narrow area (shape on green). The coverage path
plan in red and the position of the AgBot in blue. The AgBot is halfway the
coverage task. The starting point is near the left edge of the area.

##  IV Conclusions & Future work

In this work, we introduced Fields2Cover, a Coverage Path Planning open-source
library for agricultural vehicles. Fields2Cover was implemented to bundle the
research knowledge on this topic and to help other developers to accelerate
their projects. Currently, it supports the creation of coverage paths for
convex fields, with a flexible and simple structure thanks to its modular
design. The library has four modules, which are: the headlands generator, with
a constant width headlands generator; the swath generator, with a brute force
optimizer; the route planner, with three types of patterns; and the path
planner, with Dubins’ and Reed-Shepp’s curves. The last three modules have
their own objective functions specific to their domains. Fields2Cover was
tested using simulation with a public benchmark and in a real field.

Fields2Cover is an on-going project, which means the functionality of the
library will be expanded in the coming years. Future developments are
supported and maintained by the first author of this paper, with the
collaboration of the open-source community. Concave fields, 2.5D terrains and
capacitated vehicles provide a good starting point for discussion and further
research.

## Acknowledgement

This publication is part of the project "Fields2Cover: Robust and efficient
coverage paths for autonomous agricultural vehicles" (with project number
ENPPS.LIFT.019.019 of the research programme Science PPP Fund for the top
sectors which is (partly) financed by the Dutch Research Council (NWO).

## References

  * [1] L. Christiaensen, Z. Rutledge, and J. E. Taylor, “The future of work in agriculture: Some reflections,” World Bank Policy Research Working Paper, no. 9193, 2020.
  * [2] R. Bormann, F. Jordan, J. Hampp, and M. Hägele, “Indoor coverage path planning: Survey, implementation, analysis,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1718–1725, IEEE, 2018.  https://doi.org/10.1109/ICRA.2018.8460566.
  * [3] K. R. Jensen-Nau, T. Hermans, and K. K. Leang, “Near-optimal area-coverage path planning of energy-constrained aerial robots with application in autonomous environmental monitoring,” IEEE Transactions on Automation Science and Engineering, vol. 18, no. 3, pp. 1453–1468, 2020.  https://doi.org/10.1109/TASE.2020.3016276.
  * [4] I. A. Hameed, “Coverage path planning software for autonomous robotic lawn mower using dubins’ curve,” in 2017 IEEE International Conference on Real-time Computing and Robotics (RCAR), pp. 517–522, IEEE, 2017.  https://doi.org/10.1109/RCAR.2017.8311915.
  * [5] T. Oksanen and A. Visala, “Coverage path planning algorithms for agricultural field machines,” Journal of field robotics, vol. 26, no. 8, pp. 651–668, 2009.  https://doi.org/10.1002/rob.20300.
  * [6] R. Jiaping, “Rjjxp/coverageplanning github.” https://github.com/RJJxp/CoveragePlanning.  Accessed: .
  * [7] T. Clephas, C. López, and Nobleo, “Nobleo/full_coverage_path_planner github.” https://github.com/nobleo/full_coverage_path_planner.  Accessed: .
  * [8] F. IPA, “Ipa320/ipa_coverage_planning github.” https://github.com/ipa320/ipa_coverage_planning.  Accessed: .
  * [9] R. Bormann, F. Jordan, W. Li, J. Hampp, and M. Hägele, “Room segmentation: Survey, implementation, and analysis,” in 2016 IEEE international conference on robotics and automation (ICRA), pp. 1019–1026, IEEE, 2016.  https://doi.org/10.1109/ICRA.2016.7487234.
  * [10] R. Baehnemann, L. Liu, and D. Kleiser, “Ethz-asl/polygon_coverage_planning github.” https://github.com/ethz-asl/polygon_coverage_planning.  Accessed: .
  * [11] R. Bähnemann, N. Lawrance, J. J. Chung, M. Pantic, R. Siegwart, and J. Nieto, “Revisiting boustrophedon coverage path planning as a generalized traveling salesman problem,” in Field and service robotics, pp. 277–290, Springer, 2021.  https://doi.org/10.1007/978-981-15-9460-1_20.
  * [12] J. I. Vasquez, “Irvingvasquez/ocpp github.” https://github.com/irvingvasquez/ocpp.  Accessed: .
  * [13] J. I. V. Gomez, M. M. Melchor, and J. C. H. Lozada, “Optimal coverage path planning based on the rotating calipers algorithm,” in 2017 International Conference on Mechatronics, Electronics and Automotive Engineering (ICMEAE), pp. 140–144, IEEE, 2017.  https://doi.org/10.1109/ICMEAE.2017.11.
  * [14] Greenzie, “Greenzie/boustrophedon_planner github.” https://github.com/Greenzie/boustrophedon_planner.  Accessed: .
  * [15] A. Stelter, “Ipiano/coverage-planning github: Quickopp implementation.” https://github.com/Ipiano/coverage-planning.  Accessed: .
  * [16] T. M. Driscoll, Complete coverage path planning in an agricultural environment.  PhD thesis, Iowa State University, 2011.
  * [17] J. Jin, “Optimal field coverage path planning on 2d and 3d surfaces,” , 2009.  https://doi.org/10.31274/etd-180810-3122.
  * [18] S. de Bruin, P. Lerink, I. J. La Riviere, and B. Vanmeulebrouk, “Systematic planning and cultivation of agricultural fields using a geo-spatial arable field optimization service: Opportunities and obstacles,” Biosystems Engineering, vol. 120, pp. 15–24, 2014.  https://doi.org/10.1016/j.biosystemseng.2013.07.009.
  * [19] R. J. Meuth and D. C. Wunsch, “Divide and conquer evolutionary tsp solution for vehicle path planning,” in 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 676–681, IEEE, 2008.  https://doi.org/10.1109/CEC.2008.4630868.
  * [20] K. Zhou, A. L. Jensen, D. D. Bochtis, and C. G. Sørensen, “Quantifying the benefits of alternative fieldwork patterns in a potato cultivation system,” Computers and Electronics in Agriculture, vol. 119, pp. 228–240, 2015.  https://doi.org/10.1016/j.compag.2015.10.012.
  * [21] M. Spekken, S. De Bruin, J. P. Molin, and G. Sparovek, “Planning machine paths and row crop patterns on steep surfaces to minimize soil erosion,” Computers and Electronics in Agriculture, vol. 124, pp. 194–210, 2016.  https://doi.org/10.1016/j.compag.2016.03.013.
  * [22] L. E. Dubins, “On curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents,” American Journal of mathematics, vol. 79, no. 3, pp. 497–516, 1957.  https://doi.org/10.2307/2372560.
  * [23] J. Reeds and L. Shepp, “Optimal paths for a car that goes both forwards and backwards,” Pacific journal of mathematics, vol. 145, no. 2, pp. 367–393, 1990.  https://doi.org/10.2140/pjm.1990.145.367.
  * [24] J. Backman, P. Piirainen, and T. Oksanen, “Smooth turning path generation for agricultural vehicles in headlands,” Biosystems Engineering, vol. 139, pp. 76–86, 2015.  https://doi.org/10.1016/j.biosystemseng.2015.08.005.
  * [25] D. Sabelhaus, F. Röben, L. P. M. zu Helligen, and P. S. Lammers, “Using continuous-curvature paths to generate feasible headland turn manoeuvres,” Biosystems engineering, vol. 116, no. 4, pp. 399–409, 2013.  https://doi.org/10.1016/j.biosystemseng.2013.08.012.
  * [26] M. Höffmann, S. Patel, and C. Büskens, “Weight-optimized nurbs curves: Headland paths for nonholonomic field robots,” in 2022 8th International Conference on Automation, Robotics and Applications (ICARA), pp. 81–85, IEEE, 2022.  https://doi.org/10.1109/ICARA55094.2022.9738525.
  * [27] R. S. Nilsson and K. Zhou, “Method and bench-marking framework for coverage path planning in arable farming,” Biosystems Engineering, vol. 198, pp. 248–265, 2020.  https://doi.org/10.1016/j.biosystemseng.2020.08.007.
  * [28] M. Nørremark, R. S. Nilsson, and C. A. G. Sørensen, “In-field route planning optimisation and performance indicators of grain harvest operations,” Agronomy, vol. 12, no. 5, p. 1151, 2022.  https://doi.org/10.3390/agronomy12051151.
  * [29] Intel, “oneapi threading building blocks.” https://github.com/oneapi-src/oneTBB.  Accessed: .
  * [30] D. M. Beazley et al., “Swig: An easy to use tool for integrating scripting languages with c and c++.,” in Tcl/Tk Workshop, vol. 43, p. 74, 1996.
  * [31] Google, “Protobuf on c++.” https://developers.google.com/protocol-buffers.  Accessed: .
  * [32] T. Field, J. Leibs, J. Bowman, and D. Thomas, “Rosbag package.” http://wiki.ros.org/rosbag.  Accessed: .
  * [33] A. Janulevičius and K. Giedra, “The slippage of the driving wheels of a tractor in a cultivated soil and stubble,” Transport, vol. 24, no. 1, pp. 14–20, 2009.  https://doi.org/10.3846/1648-4142.2009.24.14-20.