# Evaluating Plant Disease Detection Mobile Applications: Quality and
Limitations

Ayesha Siddiqua
Department of Computer Science and Engineering
Central Women’s University
Dhaka, 1203, Bangladesh
<EMAIL_ADDRESS>
Muhammad Ashad Kabir
School of Computing, Mathematics and Engineering
Charles sturt Univerity
Bathurst, NSW 2795, Australia
<EMAIL_ADDRESS>Tanzina Ferdous
Department of Computer Science and Engineering
City University
Dhaka, Bangladesh
<EMAIL_ADDRESS>
Israt Bintea Ali
Divine IT Limited
Dhaka, Bangladesh
<EMAIL_ADDRESS>
Leslie A. Weston
Gulbali Institute for Agriculture, Water and Environment
Charles Sturt University
Wagga Wagga, NSW, 2678, Australia
<EMAIL_ADDRESS>
###### Abstract

In this technologically advanced era, with the proliferation of artificial
intelligence, many mobile apps are available for plant disease detection,
diagnosis and treatment, each with a variety of features. These apps need to
be categorized and reviewed following a proper framework that ensures their
quality. This study aims to present an approach to evaluating plant disease
detection mobile apps, this includes providing ratings of distinct features of
the apps and insights into the exploitation of artificial intelligence used in
plant disease detection. The applicability of these apps for pathogen or
disease detection, identification, and treatment will be assessed along with
significant insights garnered. For this purpose, plant disease detection apps
were searched in three prominent app stores (the Google Play store, Apple App
store, and Microsoft store) using a set of keywords. A total of 606 apps were
found and from them 17 relevant apps were identified based on inclusion and
exclusion criteria. The selected apps were reviewed by three raters using our
devised app rating scale. To validate the raters’ agreement on the ratings,
inter-rater reliability is computed alongside their intra-rater reliability,
ensuring their rating consistency. Also, the internal consistency of our
rating scale was evaluated against all selected apps. User comments from the
app stores are collected and analyzed to understand their expectations and
views. Following the rating procedure, most apps earned acceptable ratings in
software quality characteristics such as aesthetics, usability and
performance, but gained poor ratings in AI-based advanced functionality, which
is the key aspect of this study. However, most of the apps cannot be used as a
complete solution to plant disease detection, diagnosis and treatment. Only
one app, Plantix - your crop doctor, could successfully identify plants from
images, detect diseases, maintain a rich plant database, and suggest potential
treatment for the disease presented. It also provides a community where plant
lovers can communicate with each other to gain additional benefits. In
general, all existing apps need to improve functionalities, user experience,
and software quality. Therefore, a set of design considerations has been
proposed for future app improvements.

_K_ eywords app evaluation $\cdot$ app design $\cdot$ artificial intelligence
$\cdot$ disease detection $\cdot$ plant disease $\cdot$ pathogen $\cdot$
mobile app $\cdot$ machine learning $\cdot$ smartphone

##  1 Introduction

Plant diseases are a naturally occurring phenomenon limiting plant growth,
development and reproduction [1]. Plant diseases and the pathogens causing
them are a direct threat to the global economy and food security [2]. Plant
diseases are typically caused by pests, including viral, bacterial, and
fungal-like organisms that hinder the normal growth of plants and cause
variations in their vital functions [3]. For example, Leaf Blast, Brown Spot,
Leaf Blight, Sheath Rot, and Stem Rot are common rice diseases which reduce
grain quality and destroy the majority of rice seedlings when badly infested
[4]. Phytophthora infestans or Late Blight, and Alternaria solani or Early
Blight, are two well-characterised diseases that cause significant yield
losses in potatoes [5]. Stripe Rust, Powdery Mildew, Leaf Rust, and other
diseases harm winter wheat in central Asia, causing significant grain yield
losses [6, 7].

In contrast, severe grape leaf diseases like Black Rot and Powdery Mildew have
a significant impact on grape yield and quality during the growth process,
particularly in wet or moist humid conditions [8, 9, 10, 11]. Soybean exhibits
numerous leaf diseases such as Anthracnose, Bacterial Blight, Soybean Mosaic
Virus, Copper Phytotoxicity, Charcoal Rot, Leaf Cercospora, Rhizoctonia Aerial
Blight, and Downy Mildew. Plant pathogens causing disease have become a great
menace when it comes to crops or other cultivated plat materials as our
livelihood is dependent on them [12]. Therefore, identifying plant diseases is
crucial for avoiding long-term or periodic harm to harvested produce, and
living landscape plants [13]. Early diagnosis of plant diseases allows for the
implementation of preventive measures and the reduction of economic and
production damage [5]. For sustainable agriculture, plant health observation
and predictive maintenance are essential for reducing losses related to
agricultural commodity yields and quality [14].

A traditional way of treating plant diseases is through the direct
consultation of farmers and horticulturists with plant pathologists or
outreach specialists in person. Experts physically examine different parts of
the symptomatic plant or test the soil to evaluate soil properties, nutrients
or pH and treatments are then suggested. However, the traditional method of
diagnosing plant diseases can be unreliable and limited by available expertise
[15, 16]. Continuous plant surveillance by land managers is not always
practical or cost-effective [17]. For this reason, mobile and other
technologies can provide a low-cost, high-precision alternative [18]. There
are various symptoms that can be used to identify plant pathogens and related
diseases [19]. Modern technology has been brought into the picture to enhance
the diagnostic process. This sector is modernized now with the advent of
technology as smartphones have made their way into everyone’s lives, including
those of rural farmers. As a result, many applications related to plants and
their pathogens/diseases have been built on many platforms in recent years
[20, 21, 22].

Existing plant and plant disease-related apps have a broad range of
functionalities, such as identifying of plants and diseases, disease severity
estimation, biological knowledge, agricultural solutions, marketplace
parameters, and more. Plant identification and disease detection are the most
critical and beneficial features of the apps as stakeholders take the required
steps based on the result of plant disease detection. To provide these
functionalities, image processing techniques, the grading method, artificial
intelligence (AI), machine learning (ML), and deep learning (DL) techniques
have been applied, supported by a large image data-set of plant varieties and
their diseases [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33].

Several apps claim to effectively recognize and treat plant diseases on
various platforms, but unfortunately, there is no accreditation of apps, which
is why individuals may face difficulty selecting the best app. Also,
verification of the functionality does not mean the app is user-friendly and
effective, as there are many evaluation metrics, such as user interface, error
rates, response time, transparency, etc., other than user ratings. So, plant
disease detection apps require a thorough analysis using a proper methodology.

To the best of our knowledge, no such structured study has yet been conducted
on plant disease detection, severity estimation and treatment related mobile
apps. A brief description of agricultural apps and plant disease apps has been
provided by researchers [34, 35, 36, 37], however, they did not focus on plant
disease detection, diagnosis and care related criteria. In contrast, this
study aims to review plant disease detection applications from three different
platforms (the Google Play store, Apple App store and Microsoft store) by
giving equal importance to the disease identification specific functionalities
and software quality ensuring criteria. In this paper, we have made the
following five major contributions:

  1. 1.

We have examined the existing apps accessible in the three major app stores
(i.e., Google Play store, Apple App store, and Microsoft store).

  2. 2.

We have devised an app rating scale for analyzing plant disease detection apps
by adopting and extending existing rating scales.

  3. 3.

We have evaluated the selected apps through our devised app rating scale and
and identified their design issues.

  4. 4.

We have analyzed app store user comments to better understand users’
expectations and perspectives.

  5. 5.

We have provided design guidelines that emphasize using artificial
intelligence for better plant disease detection app development.

The paper is organized as follows. Section 2 describes the methodology and
Section 3 presents our evaluation results. Section 4 discusses the limitations
of the reviewed apps and design considerations for better app development.
Finally, Section 5 concludes the paper.

##  2 Methodology

###  2.1 App search strategy

Apps were searched in the three most popular app stores: Google Play, Apple
App and Microsoft. A keyword-based search process has been employed following
a similar approach used in previous studies [38, 39]. The keywords that were
used included: “plant disease", “leaf disease", “plant disease detection", and
“leaf disease detection". To maintain transparency and clarity in reporting,
as well as the opportunity for other researchers to recreate the systematic
search procedure for apps, a structured strategy PRISMA was followed [40, 41].
The complete process of app searching, screening, and selection is presented
using the PRISMA diagram in Fig. 1. After removing the duplicate apps,
multiple screenings were performed to remove the redundancies and
irrelevancies by mutual consensus of the researchers of this paper. The
selected apps were further assessed quality-wise to avoid any bias. This
extensive search procedure was executed twice to ensure no relevant apps are
missed in the screening process.

Figure 1: PRISMA diagram of study

Inclusion and exclusion criteria were defined to select relevant apps. In the
screening process, an app was included if it is related to plant disease
detection and mentions this feature in the description or about section of the
app in the respective app store (inclusion criteria). The exclusion criteria
were: (i) apps that are not relevant to plant or plant disease detection; (ii)
apps that are only information based or educational about plant diseases;
(iii) apps under the category of games since they do not have any relevance
with the scope of our review; (iv) apps that provide a marketplace for farming
essentials such as fertilizers or pesticides, or provide expert farming
consultancy or advice; and (v) apps that are not in English as other languages
are not comprehensible to us.

The investigators collaborated on the inquiry, screening, and retrieval phases
of the app searching and collection processes. Each kept a separate list of
apps they discovered in app stores using the finalized procedure’s inclusion
and elimination parameters. The investigators used their personal devices to
decide which apps are suitable for selection. Some problems were faced during
the accumulation of the individual app lists, for instance an app was excluded
by one investigator but added by another investigator. Disagreements among the
investigators were settled by discussion before consensus was achieved. The
resulting app lists were combined to create the final list of apps for review
(n=606). Multiple screenings were performed to reach a consensus among
researchers to rule out apps that were irrelevant to the review. After
removing duplicate apps (n=12), title and description-based screening was
performed, which identified eligible apps (n=45) for the review, considering
the exclusion criteria. Those 45 apps were installed and screened, and in that
process, 28 apps were excluded for various reasons. This screening process
resulted in selecting 17 apps for our review.

###  2.2 Raters

The selected apps were evaluated by three raters using our devised mobile app
rating scale (see Section 2.3). The raters reviewed all the apps independently
in a Google form and then information was collected in a Google sheet and
processed later for assessing different parameters. The three expert raters
include one software developer with over four years of mobile app development
experience in a reputed company and two computer science graduates with two
years of mobile apps development experience. In addition, the raters were
under the guidance of a PhD graduate in software engineering with ten years of
mobile app research experience. He also acted as a fourth rater whenever there
was any discrepancy among the three raters and required to resolve the matter.

###  2.3 App rating scale for plant disease detection

A standardized approach was necessary to assess the mobile apps based on
various aspects of the mobile application. Moreover, such an approach allowed
us to assess and compare apps objectively, considering their functionality and
different software quality criteria. The Mobile App Rating Scale (MARS) for
mobile health apps [42] is a popular rating scale that provides a
multidimensional measure of the app quality assessment. We adopted and
extended the existing app rating scales [39, 43, 44, 42, 45, 46] for devising
our app rating scale for plant disease detection.

The set of sub-scales that were adopted and extended to our plant disease
detection apps’ rating scale from the mentioned tools was as follows: (i) app
classification; (ii) aesthetics; (iii) general features; (iv) performance and
efficiency; (v) usability; (vi) functionality (plant disease detection
related); (vii) transparency; (viii) subjective quality; and (ix) the app’s
perceived impacts on users. Some of the sub-scales for rating were adjusted
(e.g., app classification) or newly defined (e.g., functionality) as needed to
satisfy the objectives of plant disease detection apps. The rating scale was
constructed using the app quality standards clustered around the domains,
excluding the metadata portion. Each question was scored in a range of 1 to 5
using a Likert scale [47] or binary answer, depending on the type of question
presented to determine the criterion.

####  2.3.1 App metadata

Metadata provides basic information about the subject of interest. Table 1
lists the metadata such as platform, country of origin, business model
(free/paid), and the number of downloads for each of our evaluated apps.
However, this information has not been used in the app scoring process. For
our selected plant disease detection apps, metadata are extracted from the
respective app stores. The iOS app store does not provide download numbers and
thus 5 apps’ download numbers are missing in the table. We have found 5 apps
with 10K+ downloads. All the 5 iOS apps were pay to use, and the remaining 12
Android apps were free to download and use.

Table 1: Apps meta data. App name | Country of origin | Free/paid | Downloads | Platform
---|---|---|---|---
Pestoz Idenify Plant Diseases | India | Free | 10K+ | Android
AgroAI - Plant Diseases Diagnosis (Early Access) | Africa | Free | 10+ | Android
Cropalyser | Netherlands | Free | 10K+ | Android
PDDApp: Plant Disease Detection | Russia | Free | 1K+ | Android
Leafy | India | Free | 100+ | Android
PlantDoctor | India | Free | 1K+ | Android
Plantix - your crop doctor | Germany | Free | 100K+ | Android
Plant Disease Detector | Unknown | Free | 10+ | Android
Riceye | Unknown | Free | 5+ | Android
Cassava Plant Disease Identify | Unknown | Paid | – | iOS
Plants Disease Identification | Unknown | Paid | – | iOS
Garden Plants Diseases Detector | Unknown | Paid | – | iOS
Plant Disease Identifier | Unknown | Paid | – | iOS
Plant Diseases and Pests | Unknown | Paid | – | iOS
PlantifyDr | United States | Free | 10+ | Android and iOS
Leaf Doctor | United States | Free | 10K+ | Android and iOS
Agrio | United States | Free | 100K+ | Android and iOS
– Apple app store does not provide download information

####  2.3.2 Aesthetics

The aesthetics of an app plays an important role in a good user experience.
The attractiveness or look of an interactive user interface is referred to as
visual aesthetics, which is important for the functionality and usability of
an app since it affects users’ first impressions. Users’ emotions and
emotional reactions can be generated by the visual components of technology
products [48]. It is a critical component in the user interface’s effective
usability, satisfaction, trust and credibility, and preference [49, 50, 51,
52].

Existing plant disease detection apps have been developed for gardeners,
students and producers. So, the buttons, icons, or menus on the screen must be
appropriate and zoomable. The resolution of graphics must also be good.
Producers from rural or urban areas use such apps, so the apps’ visual
information and design must be appropriate for the target audience. Thus,
aesthetics was rated on these criteria for each app – layout, graphics, visual
appeal, and appropriateness for the target user group.

####  2.3.3 General features

Several general features are expected in a plant disease detection app,
including data export option, login or sign-up, notification, multi-languages
support, and on-board tutorial for a smooth user experience. Some of these
features are more objective than subjective. An app must provide basic
functionality, like social sharing and data exporting. These capabilities
allow users to save information for later use and share it with others. Thus,
the absence of these characteristics was rated as a low point. A premium
subscription option is also an important aspect of an app as consumer spending
on premium apps is expected to reach $270 billion by 2025, over 2.5 times what
it was in 2020 [53].

A plant disease detection app must also provide an on-board tutorial so that a
new gardener/user can understand clearly how to use the app (e.g., taking
picture, etc.). If the app provides customization features, it helps users to
set their preferred language and use it with more comfort. Notifications are
designed to make it easier for users to remember new features or messages and
improve the app’s usage frequency. It can also alert users to take routine
care of their plants or crops. Thus, all these above-mentioned criteria were
included in the general features sub-scale of our rating scale.

####  2.3.4 Performance and efficiency

A plant disease detection app requires a compatible device to function
properly. For example, the device must have a resonable camera to take high
quality photos of plants or leaves and roots. The app should also offer a
quick cure or suggestions for users facing a problem with their plants. This
sub-scale includes CPU performance, memory usage, battery consumption, the
extent of heating up of the device, smooth working UI components, whether the
app crashes while running, etc. Thus, these criteria were included in our
rating scale’s the performance and efficiency sub-scale.

####  2.3.5 Usability

The term “usability" refers to the entire relationship between the user and
the product [54]. It directly impacts how users feel about an app and may help
them become long-term users. Even if people run into issues, a solution should
be simple to discover. The usability attribute analyzes how simple a system
interface is to use [55]. Usability testing is critical in determining if an
app is of sufficient quality to attract the attention of its desired user
groups [56]. We consider ease of use, ease of navigation between components,
and whether interactions between components are intuitive in the usability
sub-scale of our app rating scale.

####  2.3.6 Functionality

Functionality validates the goal of an app and its purpose. People search for
apps in app stores using essential functionality as a keyword, demonstrating
how significant this criterion is for an app [57]. The functionality
assessment criteria for plant disease detection apps included plant
identification, plant coverage, disease detection, infected area
visualization, disease severity estimation, treatment and expert/community
support.

To define the rating scale of those assessment criteria, we emphasize
technological advancement, such as the adoption of artificial intelligence
that provides automation. Manual techniques in conventional planting processes
cannot cover huge plantations or give critical early instructions in decision-
making processes [58]. As a result, it is important to create automated
solutions that are practical, reliable, and cost-effective in monitoring plant
health and delivering useful information to decision-makers, such as disease
detection, estimating its severity and recommending the correct dose of
pesticides in the treatment [59]. The functionality sub-scale criteria and the
definitions of the rating scores to evaluate those criteria are presented in
Table 2. We discuss each of those assessment criteria below.

Table 2: Functionality assessment criteria definition. Functionality assessment criteria | Rating 5 | Rating 3 | Rating 1
---|---|---|---
Plant identification | Automatically from image | Manually | Not at all
Plant coverage | 10+ | 1 to 10 | 0
Disease detection | Automatically from image | From questionnaire | Not at all
Visualize infected area | Automatically | Required user input | Not at all
Disease severity estimation | Automatically | Required user input | Not at all
Treatment | Up to date suggestions | Fixed suggestions | Not at all
Expert/community support | Expert and community | Expert or community | Not at all

There is a strong demand for automated plant identification that aids amateur
users who lack specialized knowledge in plant science in identifying plant
species [60]. In recent years, we have witnessed ample research on artificial
intelligent-based automatic plant identification from plant images [24, 61].
The leaf is an appropriate organ for this identification system as it has
several distinct features, such as shape, vein, and texture [61, 62]. Thus,
leaf identification is critical in plant classification [63, 62]. [61]
provided a thorough overview of image processing techniques for this automatic
plant identification from the leaf along with some frequently used machine
learning algorithms such as SVM, KNN, RF, K-Means, and Hypersphere
classifiers. For large-scale plant classification, the deep learning method
that uses a convolutional neural network (CNN) to generate discriminating
features for plant classification has become more prominent recently [24, 64,
65, 66]. Hence, a rating of 5 has been given for the automation of plant
identification, which means the app will identify the plant from the given
picture. A rating of 3 has been assigned for the manual selection of the plant
as plant identification is crucial for disease detection. Here, users will
select a plant from the given list of plants in the app, or they will have to
mention the name of the plant. The lowest rating of 1 has been given to the
lack of plant identification.

The range of different plants for pathogen diagnosis is also a critical
aspect, as with the inclusion of various plants, apps can cover a much broader
range of diseases. The range of plants was established by examining the apps
and appropriate values were assigned on the rating scale. Thelarger the plant
coverage, the higher its value on the rating scale and lowest value has been
given for zero plant coverage which means app does not cover any single plant.

Deep learning is a popular machine learning technique for detecting plant
disease due to its automatic prediction from large amounts of data with high
accuracy. One such commonly used dataset is “PlantVillage" [67], which
contains 54K+ healthy and unhealthy leaf photos in 38 groups based on species
and disease [68]. Using leaf images of healthy and sick plants, customized
deep learning models based on unique convolutional neural network
architectures have been constructed to diagnose of plant diseases [66]. Many
conventional deep learning architectures are paired with optimization and
customization approaches to provide considerable accuracy using technology in
plant disease detection [69, 70, 66]. The performance of several deep learning
techniques has been studied and compared [71]. Therefore, it is expected that
mobile apps should implement them to automate disease detection. Thus, the
highest value of 5 has been assigned in the rating scale for automatic disease
identification from the plant image. A rating of 3 has been assigned if any
app identifies the disease through a questionnaire, and a rating of 1 has been
given for the inability to detect disease.

After recognizing the plant disease, identifying the infected portion and its
severity estimation are crucial tasks as they will inform the user about the
disease’s threat level. Severity estimation refers to the precise measurement
of the leaf area showing disease symptoms [72]. The unhealthy region of the
plant leaves can be detected by the color space conversion, then masking the
green pixels (as they indicate the healthy areas of leaves) and finally
applying a threshold value to remove the green pixels so that the image
contains only diseased areas [73]. A novel network, PD2SE-Net [74], has been
introduced to diagnose and estimate the severity of plant diseases. A U-Net
architecture has also been utilized to segment sick leaf areas [72]. So, if
the result can be gained automatically from the image for both the
visualization and severity estimation of diseased areas, the highest value of
5 has been assigned on the rating scale. On the other hand, if the user needs
to point out the diseased portion of a leaf before the result is obtained, it
has been rated with a moderate value of 3. A value of 1 is assigned for the
absence of these features.

Recognizing a critical issue is important, but solving the problem is even
more important. The same may be said for apps for identifying plant diseases.
After the disease has been identified, the user must know what to do for the
cure. Apps can recommend a cure from their own solutions or link users to
Google’s solutions. Because the information provided by Google is recent, the
highest rating of 5 has been assigned for it. A rating of 3 has been assigned
for the fixed treatment information.

Even after all the solutions, getting aid from professionals or sharing any
sort of problem with fellow users feels fantastic. As a result, community
support is a valuable resource, and users become more engaged to the app
because of their sharing. It also serves as a link between users and
specialists. So, the highest rating of 5 has been assigned for any sort of
expert and community assistance, and a lower rating of 3 for solely community
assistance.

####  2.3.7 Transparency

In any development, transparency refers to the open and honest sharing of a
project’s status in real-time. The transparency of mobile apps has been a
significant concern for consumers in recent times [75]. Nowadays, software
developers are dedicated to gaining and maintaining user trust, which is
critical to the industry’s growth and development. However, users are
uninformed of how information is stored and disseminated because data
transmission occurs in the background of apps [76]. Also, trust is earned by
providing explicit descriptions of what each app performs and acting in
accordance with those explanations. Consumers’ sharing of data between apps
will be available to all users, regardless of the app’s ability to work or not
work. Consumers will choose when and how much they can share with other
people. The following factors were examined while evaluating the transparency
requirements of the selected apps: (1) validity of the information provided in
the app store description; (2) authenticity of the publisher or developer and
the app source; (3) evidence-based app trial; (4) determining whether the app
is feasible to achieve the developer’s stated goals; and (5) a general alert
to ask users’ permission before collecting personal or private data and
location information [75].

####  2.3.8 Subjective quality

An app’s subjective quality is related to the user’s opinion, satisfaction,
dissatisfaction, or overall view of the app. To assess the subjective quality,
the raters looked at how satisfied they are with an app, how likely they are
to pay for it, and how they would rate it overall. This sub-scale considers
the user’s willingness to pay for the app, the user’s recommendation for the
app, and the user’s rating of the app.

####  2.3.9 Perceived impact of users

The use of an app can influence a person’s attitude toward relevant areas of
their life. Considering plant disease detection apps can influence a user’s
knowledge of plant diseases, a well-informed app can make a person’s gardening
life easier and improve a user’s knowledge about relevant areas. Furthermore,
such apps can influence people to garden or gain a better understanding of
plant care issues. Therefore, the criteria to evaluate an app’s perceived
impact on its users are defined as: (1) improve awareness of the necessity of
addressing plant diseases; (2) increase knowledge or understanding of plant
diseases; (3) change people’s minds about how to improve plantation and care;
(4) improve intentions or motivation to address gardening; (5) stimulate
additional plant management help seeking; and (6) increase plant disease
treatment.

##  3 Results

###  3.1 Internal consistency of our app rating scale

Internal consistency describes the extent to which all the items in a test
measure the same concept or construct (in our case the questions/items used in
a sub-scale/assessment criteria), such that the items are consistent with one
another and measuring the same thing [77]. We used Cronbach’s alpha, the most
popular means of calculating internal consistency [78]. Cronbach’s alpha
($\alpha$) reliability coefficient indicates internal consistency that ranges
between 0 and 1, with $\alpha\geq 0.9$ excellent, $0.8\leq\alpha<0.9$ good,
$0.7\leq\alpha<0.8$ acceptable, $0.6\leq\alpha<0.7$ questionable,
$0.5\leq\alpha<0.6$ poor, and $\alpha<0.5$ unacceptable [79]. The closer the
value to 1, the higher the internal consistency.

For determining internal consistency, we placed all the apps we rated using
our proposed rating scale. Table 3 reports the internal consistency of the
sub-scales of our devised rating scale – aesthetics, performance, usability,
subjective quality, transparency and perceived impact. We excluded two sub-
scales – general and functionality, as their items are not meant to be
collective measures of the construct. The overall internal consistency of our
modified scale was high at alpha 0.97, which is excellent [79].

Table 3: Internal consistency of the devised app rating scale. App rating sub-scale |  Cronbach’s alpha ($\alpha$) |  Internal consistency
---|---|---
Aesthetics |  0.93 |  Excellent ($0.9\leq\alpha$)
Usability |  0.92 |  Excellent ($0.9\leq\alpha$)
Performance |  0.85 |  Good ($0.8\leq\alpha<0.9$)
Subjective |  0.94 |  Excellent ($0.9\leq\alpha$)
Transparency |  0.85 |  Good ($0.8\leq\alpha<0.9$)
Perceived impact |  0.95 |  Excellent ($0.9\leq\alpha$)
Overall |  0.97 |  Excellent ($0.9\leq\alpha$)

###  3.2 Inter-rater and intra-rater reliability

Inter-rater reliability is a way of quantifying the level of agreement between
two or more raters who rate an item (in this case, an app) independently based
on a set of criteria [80]. We used the intra-class correlation (ICC) method to
assess inter-rater reliability. ICC is one of the most widely used statistics
for evaluating inter-rater reliability if a study includes two or more raters
[81]. In our study, all apps were rated by the same three raters. Thus, we
used the ICC two-way mixed model as it is recommended when the raters are
fixed and each of the apps is rated by all raters [82]. Depending on the 95%
confidence interim of the ICC estimation, values smaller than 0.5, within 0.5
and 0.75, within 0.75 and 0.9, and higher than 0.90 suggest poor, moderate,
good, and excellent reliability, respectively [82]. The ICC score of our 17
apps was calculated as 0.869 (95% CI ranging from 0.847 to 0.888), showing a
good level of inter-rater reliability.

Intra-rater reliability is estimated to measure how consistent an individual
is at measuring a set of criteria. This is a reliability estimation in which
the same evaluation is performed by the same rater on more than one occasion.
To measure the intra-rater reliability of the three raters, the three raters
reviewed all 17 apps twice in a two-months interval. All three raters showed a
significant good level of intra-rater reliability between their two ratings;
their two-way mixed ICC values were 0.921 (95% CI 0.851 – 0.958), 0.985 (95%
CI 0.972 – 0.992), and 0.869 (95% CI 0.847 – 0.888), respectively.

###  3.3 Overall assessment of evaluated apps

The assessment scores for all 17 selected apps are reported in Table 4, with
sub-scale ratings and their mean, standard deviation. The mean of the scores
of all criteria in a sub-scale is used to calculate the rating of that sub-
scale. The overall mean and standard deviation of an app’s scores in all sub-
scales were used to evaluate its overall quality.

Table 4: Assessment scores for plant disease detection apps.

App name |  Aesth etics |  Gene ral |  Perfor mance |  Usabi lity |  Functio nality |  Subje ctive |  Transp arency | Impact |  Mean (Std dev)
---|---|---|---|---|---|---|---|---|---
Agrio | 4.75 | 2.25 | 4.67 | 4.00 | 3.57 | 2.83 | 2.80 | 3.83 | 3.53 (1.56)
AgroAI - Plant Diseases Diagnosis (Early Access) | 4.75 | 2.13 | 4.83 | 3.67 | 3.00 | 4.83 | 3.80 | 4.15 | 3.81 (1.50)
Cassava Plant Disease Identify | 2.25 | 3.25 | 4.33 | 1.83 | 1.67 | 1.33 | 2.20 | 3.00 | 2.48 (0.99)
Cropalyser | 5.00 | 3.13 | 4.83 | 3.17 | 2.14 | 3.33 | 4.40 | 3.83 | 3.65 (1.41)
Garden Plants Diseases Detector | 3.00 | 3.25 | 4.33 | 2.83 | 2.71 | 2.50 | 3.40 | 3.17 | 3.19 (1.47)
Leaf Doctor | 5.00 | 3.50 | 4.83 | 3.00 | 1.86 | 3.00 | 4.40 | 1.17 | 3.33 (1.77)
Leafy | 3.75 | 2.38 | 5.00 | 4.00 | 3.29 | 3.00 | 2.80 | 2.33 | 3.26 (1.69)
PDDApp: plant disease detection | 3.00 | 2.38 | 3.50 | 2.83 | 2.43 | 2.17 | 2.80 | 1.00 | 2.44 (1.55)
Pestoz Identify Plant diseases | 4.75 | 2.38 | 4.67 | 5.00 | 2.71 | 4.00 | 3.40 | 3.33 | 3.56 (1.48)
Plant Diseases and Pests | 1.00 | 2.75 | 4.50 | 1.50 | 3.00 | 1.83 | 2.60 | 1.67 | 2.42 (1.69)
Plant Disease Detector | 4.75 | 2.50 | 3.33 | 3.00 | 2.71 | 1.67 | 1.80 | 1.00 | 2.63 (1.75)
Plant Disease Identifier | 3.00 | 3.25 | 4.33 | 2.67 | 2.71 | 2.67 | 3.40 | 3.17 | 3.19 (1.47)
PlantDoctor | 2.50 | 2.00 | 3.00 | 1.83 | 1.57 | 1.00 | 1.00 | 1.00 | 1.81 (1.35)
PlantifyDr | 3.50 | 2.00 | 4.83 | 4.17 | 3.00 | 3.67 | 2.80 | 3.67 | 3.37 (1.65)
Plantix - your crop doctor | 5.00 | 4.13 | 5.00 | 4.33 | 3.29 | 5.00 | 5.00 | 5.00 | 4.56 (1.16)
Plants Disease Identification | 2.00 | 2.50 | 4.50 | 3.50 | 2.71 | 1.83 | 2.80 | 3.00 | 3.00 (1.53)
Riceye | 3.00 | 2.25 | 4.83 | 2.50 | 2.43 | 1.83 | 2.20 | 1.00 | 2.53 (1.65)

In aesthetics, 41.18% (7/17) of the apps are rated above 4 out of 5. Among
them three apps, Cropalyser, Plantix - your crop doctor, and Leaf Doctor,
receive the highest score, i.e., 5 out of 5. Only one app, Plant Diseases and
Pests, receives the lowest score of 1.

In general characteristics, all the apps except Plantix - your crop doctor,
received a score below 4. In terms of login or sign-up, 82.36% (14/17) of the
apps do not require users to login or sign-up to use - this allows the rural
farmers and gardeners to easily benefit from the app. However, only three apps
(17.67%) send users notifications about their regular tasks. Most of the apps
94.11% (16/17) do not have a premium subscription. The user cannot export
their data in 70.59% (12/17) of the apps. 52.94% (9/17) of the apps do not
present information visually through charts, graphs, images or videos and do
not have any tutorial or on-boarding facilities to help users operate the
apps. Also, 64.71% (11/80) of the apps do not have customization features. In
this sub-scale, the app Plantix - your crop doctor received the highest score
of 4.13.

In the performance sub-scale, 14 out of 17 apps (82.36%) received a high score
of between 4.33 and 5, which means they are responsive, their components are
working well, the apps do not crash, they consume low battery power and have a
reasonable memory. Among those 14 apps, Leafy and Plantix - your crop doctor,
scored 5 out of 5, which indicates that these apps show excellent performance
while being used.

In the usability sub-scale, 82.36% (14/17) of the apps were very easy to use
and scored 4 to 5, 70.59% (12/17) had high navigational accuracy, and 58.82%
(10/17) depict a high quality of gestural design. Overall, only a few apps
29.41% (5/17) scored high (between 4 and 5) in this sub-scale. The lowest
rated app in this sub-scale is Plant Diseases and Pests, which scored 1.50.
Only the app Pestoz - Identify Plant diseases scored 5 out of 5 in usability.

The functionality sub-scale is the most significant sub-scale as it evaluates
the features an app provides for disease identification and management. In our
review, only 6 out 17 apps (35.29%) scored between 3 and 3.57 and the rest
scored below 3. The results indicate that apps with proper automation of
identifying plants and diseases along with disease severity estimation are
mostly missing in the current apps.

The transparency sub-scale is assessed based on an app’s description in the
app store, credibility through a legitimate source, verified by evidence,
goals and policy in accessing and sharing user data. In this sub-scale, only 3
out of 17 apps (17.65%) scored above 4. Among them, Plantix - your crop doctor
scored 5 out of 5. Two apps, Cropalyser and Leaf Doctor, each scored 4.40.

The subjective quality sub-scale is measured based on an individual’s
willingness to use, recommend people and pay for the apps, and the overall
star rating given by raters. In this sub-scale, 3 out of 17 apps (17.65%)
scored 4 and higher. Among them, Plantix - your crop doctor scored the highest
value of 5. PlantDoctor scored the lowest value of 1.

The perceived impact on user sub-scale measures the effectiveness of an app in
changing the attitudes of the farmers and plant lovers who use the app. This
was evaluated based on whether the app provides a community or forum to share
info and seek help. Eight out of 17 apps (47.05%) help enormously to increase
awareness of the importance of addressing plant diseases and behavior change
toward crop illness treatment, 58.82% (10/17) are likely to increase knowledge
or understanding of infected plants; 29.41% (5/17) do not have concern about
these observations. Two out of 17 apps (11.76%) (AgroAI - Plant Diseases
Diagnosis and Plantix - your crop doctor) rated 4.15 and 5, respectively. They
can change attitudes toward improving plantations and care. The app, Plantix -
your crop doctor, provides strong motivation for users in their gardening and
farming. On the other hand, 23.53% (4/17) of the apps scored 1 in this sub-
scale.

From the overall screening of the mean values of the evaluated apps, Plantix -
your crop doctor, scored the highest (4.56), but the functionality score for
this app was 3.29. Most of the apps (10/17) got mean values ranging from 3 to
4, indicating that the quality of the apps is not up to the mark and they are
specifically lagging in functionality.

Fig. 2 shows the average sub-scale specific scores and the total mean score of
all 17 apps. The overall performance and usability of the evaluated apps are
very good (scored above 4.5), which means they are easy to use and fast to
open. However, the overall aesthetics value is average because few apps are
outstanding and eye-catchy to look at, and most are not so attractive. Our
focus is the functionality sub-scale, which scored below 2.5. This value
indicates that we have not found any quality apps that can do all the desired
tasks regarding plant disease identification.

AestheticsGeneralPerformanceUsabilityFunctionalitySubjectiveTransparencyImpactTotal
mean$0$$0.5$$1$$1.5$$2$$2.5$$3$$3.5$$4$$4.5$$5$Sub-scalesRating

Figure 2: Sub-scale specific ratings and overall rating.

###  3.4 Assessment of app functionality

Plant disease identification apps should contain some major criteria for
fulfilling their sole intention. These apps are meant to help farmers and
gardeners by providing solutions regarding their plants. However, the reality
is different because the apps lack the functionalities to provide necessary to
help the target users. We have already discussed seven critical
characteristics for plant disease detection apps in Section 2.3.6. They are
(1) plant identification; (2) plant coverage; (3) disease detection; (4)
visualize infected area; (5) disease severity estimation; (6) treatment; and
(7) expert/community support. The outcomes of our functionality analysis of 17
evaluated applications based on the seven assessment criteria are shown in
Fig. 3.

(a) Plant identification

(b) Plant coverage

(c) Disease detection

(d) Visualize infected area

(e) Disease severity estimation

(f) Treatment

(g) Expert/community support

Figure 3: Representation of specific functionality rating ratio.

First, identifying the plants is a preliminary criterion for plant disease
detection apps. Among the 17 reviewed apps, 47.06% (8/17) can automatically
identify plants from the given image and 41.18% (7/17) kept the option of
choosing the plants manually before diagnosing any disease. Only two apps,
Leaf Doctor and Riceye, do not fulfill this criteria as Leaf Doctor focuses
only on disease severity and visualizing the infected area of the leaf, where
Riceye is dedicated for rice crops only. Different apps have adopted different
technologies to identify plants. For example, the PlantifyDr app uses ML
algorithms to identify a plant and detect whether a specific plant has a
disease or not. Plant identification by PlantifyDr is shown in the Fig. 4.
Another key criterion in our study was to identify the plant coverage, i.e.,
the number of plants covered by the disease detection apps. The only app that
missed this criterion is Leaf Doctor. Most of the apps, nearly 53% (9/17), can
identify diseases of more than 10 plants.

Figure 4: Plant identification by PlantifyDr.

Our most significant functionality is disease detection where it is expected
that an app would automatically recognize the disease from the photos of
affected plants or leaves. Fortunately, 82% (14/17) of the apps have fulfilled
that expectation. Only one app, Cropalyser, identifies diseases based on
answers given by the user from a series of questionnaires. On the other hand,
the Leaf Doctor and Plant Doctor apps do not provide this functionality. Our
reviewed apps use a variety of techniques to detect diseases. For instance,
Plants Disease Identification app uses ML Apple technology to classify the
plant diseases productively. The apps Cassava Plant Disease Identify and
Garden Plant Diseases Detector use computer vision techniques to classify the
disease and monitor severity. To get the maximum accuracy in disease
detection, a large image database containing thousands of images has been used
for developing a model using artificial neural networks (ANN). The app Cassava
Plant Disease Identify allows user to report photos that are not properly
captured. As more photographs are uploaded, the accuracy of the database
improves. User can notify the developer of the app by email, if a photograph
is not recognized. The app is improving its accuracy over time by using ML.
The app Pestoz- Identify Plant diseases also applies advanced computer vision
techniques to identify the diseases. Fig. 5 shows screenshots of the disease
detection UI of the Plantix and AgroAI apps.

(a) Plantix - your crop doctor

(b) AgroAI - Plant Diseases Diagnosis (Early Access)

Figure 5: Disease detection.

Another important criterion is disease severity estimation which is provided
by only four apps, AgroAI - Plant Diseases Diagnosis (Early Access), Leaf
Doctor, Riceye, and Cassava Plant Disease Identify. Visualization is also a
useful measure to assess the severity of disease and identify the infected
area of a plant leaf.

Only Leaf Doctor provides this functionality. This app helps users
differentiate between damaged and healthy leaf tissues and calculates disease
severity as a percentage, as shown in Fig. 6. However, the app requires user
knowledge and input. The user needs to select up to 8 distinct colors that
indicate healthy tissues on the app’s touch panel and subsequently moves a
threshold slider until only symptomatic tissues are changed into a bluish
shade. Afterwards, the app calculates the diseased percentage using that
pixelated photo. The precision, accuracy, and stability of the Leaf Doctor app
have been tested using six diseases and typical lesions of increasing severity
[83].

Figure 6: Severity detection and visualizing the infected area of diseased
leaf by Leaf Doctor.

Providing treatment and care suggestions for plants is an essential
functionality to motivate farmers and gardeners to use plant disease detection
apps. Two apps, Leafy and PlantifyDr, detect the plant disease and
subsequently take users directly to Google to get the latest information about
the treatment (see Fig. 7(a) for UI screenshots of PlantifyDr). Nine apps
provide treatment information from their own database. Only the app Agrio
provides treatment through a community (see Fig. 7(b)).

(a) PlantifyDr

(b) Agrio

Figure 7: Treatment.

Local farmers can contact with field investigators through Agrio app to
receive benefits. This app uses drones for remote field investigation. Users
of Agrio app can look at remote sensing insights, alerts, and data gathered
during the inspection process. They have a community where farmers can share
their problems and intelligent sources quickly provide necessary solutions or
treatments. Within the groups, discussions about the necessary interventions
are becoming simpler. This app also uses AI which makes precise hyper-local
weather forecasts easily available to all growers. Growing degree days (GDD)
is a measure used to calculate the amount of heat required for the development
of organisms (such as insects) in each stage of their growth. Monitoring GDD
helps eliminate the guesswork in determining the time required for control
measures. One of the exciting possibilities is the alignment of treatment
schedules in different farms and gardens. Area-wide integrated pest management
is the paradigm in which pest control decisions and timing are coordinated
across many fields.

Agrio’s daily briefing informs growers of the required scouting operations and
interventions in their fields and also uses big-data to optimize predictions
and offer phenology models specific to the different locations. They
coordinate area-wide integrated pest management operations and present users
with the optimal treatment on time using AI. Growers can use image
identification capabilities if help is needed with the trap analysis.

One extraordinary feature to connect the user with fellow gardeners, farmers
and experts is the creation of a community. Only two apps provide this
feature, Plantix - your crop doctor and Agrio. Fig. 8 shows the community
support UI screenshots of these two apps.

(a) Plantix - your crop doctor

(b) Agrio

Figure 8: Expert/community support.

The Plantix - your crop doctor community is the world’s largest online
community of farmers and agricultural specialists where users can take
advantage of agricultural professionals’ knowledge and share their crop
related problems to get solutions. The Agrio app provides a community with
experts who respond to the users’ problems more individually and provide
treatment based on the issue, as discussed above. In conclusion, this
expert/community support feature should be included in all the apps for better
user connectivity.

Our overall assessment of the functionality criteria for the apps is
illustrated in Table 5. Here, we can see that 88.23% of the apps can identify
plants and detect diseases. In these sectors, apps from both app stores showed
almost equal contributions. The most important criterion for our apps was
disease detection but two of the apps Leaf Doctor and PlantDoctor did not
provide this functionality. About 94% of the apps cover more than 10 plants,
and, in this case, Android apps have a richer plant disease database than iOS
apps. Sixty four percent of the apps provide treatment after diagnosing plant
disease. Most of our reviewed apps lack three functionalities - disease
severity estimation, visualizing the infected area, and expert/community
support. Of all the apps, 23.52% can estimate the severity of the disease from
a leaf, 11.76% have expert/community support, and only one app can visualize
the infected area from an infected leaf.

Among the 17 reviewed apps, Agrio, AgroAI - Plant Diseases Diagnosis, Cassava
Plant Disease Identify and Plantix - your crop doctor, contain a total of five
out of seven functionalities. Seven apps, Pestoz Identify Plant diseases,
Cropalyser, PDDApp: plant disease detection , Leafy, Plant Disease
Detector,PlantifyDr, and Plant Diseases and Pests, contain four
functionalities. The remaining six apps contained less than half the
functionalities outlined. In our study, no app was found to have all seven
functionalities for the best plant disease detection and management.

Table 5: Assessment criteria for the functionality of apps. Functionality assessment criteria |  Google Play (n=9) n (%) |  Apple App (n=8) n (%) |  Total (N=17) N (%)
---|---|---|---
Plant identification |  8 (88.89) |  7 (87.50) |  15 (88.23)
Plant coverage |  9 (100) |  7 (87.50) |  16 (94.12)
Disease detection |  8 (88.89) |  7 (87.50) |  15 (88.23)
Disease severity estimation |  2 (22.22) |  2 (25) |  4 (23.52)
Visualize infected area |  0 (0) |  1 (12.50) |  1 (5.88)
Treatment |  7 (77.78) |  4 (50) |  11 (64.71)
Expert/Community support |  1 (11.11) |  1 (12.50) |  2 (11.76)

###  3.5 Analysis of app ratings from the app store and the developed rating
scale

The app stores’ ratings and our rating scale’s ratings of the evaluated apps
were analyzed. Reviews and ratings help users decide whether to download an
app. According to a study [84], mobile consumers do not even bother
downloading an app with a rating of fewer than three stars. Additionally, 79%
of users will check at least one review before downloading the application.
Good ratings indicate that the app provides an advantage to its users.
Developers can do everything to improve the app’s rating and make it more
successful in the future. During this survey period, AgroAI - Plant Diseases
Diagnosis (Early Access), Leafy, Plant Disease Detector, Riceye, Cassava Plant
Disease Identify, Garden Plants Diseases Detector, Plant Diseases and Pests
and Agrio apps dis not have any star ratings. Thus, these seven apps were
excluded from this analysis. Fig. 9 shows the app store rating and our
measured app rating of ten apps.

$0$$1$$2$$3$$4$$5$PDDApp: plant disease detectionPlants Disease
IdentificationPlantDoctorLeaf DoctorPlant Disease IdentifierPestoz Identify
Plant diseasesAgrioCropalyserPlantix - your crop doctorPlatifyDrApp
rating/scoreApp NameMeasured scoreApp store rating

Figure 9: App store ratings (omitted apps those ratings were not available)
and measured ratings of the apps.

Three apps (PlatifyDr, Cropalyser, and Plant Disease Identifier) got a higher
star rating from app store users, but some limitations were found in our
analysis, they did not perform well in our devised rating scale. Consequently,
these apps achieved a lower rating from our raters. The measured score and app
store ratings are almost the same for four apps (Agrio, Pestoz Identify Plant
diseases, PlantDoctor, and Plantix - your crop doctor), which shows that our
raters agree with the users’ ratings found from the app stores. Finally, three
apps (Leaf Doctor, Plants Disease Identification, and PDDApp: plant disease
detection) received a slightly higher score compared to the app stores’
ratings.

###  3.6 Analysis of user reviews from the app stores

Text reviews are more reliable than star ratings in describing users’ actual
experiences. After using a mobile app, users can add comments in the user
reviews about that app. The user review section of an app store represents the
experience of using the app. Users’ feedback helps potential new users and
developers to understand an app’s quality. The developer benefits from the
evaluation because it focuses on the app’s limitations and technical flaws,
allowing them to create a better version of the app.

In this research, to understand the overall user experience of those apps, two
word-clouds have been created, as shown in Fig. 10. Word-clouds are graphical
representations of word frequency. The larger the word in the graphic, the
more frequently that word appear in the text [85].

(a) Positive comments

(b) Negative comments

Figure 10: Word-cloud of positive and negative comments.

User comments from the app listing from the respective app stores along with
the app metadata were collected for this study. Based on the users’ review
ratings of the app, the consumer comment sections were split into two kinds -
the comment is positive if its rating is 4 stars or above, otherwise it is
considered negative. The study of user feedback from both app stores on plant
disease detection apps reveals that most apps have little or no strong
feedback or favorable ratings from smartphone users because the count of apps
installed was generally poor for most of the relevant applications. Moreover,
Apple App store does not show the number of downloads of an app.

However, 35.30% (6/17) of the apps contain informative comments and have 10K+
to 10M+ installations. A word-cloud is generated in Fig. 10(a) using the
positive comments collected from the selected apps. One of the most optimistic
collections of user comments has been noticed for Plantix-your crop doctor.
This app has already received 60K+ comments, with 65% being 5-star reviews.
Among the 35K+ highest star rating comments, people mentioned the good points
of this app. Users have used several words to express their satisfaction, such
as “helpful", “wonderful", “amazing", “fantastic", “awesome", “very good",
“great", “best", “excellent", “useful", “easy to use", “accurate", “nice" and
many more. Many users gave “thanks" and praised the expert team of this app
community. According to some users, the finest plant disease diagnosis app is
Plantix - your crop doctor. The app has several wonderful features such as
detection ability for crop diseases, a fertilizer calculator, and pest and
disease-related information with high-resolution pictures. The app provides a
community support where plant lovers can share plant-related problems and get
suggestions from other users and experts. One user has commented “Hello team
plantix You people are doing wonderful job… I personally loved this app this
app gives complete guidance for farming a variety of crops. Gives details not
only about disease but also about its treatment also provide guidance about
fertilization of crop. It would be great if more people install and give it a
shot you wont regret of installing it . Best luck team plantix and thanks a
lot.". This comment is supported by many users. Another app, Agrio, has
received the second-highest number of positive reviews from users.

A word-cloud is generated in Fig. 10(b) using the negative comments collected
from the selected apps. Some users provided their damaged or diseased leaves
images to the app, but could not detect the disease, or sometimes even the
plants, prompting some comments to include the word “unable". The word “stuck”
is used to indicate that an app had been trying to load a certain step or was
being so sluggish that it seemed to have halted its task. Some apps have a
lengthy lag, most often unable to load an image from their servers, so they
can take a considerable time to load affected crop images. People use the
words “unfortunately" and “worst" to express their dissatisfaction in these
situations. Some apps’ databases are limited and require more data. Sometimes
apps show an unexpected error message. Other problems include app “crashes”
where the site does not function correctly. That is why users use the word
“useless” many times.

Users’ dissatisfaction related to the Plant Disease Detection and PlantifyDr
apps was focused on several points, including limited crop and plant
availability, an inability to detect plants, failure to login, the system
displaying unknown errors, incorrect weather reports, and failure to capture
any pictures with a high-resolution camera. A user posted a comment that was
supported by 1k+ other users. He wrote: “Useless app for me. The app asks you
to pick the top 4 crops you are interested in and it is really frustrating
that you are forced to select from a VERY limited collection of crops that
don’t include the ones I want, like kale, spinach, garlic, broccoli, beets."

The Leaf Doctor app can not always determine the correct threshold of the
given leaf. It indicates inaccurate thresholds in most situations. Riceye
provides garbage threshold every time. When a picture is imported into the
PlantDoctor app, it always responds that the image is unclear, despite the
fact even if the image is in HD quality. The Pestoz app has poor service and
needs major upgrades. It has a limited database of diseased plant images.
Sometimes it shows an unexpected error message. It “crashes" and the site does
not function correctly. Plantix - your crop doctor covers 45+ crops and 600+
diseases related to them, does not support all ornamental plants and also has
a community to share queries with experts. The Agrio app is sometimes stuck on
the home interface. Consumers used the “worst" word several times to describe
these incidents. Users put their damaged crops to the “Agrio community" but no
help or suggestion was found from the team. The Cropalyser app takes a
considerable time to load crop images and is most often unable to load an
image from its servers. PDDApp: Plant Disease Detection shows a “Check
Connection" message every time through email, even with a strong internet
connection, it can not run a single job. Plant Disease Detector is a flawed
app that can not detect plants or leaves as it shows two random thresholds of
any thing.

In summary, most of the currently available apps suffer from serious system
design flaws and development issues and need improvement.

##  4 Discussions

###  4.1 Principal findings

We found many apps in the app stores when we searched by the keywords “plant
disease", “leaf disease detection", etc., but only a few of them were well
designed and developed to successfully detect plant diseases. Unfortunately,
those few apps of key interest also lacked the seven basic functionalities
related to plant disease detection. Both plant identification and disease
detection are present in 88.23% (15/17) of the apps. While selecting a plant
from a pre-specified category, sometime apps cannot identify it correctly.
Plant disease detection is the central feature of all these apps, yet Leaf
Doctor and PlantDoctor, do not provide this feature. PlantDoctor gives the
option of taking pictures through the device camera and every time it shows
the message “The image is not clear to be identified” even for a high quality
image. Another app, Plant Disease Detector, gives a verdict for both diseased
and healthy plants by saying only “Diseased”. Severity assessment helps the
farmers to decide whether their plant needs treatment and to what extent. This
feature is present in only 23.52% (4/17) of the apps, where two of them (Plant
Disease Detector and Riceye) display a random threshold value.

We have noticed that some apps’ designs are troublesome. First, 35% (6/17) of
the apps need an uninterrupted internet connection to provide the complete
service, which is difficult to manage for village farmers or users living in
remote areas where the internet service is often poor. Second, a handful of
apps cannot upload pictures from the phone photo gallery, causing problems for
users with low quality device cameras or who may not have the proper
illumination to take clear pictures while using the app. Third, most of the
apps we reviewed were not multilingual, and thus, users of these apps must
have a basic knowledge of English.

We have found several beneficial features in some of the reviewed apps.
Plantix - your crop doctor is the best for plant disease identification. This
app detects pests and diseases on crops and recommends the necessary
treatment, gives disease alerts, and contains a fertilizer calculator for
crops based on the farmer’s plot measurements. It also gives a weather
forecast to let gardeners know the best time to weed, spray and harvest along
with cultivation tips. Users can interact with researchers, farmers, and plant
professionals in an online forum to share plant health issues. They have more
than 30 million images in their data set, which contains more than 400 crop
diseases. This app applies ML algorithms and AI to photos to diagnose crops.
It also allows for real-time disease and pest monitoring. The user’s images
are submitted to the servers and are immediately examined to identify the
disorder. The user is given essential information about the symptoms,
triggers, agrochemicals, and biological remedies. The app includes a
meteorological information system customized to farmers’ locations, and a
community option that allows them to communicate with other plant protection
groups and thus, increases productivity [86].

Another app, Leaf doctor, is slightly advanced in that users can select
healthy tissues using up to eight colors in the photograph and then be
provided with the disease severity estimation. Also, users can fix the
threshold by using a slider provided in the app. A significant flaw is that
this app gives the severity estimation for any type of object which is not
necessarily related to plants or leaves. A unique feature provided by the app
AgroAI - Plant Diseases Diagnosis (Early Access) is the soil fertility test,
where landowners can input the pH value of soil and see the proportion of
different soil elements such as calcium, potassium, clay, silt, etc. but this
feature needs more development. Another app with only the plant disease
identification feature is Leafy, which works offline and identifies the
correct plant and its disease, though it has a limited diseases list (only
30).

Providing treatment or prevention alongside disease detection is a helpful
feature provided by 64.71% (11/17) of the apps. Also, many apps shared disease
information with accompanying disease images, which is another useful feature
for information gathering related to plant diseases. Community help is
provided by two apps Plantix - your crop doctor and Agrio. In the Plantix -
your crop doctor community, all gardeners, botanists and agricultural experts
can communicate and exchange suggestions - this is highly beneficial. However,
in the Agrio community, plant disease detection is manually handled by their
own expert team commenting in posts and suggesting treatments. Moreover, any
other people can also comment in the community forum. This is highly
beneficial for increasing farmers’ knowledge, improving their attitudes toward
plant management, and increasing their help-seeking mentality. However, the
number of such apps is minimal. Most of the apps lack basic functionality and
do not benefit the users.

Automatic visualization of disease in an infected leaf helps to identify the
diseased portion of a plant and this helpful functionality is missing from all
the apps. The existing apps need their basic features improvement where basic
features will be present andd to they will work more accurately.

###  4.2 Design considerations

There are many aspects of existing plant disease detection apps that require
significant technical improvements. A simple yet attractive user interface
with high-resolution graphics will attract users and increase their app
exploring time. Multilingual apps will encourage users from different
countries to use the app and also give a better understanding of the apps’
functionalities. Apps that provide tutorials at the beginning will help users
operate the app easily and correctly. Another simple but effective feature of
an app that will keep users up to date on the status of their plants is
regular notifications. Any efficient app will have low battery power
consumption and low memory usage. Many of our reviewed apps frequently crashed
while in use. This issue should be considered while developing apps, as it has
a large negative impact on users. Also, the user interface components should
work smoothly and be consistent.

From the functionality perspective, apps should perform all the operations
accurately. Apps that cannot identify plants can keep major categories of
plants such as crops, vegetables, ornamental plants, medicinal plants, etc.,
covering many different plant types so that users can easily find their
plants. The most important feature is plant disease detection, which is the
apps’ focus. Apps should detect the diseases accurately and, in the case of
similar diseases, they should be able to give the likelihood of each disease.
Disease severity detection is another basic criterion to understand the threat
level of the disease that app should provide. To perform the disease severity
estimation properly, visualizing the infected area of the diseased portion is
a prerequisite. Therefore, visualization has become a crucial task that should
be performed by the apps using existing image processing and ML technologies.
Additionally, more information about the disease and its control measures
benefits users, which encourages them to take further disease management steps
from the apps.

###  4.3 Limitations of this study

Several apps were unfortunately discarded due to regional restrictions,
specific language restrictions, and the need to have a license key to use the
app. Furthermore, since our app search and evaluation, new apps may have been
added, and some of the apps might have been updated with enhanced
functionality, or been removed from the app stores. In this paper, we did not
evaluate the accuracy of the disease detection performed by those reviewed
apps but rather focused on the software quality aspects, app features, and to
what extent artificial intelligence is being adopted to provide advanced
disease detection functionality.

##  5 Conclusion and future work

This paper features a survey on plant disease detection apps from three
popular app stores. Our search results identified a total of 606 apps, from
which we selected 17 based on a defined set of inclusion and exclusion
criteria. We further evaluated those 17 apps using our devised app rating
scale. Our app rating scale for plant disease detection evaluated the AI-based
functionality and software quality of apps. According to our findings, most of
the apps fell significantly below standard. They tended to lack many basic and
important functionalities to detect plant disease, and many of them give users
an inappropriate or incorrect verdict or solution. Although a few apps had
some of the expected features, none of those under review met all the required
functionalities. We also observed that there were very few evidence-based
apps. As there have been numerous studies about plant disease detection and
identification, disease severity estimation, localization and visualization,
such technological advancements must be included in modern plant disease
detection apps. We hope that the developers mitigate the shortcomings found in
existing apps so that the upcoming or updated apps will contain suggested new
functionalities and perform those features correctly. Producers and botanists
will soon be able to use these improved plant disease identification apps in
the field. Users will be inspired to use these apps for detection of any type
of disease or plant pathogen, while others will be motivated to use these apps
for further plant management assistance.

In the future, we plan to address the identified limitations and study
numerous apps from different app stores. Also, our rating scale will be
extended by incorporating the accuracy metrics related to plant disease
detection. We will conduct a formal sentiment analysis on user reviews in app
stores. We will also improve the reviewer panel so that different aspects of
the apps, such as software quality and plant-specific functionalities, get
appropriate attention and improved evaluation.

## References

  * [1] George N Agrios.  Plant pathology.  Academic Press, fifth edition edition, 2005.
  * [2] Malusi Sibiya and Mbuyu Sumbwanyambe.  An algorithm for severity estimation of plant leaf diseases by the use of colour threshold image segmentation and fuzzy logic inference: a proposed algorithm to update a “leaf doctor" application.  AgriEngineering, 1(2):205–219, 2019.
  * [3] Malcolm C. Shurtleff, Rita M. Pelczar, Michael J. Pelczar, and Arthur Kelman.  Plant disease.  Encyclopedia Britannica, 2021.
  * [4] Santanu Phadikar and Jaya Sil.  Rice disease identification using pattern recognition techniques.  In 2008 11th International Conference on Computer and Information Technology, pages 420–423. IEEE, 2008.
  * [5] Monzurul Islam, Anh Dinh, Khan Wahid, and Pankaj Bhowmik.  Detection of potato diseases using image segmentation and multiclass support vector machine.  In IEEE 30th canadian conference on electrical and computer engineering (CCECE), pages 1–4. IEEE, 2017.
  * [6] Wenjiang Huang, Qingsong Guan, Juhua Luo, Jingcheng Zhang, Jinling Zhao, Dong Liang, Linsheng Huang, and Dongyan Zhang.  New optimized spectral indices for identifying and monitoring winter wheat diseases.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6):2516–2524, 2014.
  * [7] Sharma Ram, Kumarse Nazari, Amir Amanov, Zafarjon Ziyaev, and Anwar Jalilov.  Reduction of winter wheat yield losses caused by stripe rust through fungicide management.  Journal of Phytopathology, 164, 2016.
  * [8] A Meunkaewjinda, P Kumsawat, K Attakitmongcol, and A Srikaew.  Grape leaf disease detection from color imagery using hybrid intelligent system.  In 5th international conference on electrical engineering/electronics, computer, telecommunications and information technology, volume 1, pages 513–516. IEEE, 2008.
  * [9] Sanjeev S Sannakki, Vijay S Rajpurohit, VB Nargund, and Pallavi Kulkarni.  Diagnosis and classification of grape leaf diseases using neural networks.  In 2013 Fourth International Conference on Computing, Communications and Networking Technologies (ICCCNT), pages 1–5. IEEE, 2013.
  * [10] Monika Jhuria, Ashwani Kumar, and Rushikesh Borse.  Image processing for smart farming: Detection of disease and fruit grading.  In 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013), pages 521–526, 2013.
  * [11] Xiaoyue Xie, Yuan Ma, Liu Bin, Jinrong He, Shuqin Li, and Hongyan Wang.  A deep-learning-based real-time detector for grape leaf diseases using improved convolutional neural networks.  Frontiers in Plant Science, 11, 2020.
  * [12] Aditya Karlekar and Ayan Seal.  Soynet: Soybean leaf diseases classification.  Computers and Electronics in Agriculture, 172:105342, 2020.
  * [13] Heba Al-Hiary, Sulieman Bani-Ahmad, M Reyalat, Malik Braik, and Zainab Alrahamneh.  Fast and accurate detection and classification of plant diseases.  International Journal of Computer Applications, 17(1):31–38, 2011\.
  * [14] Sachin D Khirade and AB Patil.  Plant disease detection using image processing.  In 2015 International conference on computing communication control and automation, pages 768–771. IEEE, 2015.
  * [15] Amanda Ramcharan, Peter McCloskey, Kelsee Baranowski, Neema Mbilinyi, Latifa Mrisho, Mathias Ndalahwa, James Legg, and David P Hughes.  A mobile-based deep learning model for cassava disease diagnosis.  Frontiers in plant science, 10:272, 2019.
  * [16] M. Dutot, L. Nelson, and Rebecca Tyson.  Predicting the spread of postharvest disease in stored fruit, with application to apples.  Postharvest Biology and Technology, 85:45–56, 11 2013.
  * [17] Jobin Francis, BK Anoop, et al.  Identification of leaf diseases in pepper plants using soft computing techniques.  In 2016 conference on emerging devices and smart systems (ICEDSS), pages 168–173. IEEE, 2016.
  * [18] Nikos Petrellis.  Mobile application for plant disease classification based on symptom signatures.  In 21st Pan-Hellenic Conference on Informatics, pages 1–6, 2017\.
  * [19] Melissa B Riley, Margaret R Williamson, and Otis Maloy.  Plant disease diagnosis.  The Plant Health Instructor, 10, 2002.
  * [20] Punam Bedi and Pushkar Gole.  Plant disease detection using hybrid model based on convolutional autoencoder and convolutional neural network.  Artificial Intelligence in Agriculture, 5:90–101, 2021.
  * [21] Ahmed Abdelmoamen Ahmed and Gopireddy Harshavardhan Reddy.  A mobile-based system for detecting plant leaf diseases using deep learning.  AgriEngineering, 3(3):478–493, 2021.
  * [22] Shitala Prasad, Sateesh K Peddoju, and Debashis Ghosh.  Agromobile: a cloud-based framework for agriculturists on mobile platform.  International Journal of advanced science and technology, 59:41–52, 2013.
  * [23] Asheesh Kumar Singh, Baskar Ganapathysubramanian, Soumik Sarkar, and Arti Singh.  Deep learning for plant stress phenotyping: trends and future perspectives.  Trends in plant science, 23(10):883–898, 2018.
  * [24] Lili Li, Shujuan Zhang, and Bin Wang.  Plant disease detection and classification by deep learning—a review.  IEEE Access, 9:56683–56698, 2021.
  * [25] Cotton Farming.  Plant disease detection using smartphone goes high tech.  https://www.cottonfarming.com/production-2/plant-disease-detection-using-smartphone-goes-high-tech/, 2019\.  (accessed 10 June 2021).
  * [26] Nikos Petrellis.  Plant disease diagnosis for smart phone applications with extensible set of diseases.  Applied Sciences, 9(9):1952, 2019.
  * [27] Surampalli Ashok, Gemini Kishore, Velpula Rajesh, S Suchitra, SG Gino Sophia, and B Pavithra.  Tomato leaf disease detection using deep learning techniques.  In 2020 5th International Conference on Communication and Electronics Systems (ICCES), pages 979–983. IEEE, 2020.
  * [28] Shen Weizheng, Wu Yachun, Chen Zhanliang, and Wei Hongda.  Grading method of leaf spot disease based on image processing.  In 2008 international conference on computer science and software engineering, volume 6, pages 491–494. IEEE, 2008.
  * [29] Dheeb Al Bashish, Malik Braik, and Sulieman Bani-Ahmad.  A framework for detection and classification of plant leaf and stem diseases.  In 2010 International Conference on Signal and Image Processing, pages 113–118, 2010.
  * [30] Marko Arsenovic, Mirjana Karanovic, Srdjan Sladojevic, Andras Anderla, and Darko Stefanovic.  Solving current limitations of deep learning based approaches for plant disease detection.  Symmetry, 11(7):939, 2019.
  * [31] Houda Orchi, Mohamed Sadik, and Mohammed Khaldoun.  On using artificial intelligence and the internet of things for crop disease detection: A contemporary survey.  Agriculture, 12(1), 2022.
  * [32] Artzai Picon, Aitor Alvarez-Gila, Maximiliam Seitz, Amaia Ortiz-Barredo, Jone Echazarra, and Alexander Johannes.  Deep convolutional neural networks for mobile capture device-based crop disease classification in the wild.  Computers and Electronics in Agriculture, 161:280–290, 2019.
  * [33] Yuan Yuan, Lei Chen, Huarui Wu, and Lin Li.  Advanced agricultural disease image recognition technologies: A review.  Information Processing in Agriculture, 9(1):48–59, 2022.
  * [34] Nik Norasma Che’Ya, Nur Adibah Mohidem, Nor Athirah Roslin, Mohammadmehdi Saberioon, Mohammad Zakri Tarmidi, Jasmin Arif Shah, Wan Fazilah Fazlil Ilahi, and Norsida Man.  Mobile computing for pest and disease management using spectral signature analysis: A review.  Agronomy, 12(4):967, 2022.
  * [35] Blessing K Sibanda, Gloria E Iyawa, and Attlee M Gamundani.  Systematic review of plant pest and disease identification strategies and techniques in mobile apps.  In World Conference on Information Systems and Technologies, pages 491–502. Springer, 2021.
  * [36] Hetal Patel and Dharmendra Patel.  Survey of android apps for agriculture sector.  International Journal of Information Sciences and Techniques, 6(1-2):61–67, 2016.
  * [37] Bhavsar, Shreyaben. B.  Multilingual multimedia based crop disease management system using mobile technology.  PhD thesis, Sardar Patel University, Gujarat, India, 2019.
  * [38] Jordan Rivera, Amy McPherson, Jill Hamilton, Catherine Birken, Michael Coons, Sindoora Iyer, Arnav Agarwal, Chitra Lalloo, and Jennifer Stinson.  Mobile apps for weight management: a scoping review.  JMIR mHealth and uHealth, 4(3):e5115, 2016.
  * [39] Muhammad Ashad Kabir, Sheikh Sowmen Rahman, Mohammad Mainul Islam, Sayed Ahmed, and Craig Laird.  Mobile apps for foot measurement in pedorthic practice: Scoping review.  JMIR Mhealth Uhealth, 9(3), Mar 2021.
  * [40] Andrea C Tricco, Erin Lillie, Wasifa Zarin, Kelly K O’Brien, Heather Colquhoun, Danielle Levac, David Moher, Micah DJ Peters, Tanya Horsley, Laura Weeks, et al.  Prisma extension for scoping reviews (prisma-scr): checklist and explanation.  Annals of internal medicine, 169(7):467–473, 2018.
  * [41] Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John PA Ioannidis, Mike Clarke, Philip J Devereaux, Jos Kleijnen, and David Moher.  The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.  Journal of clinical epidemiology, 62(10):e1–e34, 2009.
  * [42] Stoyan R Stoyanov, Leanne Hides, David J Kavanagh, Oksana Zelenko, Dian Tjondronegoro, and Madhavan Mani.  Mobile app rating scale: a new tool for assessing the quality of health mobile apps.  JMIR mHealth and uHealth, 3(1):e27, 2015.
  * [43] Sadia Tasnuva Pritha, Rahnuma Tasnim, Ashad Kabir, Sumaiya Amin, and Anik Das.  Smartphone apps for child sexual abuse education: Gaps and design considerations.  International Journal of Mobile Learning and Organisation, 2022\.
  * [44] Sabiha Samad, Fahmida Ahmed, Samsun Naher, Muhammad Ashad Kabir, Anik Das, Sumaiya Amin, and Sheikh Mohammed Shariful Islam.  Smartphone apps for tracking food consumption and recommendations: Evaluating artificial intelligence-based functionalities, features and quality of current apps.  Intelligent Systems with Applications, 15:200103, 2022.
  * [45] Stoyan R Stoyanov, Leanne Hides, David J Kavanagh, and Hollie Wilson.  Development and validation of the user version of the mobile application rating scale (umars).  JMIR mHealth and uHealth, 4(2):e72, 2016.
  * [46] Johannes Huebner, Carlo Schmid, Mehdi Bouguerra, and Alexander Ilic.  Finmars: A mobile app rating scale for finance apps.  In 9th International Conference on Information Communication and Management, pages 6–11, 2019.
  * [47] Chien-Ho Wu.  An empirical study on the transformation of likert-scale data to numerical scores.  Applied Mathematical Sciences, 1(58):2851–2862, 2007.
  * [48] Experiencing visual usability and aesthetics in two mobile application contexts.  Journal of usability studies, 10(1), 2014.
  * [49] Christiane G von Wangenheim, João V Araujo Porto, Jean CR Hauck, and Adriano F Borgatto.  Do we agree on user interface aesthetics of android apps?  arXiv preprint arXiv:1812.09049, 2018.
  * [50] ISO.  Iso/iec 25010:2011 systems and software engineering — systems and software quality requirements and evaluation (square) — system and software quality models.  https://www.iso.org/standard/35733.html, 2011.  (accessed 10 June 2021).
  * [51] Mathieu Zen and Jean Vanderdonckt.  Assessing user interface aesthetics based on the inter-subjectivity of judgment.  In 30th International BCS Human Computer Interaction Conference 30, pages 1–12, 2016.
  * [52] Noam Tractinsky, Adi S Katz, and Dror Ikar.  What is beautiful is usable.  Interacting with computers, 13(2):127–145, 2000.
  * [53] Stephanie Chan.  5-year market forecast: App spending will climb to $270 billion by 2025\.  https://sensortower.com/blog/sensor-tower-app-market-forecast-2025, 2021\.  (accessed 15 July 2021).
  * [54] Qunyi Wei, Zhaoxin Chang, and Qin Cheng.  Usability study of the mobile library app: an example from chongqing university.  Library Hi Tech, 2015.
  * [55] Hannah E Payne, Cameron Lister, Joshua H West, and Jay M Bernhardt.  Behavioral functionality of mobile apps in health interventions: a systematic review of the literature.  JMIR mHealth and uHealth, 3(1):e3335, 2015.
  * [56] Titti Kallio, Anne Kaikkonen, et al.  Usability testing of mobile applications: A comparison between laboratory and field testing.  Journal of Usability studies, 1(4-16):23–28, 2005.
  * [57] Maleknaz Nayebi, Mahshid Marbouti, Rache Quapp, Frank Maurer, and Guenther Ruhe.  Crowdsourced exploration of mobile app features: A case study of the fort mcmurray wildfire.  In IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Society Track (ICSE-SEIS), pages 57–66, 2017.
  * [58] Sally A Miller, Fen D Beed, and Carrie Lapaire Harmon.  Plant disease diagnostic capabilities and networks.  Annual review of phytopathology, 47:15–38, 2009.
  * [59] Anne-Katrin Mahlein.  Plant disease detection by imaging sensors–parallels and specific demands for precision agriculture and plant phenotyping.  Plant disease, 100(2):241–251, 2016.
  * [60] Jana Wäldchen, Michael Rzanny, Marco Seeland, and Patrick Mäder.  Automated plant species identification—trends and future directions.  PLoS computational biology, 14(4):e1005993, 2018.
  * [61] Zhaobin Wang, Huale Li, Ying Zhu, and TianFang Xu.  Review of plant identification based on image processing.  Archives of Computational Methods in Engineering, 24(3):637–654, 2017.
  * [62] Pradeep Nijalingappa and VJ Madhumathi.  Plant identification system using its leaf features.  In 2015 International Conference on Applied and Theoretical Computing and Communication Technology (iCATccT), pages 338–343. IEEE, 2015\.
  * [63] Sonali Agarwal, Anand Singh Jalal, Mohd Khan, et al.  Plant identification using leaf image analysis.  In 3rd International Conference on Internet of Things and Connected Technologies (ICIoTCT), pages 26–27, 2018.
  * [64] Ngo Le Huy Hien and Nguyen V.H.  Recognition of plant species using deep convolutional feature extraction.  International Journal on Emerging Technologies, 11:904–910, 06 2020\.
  * [65] Vinit Bodhwani, DP Acharjya, and Umesh Bodhwani.  Deep residual networks for plant identification.  Procedia Computer Science, 152:186–194, 2019.
  * [66] Konstantinos P. Ferentinos.  Deep learning models for plant disease detection and diagnosis.  Computers and Electronics in Agriculture, 145:311–318, 2018.
  * [67] David Hughes and Marcel Salathé.  An open access repository of images on plant health to enable the development of mobile disease diagnostics.  arXiv preprint arXiv:1511.08060, 2015.
  * [68] G Geetharamani and Arun Pandian.  Identification of plant leaf diseases using a nine-layer deep convolutional neural network.  Computers & Electrical Engineering, 76:323–338, 2019.
  * [69] André Abade, Paulo Afonso Ferreira, and Flavio de Barros Vidal.  Plant diseases recognition on images using convolutional neural networks: A systematic review.  Computers and Electronics in Agriculture, 185:106125, 2021.
  * [70] Vinicius Bischoff, Kleinner Farias, Juliano Paulo Menzen, and Gustavo Pessin.  Technological support for detection and prediction of plant diseases: A systematic mapping study.  Computers and Electronics in Agriculture, 181:105922, 2021.
  * [71] Sue Han Lee, Hervé Goëau, Pierre Bonnet, and Alexis Joly.  New perspectives on plant disease characterization based on deep learning.  Computers and Electronics in Agriculture, 170:105220, 2020.
  * [72] Patrick Wspanialy and Medhat Moussa.  A detection and severity estimation system for generic diseases of tomato greenhouse plants.  Computers and Electronics in Agriculture, 178:105701, 2020.
  * [73] Sai Arivazhagan, R Newlin Shebiah, S Ananthi, and S Vishnu Varthini.  Detection of unhealthy region of plant leaves and classification of plant leaf diseases using texture features.  Agricultural Engineering International: CIGR Journal, 15(1):211–217, 2013.
  * [74] Qiaokang Liang, Shao Xiang, Yucheng Hu, Gianmarc Coppola, Dan Zhang, and Wei Sun.  Pd2se-net: Computer-assisted plant disease diagnosis and severity estimation network.  Computers and Electronics in Agriculture, 157:518–529, 2019.
  * [75] Jan Hendrik Betzing, Matthias Tietz, Jan vom Brocke, and Jörg Becker.  The impact of transparency on mobile privacy decision making.  Electronic Markets, 30(3):607–625, 2020.
  * [76] Alastair Beresford, Andrew Rice, Nicholas Skehin, and Ripduman Sohan.  Mockdroid: Trading privacy for application functionality on smartphones.  HotMobile 2011: The 12th Workshop on Mobile Computing Systems and Applications, 03 2011.
  * [77] Andreas Christmann and Stefan Van Aelst.  Robust estimation of cronbach’s alpha.  Journal of Multivariate Analysis, 97(7):1660–1674, 2006.
  * [78] Lee J Cronbach.  Coefficient alpha and the internal structure of tests.  psychometrika, 16(3):297–334, 1951.
  * [79] Joseph Gliem and Rosemary Gliem.  Calculating, interpreting, and reporting cronbach’s alpha reliability coefficient for likert-type scales.  2003 Midwest Research to Practice Conference in Adult, Continuing, and Community Education, 2003.
  * [80] RT Lange.  Inter-rater reliability.  Encyclopedia of Clinical Neuropsychology. New York, NY: Springer New York, page 1348, 2011.
  * [81] Jungo Sawa and Toshihiko Morikawa.  Interrater reliability for multiple raters in clinical trials of ordinal scale.  Drug information journal: DIJ/Drug Information Association, 41(5):595–605, 2007.
  * [82] Terry K Koo and Mae Y Li.  A guideline of selecting and reporting intraclass correlation coefficients for reliability research.  Journal of chiropractic medicine, 15(2):155–163, 2016.
  * [83] Sarah J Pethybridge and Scot C Nelson.  Leaf doctor: A new portable application for quantifying plant disease severity.  Plant disease, 99(10):1310–1316, 2015.
  * [84] Dennis Pagano and Walid Maalej.  User feedback in the appstore: An empirical study.  In 2013 21st IEEE International Requirements Engineering Conference (RE), pages 125–134, 2013.
  * [85] Lisa Lepki.  How to use word clouds for business, fiction and copywriting.  https://prowritingaid.com/art/425/What-the-Heck-is-a-Word-Cloud-and-Why-Would-I-Use-One.aspx, 2020\.  (accessed 5 July 2021).
  * [86] CGIAR.  Plant disease diagnosis using artificial intelligence: A case study on plantix.  https://bigdata.cgiar.org/digital-intervention/plant-disease-diagnosis-using-artificial-intelligence-a-case-study-on-plantix/, 2021\.  (accessed 20 October 2021).