# Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality
Assurance

Tinghui Ouyang1, AprilPyone MaungMaung1, Koichi Konishi2,
Yoshiki Seo2, and Isao Echizen1
1National Institute of Informatics
2National Institute of Advanced Industrial Science and Technology, Japan
E-mail: {thouyang, pyone<EMAIL_ADDRESS>{k.konishi<EMAIL_ADDRESS>
###### Abstract

In the era of large AI models, the complex architecture and vast parameters
present substantial challenges for effective AI quality management (AIQM),
e.g. large language model (LLM). This paper focuses on investigating the
quality assurance of a specific LLM-based AI product—a ChatGPT-based sentiment
analysis system. The study delves into stability issues related to both the
operation and robustness of the expansive AI model on which ChatGPT is based.
Experimental analysis is conducted using benchmark datasets for sentiment
analysis. The results reveal that the constructed ChatGPT-based sentiment
analysis system exhibits uncertainty, which is attributed to various
operational factors. It demonstrated that the system also exhibits stability
issues in handling conventional small text attacks involving robustness.

###### Index Terms:

AI quality management, large language model (LLM), ChatGPT-based sentiment
analysis, stability analysis,

##  I Introduction

Advances in machine learning, especially in deep learning (DL), have led to
the application of artificial intelligence (AI) to a broad range of areas,
e.g., autonomous driving, e-commerce, and robotics [1]-[3]. However, due to
the black-box property of DL models and the lack of effective evaluation and
testing techniques, it is difficult to identify the cause of malfunctions in
AI-based products. The study of AI quality management (AIQM) [4] is aimed at
providing useful techniques for AI-based product evaluation, testing, and
improvement. It typically encompasses various aspects related to internal and
external qualities, such as data- and model-related qualities. Therefore, AIQM
is critical for ensuring the effectiveness and reliability of AI-based
products.

The increasing power of computer devices has made possible the development of
complex AI architectures with a great number of parameters. Thus, many large
models have been developed in the fields of computer vision [5] and natural
language processing (NLP) [6]. The most well-known example is ChatGPT [7],
which was released by OpenAI in 2023. It can generate human-like text with
context and coherence. Its ability to understand natural language, even the
nuances of language, and to generate highly accurate responses is the result
of pretraining on a massive amount of text data using an advanced neural
architecture (i.e., a Generative-Pretrained–Transformer (GPT) [8]) with a
great number of parameters. As an advanced cutting-edge AI product currently
in the limelight, ChatGPT has attracted much attention due to its superiority
in addressing various traditional NLP tasks as well as its generation ability.
For example, ChatGPT can deal with various languages besides English, so it
was used as a translator in [9]. By making use of its ability for semantic
understanding, ChatGPT was also used as a grammatical checker [10]. Its most
widely used ability is conversation ability, enabling it to be used as a
chatbot for question answering, or as a tool to guide educational exams [11],
and to assist diagnosis in medical counseling [12]. Moreover, by making use of
ChatGPT’s generation ability, cheating in writing is a serious problem in
education [13]. In [14], ChatGPT was also used as a tool to augment text data
for other research.

Along with the convenience and benefits of ChatGPT, it also poses significant
challenges to our society, including not only the agnosticism and concerns to
everyday people but also the difficulties and complexities presented to
researchers and developers. Especially, researchers in software engineering
and quality management in particular need to understand how ChatGPT works, how
to evaluate it, and how to maintain it. Even though it has been accessible to
users for only a short while, much research has been conducted on the quality
management of ChatGPT due to its prominence and wide usage. For example, in
terms of the correctness study, ChatGPT’s performances have been evaluated on
several traditional NLP tasks, like translation, grammar checking, text
animation, and classification, and compared with that of conventional
products. Moreover, Other studies have evaluated its performance on novel
tasks like mathematical capabilities [15] and logical reasoning in coding
[16]. For the robustness study, some adversarial text data in various NLP
tasks were used to evaluate ChatGPT’s performance [17]-[18]. Some researchers
also tried to attack LLM for robustness analysis, for example, using the
gradient-based method for a prompt attack on Llama and other open-sourced LLMs
[19], but it is hard to perform such analysis on ChatGPT because of its black-
box property. In [20], an interesting idea was proposed to utilize LLM to
generate attacks and then test its robustness on these attacks. Moreover, some
other quality studies have also been performed. For example, the reliability
of ChatGPT on text classification was studied in [21]. The limitations and
weaknesses of ChatGPT faced with some questions were also reported, like
political questions [22], failure cases [23], and so on [24]-[25].

However, due to ChatGPT’s complex architecture and black-box properties, there
are still many difficulties. Evaluating ChatGPT directly on the basis of these
related studies is challenging [26]. Since ChatGPT can deal with many tasks
based on the foundation model, it has no clear function requirements as
addressed in traditional software quality management. A useful way is to
concentrate on a specific NLP task for quality analysis, e.g., accuracy on
sentiment analysis, grammar checking, and translation. The other possible way
is to use some surrogate datasets to evaluate ChatGPT’s performance on
specific qualities, e.g., applying an existing adversarial dataset, AdvGLUE
[27], to test the robustness of ChatGPT. These solutions can evaluate and find
some weaknesses of ChatGPT, however its shortages are not analyzed directly as
a software product.

Given these considerations, this paper proposes to study the stability quality
of a ChatGPT-based software product, namely a ChatGPT-based sentiment analysis
system. Sentiment analysis was selected because it is the simplest task in NLP
and because it has a clear problem formulation and evaluation metrics for
quality assurance [28]-[29]. Moreover, this study adopts an architecture
oriented to a specific task and a specific quality. It focused on the causes
of instability from the perspective of software engineering and on robustness
against attacks. Experiments using benchmark sentiment analysis datasets
demonstrated that ChatGPT is robust against attacks when used for sentiment
analysis although weaknesses were revealed for synonym perturbations.

The rest of this paper is organized as follows. Section 2 gives an overview of
using ChatGPT for sentiment analysis and describes the two aspects of
stability in AIQM. Based on the division of stability studies, Section 3 and 4
analyze the stability issues of ChatGPT from different aspects, such as
uncertainty causes of ChatGPT operation in Section 3 and the AI model’s
robustness issues in Section 4. Section 5 concludes the work of this paper.

##  II Overview

###  II-A ChatGPT-based sentiment analysis system

As described above, our major objective was to investigate the AI quality
assurance of a ChatGPT-based sentiment analysis system. It is known that the
product’s functions are clear in traditional software testing. However,
ChatGPT is based on the foundation model which can address several practical
tasks e.g., question answering, chatting, text generation, grammar checking,
and translation. To evaluate the quality of ChatGPT as an AI product, we need
to investigate multiple aspects. This is a huge time-consuming task. To
simplify this AIQM task, we focused on evaluating ChatGPT’s ability on a
specific NLP task, namely sentiment analysis. In this paper, sentiment
analysis is selected because it is a simple and typical NLP task. Moreover, it
is a classification problem for which there are standard evaluation metrics.
Therefore, the objective of this paper in AIQM is to evaluate the quality of
the ChatGPT-based sentiment analysis system.

Figure 1: Diagram of using ChatGPT for sentiment analysis

The diagram in Fig. 1 illustrates how using ChatGPT for sentiment analysis.
Compared with conventional testing based on input and output data, e.g.,
review comments and sentiment labels in sentiment analysis, it is found that
some differences happen in the process of using ChatGPT in Fig. 1. Since
ChatGPT operates in conversation mode, extra editing on prompt and response is
usually needed to make ChatGPT focus on a specific task and produce the
desired output. An understanding of this difference, and with the help of the
ChatGPT API provided by OpenAI, it becomes easy for users to develop a
specific AI product based on ChatGPT. The following are details about the
settings for developing a ChatGPT-based sentiment analysis system in our AIQM
study.

  * •

PromptSetting: Analyze the following product review and determine if the
sentiment is POSITIVE, NEGATIVE or NEUTRAL: {ReviewText}

  * •

OutputControl: Return only a single word, such as POSITIVE, NEGATIVE or
NEUTRAL.

###  II-B Stability of AI

According to the Machine Learning Quality Management guideline [4], among the
diverse AI qualities, stability is a crucial quality that deserves special
attention. In this context, stability refers to an AI product performing
consistently and reliably. It encompasses the ability of the product to
operate seamlessly under varying conditions and to resist disruptions. This
means that the product must be less prone to errors, crashes, or unexpected
behaviors. It is thus crucial to achieve and maintain AI product stability to
ensure a positive user experience and to minimize the effect of potential
problems.

Moreover, with respect to AI-based software products, the stability study can
be approached in two ways. One is to focus on system stability, specifically
the intricacies of operational stability and the challenges posed by
uncertainty. The second is to focus on model stability, mainly related to the
AI model used in the software product. In other words, the second type of
stability is about the robustness analysis of trained AI models.In the
subsequent sections, we will explore these two stability studies in the
context of ChatGPT in detail.

##  III Uncertainty analysis

###  III-A Model architecture design

Currently, it is widely discussed that uncertainty issues exist in the running
of ChatGPT. Both with the advanced GPT4 or the widely used GPT-3.5-turbo
model, the responses are non-deterministic even for a temperature setting of
0.0. For example, repeated inputs of a question to ChatGPT generally produces
different responses. One possible reason for this is that ChatGPT is
continually being updated on the basis of data collected from global
customers. However, as shown in Fig. 2, the responses on two different devices
for the same input at the same time differed. Although the responses could be
affected by randomness in the text generation process, the major reason for
the non-determinism of responses is the sparse MoE (mixture of expert models)
architecture in ChatGPT [30]. This architecture organizes tokens into fixed-
size groups due to capacity constraints and emphasizes balance within each
group. When groups incorporate tokens from diverse sequences or inputs, there
is often competition between expert buffers among these tokens, leading to a
failure in enforcing per-sequence determinism.

Figure 2: ChatGPT’s responses on two devices at same time

###  III-B Difference on using ChatGPT and ChatGPT API

Developers using ChatGPT have reported differences between using ChatGPT on
the web and the ChatGPT API. A general finding is that the web version
performs better than the API even when using the same model. Several reasons
for this have been suggested [31]–[32]. One is that the web version is
continually being updated on the basis of the huge amounts of data being
received from global customers, whereas the API version is fixed for a certain
period of time. Another is related to the system prompt. ChatGPT has a default
system prompt, namely ” You are an LLM-based AI system created by OpenAI …” or
what model it is using, or something else . However, when using the API for
testing, developers must construct their own system prompt to specify how
ChatGPT should behave. For example, when using the API to develop a sentiment
analysis system, the system prompt can be set as

”You are an AI language model trained to analyze and detect the sentiment of
product reviews”

Moreover, whereas ChatGPT is a conversation-based system which can collect
historical data for AI to produce the most satisfactory results, the API is a
simple question-answer chat robot. A conversation management system must be
created to enable history-based conversation. Another possible reason that the
web version performs better than the API even when using the same model is a
difference in the setting of the undisclosed “temperature” parameter in
ChatGPT. This parameter controls the degree of creativity or unpredictability
in response generation. Generally, the higher the value, the more uncertain
the result. To obtain a stable result, a temperature setting of 0.0 should be
used.

###  III-C Variance due to timing

As discussed above, ChatGPT continuously updates itself on the basis of newly
collected data and design changes. For example, OpenAI has made both GPT-3 and
GPT-4 models available for use in ChatGPT and reports that GPT-4 performs
better than GPT-3 on most tasks. Even for two identical ChatGPT systems
processing the same input at the same time, there is uncertainty about the
output. Since the process for updating the models is not transparent, the
effect of each update on model behavior is unclear. These uncertainties pose
challenges in studying the stability of ChatGPT. For example, if there is a
sudden change in a model’s response to a prompt (e.g., its accuracy or
formatting), downstream processing may be disrupted. Furthermore, it is
challenging to reproduce a model’s results even with the same settings. This
issue has been discussed on the basis of comparative experiments on several
NLP downstream tasks [33]. In this paper, using the same set-up, performances
of using ChatGPT for sentiment analysis at different time slots are also
experimented with, as shown in Fig. 3.

Figure 3: ChatGPT for sentiment analysis at different time

A comparison of the results using the ChatGPT API for sentiment analysis in
June and December 2023 is shown in Fig. 3. The dataset was constructed using
Amazon.com review data, and the settings were the same. The confusion matrices
show that the June version was better able to comprehend the prompts and thus
better able to distinguish all positive, negative, and neutral reviews.
Updating appears to have affected the performance of the December version: it
focused more on polar classification (positive or negative), reflecting
weakness in recognizing neutral review comments, as shown in Fig. 3. This
change in ChatGPT’s ability to perform sentiment analysis means that it is
essential to determine what happens when the model version of ChatGPT is
updated.

###  III-D Prompt engineering

In the operation of ChatGPT, another issue that affects the stability of
ChatGPT operation is the prompts. Prompt engineering is an essential topic in
the study of LLMs. Prompts should usually be carefully designed to optimize
the output. To investigate how the prompts affect the stability of ChatGPT,
this paper designed several prompt settings. By making use of the in-context
learning ability of ChatGPT, the prompt engineering here is set based on the
criterion of zero-shot, one-shot, and few-shot, as shown in Fig. 4.

Figure 4: Designs of zero-, one-, and few-shot prompts for sentiment analysis

In Fig. 4, three example prompts are given for using ChatGPT in sentiment
analysis. It is seen that with a zero-shot prompt, no example information is
provided, and with a few-shot prompt, several examples are provided for
guidance. A one-shot prompt is a special case of a few-shot prompt where only
one example is provided for guidance. Therefore, based on this idea, three
sub-designs providing positive, neutral, and negative examples, respectively,
are also considered in this paper. Also, based on Amazon.com review data for
sentiment analysis, the evaluation results are presented via the confusion
matrix, as shown in Fig. 5.

Figure 5: Sentiment analysis results for different prompt settings

As shown by the confusion matrices in Fig. 5, different prompts resulted in
different sentiment classification accuracies for the same dataset, indicating
that prompt engineering indeed affects the stability of ChatGPT performance.
The accuracy with the zero-shot prompt was slightly better than that with the
few-shot prompt. The main difference was that the few-shot prompt reduced
prediction accuracy for positive reviews. Looking at the matrices for the
three one-shot prompts with positive/neutral/negative examples, we see that
using the ones with neutral and negative examples seems to have improved
accuracy, indicating that the performance of ChatGPT for sentiment analysis
can be improved by careful engineering of the prompts.

##  IV robustness testing

According to the above description, the second type of stability analysis is
model stability, specifically the robustness of the trained AI model. Given
that the focus of our study was the use of ChatGPT for sentiment analysis, we
investigated the robustness of the AI model on which it is based.

The IEEE standard notation in software engineering terminology [34] indicates
that robustness is the degree to which a system or component can function
correctly in the presence of invalid inputs and/or stressful environmental
conditions. In general, invalid inputs are data generated by perturbation or
mutation, and a stressful environmental condition would be, for example, an
attack or threat. Therefore, investigation of an AI model’s robustness
involves an in-depth analysis of its resilience to perturbations. With respect
to the ChatGPT-based model, we comprehensively evaluated model robustness by
taking into account potential vulnerabilities arising from different types of
perturbation. Such an evaluation contributes to a deeper understanding of the
interplay between AIQM and provides insights into ways to enhance the
robustness and reliability of AI systems.

###  IV-A Data preparation

In this paper, two commonly used datasets for sentiment analysis, namely the
Amazon.com review dataset [35] and the Stanford Sentiment Treebank (SST)
dataset [36], are used for the evaluation.

  * •

Amazon.com review dataset: This dataset is a collection of a large number of
product reviews from Amazon.com. The raw data contains 82.83 million unique
reviews and includes product and user information, a rating score (1–5 stars),
and a plain text review. For sentiment analysis, researchers usually take a
review score of 1 or 2 as negative, 4 or 5 as positive, and 3 as neutral. We
did likewise.

  * •

SST dataset: This dataset consists of 11,855 individual sentences extracted
from movie reviews by Pang and Lee [36]. Applying the Stanford parser to this
dataset enables a comprehensive examination of the compositional effects of
sentiment in language. In this paper, we used an extension of this dataset
with fine-grained labels (very positive, positive, neutral, negative, very
negative) and roughly categorized the review sentiments as positive, neutral,
or negative.

In this paper, only a small amount of samples selected from the original
benchmark datasets are taken as the testing sets in evaluation. Table I
presents the information of selected testing sets.

TABLE I: Dataset information Dataset | No. of samples | Distribution (Pos./Neu./Neg.) | Avg. text length
---|---|---|---
Amazon | 983 | 0.8993/0.0264/0.0743 | 49.6185
SST | 1101 | 0.4033/0.2080/0.3887 | 19.3224

In this paper, only a small number of samples selected from the original
datasets were taken as the test sets. As shown in Table I, 983 samples from
the Amazon.com review dataset and 1101 samples from the SST dataset were used
for testing. Looking at the distribution of positive/neutral/negative reviews,
we see that the SST dataset has a more balanced distribution than the
Amazon.com one, which contains a larger number of positive reviews and a fewer
number of negative and neutral reviews. Moreover, Table I also shows the
average text length of these two testing sets. Amazon.com review data has a
longer average text length, and the SST dataset usually has short reviews. To
further discuss the two datasets, the statistics of their review text lengths
are plotted in Fig. 6.

Figure 6: Length of reviews in Amazon.com review and SST datasets

From Fig. 6, it is more clear that these two datasets have different
distributions on review text length. Combined with the results in Table I, we
can say these two datasets consider both data balance and text length for
testing the given AI product, which would be helpful for the complete
evaluation in AIQM.

###  IV-B Evaluation metrics

We evaluated the robustness of the AI model by using the definition of an
adversarial example in which a small perturbation in the data fools the model.
Since the evaluation of using ChatGPT for sentiment analysis is essentially
classification problem, we can apply two traditional classification metrics:
accuracy ($Acc$) and attack success rate (ASR).

$Acc=\frac{\text{\\# of correctly classified samples}}{\text{\\# of total
testing samples}}\times 100\%$ (1) $ASR=\frac{\text{\\# of successfully
attaked samples}}{\text{\\# of total testing samples}}\times 100\%$ (2)

Although these two metrics are similar, they have different meanings. $Acc$
gives the percentage of samples for which the decision matched the ground
truth, and $ASR$ gives the difference in accuracy between before and after
perturbation. Therefore, $ASR$ can be directly used for robustness analysis,
meanwhile we can also compare accuracy before and after perturbation for
robustness analysis.

###  IV-C Perturbation and robustness analysis

As shown in Fig. 7, this paper considered four types of perturbation from
different perspectives, such as typo, synonym, homoglyph, and homophone
perturbation. They mainly involve character- and word-level perturbation. To
maintain naturalness and readability, the adversarial texts were generated in
accordance with the given perturbation and kept within an edit distance of one
word, as seen in the examples in Fig. 7, where successfully attacked review
comments via the given four perturbations are presented.

Figure 7: The homoglyph of alphabets

(1) typo perturbation

Psychological studies have shown that a sentence with one or more typos can
often be still comprehended by a person [37]. However, in the information era
with computers, words with typo perturbation are encoded differently, which
may lead to incorrect machine processing. Therefore, typo perturbation is
regarded as a common attack in NLP studies, mainly because typing errors are
common in computer input. In this paper, we used the four kinds of common typo
perturbations for words introduced in the TextAttack package [38]: swapping
two adjacent letters in a word, substituting a letter with a random letter,
randomly deleting a letter, and randomly inserting a letter. Adversarial texts
with these perturbations were generated by restricting the edit distance in
the sentence to maintain semantic understanding, which we refer to as “one-
word perturbation”. Typos were accordingly added to review comments in the
datasets, and the performance of the AI model’s robustness was tested,
presented in Table II.

TABLE II: ROBUSTNESS AGAINST TYPO PERTURBATION | $ori_{acc}$ | $pert_{acc}$ | $\Delta_{diff}$ | $ASR$
---|---|---|---|---
Amazon | 0.8942 | 0.7636 | 0.1306 | 0.1273
SST | 0.8065 | 0.6129 | 0.1936 | 0.1935

The resulting performance is summarized in Table II, where $ori_{acc}$ and
$pert_{acc}$ represent accuracy before and after perturbation, respectively,
and their difference is expressed as $\Delta_{diff}$. These results show that
the ChatGPT-based sentiment analysis system performs relatively better on
Amazon.com review data than on SST data. Moreover, the ASRs for the two
datasets were close to the drops in accuracy. The larger $ASR$ for the SST
dataset, along with the text length analysis results shown in Fig. 6,
indicates that short review texts are more easily attacked by typo
perturbation. This is consistent with the common understanding that longer
texts are more robust against attacks.

(2) synonym perturbation

Another common attack used in NLP studies is synonym perturbation in which a
word is replaced with a synonym. The text with the replacement word should
have readability and meaning similar to those of the original text. However,
in ML/DL-based NLP studies, an NLP model may behave differently on synonyms
because the different words have different encodings in the tokenization.As a
result, synonym perturbation has generally worked well in conventional
adversarial text generation studies, especially in sentiment analysis where
sentiment words play a crucial role. This paper adopts the provided synonym
perturbation algorithm in [39] to create adversarial text in both the
Amazon.com review dataset and the SST dataset. Results of robustness analysis
are presented in Table III.

TABLE III: ROBUSTNESS AGAINST SYNONYM PERTURBATION  | $ori_{acc}$ | $pert_{acc}$ | $\Delta_{diff}$ | $ASR$
---|---|---|---|---
Amazon | 0.8942 | 0.5781 | 0.3161 | 0.3642
SST | 0.8065 | 0.3871 | 0.4194 | 0.5200

As shown in Table III, the ASR values for synonym perturbation were much
larger than those for typo perturbation, indicating that synonym perturbation
is a stronger attack than a typo one. Again, the longer texts in the
Amazon.com review dataset were more robust than the shorter ones in the SST
dataset against synonym perturbation.

(3) homoglyph perturbation

In [38], Gao et al. also proposed using homoglyph perturbation for adversarial
text attacks. The idea is to replace a character with a similar-looking
character, e.g., using a symbol with an identical shape but with a different
ASCII code, as illustrated by the examples in Fig. 8. This perturbation was
demonstrated to be useful in adversarial text attacks. In accordance with the
number of replaced characters, homoglyph perturbation can be categorized as
character-level (transforming a few characters) or word-level (transforming
all characters in a word). Since homoglyph perturbation retains a look similar
to that of the original word, it does not affect readability or the semantic
meaning of the original text, which is useful in adversarial robustness
studies. Therefore, in this paper, word-level homoglyph perturbation is
adopted first. On the basis of word importance in sentiment analysis as
determined using the Vader analyzer [40], adversarial texts were generated to
test the performance of ChatGPT on sentiment analysis. Table IV summarizes
performance on the two datasets.

Figure 8: Example homoglyphs (hom.) of 26 letters in English alphabet TABLE IV: PROBUSTNESS AGAINST HOMOGLYPH PERTURBATION | $ori_{acc}$ | $pert_{acc}$ | $\Delta_{diff}$ | $ASR$
---|---|---|---|---
Amazon | 0.8942 | 0.6536 | 0.2406 | 0.2397
SST | 0.8065 | 0.7419 | 0.0646 | 0.1290

A comparison of the results in Table II -IV shows that homoglyph perturbation
is a stronger attack than typo perturbation but weaker than synonym
perturbation. ChatGPT thus achieved relatively good robustness against
homoglyph perturbation when used for sentiment analysis.

(4) homophone perturbation

In contrast to homoglyph perturbation in which text appearance is transformed,
homophone perturbation [41] transforms text on the basis of pronunciation.
Homophones, which are words with similar sounds but are spelled differently
and have different meanings, can cause the system to misclassify the
sentiment. It has been demonstrated to be useful, especially in adversarial
Chinese text generation [41]. In this paper, we use this perturbation in
English text perturbation. First, word importance for sentiment analysis was
determined using the Vader analyzer. Then, homophone perturbation was used to
generate adversarial texts based on the order of word importance. Robustness
analysis based on these adversarial texts is done on two datasets, and results
are presented in Table V.

TABLE V: PROBUSTNESS AGAINST homophone perturbation | $ori_{acc}$ | $pert_{acc}$ | $\Delta_{diff}$ | $ASR$
---|---|---|---|---
Amazon | 0.8942 | 0.8445 | 0.0497 | 0.0497
SST | 0.8065 | 0.7419 | 0.0646 | 0.0645

As shown in Table V, the accuracy drop and ASR values for homophone
perturbation were quite low, indicating that homophone perturbation is not a
strong attack against a ChatGPT-based system and that ChatGPT is robust
against homophone perturbation when used for sentiment analysis.

(5) discussion

Looking at the results in Tables II–V, we see that synonym perturbation was
the strongest type of attack as it resulted in the largest reduction in
accuracy and largest ASR values on adversarial texts. A comprehensive
evaluation shows that it is difficult to generate a strong attack against a
ChatGPT-based system by using conventional methods, including character- and
word-level. Therefore, we can say that the ChatGPT-based sentiment analysis
system can be robust against adversarial text perturbations in real
applications of sentiment analysis.

##  V Conclusion

This paper introduces a sentiment-analysis system based on ChatGPT as the
studied product for AI quality assurance. Meanwhile, the study mainly focuses
on the specific quality assurance of stability. Two topics are discussed to
examine the stability issue. One is based on the operation uncertainty, and
several factors are analyzed and discussed. The second one is based on
ChatGPT’s robustness against four types of perturbations. Two benchmark
datasets were used for the evaluation. Results demonstrated that the ChatGPT-
based sentiment analysis system is robust against all four perturbations
although a bit weaker against synonym perturbation. Through the results, we
can conclude that it is reasonable to get a robust sentiment analysis product
based on ChatGPT. Still, it is also necessary to notice the operation
uncertainty due to ChatGPT continuously updating itself, the effect of
differences in the time of operation, and some other factors.

##  VI Acknowledgement

This research is supported by the project ’JPNP20006’, commissioned by the New
Energy and Industrial Technology Development Organization (NEDO). and partly
supported by JSPS Grant-in-Aid for Early-Career Scientists (Grant Number
22K17961)

## References

  * [1] Ouyang, Tinghui, Yoshinao Isobe, Saima Sultana, Yoshiki Seo, and Yutaka Oiwa. ”Autonomous driving quality assurance with data uncertainty analysis.” In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2022.
  * [2] Shinde, Pramila P., and Seema Shah. ”A review of machine learning and deep learning applications.” In 2018 Fourth international conference on computing communication control and automation (ICCUBEA), pp. 1-6. IEEE, 2018.
  * [3] Hordri, Nur Farhana, Siti Sophiayati Yuhaniz, and Siti Mariyam Shamsuddin. ”Deep learning and its applications: a review.” In Conference on Postgraduate Annual Research on Informatics Seminar, pp. 1-5. 2016.
  * [4] Machine Learning Quality Management Guideline$(https://www.digiarc.aist.go.jp/en/publication/aiqm/)$
  * [5] Wang, Jiaqi, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai et al. ”Review of large vision models and visual prompt engineering.” arXiv preprint arXiv:2307.00855, 2023.
  * [6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  * [7] OpenAI. $https://chat.openai.com.chat$, 2023.
  * [8] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
  * [9] Jiao, Wenxiang, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. ”Is ChatGPT a good translator? A preliminary study.” arXiv preprint arXiv:2301.08745, 2023.
  * [10] Wu, Haoran, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. ”Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark.” arXiv preprint arXiv:2303.13648, 2023.
  * [11] Bordt, Sebastian, and Ulrike von Luxburg. ”Chatgpt participates in a computer science exam.” arXiv preprint arXiv:2303.09461 , 2023.
  * [12] Selivanov, Alexander, Oleg Y. Rogov, Daniil Chesakov, Artem Shelmanov, Irina Fedulova, and Dmitry V. Dylov. ”Medical image captioning via generative pretrained transformers.” Scientific Reports 13, no. 1 : 4171, 2023.
  * [13] Mitrović, Sandra, Davide Andreoletti, and Omran Ayoub. ”Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text.” arXiv preprint arXiv:2301.13852, 2023.
  * [14] Dai, Haixing, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu et al. ”Chataug: Leveraging chatgpt for text data augmentation.” arXiv preprint arXiv:2302.13007, 2023.
  * [15] Frieder, Simon, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. ”Mathematical capabilities of ChatGPT.” arXiv preprint arXiv:2301.13867, 2023.
  * [16] Liu, Hanmeng, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. ”Evaluating the logical reasoning ability of chatgpt and gpt-4.” arXiv preprint arXiv:2304.03439, 2023.
  * [17] Wang, Jindong, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang et al. ”On the robustness of chatgpt: An adversarial and out-of-distribution perspective.” arXiv preprint arXiv:2302.12095, 2023.
  * [18] Wang, Boxin, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu et al. ”DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models.” arXiv preprint arXiv:2306.11698 , 2023.
  * [19] Jones, Erik, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. ”Automatically Auditing Large Language Models via Discrete Optimization.” arXiv preprint arXiv:2303.04381, 2023.
  * [20] Xu, Xilie, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. ”An LLM can Fool Itself: A Prompt-Based Adversarial Attack.” arXiv preprint arXiv:2310.13345, 2023.
  * [21] Reiss, Michael V. ”Testing the reliability of chatgpt for text annotation and classification: A cautionary remark.” arXiv preprint arXiv:2304.11085, 2023.
  * [22] Rozado, David. ”The political biases of chatgpt.” Social Sciences 12, no. 3 : 148, 2023.
  * [23] Li, Tsz-On, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, and Shing-Chi Cheung. ”Finding Failure-Inducing Test Cases with ChatGPT.” arXiv preprint arXiv:2304.11686, 2023.
  * [24] Borji, Ali. ”A categorical archive of ChatGPT failures.” arXiv preprint arXiv:2302.03494, 2023.
  * [25] Yang X, Li Y, Zhang X, Chen H, Cheng W. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081, 2023 Feb 16.
  * [26] Jalil, Sajed, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. ”Chatgpt and software testing education: Promises and perils.” arXiv preprint arXiv:2302.03287, 2023.
  * [27] Wang, Boxin, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. ”Adversarial glue: A multi-task benchmark for robustness evaluation of language models.” arXiv preprint arXiv:2111.02840, 2021.
  * [28] Ouyang, Tinghui, Yoshiki Seo, and Yutaka Oiwa. ”Quality assurance study with mismatched data in sentiment analysis.” In 2022 29th Asia-Pacific Software Engineering Conference (APSEC), pp. 442-446. IEEE, 2022.
  * [29] Zhang, Wenxuan, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. ”Sentiment Analysis in the Era of Large Language Models: A Reality Check.” arXiv preprint arXiv:2305.15005, 2023.
  * [30] Puigcerver, Joan, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. ”From sparse to soft mixtures of experts.” arXiv preprint arXiv:2308.00951, 2023.
  * [31] $https://community.openai.com/t/chatgpt-results-much-better-than-api/336749$
  * [32] $https://community.openai.com/t/different-output-generated-for-same-prompt-in-chat-mode-and-api-mode-using-gpt-3-5-turbo/318246$
  * [33] Chen, Lingjiao, Matei Zaharia, and James Zou. ”How is ChatGPT’s behavior changing over time?.” arXiv preprint arXiv:2307.09009, 2023.
  * [34] IEEE standard glossary of software engineering terminology. IEEE Std 610.12-1990, pages 1–84, Dec 1990.
  * [35] McAuley, Julian, and Jure Leskovec. ”Hidden factors and hidden topics: understanding rating dimensions with review text.” In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172. 2013.
  * [36] Pang, Bo, and Lillian Lee. ”Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.” arXiv preprint cs/0506075 , 2005.
  * [37] Rayner, Keith, Sarah J. White, and S. P. Liversedge. ”Raeding wrods with jubmled lettres: There is a cost.”, 2006.
  * [38] Gao, Ji, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. ”Black-box generation of adversarial text sequences to evade deep learning classifiers.” In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50-56. IEEE, 2018.
  * [39] Ren, Shuhuai, Yihe Deng, Kun He, and Wanxiang Che. ”Generating natural language adversarial examples through probability weighted word saliency.” In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085-1097. 2019.
  * [40] Hutto, Clayton, and Eric Gilbert. ”Vader: A parsimonious rule-based model for sentiment analysis of social media text.” In Proceedings of the international AAAI conference on web and social media, vol. 8, no. 1, pp. 216-225. 2014.
  * [41] Xu, En-Hui, Xiao-Lin Zhang, Yong-Ping Wang, Shuai Zhang, Li-Xin Liu, and Li Xu. ”Adversarial Examples Generation Method for Chinese Text Classification.” International Journal of Network Security 24, no. 4 (2022): 587-596.


# IoT Solution for Winter Survival of Indoor Plants

Md Saroar Jahan
University of Oulu, CMVS, BP 4500, 90014 Finland
<EMAIL_ADDRESS>
Jhuma kabir Mim
LUT University, 53850 Lappeenranta, Finland
<EMAIL_ADDRESS>
Sampo Niittyviita, Santeri Moberg, Murad Ahmad, Nijar Hossain
University of Oulu, UBICOMP, 90014 Finland
{saniitty, santeri.moberg, murad.ahmad, nhossain18<EMAIL_ADDRESS>

###### Abstract

Not only does cold climate pose a problem for outdoor plants during winter in
the northern hemisphere, but for indoor plants as well: low sunlight, low
humidity, and simultaneous cold breezes from windows and heat from radiators
all cause problems for indoor plants. People often treat their indoor plants
like mere decoration, which can often lead to health issues for the plant or
even death of the plant, especially during winter. A plant monitoring system
was developed to solve this problem, collecting information on plants’ indoor
environmental conditions (light, humidity, and temperature) and providing this
information in an accessible format for the user. Preliminary functional tests
were conducted in similar settings where the system would be used. In
addition, the concept was evaluated by interviewing an expert in the field of
horticulture. The evaluation results indicate that this kind of system could
prove useful; however, the tests indicated that the system requires further
development to achieve more practical value and wider usage.

_K_ eywords Smart agriculture $\cdot$ Sensor for plant $\cdot$ Wireless sensor
network $\cdot$ Precision agriculture $\cdot$ IoT plant health $\cdot$ Smart
home

##  1 Introduction

Several factors present challenges when growing plants at home in Finland. One
of them being Finland’s climate with long and dark winters and short summers.
Also, during spring and autumn, while the average temperature is typically
above zero degrees during the day, the temperature can drop under zero degrees
celsius during the night, which can inflict damage on plants outside. In
addition, during the winter, there are only a few hours of daylight available
which can be inadequate for some plants. The Nordic climate also presents
challenges for indoor plants. Especially in the winter, some of the indoor
plants must be placed next to the window where they can get enough light, yet
that could also expose them to colder temperatures. A study by Nguyen et al.
shows that absolute humidity correlates between outdoor and indoor air, which
then makes the relative humidity drop in the indoors during cold weather [1].
This drop-in relative humidity in winter should also be considered when
growing plants indoors.

For greenhouses and indoor home environments, controlling the growing
environment is crucial since the environment entirely relies on human control.
The moisture of the soil, temperature, illuminance, and humidity are some of
the environmental variables that a person can control for growing plants.
However, determining the optimal growing conditions for certain plants can be
tricky. This is especially the case for people who lack experience. Sometimes
growing plants indoors is challenging because of the lack of proper knowledge
about the plants the person is growing. To help small-scale indoor farming,
including large-scale farming, the use of modern sensors and mobile
applications could be a great addition.

In this research, a sensor system with an Android user interface was
developed, which can be used to measure the environmental conditions of indoor
plants. These sensors are able to collect temperature, humidity, and
illuminance data surrounding the plant and show them in real-time to the user.
Ìf the environmental conditions are outside the recommended range for a
certain plant, the app will notify the user.

The use of a convenient mobile application could also help to promote small-
scale agriculture by making it easier for new home growers to be successful in
gardening. It could also improve the level of localization of food production.

##  2 Related Work

There are a number of different terminologies in use for smart agriculture
(SA), including precision agriculture, variable rate technology, Global
positioning system (GPS) agriculture, farming by inch, information-intensive
agriculture, and site-specific crop management [2, 3]. The topic of smart
agriculture has been researched since the 1990s [4]. It is a broad topic
covering many different kinds of areas of research, from wireless sensor
networks (WSN) to unmanned aerial systems (UAVs) [5].

###  2.1 Sensing in Smart Agriculture

According to Ruiz-Garzia L. et al., the first wireless sensor network
application for greenhouses was published in 2003 which included sensors for
wind flow, wind direction, ambient light, temperature, ambient pressure,
humidity and CO2 percentage [6, 7]. These are some of the variables that are
commonly detected by sensors in SA solutions. Table 1 displays common
variables that can be detected by sensors and are available in the market [2,
8]. There are a wide range of market available sensors; many of which can
collect data from multiple environmental variables [2]. Various systems have
been developed to account for these variables. Examples of different
applications are provided in the next chapter.

Table 1: Different environmental factors measurable by market available
sensors. Soil
---
Temperature | Moisture |  Dielectric permittivity |  Rain/water flow | Water level | Conductivity | Salinity
Plant
Photosynthesis | Moisture |  Hydrogen |  Wetness | CO2 | Temperature | -
Weather
Temperature | Humidity |  Atmospheric pressure |  Wind speed | Wind direction | Solar radiation | Rainfall

###  2.2 Smart Agriculture in Practice

A work by Khurshid Aliev and Eros Pasero presented a practical approach to
acquiring data of temperature, humidity, and soil moisture of plants for smart
agriculture system [9]. They built a prototype device and an Android
application that obtains physical information and sends it to the cloud to
achieve the goal of their project. Moreover, they have focused on a
temperature forecasting application in the subsequent part of their current
research work.

Lashitha et al. presented an efficient sensor, and remote systems coordination
of IoT in an actual agricultural circumstance [10]. By gathering ongoing
information about agriculture that gives simple access to the farmer, their
project’s main objective. They developed task screening that utilizes
different parameters so that it will help the farmer to screen more than one
rural field in the meantime while doing their smart farming. Additionally, the
vast majority of the observation is done remotely; it will help the person
pick up data. As a result, observing through their framework requires less
labor; individuals with physical handicaps can be utilized for checking
fields.

G. Vellidis and M. Tucker have described the smart sensor array, and testing
in a cotton crop [11]. Inputs from the smart sensor array will determine
timing and amounts for real-time site-specific irrigation applications while
doing the integration of the sensors with precision irrigation technologies
will provide a closed-loop irrigation system. That means the array consists of
a centrally located receiver connected to a laptop computer and multiple
sensor nodes installed in the field, a specially designed circuit board and a
Radio Frequency Identification (RFID) tag which transmits data to the receiver
for reliably monitoring spatially variable soil water status in crop fields.

Shailendra Wadje and Mahindra Gadakh proposed a Smart Farming (SF) System to
increase the Production of plants [12]. SF focuses on three main parts as
Sensors, Microcontroller, and a system. They have focused on control, such as
watering capturing the images based on statistical data sensed from sensor
systems (including temperature, humidity, moisture, and light intensity
sensors), by applying Kalman filtering to take more précised data before using
an input into their decision-making process. A decision tree model has been
used to predict data about the weather condition. After that, the set of
decision-based data on both sensed data and weather conditions is developed
automatically to decide on whether the watering and light system should be on
or off. They also provided the function to manually control the watering and
light system through a mobile application for smart farming.

###  2.3 Issues in Smart Agriculture

Smart agriculture is used for its potential toward improving agriculture. For
example, automated agriculture sensing systems can be very effective compared
to traditional methods; for instance, Gutiérrez J. et al. created a
computerized system and tested it in a sage crop field for 136, which resulted
in water saving of up to 90% [13]. Despite efficacy like this, there is yet
room for improvement.

Review papers that focus on different aspects of SA state a wide range of
issues related to the application of SA. Issues derive from the state of
technology associated with SA, currently available systems in the market, and
the farmers themselves. As stated in the paper by Mare Srbinovska et al.,
current issues include poor utilization of new technologies related to smart
agriculture, which is caused by lack of investments and unfavorable decision-
making [4]. According to Srbinovska et al., poor decision-making is due to a
lack of information regarding the potential of using intelligent agricultural
solutions for farming. Many issues also arise from the use of big data in
agricultural systems, which S. Wolfert et al. state in their review paper of
big data in smart farming:

  1. a)

Data ownership and related privacy and security issues,

  2. b)

data quality,

  3. c)

intelligent processing and analytics,

  4. d)

sustainable integration of big data,

  5. e)

attractive business models which enable fair share between different
stakeholders, and

  6. f)

openness of platforms [14].

As current issues related to the state of the technology, low battery power,
limited computation capability, and small memory of the sensor nodes are
mentioned [8]. Most of these issues also affect individuals outside the
farming industry if they wished to start using similar solutions in their
gardening or farming. Currently, the focus of SA applications are on large
scale agriculture; thus, SA could be effectively applied to smaller scale as
well. One area of improvement in the management of the plants in SA is to have
an individualized approach for each plant to meet their needs in different
environmental conditions [15].

Figure 1: Application architecture.

##  3 Design

In this section, the design of the application and its parameters are
presented. While this application can be beneficial even for an experienced
gardener, the main target group for this application is the general public who
are interested in growing plants as a hobby but do not have sufficient
information about plants or their optimal living environmental conditions. The
basic idea of the application is to help plants survive winter in indoor
locations in good health and provide suggestions to improve the conditions of
each plant species that the user might be growing. The system consists of a
bluetooth sensor device called ruuvitag and an Android smartphone to which it
is connected. The application is designed to get temperature and humidity
values from the ruuvitag and illuminance data from the smartphone’s built-in
light sensor, showing the values in real-time to the user. Additionally, the
past sensor data can be viewed in the form of a line graph and a CSV file.
Based on the sensor data, the application sends notifications to the user if
their added plants are out of their optimal range in terms of temperature,
humidity, or light. The software architecture for the application is presented
in Figure 1.

##  4 Implementation

An Android application, which provides feedback on how the user could improve
the living conditions of their plants, was developed for the Android platform.
Since the user could have multiple plants in a single plant pot, the
application has support for multiple plants. The purpose for that is to give
specialized information for each of the plants the user might have. For
example, some plants may cope with less sunlight during the winter; some
plants are more adapted to winter conditions. A notification system was
implemented to inform the user if there are actions that they should take to
ensure a healthy environment for their plants. For each plant, there is set an
optimal range for temperature, relative humidity, and illuminance. If the
values that are sensed by the sensors go out of the optimal range for a more
extended period, the user receives a warning notification. These ranges are
different for each plant species because every species have their own optimal
range of suitable growing conditions. The data for the optimal living
conditions for a specific plant species is stored in a cloud database, where
it can be fetched through the internet or exported in the desired format and
manually transferred to the mobile device.

###  4.1 Equipment and Software

Android Studio was used for developing the application. Android mobile
Motorola Moto G (3rd gen) and Google Pixel were used to test the software
(Motorola Moto G (3rd gen) in evaluation). Java programming language was used
for development (Android programming). The system was developed on Android
version 8.0.0. The compile SDK version was 28, and the minimum SDK version was
18, along with some libraries including android-beacon, bluetooth, gson, and
volley. Ruuvitag was used for sensing relative humidity and temperature.
SQLite was used to store the sensor data and information regarding the plants
locally on the mobile device.

###  4.2 Application

Sensor data is transferred directly via bluetooth from the ruuvitag to an
Android smartphone. Based on the data, the user gets real-time information
regarding the precise environmental conditions of the plants in the sensor
device’s close proximity. The user has added in the app for each individual
plant species; he/she will get notified if the environmental conditions are
not within their optimal range. Users can remove or add new plants to be
monitored by the app. Based on the notifications the user gets from the
application, the user can then do the corrections to the plants’ environment.

Data logging was implemented by writing the values of temperature, humidity,
and illuminance along with a timestamp to a CSV file every few seconds. This
data was later used in the evaluation of the application.

The application has the following components:

  1. a)

UI,

  2. b)

API for Ruuvitag,

  3. c)

data logging,

  4. d)

plant management,

  5. e)

database and

  6. f)

notification system.

###  4.3 User Interface (UI)

There are three main screens of user interface as shows in Figure 2:

  1. a)

Main screen,

  2. b)

Add plant, and

  3. c)

Plant management

On the main screen, the current environmental conditions are displayed, and
bottom navigation is used for other screens. When the application is launched,
the main screen displays the real-time temperature values, humidity, and
illuminance. Below the sensor data on the main screen, a graph shows the
history of the sensor values to the user.

In the bottom navigation, there are three buttons to access each of the
screens: "Dashboard" for the main screen, "Add plant" for adding the plants,
and "Plants" for the plant management. In the "Add plant" screen, the plants
are divided into categories, and different plants are shown, which are stored
in the database. Users can add a plant from the list of plants to their
existing plants by tapping on the plant’s name. The user can see the existing
plants that they have added to the application in the plant management screen.
Users can also remove a specific plant from their existing plants by pressing
the minus-icon.

Figure 2: Screenshots of the smartphone UI.

###  4.4 Plant Database

The database was implemented to contain eight properties for each plant, which
include:

  1. i)

Name of the plant,

  2. ii)

Maximum temperature (Celsius),

  3. iii)

Minimum temperature (Celsius),

  4. iv)

Maximum humidity (relative humidity),

  5. v)

Minimum humidity (relative humidity),

  6. vi)

Maximum illuminance (lux),

  7. vii)

Minimum illuminance (lux), and

  8. viii)

Description of the plant.

The maximum and minimum values determine the optimal range for the variable in
question regarding the plant’s growing condition.

The plan was to gather data for each plant, searching the internet for guides
and advice about growing the selected plant. Because of time constraints, data
only for one plant species was gathered, Peace Lily (Spathiphyllum) plant,
used in the evaluation part. Figuring the optimal growing conditions for Peace
Lily was done by searching the internet for information on how to treat a
Peace Lily plant. Multiple sites stated that the optimal temperature for Peace
Lily is between 18 to 25 degrees Celsius. Since Peace Lily is a tropical
plant, it also likes high humidity (40-90%). If the humidity is under 40%, it
might cause health problems for the plant. Determining the proper lux values
for the plant proved to be infeasible; thus, only textual information was
recorded for the amount of light the Peace Lily needs. Which was "low to
moderate light". Data for the Peace Lily plant was stored locally on the
installed app (hardcoded) using SQLite.

##  5 Evaluation

The purpose of the application is to develop a tool for helping plants survive
winter in living apartments. The evaluation was performed to see how the
application performs in this task; thus, the goal of the evaluation is to
measure: how accurate and meaningful data is presented for the user. The
assessment consisted of two parts: (i) concept evaluation and (ii) functional
evaluation. The concept evaluation was performed in the form of an interview
for a single professional in the field of horticulture to remediate potential
issues that were not taken into account in the implementation and get some
feedback on the aspects of the system. The functional evaluation was conducted
to test the functionalities of the core system features and data reliability.

###  5.1 Interview Setup

The interview was conducted at the University of Oulu for a professional
horticulturist with 33 years of experience with plants. The language used in
the interview was Finnish since that was the native language of the
interviewee. The whole interview was voice recorded and later analyzed and
transcribed into English. The interview was conducted in two parts. The first
part consisted of questions regarding the interviewee’s background, and in the
second part, the concept of the application and the work-in-progress
application were presented to the interviewee. After that, the interviewee was
asked to fill an AttrakDiff questionnaire. This was followed by questions
regarding the concept and more general questions about plant care during
winter.

###  5.2 Functional Evaluation Setup

Two peace lily (Spathiphyllum) plants which are common house plants in
Finland, were used in two different locations to conduct the evaluation. This
was done to give variability to the potential plant effects which might affect
the measurements. Real plants were used because the plants and their pot might
affect the measurements; for example, the plant might provide shade for the
measurement device and affect the measurements’ moisture levels or
temperature. The plants were bought from a local nursery garden. Also, the
well-being of the plant might give some indications regarding the plants’
conditions. The plant pots’ original material was plastic which was the
material of the pots the plants were sold in. This plant pot was used in the
evaluation.

Figure 3: Two locations used in the experiment.

The experiment setup consisted of both plants, where the sensors were placed
next to the plants. The plants were located in two different rooms in the same
apartment. Figure 3 presents the experimental setup in the locations: the
corner location and the window location. The windows of each room were
directed towards different compass points, and the plants were located at a
slightly different height from the floor. For sensing illuminance, the
smartphone was placed at an angle of 45 degrees so that the sensor would
receive a similar amount of light as the plant next to it would. The ruuvitag
sensor device was placed under the leaves to mimic the humidity and
temperature that the plants’ experience (the plants change air under their
leaves). The evaluation was conducted between 24.11.2018 and 8.12.2018 (14
days). For the whole period, the Android smartphone (Motorola Moto G 3rd Gen)
was constantly charged and within the bluetooth range of the sensors.

In the evaluation, the following actions were taken to gather data from the
evaluated system:

  * •

Displayed sensor data every few seconds (humidity, illuminance and
temperature),

  * •

Pictures of the plants.

Thus, the focus of the evaluation was on the sensor data and not on the
usability of the system. The pictures of the plants were taken every day from
the same time of the day from the same angle and distance to see if the plants
had any observable changes in their appearance during the experiment. The
collected sensor data from the devices was uploaded every day to external
storage in CSV format; moreover, the plants were observed carefully on a
regular basis, and the living environment was kept the same.

Before the start of the experiment, the leaves of both of the plants were
cleaned with a wet towel to remove any dust which might have an effect on
their photosynthesis. Camera and sensor locations were marked with tape on the
ground to ensure that they were not moved during the experiment. Plant care
guidelines were followed to ensure plant health: fertilizers were not used
since fertilizing during winter could potentially kill the plant, and each
plant was given ten dl amount of water every week.

##  6 Evaluation Results

The concept of the application appeared to be promising based on the feedback
gained from the interview. For functional evaluation, sensor data during the
experiment was gathered. Data showed that the system can be used to measure
temperature, humidity, and illuminance within the plants vicinity and that
there is a meaningful difference in the sensor values between two different
locations in the same apartment. Also, the sensor values being outside the
optimal range for the test plants showed how this kind of application could be
helpful.

###  6.1 Interview Results

The interview gave insight into how smart agriculture applications mostly
focus on the greenhouse environment. There are not many, if any, applications
that are for the hobbyist to use, which was also hypothesized based on the
state-of-the-art research. Also, sensor systems used in a greenhouse do not
fit well in a home environment because the spectrum of different kinds of
house plants is much larger than in a greenhouse where there are many plants
of the same species, to which the system is calibrated.

Based on the interview, the software concept appeared to be promising with
its’ intended features. As an improvement suggestion, the interviewee stated
that the concept could be broadened to be used even in summer, yet was this to
be implemented; changes should be made to enable support for summer conditions
and are considerably different. If there were more than one plant in a room,
more sensors would be helpful, especially more light sensors.

The application could be useful for both hobbyists and casual plant owners.
Dedicated hobbyists may require precise plant information from the plants day
and night, especially in winter, while casual plant owners often neglect their
plants due to a lack of information. The interviewee often receives customers
who tell that their plants have died during the winter, which is not usually
the case for more dedicated hobbyists.

The interviewee showed interest in using an application of this kind: he/she
has not encountered this kind of application before. According to his/her, the
existing greenhouse applications are very complex and cannot be used in
practice by the average person. In addition to the measured conditions, light,
humidity, and temperature, information regarding plant watering could be
helpful since watering requirements are not the same during winter and summer.

The AttrakDiff questionnaire result was deemed inaccurate - the interviewee
expressed his/her positive outlook on the software, yet without any nuances
that might be lacking. He/she rated the software with the highest scores in
the questionnaire. The AttrakDiff would be more useful with a larger sample
size and in a situation where the participant could perform the questionnaire
anonymously. Since he/she was the only participant in the concept evaluation,
the participant might feel more pressured or vital, which might affect the
results.

###  6.2 Functional Evaluation Results

The results show that the system can be used to measure light level, humidity,
and temperature in the plants’ vicinity. The evaluation results were for this
short period; however, the pictures are taken every day during the evaluation
every day and plants showed no sign of observable changes in either of the
plants. This would indicate that at least for the period of the evaluation,
the plants’ health did not deteriorate significantly, even though conditions
were not optimal for a Peace Lily plant. The temperature in the window
location was under the recommended 18-25 degrees celsius, and humidity was
under the recommended 40-90% humidity in both locations. The sensor data from
both locations consisted of temperature, humidity, and illuminance values and
a timestamp that was saved in a CSV file. This file was used for the analysis.

Figure 4: Line graphs from the collected sensor data.

Figure 4 presents line graphs of temperature and humidity values during the
experiment in both locations, comparing the illuminance in both locations and
illuminance during daytime in one day. We can see from the collected data that
the temperature was the steadiest of the three variables; there is only little
variance, and only one spike of cold temperature can be seen during the
experiment. That could have been due to a cold draft made by opening a window,
for example. Relative humidity varied between 30 to 45 in both of the
locations, which is outside of the recommended humidity for the Spathiphyllum
plant. The illuminance was near zero during most of the experiment and only
spiked during the daytime. The highest value of illuminance recorded was 871
luxes. During this peak, the illuminance value in the location further away
from the window was only 70 luxes, which shows that distance from the window
affects a lot on how much light the plant gets.

Average sensor values during the experiment in both of the locations are
presented in Table 2. The temperature was almost two degrees warmer in the
window corner location, which was further away from the window. The difference
is significant because the temperature in the window location lower than the
recommended temperature for the Peace Lily plants that were used in the
experiment. The difference in location does not seem to affect much in terms
of humidity, the only difference of 1,62 percentage points. Humidity was also
lower than the recommended humidity for Peace Lily plants in both of the
locations. Illuminance had the biggest difference between the locations. The
average illuminance in the window locations was over seven times as much as in
the corner location.

Table 2: Average sensor values during experiment. Location |  Temperature |  Humidity |  Illuminance
---|---|---|---
Corner |  19.48 °C |  34.24 |  10.36 lux
Window |  17.59 °C |  35.86 |  75.55 lux

###  6.3 Limitations

During the two-week experiment, the application that was running on both of
the smartphones crashed multiple times. While the application was down, no
sensor data was gathered, and the application had to be restarted manually.
This resulted in significant gaps in the collected sensor data. Also, it is to
be noted that the evaluation only had two plants in two locations - it would
be good to have more plants and more locations to get more reliable and
consistent. Due to these points, no generalization should be made from the
gathered data.

The tested system required a constant connection between the Android
smartphone and the sensor; however, this proved to be an insufficient
solution; instead, the system should be implemented using either an external
server node which would send data to a cloud database or the system would
store the data from longer periods until it can be synced to the Android
smartphone.

There are advantages and disadvantages for both multiple and single sensor
approaches. A single sensor approach for all variables would be easier to
manage yet slightly more inaccurate due to un-optimal placing (e.g.,
optimally, the illuminance is measured from the top of the plant and humidity
under the plant’s leaves). The system should be tested with both approaches on
actual end-users to determine each approach’s applicability. While, in theory,
multiple sensors would give more accurate data - too many sensors might be too
cumbersome to handle for normal users.

Since the experimenters did not have a background in gardening, the
mishandling of the plants was an issue for consideration. The initial
condition of the plants was that of the plant’s condition, which they were
available from the plant nursery. While it can be expected that the plants are
well taken care of in the plant nursery, this might not always be the case,
yet the plants appeared to be in good condition upon purchase. The condition
of the plants is displayed in Figure 3. Also, in an actual situation, the
mobile device may not necessarily constantly be within the range of the
sensors; this would be relevant to include in the case of an available
product.

##  7 Future Works

Another option for the implementation would be to include a network node that
would have access to the internet. This would allow remote and constant access
to the plants’ conditions; however, it would make the system more expensive.
The number of devices required for this would increase from two to three,
which would make the system more complicated to use and upkeep.

Instead of only one set of sensors, multiple sensors of the same type could be
used. Yet, there are advantages and disadvantages for multiple and single
sensor approaches. A single sensor approach for all variables would be easier
to manage, yet potentially slightly more inaccurate due to un-optimal placing
in relation to some of the plants. It would be useful to test the system with
both approaches on actual end-users to determine the applicability of each of
the approaches. The results also suggest that the use of multiple sensors
would be beneficial if there are plants in multiple locations in the same
apartment, as the measured locations had clearly different conditions.

As there are a large number of different plant species, the plant database
would have to be very large to be all-encompassing. Optimally, for saving
plant data and analyzing user data more effectively, a cloud server would be a
good solution. Since gathering plant data would be very difficult to achieve,
another solution could be investigated. One option for this would be to
implement a parallel crowdsourced database, where the users would themselves
add relevant plant information to the database. Additionally, other
environmental variables which might affect the measurement include spatial
requirements of the plants and watering of the plants and soil conditions. Of
these measurements, the watering of the plants came up in the interview. While
data from these variables would be interesting, these are not the most
relevant variables for winter survival. It would make the system even more
complex, less approachable, and expensive for a layperson.

The messaging system in the implementation included two thresholds for
messages; a low and a high, as discussed in plant database section 4.4. For
each measured variable, it would be beneficial to have more thresholds instead
of only low and high, for example, very low and very high in addition.

##  8 Conclusion

The results indicate that the system 1) can be used to measure light level,
humidity, and temperature in the plant’s vicinity, and 2) the measurement of
these variables is useful. These preliminary results promise to conduct
further research on the system and the area related to the system. The
developed system would require more iterations. The information for optimal
growing conditions would have to be accurate for a larger number of plants
before the system would be sufficient for commercial use.

The results also suggest that using multiple sensors would benefit plants in
multiple locations in the same apartment, as the measured locations had
clearly different conditions. The interview results also suggest this
improvement. In addition, the concept could be broadened and some changes to
be implemented, for example, for summer usage as the interview results
suggested, yet, were this to be implemented, changes should be made to enable
support for summer conditions as they would significantly differ from winter
conditions.

## Acknowledgments

We would like to thank University of Oulu Ubicom Department for providing
research material.

## References

  * [1] Jennifer L Nguyen, Joel Schwartz, and Douglas W Dockery.  The relationship between indoor and outdoor temperature, apparent temperature, relative humidity, and absolute humidity.  Indoor air, 24(1):103–112, 2014.
  * [2] Abu Zafar Abbasi, Noman Islam, Zubair Ahmed Shaikh, et al.  A review of wireless sensors and networks’ applications in agriculture.  Computer Standards & Interfaces, 36(2):263–270, 2014.
  * [3] Ancha Srinivasan.  Precision agriculture: An overview.  Handbook of Precision Agriculture, pages 33–48, 2006.
  * [4] Mare Srbinovska, Cvetan Gavrovski, Vladimir Dimcev, Aleksandra Krkoleva, and Vesna Borozan.  Environmental parameters monitoring in precision agriculture using wireless sensor networks.  Journal of cleaner production, 88:297–307, 2015.
  * [5] Chunhua Zhang and John M Kovacs.  The application of small unmanned aerial systems for precision agriculture: a review.  Precision agriculture, 13(6):693–712, 2012.
  * [6] Luis Ruiz-Garcia, Loredana Lunadei, Pilar Barreiro, and Ignacio Robla.  A review of wireless sensor technologies and applications in agriculture and food industry: state of the art and current trends.  sensors, 9(6):4728–4750, 2009.
  * [7] G Liu and Y Ying.  Application of bluetooth technology in greenhouse environment, monitor and control.  J. Zhejiang Univ., Agric. Life Sci, 29(1):329–334, 2003.
  * [8] Tamoghna Ojha, Sudip Misra, and Narendra Singh Raghuwanshi.  Wireless sensor networks for agriculture: The state-of-the-art in practice and future challenges.  Computers and electronics in agriculture, 118:66–84, 2015.
  * [9] Khurshid Aliev, M Moazzam, Sanam Narejo, Eros Pasero, and Alim Pulatov.  Internet of plants application for smart agriculture.  Int. J. Adv. Comput. Sci. Appl, 9(4):421–429, 2018.
  * [10] P Lashitha Vishnu Priya, N Sai Harshith, and NVK Ramesh.  Smart agriculture monitoring system using iot.  International Journal of Engineering and Technology (UAE), 7:308–311, 2018.
  * [11] George Vellidis, Michael Tucker, Calvin Perry, Craig Kvien, and C Bednarz.  A real-time wireless smart sensor array for scheduling irrigation.  Computers and electronics in agriculture, 61(1):44–50, 2008.
  * [12] Shailendra Wadje, Machindra Gadakh, Abhijit Sangamnere, and Gaurav Tungar.  Smart farming system android app.  2017\.
  * [13] Joaquín Gutiérrez, Juan Francisco Villa-Medina, Alejandra Nieto-Garibay, and Miguel Ángel Porta-Gándara.  Automated irrigation system using a wireless sensor network and gprs module.  IEEE transactions on instrumentation and measurement, 63(1):166–176, 2013.
  * [14] Sjaak Wolfert, Lan Ge, Cor Verdouw, and Marc-Jeroen Bogaardt.  Big data in smart farming–a review.  Agricultural systems, 153:69–80, 2017.
  * [15] David J Mulla.  Twenty five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps.  Biosystems engineering, 114(4):358–371, 2013.