# Self-Supervised Backbone Framework for Diverse Agricultural Vision Tasks

Sudhir Sornapudi, Rajhans Singh
Corteva Agriscience, Indianapolis, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Computer vision in agriculture is game-changing with its ability to transform
farming into a data-driven, precise, and sustainable industry. Deep learning
has empowered agriculture vision to analyze vast, complex visual data, but
heavily rely on the availability of large annotated datasets. This remains a
bottleneck as manual labeling is error-prone, time-consuming, and expensive.
The lack of efficient labeling approaches inspired us to consider self-
supervised learning as a paradigm shift, learning meaningful feature
representations from raw agricultural image data. In this work, we explore how
self-supervised representation learning unlocks the potential applicability to
diverse agriculture vision tasks by eliminating the need for large-scale
annotated datasets. We propose a lightweight framework utilizing SimCLR, a
contrastive learning approach, to pre-train a ResNet-50 backbone on a large,
unannotated dataset of real-world agriculture field images. Our experimental
analysis and results indicate that the model learns robust features applicable
to a broad range of downstream agriculture tasks discussed in the paper.
Additionally, the reduced reliance on annotated data makes our approach more
cost-effective and accessible, paving the way for broader adoption of computer
vision in agriculture.

##  1 Introduction

Computer vision, coupled with deep learning models [14, 10, 15, 5], has
ushered in a new era of transformation for the agricultural sector. These
vision models have become integral to numerous aspects of agricultural
practices, including crop phenotyping [32, 2, 41], monitoring [23, 19],
precision agriculture [44, 18], agricultural robotics [7, 31], and weed
control [43]. These applications encompass everything from biological cell
images to vast satellite imagery, showcasing the extensive and diverse data
generated within the agricultural domain. However, the performance of deep
learning models depends on the availability of large high-quality labeled
datasets for training. Given the expensive and time-consuming nature of manual
labeling, true potential of these datasets remains largely untapped. As
agricultural datasets grow in size and complexity, the challenge of manual
annotation presents a significant obstacle to unlocking their value.

Figure 1: Wide range of applications based on representations learned from
self-supervised modeling.

The advent of self-supervised learning (SSL) approaches [17, 35], particularly
pretraining steps, has transformed the AI landscape by enabling learning from
voluminous unlabeled datasets, thereby minimizing dependency on manual
labeling. This pretraining step has significantly improved performance across
diverse tasks such as classification [16], segmentation [3], and detection
[37]. By employing self-supervised techniques, models can learn generic visual
representations encapsulating inherent data structures and patterns. Once
pretrained, these models serve as a backbone for various tasks. Moreover,
leveraging self-supervised learning with datasets from similar domains has
proven to improve transfer learning capabilities as compared to data from
other domains [24]. However, publicly available large datasets from the
internet often contain redundant data unrelated to agricultural tasks. Hence,
in this paper, we investigate contrastive learning [6], a self-supervised
approach, utilizing large agricultural datasets for various tasks as shown in
Figure 1.

While our approach isn’t the first application of self-supervised learning in
the agricultural domain, notable efforts like Güldenring et al.’s [12] self-
supervised contrastive learning on agriculture images and Ogidi et al.’s [24]
benchmarking of self-supervised techniques in agricultural phenotyping have
set important precedents. However, these works focus on a self-supervised
approach tailored for transfer learning. Despite previous endeavors, these
works overlook certain advantageous aspects of the derived feature
representations from self-supervision. Our paper aims to emphasize these
potential benefits and applications through various empirical analyses.

Ogidi et al. [24] demonstrated empirically that, in many instances, leveraging
a domain-specific yet diverse pretraining dataset yields superior performance
in transfer learning tasks. Thus, we curated a large dataset comprising
agricultural field imagery captured via mobile phones and satellites for this
paper. Encompassing diverse scenarios such as different crop stages, harvested
and unharvested fields, and ranging from close-up leaf images to expansive
field views, our dataset aims to provide a rich and varied resource for
advancing agricultural computer vision research.

This work investigates how self-supervised representation learning can empower
the agricultural sector by unlocking the value of unlabeled data. We explore
the transformative potential of the powerful feature representations from
agriculture images, leading to improved efficiency and performance across
various downstream tasks. In particular, we explore the following applications
and benefits:

  * •

Enhanced data efficiency: SSL pretrained model features facilitate insightful
interpretation of raw data and help in validating and balancing training
dataset diversity. Moreover, fine-tuning with just $1\%$ of labeled in-domain
data achieves an impressive $80.2\%$ accuracy, significantly reducing labeling
requirements.

  * •

Transfer learning: Through experiments covering classification, detection, and
segmentation tasks, we demonstrate the benefit of using self-supervised
pretrained models over fully supervised models trained from scratch,
particularly in scenarios with limited labeled training data. For these
experiments, we use publicly available datasets including rice leaf disease
identification [38], PlantVillage [22], and MinneApple [13].

  * •

Faster model convergence: Experimental results from finetuning for downstream
tasks demonstrate that utilizing SSL pretrained models leads to a faster
training convergence rate.

  * •

Outlier detection: Using representations from the SSL pretrained model with
clustering analysis enables effective identification of anomalies in
agricultural data.

  * •

Content-based image retrieval at scale: We created PixelAffinity, a web-based
tool for content-based image retrieval, harnessing SSL pretrained model
features to efficiently search for similar images within vast agroecology
databases.

  * •

Video data prepossessing: Utilizing features extracted from SSL pretrained
models on video data enables the identification of redundant frames and the
selection of relevant frames tailored to specific applications.

  * •

Guidance model for Image reconstruction/editing : By employing SSL pretrained
models as guidance networks, we can effectively reconstruct and synthesize new
images or perform image editing tasks based on given input images, using
diffusion or GAN models.

##  2 Related Work

Figure 2: Illustration of the two-stage process: self-supervised pre-training
by contrasting different views of the same image and label-light supervised
fine-tuning for downstream tasks.

Deep learning in agriculture: Deep learning based approach has proven highly
successful in various vision-based tasks such as classification, object
detection, and semantic segmentation. Its application in agriculture has
yielded significant advancements across diverse classification tasks such as
rice leaf disease classification [38], plant disease classification [22],
sorghum classification [29], and many more. Segmentation tasks tackle a vast
range of challenges such as identifying Aphid Clusters [27], Mushroom
Segmentation [30], and instance segmentation of lettuce [36]. Additionally,
detection tasks have been pursued with initiatives like Sorghum Panicle
Detection [4], pollination monitoring [28], and Oil Palm Tree Detection [39],
among others. However, the performance of deep learning models for these tasks
in agriculture is highly dependent on the availability of high-quality labeled
datasets. In our work, we explore a self-supervised approach to reduce the
burden of labeled data for training.

Transfer learning and self-supervision in agriculture: Due to the limited
availability of labeled training data in agriculture, transfer learning and
self-supervision can play pivotal roles. For example [38] for rice leaf
disease and [25] for weed species classification use ImageNet [8] pretrained
models, weed detection [25] and apple detection [13] use Coco [20] pretrained
model. These publicly available large datasets for pretraining often differ
significantly from the agricultural domain, underscoring the importance of
leveraging pretrained models from similar domains for improved performance.
Self-supervised approaches are popular in general vision tasks; however, their
application to agriculture tasks has been relatively less explored. For
instance, [24] utilizes a self-supervised approach for plant phenotyping,
while [40] applies it to fine-grained tomato disease classification. Despite
these initial efforts, we believe that the self-supervised approach holds far
greater potential than previously explored. Our paper aims to highlight these
potential benefits backed by concrete evidence.

##  3 Methodology

Representation learning is concerned with training machine learning algorithms
to learn useful features. Deep neural networks can be considered
representation learning models that typically encode information (viz. images)
that is projected into a different subspace. Our approach leverages self-
supervised learning in a two-stage process, as shown in Figure 2. In the
pretraining stage, we employ a large, unlabeled dataset and a contrastive
pretext task to train our model. This builds a foundation of generalizable
feature representations. We then freeze specific layers to preserve this
learned knowledge. Subsequently, in the fine-tuning stage, we adapt the pre-
trained model to the specific downstream task using a smaller, labeled
dataset. By strategically unfreezing relevant layers, we leverage the pre-
trained weights as a starting point while simultaneously refining the model
for our task objective. This two-stage approach aims to harness the power of
large-scale unlabeled data for improved performance on our target task.

###  3.1 Pre-training stage

For simplicity, we adopted the SimCLR framework [6] that utilizes a self-
supervised contrastive learning approach to learn high-quality feature
representations from completely unlabeled data. Figure 2 illustrates the
SimCLR framework. This is a special case of unsupervised learning in which we
create a pseudo-task to generate pseudo-labels, enabling pre-training in a
supervised manner. This is accomplished by contrastive comparison of image
pairs. A large minibatch of $N$ images is randomly augmented twice to get two
different views. This results in $2N$ data points. Given a positive pair
forming from differently augmented views of the same image, the rest $2(N-1)$
augmented views are treated as negative pairs. This way the negative pairs
were implicitly sampled. The resulting augmented pairs are passed through the
same convolutional neural network (CNN). For our study, we chose ResNet-50
($\times 1$) as the backbone CNN architecture to extract the hidden
representation and eventually maximize the agreement between the two embedding
vectors. This indirectly means minimizing the contrastive loss. In other
words, we want to maximize the similarity between the positive pairs and
minimize the similarity between the negative pairs. The representations
generated from the backbone network empirically do not generate the best
results, so Chen et. al. [6] introduced a second projection network that
generates the optimal latent space embeddings ($z$) to apply the loss
function. We used a 3-layer multi-layer perceptron projection head to project
the representation to 128-dimensional embedding. These embeddings from the
projection network are only used for optimizing the training while minimizing
the contrastive loss function ($L$). The high-quality 2048-dimensional feature
representations ($h$) from the backbone encoder network are retained for the
downstream tasks.

For a given $x_{i}$, the pretext task aims to identify its positive counter
part $x_{j}$ in $\\{x_{k}\\}_{k\neq i}$, where $\\{x_{k}\\}$ is $k$-th mini-
batch set including positive pairs $x_{i}$ and $x_{j}$. The contrastive cost
function [33] for a positive pair of examples in a minibatch is defined as

$\displaystyle
l_{ij}=-log\frac{exp(\mathbf{z_{i}}^{T}\mathbf{z_{j}}/\tau)}{\sum_{k\neq
i}exp(\mathbf{z_{i}}^{T}\mathbf{z_{k}}/\tau)},$ (1)

where $\tau$ denotes a temperature parameter, and the final loss ($L$) is
computed across all positive pairs.

We choose random crops with resize, random flip, and color jitter as data
augmentations. The loss ($\tau$ = 0.1) is optimized using LARS [42] with a
learning rate of $0.075$ and weight decay of $1e^{-4}$. The pre-training was
conducted at a batch size of $2048$ for $100$ epochs with $128$ GB Cloud TPU
using $8$ cores.

###  3.2 Fine-tuning stage

After the pretraining, the model serves as a backbone for various downstream
tasks such as classification, segmentation, feature visualization, and
analysis. For instance, in smaller classification datasets, we fine-tune the
pretrained model by appending classification heads and fine-tuning the
appended MLP layers, while freezing the rest of the model. Alternatively, we
can fine-tune the entire network parameters with a very small learning rate.
For classification fine-tuning, we employ the Adam optimizer with a learning
rate of $1e^{-5}$, trained for $100$ epochs (or $200$ for smaller datasets)
with a batch size of $64$ on a single NVIDIA T4 GPU with multiclass cross-
entropy loss. In detection and segmentation tasks, we utilize the SSL
pretrained ResNet-50 model as a backbone for the Mask R-CNN model [15],
implemented in TensorFlow’s official library. This model is trained with
classification, bounding box regression, and instance segmentation loss
functions. We fine-tuned this Mask R-CNN model with an image size of
$640\times 640$ and a batch size of $8$ on two A100 GPUs for $100k$
iterations.

##  4 Experimental Analysis

This section introduces the datasets, benefits and applications of self-
supervised representations, and discussion. The performance of the model is
evaluated through a series of applications, and the experimental results are
quantitatively and qualitatively analyzed. Finally, the effectiveness of self-
supervised representation learning in digital agriculture is verified.

###  4.1 Dataset

In our work, we utilized unlabeled Corteva’s real-world agriculture field data
for self-supervised pretraining and publicly available datasets like
PlantVillage, Rice Leaf Disease, and Minneapple for supervised fine-tuning to
demonstrate robust downstream learning.

Corteva Dataset. This is an unstructured, unlabeled raw agriculture on-field
and off-field image dataset captured with digital cameras, smartphones,
drones, satellites, and screenshots at varying resolutions (Figure 3). We
collected a total of $776377$ images and used the entire data for the
pretraining stage to learn high-quality visual representations. We held $1\%$
of the data (i.e., $7763$ images) for finetuning and an additional $1864$
images as hold-out-test. These were labeled for downstream classification
tasks. We categorized this subset of data into 8 different classes: Corn
field, Corn cob, Corn leaf, Soy field, Satellite view, harvested field,
Unknown field, and Other (screenshots, buildings, vehicles, manufacturing
plants, etc.). This is a proprietary dataset, and will not be released to the
public.

Figure 3: Corteva real-world unlabeled agriculture image data.

PlantVillage Dataset. The PlantVillage dataset [22] is a publicly available
leaf disease dataset. The images were captured at experimental research
stations in association with the Land Grant Universities. The leaves were
removed from the plant and imaged using Sony DSC 20.2 megapixel digital
camera. We used a subset of this dataset available on
Kaggle111https://www.kaggle.com/datasets/emmarex/plantdisease/data. It
consists of $20.6$K healthy and unhealthy leaf images. These images were
labeled by expert plant pathologists and were divided into 15 categories based
on 3 crop species (bell pepper, potato, and tomato) and various diseases
(bacterial, early and late blight, leaf mold, spots, 2 viral and 1 caused by
mite). We partitioned this dataset into training and test sets using an
$80-20\%$ split, ensuring a uniform selection from each class. We used this
dataset to demonstrate the transfer learning capability of our pre-trained and
fine-tuned models.

Rice Leaf Disease Identification Dataset. This dataset [38] was aggregated
from publicly available rice leaf datasets and the internet, and annotated by
researchers under the guidance of plant protection experts. The dataset
consists of $4523$ rice leaf disease images and is sorted into seven classes
(healthy, hispa, brown spot, leaf blast, bacterial blight, bacterial leaf
streak, and sheath blight). The authors released a subset of $3353$ annotated
images on Kaggle, a platform for data science competitions. We split the
dataset $80-20\%$ into training and test sets, uniformly sampling from all
classes, and we used this subset of images for our downstream classification
task.

MinneApple Dataset. The dataset [13] was created by the University of
Minnesota researchers to create a unified dataset to advance the state-of-the-
art in fruit counting, detection, and segmentation in orchard environments.
These are a variety of high-resolution red and green apple fruit on tree
images. The dataset contains $1000$ images with over $41000$ annotated object
instances. The annotations span over patch-based counting of clustered fruits,
marked bounding boxes, and masks drawn over each apple instance. We use this
dataset to study the transferability of the base ResNet-50 model for detection
and segmentation tasks.

Sentinel-2 satellite data. Sentinel-2 data is part of the Copernicus
Sentinel-2 mission [1], which aims to provide free and open access to Earth
observation data captured by two polar-orbiting satellites. Although
Sentinel-2 is a multi-spectral instrument, we focused on retrieving data that
contain RGB bands at 10 meters spatial resolution from 10 different field
locations around the globe for our experimental purposes, specifically from
Level-2A which offers atmospherically corrected surface reflectance images. We
prepared a $338$ agriculture images dataset with $256\times 256$ resolution,
containing regions with green vegetation, no vegetation, and water, with and
without cloud shadow, and a few medium and high cloud cover images. We use
this dataset to observe the outlier detection capability of the proposed
model, where images with cloud cover are outliers.

###  4.2 Benefits and Applications

Enhanced Data Efficiency
We begin by examining feature visualization of the SSL pretrained model as a
means to enhance data efficiency. In Figure 4, we present a 3D point cloud
comprising $1864$ labeled test set, where each point corresponds to a
2048-dimensional feature embedding vector representing its respective input
image (plotted in a 3D space using UMAP dimension reduction algorithm [21]).
The figure also shows the ground-truth labels of each point as color code. The
cluster formations, indicated by ground truth labels, emphasize the model’s
ability to learn robust feature representation, thereby serving as an
effective automated labeling tool. This can be extended to observe the
training data points of downstream tasks and understand the cluster formations
to evaluate the diversity and imbalance in the training data. Further, we
leverage the pretrained model for fine-tuning on a $1\%$ of labeled data from
our Corteva dataset across eight classes. Subsequently, we evaluated the model
on $1864$ held-out test images, achieving an impressive accuracy of $80.2\%$
accuracy despite utilizing only small subset of labeled information during
training.

Figure 4: 3D point cloud visualization of hold-out Corteva test data feature
representations with color-coded ground-truth labels.

Transfer Learning
For transfer learning, we used two datasets for classification and one dataset
for detection and instance segmentation. For the classification tasks, we
adopted a fine-tuning approach with pre-trained models and contrasted the
results against models trained from scratch and fine-tuned from ImageNet
weights on both datasets. Additionally, we conducted experiments using only
$25\%$ and $10\%$ of the training datasets and evaluated their performance on
the entire training set. Tables 1 and 2 show the accuracy of various models on
the rice leaf disease and plant village datasets respectively. In rice disease
classification, fine-tuning the entire parameter set yielded a $1.2\%$
improvement when using $100\%$ of the training data, and a notable $17\%$ and
$4\%$ improvement with only $10\%$ of the data, compared to models trained
from scratch and fine-tuning ImageNet, respectively. Similarly, on the plant
village dataset, while performance remained almost the same with $100\%$ of
the training data, a substantial $7\%$ and $3\%$ respective improvement was
observed when utilizing only $10\%$ of the data. Interestingly, fine-tuning
only the last layer resulted in inferior performance compared to models
trained from scratch. These tables indicate that in scenarios with limited
training data, the fine-tuning approach significantly outperforms models
trained from scratch.

Table 3 presents the mean Average Precision (mAP) results for object detection
on the minneapple dataset using the Mask R-CNN architecture. The experiments
involve training with three different setups: one starting from scratch, and
the other two utilizing pretrained weights from ImageNet and SSL pretrained
weights from the Corteva dataset. The table shows nearly equal performance
using pretrained backbone weights compared to training from scratch. However,
the backbone model pretrained on the Corteva dataset with self-supervision
outperforms the ImageNet pretrained model, underscoring the significance of
using domain-specific data for pretraining backbone models. Figure 5 shows the
visualization of detection results alongside ground truth, obtained from the
Mask R-CNN utilizing our SSL pretrained Corteva backbone model.

Table 1: Performance comparison of finetuning and training from the scratch approach on rice leaf disease classification using ResNet-50 model. Training | Training From | Finetuning | Finetuning
---|---|---|---
Data | Scratch | ImageNet | SSL Pretrained
100% | 95.99 | 96.12 | 97.18
25% | 79.49 | 85.88 | 90.49
10% | 63.00 | 76.37 | 80.83
Table 2: Performance comparison of finetuning and training from scratch approach on plant village classification using ResNet-50 model. Training | Training From | Finetuning | Finetuning
---|---|---|---
Data | Scratch | ImageNet | SSL Pretrained
100% | 99.56 | 99.54 | 99.59
25% | 97.39 | 97.77 | 98.79
10% | 90.30 | 94.44 | 97.22
Table 3: Performance comparison of finetuning and training from scratch approach on Minneapple object detection dataset using Mask R-CNN with ResNet-50 backbone. Metric | Training From | Finetuning | Finetuning
---|---|---|---
| Scratch | ImageNet | SSL Pretrained
AP[.5:.95] | 0.23 | 0.22 | 0.24
AP[.5] | 0.60 | 0.59 | 0.59
AP[.75] | 0.12 | 0.11 | 0.16
Figure 5: Instance segmentation and detection result from Mask R-CNN model.
Finetuned model from Self-supervised pretrained ResNet-50 weights (left) and
ground-truth annotation (right).

Faster Model Convergence
Using pretrained models for downstream tasks accelerates model convergence,
helping the model learn faster and more efficiently. Figure 6 shows the cross-
entropy loss values plotted against training epochs for the classification
tasks on the rice leaf disease and PlantVillage datasets, comparing models
trained from scratch and SSL finetuned. The graphs indicate that SSL finetuned
models achieve faster convergence compared to models trained from scratch. We
observed comparable convergence rates and sometimes better with SSL finetuned
when contrasted with ImageNet fine-tuned models.

(a) Rice leaf disease classification

(b) PlantVillage classification

Figure 6: Training loss convergence vs epoch for classification downstream
task. The self-supervised finetuning approach leads to faster convergence
compared to training from scratch.

Outlier Detection
Outlier detection, also known as anomaly detection, refers to the process of
identifying patterns in data that deviate significantly from the norm or
expected behavior. For instance, in agricultural contexts, anomalies could
signify diseased crops, pest infestations, cloud cover, obstacles, or
environmental stressors that require attention. Leveraging the feature
representations learned by pretrained models can effectively differentiate
between normal and anomalous patterns in agricultural data. Furthermore, this
contributes to identifying data imbalance and diversifying the training data,
thereby improving the model’s capacity to generalize and make precise
predictions on unseen data.

Figure 7: Visualization of cloud cover outlier clusters in Sentinel-2
agricultural field satellite data. This visualization shows the clustering of
partial or complete cloud covers (outlier marked in red) and also different
landscapes, including green vegetation, no vegetation, water bodies, location,
etc.

We illustrate this utility by employing the Sentinel-2 dataset, discerning the
cloud coverage from agricultural field satellite imagery data. We utilized the
SSL pretrained ResNet-50 model to derive $2048$ dimensional image
representation vectors and projected them onto a 3D plane using the UMAP
dimensionality reduction algorithm. The cluster formations can be observed
from Figure 7, facilitating the identification of a minor cluster with the
cloud cover outlier images from the Sentinel-2 dataset. Consequently, we can
efficiently exclude undesirable cloud cover images from the high-altitude
field images captured by drones, flights, and satellites.

Content-based Image Retrieval
AI applications in agriculture analysis may also encounter rare and
complicated cases that are hard for normal agronomists to analyze due to their
scarcity. Content-based image retrieval (CBIR) means AI’s ability to search
for images similar to those retrieved from large agroecology databases. CBIR
is very helpful in the agriculture analysis process because it improves the
chance of making the right analysis in challenging cases by quickly finding
examples of cases that are similar to available agroecology image databases
that have such rare and complicated cases. In this case, similarity reflects
related agroecological intrinsic feature representations rather than just
visual similarities.

Figure 8: Pixel Affinity web application user interface for content-based
image retrieval. Figure 9: Block diagram of Pixel Affinity tool for content-
based image retrieval.

We have designed PixelAffinity, a web-based CBIR search tool (Figure 8), which
leverages the proposed method to deliver accurate results. We considered the
fine-tuned ResNet-50 model to create robust image representations. These
representations form an offline database, allowing for efficient searches.
When a user uploads an image, its representation is generated online processed
by a CPU, and compared to the database using cosine similarity. This vector
similarity search is performed at scale for high throughput. The closest
matches are then displayed in the search results, offering users a fast and
effective way to find similar images. This entire process can be visualized
from Figure 9.

Video Data Analysis
Video analysis can help improve productivity in agriculture from field
monitoring to harvest automation. Drones equipped with cameras survey fields,
providing data for precision planning like targeted fertilizer application.
Robots guided by video analysis automate plant phenotyping and harvesting,
minimizing waste. This translates to better decision-making, boosting
productivity and efficiency across the farm.

Figure 10: Visualization of video frames as row vs alley clusters in 3D space
based on its respective feature representations.

Our seed research facilities conduct numerous experiments and capture on-farm
video data to be processed for automated phenotyping purposes. This can be
quite compute-intensive to process all the frames in a video. We studied corn
field videos for corn phenotyping and found a real value by cutting video
frame processing by half by separating rows and alley frames using the feature
representations generated from the proposed model. Figure 10 shows
representations plotted in a 3D space by the UMAP dimensionality reduction
technique. We can observe the formation of two clusters based on rows and
alleys in the frames. For phenotyping purposes, breeders are interested in
rows with corn plants, and alley frames were discarded cutting down the time,
resources, and storage in half.

Guidance model for Image reconstruction
We observed that self-supervised training offers powerful representations from
raw raw imagery. We exploit the representations and pre-trained encoder to
guide the decoder for image reconstruction and image editing. Designing an
effective decoder presents unique challenges, and we are investigating several
promising avenues. One approach utilizes Generative Adversarial Networks
(GANs) [11], where a discerning discriminator evaluates the reconstructed
images. Another method employs a PixelCNN decoder [34], iteratively predicting
each pixel’s value while conditioned on the representation. Additionally, we
are exploring diffusion models [9], capable of synthesizing images guided by
the extracted information. To ensure reconstructed images retain high-level
visual quality, we employ perceptual loss functions like the perceptual
similarity index (PSI) or style loss. This is currently a work in progress and
we aim to unlock the power of self-supervised learning for accurate and
visually pleasing image reconstruction.

###  4.3 Discussion

In this paper, we delve into the advantages of self-supervised learning within
the agricultural domain. Our experiments demonstrate that using pretraining
techniques significantly reduces the need for manual labeling while enhancing
downstream task performance. This is evident from Table 1 and Table 2,
especially in the case of 10% of training data. Notably, pretrained models
using agricultural domain datasets outperform those trained on general vision
datasets, highlighting the importance of domain-specific knowledge for
effective representation learning. This translates to a substantial decrease
in the need for manual labeling, a critical bottleneck in traditional
supervised learning approaches. Moreover, leveraging feature representations
from pretrained models facilitates the analysis of vast raw data, such as
videos, breathing new life into previously underutilized resources, and
identifying unknown outliers in the dataset. Furthermore, these
representations prove effective in image retrieval tasks, streamlining the
process of finding similar images across diverse databases within large
agricultural organizations and fostering collaboration across teams for
training data creation and knowledge exchange.

Challenges and future work: While pretrained models offer improved performance
for downstream tasks, they still require significant computational resources
for training. Additionally, the vast spectrum of agricultural tasks, ranging
from microscopic images to satellite imagery, poses a significant challenge in
curating large datasets for pretraining purposes. In our study, we focused
solely on the ResNet-50 model; however, it is widely known that employing
deeper or more advanced architectures, such as self-attention-based vision
transformers [10], could yield even better performance. Investigating these
architectures and exploring recent training techniques and approaches such as
DINOv2 [26] and mask autoencoder [16] in the agricultural domain would be of
great interest. Choosing the right self-supervised pretext task (contrastive
learning in our case) is crucial and challenging research problem as it
affects downstream decision-making capability because pretexts rely on
exploiting the structure of the data. It is a good practice to think of the
properties we want in our representations to decide on a pretext task.
Additionally, for specific downstream tasks like detection or segmentation,
leveraging dense prediction-based self-supervised approaches [37] prove more
effective than global feature-based methods. These avenues represent promising
directions for future research in enhancing the efficacy of self-supervised
learning within agricultural contexts.

##  5 Conclusion

This work demonstrates the significant promise of self-supervised
representation learning for agricultural applications. By leveraging readily
available unlabeled data, self-supervision proves to be a powerful backbone
framework for significantly reducing manual labeling efforts for efficient
downstream learning tasks, analyzing large-scale data through clustering,
identifying anomalies, enhancing video analysis, and facilitating image search
functionality within agricultural organizations. We believe that further
research in this area, including exploring advanced architectures, identifying
ideal self-supervised pretext tasks, tailoring approaches to specific needs,
and addressing computational limitations, will unlock the full potential of
self-supervised representation learning and empower agriculture to thrive in a
data-driven future. Finally, our study outlines and encourages more
exploration of the versatile use of self-supervised methods and their
generated meaningful low-level feature representations.

## References

  * Agency [2015] The European Space Agency.  Sentinel-2: Colour vision for copernicus.  https://www.esa.int/Applications/Observing_the_Earth/Copernicus/Sentinel-2, 2015.  Accessed: February 27, 2024.
  * Aich et al. [2018] Shubhra Aich, Anique Josuttes, Ilya Ovsyannikov, Keegan Strueby, Imran Ahmed, Hema Sudhakar Duddu, Curtis Pozniak, Steve Shirtliffe, and Ian Stavness.  Deepwheat: Estimating phenotypic traits from crop images with deep learning.  In _2018 IEEE Winter conference on applications of computer vision (WACV)_ , pages 323–332. IEEE, 2018.
  * Araslanov and Roth [2021] Nikita Araslanov and Stefan Roth.  Self-supervised augmentation consistency for adapting semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15384–15394, 2021.
  * Cai et al. [2022] Enyu Cai, Zhankun Luo, Sriram Baireddy, Jiaqi Guo, Changye Yang, and Edward J Delp.  High-resolution uav image generation for sorghum panicle detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1676–1685, 2022.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(4):834–848, 2017.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  In _Advances in Neural Information Processing Systems_ , pages 22243–22255. Curran Associates, Inc., 2020.
  * Ciarfuglia et al. [2022] Thomas A Ciarfuglia, Ionut Marian Motoi, Leonardo Saraceni, and Daniele Nardi.  Pseudo-label generation for agricultural robotics applications.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1686–1694, 2022.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Güldenring and Nalpantidis [2021] Ronja Güldenring and Lazaros Nalpantidis.  Self-supervised contrastive learning on agricultural images.  _Computers and Electronics in Agriculture_ , 191:106510, 2021.
  * Hani et al. [2020] Nicolai Hani, Pravakar Roy, and Volkan Isler.  Minneapple: A benchmark dataset for apple detection and segmentation.  _IEEE Robotics and Automation Letters_ , 5(2):852–858, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Jing and Tian [2020] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(11):4037–4058, 2020.
  * Kashyap et al. [2021] Pankaj Kumar Kashyap, Sushil Kumar, Ankita Jaiswal, Mukesh Prasad, and Amir H Gandomi.  Towards precision agriculture: Iot-enabled intelligent irrigation systems using deep learning neural network.  _IEEE Sensors Journal_ , 21(16):17479–17491, 2021.
  * Kussul et al. [2017] Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov.  Deep learning classification of land cover and crop types using remote sensing data.  _IEEE Geoscience and Remote Sensing Letters_ , 14(5):778–782, 2017.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * McInnes et al. [2020] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction, 2020.
  * Mohanty et al. [2016] Sharada P Mohanty, David P Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection.  _Frontiers in plant science_ , 7:1419, 2016.
  * Nguyen et al. [2020] Thanh Tam Nguyen, Thanh Dat Hoang, Minh Tam Pham, Tuyet Trinh Vu, Thanh Hung Nguyen, Quyet-Thang Huynh, and Jun Jo.  Monitoring agriculture areas with satellite images and deep learning.  _Applied Soft Computing_ , 95:106565, 2020.
  * Ogidi et al. [2023] Franklin C Ogidi, Mark G Eramian, and Ian Stavness.  Benchmarking self-supervised contrastive learning methods for image-based plant phenotyping.  _Plant Phenomics_ , 5:0037, 2023.
  * Olsen et al. [2019] Alex Olsen, Dmitry A Konovalov, Bronson Philippa, Peter Ridd, Jake C Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, et al.  Deepweeds: A multiclass weed species image dataset for deep learning.  _Scientific reports_ , 9(1):2058, 2019.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Rahman et al. [2023] Raiyan Rahman, Christopher Indris, Tianxiao Zhang, Kaidong Li, Brian McCornack, Daniel Flippo, Ajay Sharda, and Guanghui Wang.  On the real-time semantic segmentation of aphid clusters in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6298–6305, 2023.
  * Ratnayake et al. [2021] Malika Nisal Ratnayake, Adrian G Dyer, and Alan Dorin.  Towards computer vision and deep learning facilitated pollination monitoring for agriculture.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2921–2930, 2021.
  * Ren et al. [2021] Chao Ren, Justin Dulay, Gregory Rolwes, Duke Pauli, Nadia Shakoor, and Abby Stylianou.  Multi-resolution outlier pooling for sorghum classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2931–2939, 2021.
  * Retsinas et al. [2023] George Retsinas, Niki Efthymiou, and Petros Maragos.  Mushroom segmentation and 3d pose estimation from point clouds using fully convolutional geometric features and implicit pose encoding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6263–6270, 2023.
  * Saleem et al. [2021] Muhammad Hammad Saleem, Johan Potgieter, and Khalid Mahmood Arif.  Automation in agriculture by machine and deep learning techniques: A review of recent developments.  _Precision Agriculture_ , 22:2053–2091, 2021.
  * Shakoor et al. [2017] Nadia Shakoor, Scott Lee, and Todd C Mockler.  High throughput phenotyping to accelerate crop breeding and monitoring of diseases in the field.  _Current opinion in plant biology_ , 38:184–192, 2017.
  * Sohn [2016] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2016.
  * Van den Oord et al. [2016] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  _Advances in neural information processing systems_ , 29, 2016.
  * Wang and Liu [2021] Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2495–2504, 2021.
  * Wang et al. [2022] Luhan Wang, Lihua Zheng, and Minjuan Wang.  3d point cloud instance segmentation of lettuce based on partnet.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1647–1655, 2022.
  * Wang et al. [2021] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3024–3033, 2021.
  * Wang et al. [2023] Xiaoqi Wang, Yaojun Wang, Jingbo Zhao, and Jing Niu.  Eca-convnext: A rice leaf disease identification model based on convnext.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 6235–6243, 2023.
  * Wu et al. [2020] Wenzhao Wu, Juepeng Zheng, Haohuan Fu, Weijia Li, and Le Yu.  Cross-regional oil palm tree detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 56–57, 2020.
  * Yang et al. [2020a] Guofeng Yang, Guipeng Chen, Yong He, Zhiyan Yan, Yang Guo, and Jian Ding.  Self-supervised collaborative multi-network for fine-grained visual categorization of tomato diseases.  _IEEE Access_ , 8:211912–211923, 2020a.
  * Yang et al. [2020b] Qi Yang, Liangsheng Shi, Jingye Han, Jin Yu, and Kai Huang.  A near real-time deep learning approach for detecting rice phenology based on uav images.  _Agricultural and Forest Meteorology_ , 287:107938, 2020b.
  * You et al. [2017] Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks, 2017.
  * Yu et al. [2019] Jialin Yu, Arnold W Schumann, Zhe Cao, Shaun M Sharpe, and Nathan S Boyd.  Weed detection in perennial ryegrass with deep learning convolutional neural network.  _Frontiers in plant science_ , 10:1422, 2019.
  * Zheng et al. [2019] Yang-Yang Zheng, Jian-Lei Kong, Xue-Bo Jin, Xiao-Yi Wang, Ting-Li Su, and Min Zuo.  Cropdeep: The crop vision dataset for deep-learning-based classification and detection in precision agriculture.  _Sensors_ , 19(5):1058, 2019.