∎ 500

11institutetext: Z. Zhao 22institutetext: School of Computing, Queen’s
University, Kingston, ON, Canada
22email<EMAIL_ADDRESS>33institutetext: Y. Chen 44institutetext: School of
Computing, Queen’s University, Kingston, ON, Canada
44email<EMAIL_ADDRESS>55institutetext: A.A. Bangash 66institutetext:
School of Computing, Queen’s University, Kingston, ON, Canada
66email<EMAIL_ADDRESS>77institutetext: B. Adams 88institutetext:
School of Computing, Queen’s University, Kingston, ON, Canada
88email<EMAIL_ADDRESS>99institutetext: A.E. Hassan 1010institutetext:
School of Computing, Queen’s University, Kingston, ON, Canada
1010email<EMAIL_ADDRESS>
# An Empirical Study of Challenges in Machine Learning Asset Management

Zhimin Zhao  Yihao Chen  Abdul Ali Bangash  Bram Adams  Ahmed E. Hassan

(Received: date / Accepted: date)

###### Abstract

Context: In machine learning (ML) applications, assets include not only the ML
models themselves, but also the datasets, algorithms, and deployment tools
that are essential in the development, training, and implementation of these
models. Efficient management of ML assets is critical to ensure optimal
resource utilization, consistent model performance, and a streamlined ML
development lifecycle. This practice contributes to faster iterations,
adaptability, reduced time from model development to deployment, and the
delivery of reliable and timely outputs. Objective: Despite research on ML
asset management, there is still a significant knowledge gap on operational
challenges, such as model versioning, data traceability, and collaboration
issues, faced by asset management tool users. These challenges are crucial
because they could directly impact the efficiency, reproducibility, and
overall success of machine learning projects. Our study aims to bridge this
empirical gap by analyzing user experience, feedback, and needs from Q&A
posts, shedding light on the real-world challenges they face and the solutions
they have found. Methodology: We examine $15,065$ Q&A posts from multiple
developer discussion platforms, including Stack Overflow, tool-specific
forums, and GitHub/GitLab. Using a mixed-method approach, we classify the
posts into knowledge inquiries and problem inquiries. We then apply BERTopic
to extract challenge topics and compare their prevalence. Finally, we use the
open card sorting approach to summarize solutions from solved inquiries, then
cluster them with BERTopic, and analyze the relationship between challenges
and solutions. Results: We identify $133$ distinct topics in ML asset
management-related inquiries, grouped into $16$ macro-topics, with software
environment and dependency, model deployment and service, and model creation
and training emerging as the most discussed. Additionally, we identify $79$
distinct solution topics, classified under $18$ macro-topics, with software
environment and dependency, feature and component development, and file and
directory management as the most proposed. Conclusion: This study highlights
critical areas within ML asset management that need further exploration,
particularly around prevalent macro-topics identified as pain points for ML
practitioners, emphasizing the need for collaborative efforts between
academia, industry, and the broader research community.

††journal: Empirical Software Engineering

Keywords Machine Learning · Asset Management · Mining Software Repositories ·
Stack Overflow · Topic Modeling

##  1 Introduction

From personalizing content recommendations on entertainment platforms
lapan2018deep  to enabling predictive diagnostics in healthcare miotto2018deep
, and optimizing traffic flow in urban planning nagy2018survey , machine
learning (ML) has transformed industries and our daily experiences. Despite
these advances, the practical implementation of an ML-driven solution in a
real-world scenario is fraught with challenges arising from the complexities
of dynamic environments, such as changing data distributions, evolving
requirements, and the need to constantly update and optimize models to
maintain performance and accuracy.

To help address these challenges, an intricate, yet essential task is asset
management in ML-driven solutions. Asset management encompasses organizing,
tracking, evaluating, and monitoring ML assets such as code, models, data
sets, and other artifacts, which are crucial to the effective implementation
of ML-driven solutions idowu2021asset . Although traditional software
management tools, such as Git, are designed to handle software assets such as
source code and documentat ion, these tools lack native capabilities to
effectively track ML-specific assets, such as model architectures,
hyperparameters, and data splits sculley2015hidden . Therefore, it is
imperative to have modern tools specifically designed to manage ML assets
throughout the development lifecycle of ML-driven solutions.

In recent years, there has been a significant increase in the development of
ML asset management tools. This growth can be attributed to joint efforts from
academia and industry. In academia, tools such as CANDLE wozniak2018candle ,
DLHub chard2019dlhub , MLflow zaharia2018accelerating , ModelDB
vartak2018modeldb , Pdmdims peili2018deep , and Runway tsay2018runway  have
emerged as prototypes for the management of machine learning experiments.
Meanwhile, the industry has introduced tools such as
Aim111https://aimstack.io, ClearML222https://clear.ml, DVC333https://dvc.org,
Kedro444https://kedro.org, and Polyaxon555https://polyaxon.com, among others.
These innovative tools address and mitigate many of the challenges previously
associated with traditional software management tools idowu2021asset .

However, several factors compromise the efficiency of ML asset management
tools, leading to a myriad of challenges. For example, Kumar et al.
kumar2017data  offer an in-depth analysis of the challenges associated with
data management in ML workloads. They further explore the primary techniques
and systems formulated to address these challenges. Schlegel et al.
schlegel2023management  emphasize the complexities of ensuring the
comparability, traceability, and reproducibility of ML artifacts. Idowu et al.
idowu2021asset  further stress that these challenges stem from the absence of
tool interoperability and standardized management practices in asset
management. While there is some preliminary research in this domain, the
literature lacks comprehensive analysis across tools, examination of user
challenges, and evaluation of the solutions to these challenges. This
underscores the critical need for more extensive empirical studies to gain a
holistic understanding from the tool users’ perspective, with a focus on
cross-tool analysis and solution evaluation.

In this paper, we conduct a comprehensive study of developer discussion forums
to identify the main challenges in ML asset management faced by ML
practitioners and the strategies they adopt to address these challenges. We
start by searching through multiple channels for tools that manage ML assets,
then collect Q&A posts about these tools from different developer discussion
forums. This includes popular sites, such as Stack Overflow and GitHub, as
well as forums set up by tool developers themselves (_i.e._ , tool-specific
forums). Based on the purpose of the inquiries, we sort these Q&A posts into
knowledge or problem types. Next, we use a topic modeling technique called
BERTopic666https://github.com/MaartenGr/BERTopic to extract the common topics
in these inquiries. Then we manually group them into macro-topics, which are
broad categories encompassing multiple related topics. We study the prevalence
of these macro-topics to gain insights into the common challenges faced by ML
practitioners. For inquiries marked as solved, we summarize the solutions
provided in the content of the posts. We also use BERTopic to cluster these
summaries to understand the patterns in the solutions. This analysis helps us
to map the strategies that ML practitioners use to tackle the challenges in ML
asset management. To our knowledge, this is the first study to use mixed
methods to systematically explore user challenges in adopting ML asset
management tools.

Our main findings and their implications include the following.

  * $\bullet$

We identify $133$ distinctive challenge topics in ML asset management, which
we subsequently group into $16$ macro-topics, _i.e._ , high-level groupings of
multiple interconnected topics with similar concerns. The most frequently
discussed macro-topics are software environment and dependency ($18.89\%$),
model deployment and serving ($10.59\%$), and model creation and training
($9\%$). The high prevalence indicates that these areas are of significant
interest and concern for ML practitioners. Also, $55\%$ of the tools
predominantly receive inquiries about the software environment and dependency
topics. Furthermore, we observe different prevalent topics in each tool of the
asset management ecosystem.

  * $\bullet$

We identify $79$ different solution topics in ML asset management, which we
subsequently group into $18$ macro-topics. Among these, the most commonly
proposed solutions are related to the software environment and dependencies
($23.31\%$), the development of features and components ($15.35\%$), and the
file and directory ($9.64\%$). For knowledge inquiries, feature and component
development methods are mainly used to solve challenges in source code
management. Similarly, issues in experiment management are primarily solved
using software environment and dependency-related solutions. On the other
hand, for problem inquiries, a significant portion ($62.5\%$) of problems are
resolved using software environment and dependency-related solutions.
Additionally, feature and component development solutions are often applied to
address challenges related to logging and monitoring. Moreover, problem
inquiries tend to be solved more effectively using skills or knowledge from
different domains, as opposed to knowledge inquiries. This observation
highlights the necessity to employ varied approaches when addressing knowledge
inquiries compared to problem ones.

  * $\bullet$

We find that Stack Overflow is the primary forum for practitioners seeking
help with asset management, accounting for $48.82\%$ of all such inquiries.
Tool-specific forums are next, contributing $34.19\%$, while repository-
specific forums have the fewest, at only $17.16\%$. Of the tools examined,
$25\%$ witness most user inquiries on tool-specific forums, $20\%$ on
repository-specific forums, and $35\%$ on Stack Overflow. This trend implies a
potential area of exploration for tool maintainers to enhance user engagement
on their respective forums. Additionally, inquiries related to software
environment and dependency are the most prevalent in both Stack Overflow,
GitHub, and tool-specific forums. This suggests that regardless of where
practitioners seek help, managing software environments and dependencies
remains a top challenge in ML asset management. Furthermore, the frequency
distribution of Q&A posts varies significantly between different discussion
forums. Developers can leverage these insights to customize content and
discussions to align with the distinct characteristics and preferences of the
audience in each forum.

##  2 Background and Related Work

In this section, we provide a comprehensive overview of the role and
challenges of asset management in the ML development lifecycle.

###  2.1 Machine learning development lifecycle

The ML development lifecycle, as shown in Figure 1, is a comprehensive process
that encapsulates various stages, each integral to the development and
deployment of ML-driven solutions. This lifecycle, as described by Amershi et
al. amershi2019software , is not a strictly linear progression, but rather a
dynamic flow with potential feedback loops. Each stage may require iterative
refinement if certain criteria are not met, necessitating a review of previous
stages.

  * $\bullet$

Problem Understanding stage is about understanding the problem at hand. It
involves identifying potential features, understanding their significance in
the context of existing or upcoming ML-driven solutions, and selecting an
appropriate model architecture tailored to the problem.

  * $\bullet$

Data Collection stage involves sourcing, integrating, and augmenting datasets,
often using generic data for initial model training and specialized data for
transfer learning.

  * $\bullet$

Data Preprocessing stage involves cleaning, labeling, and transforming the
datasets. It also emphasizes the extraction and selection of the most
informative features that serve as input to the ML models.

  * $\bullet$

Model Development stage involves training, evaluating, and testing models on
selected features. The performance of these models is typically measured using
predefined metrics in validation and test datasets.

  * $\bullet$

Model Deployment stage involves integrating the trained model into real-world
applications, allowing it to make inferences based on new data.

  * $\bullet$

Model Monitoring stage is about tracking how the ML model performs over time.
If any degradations in performance is observed, the current model might be re-
trained with newly collected data.

Figure 1: Machine learning development lifecyle and asset management.

###  2.2 Machine learning asset management

In the context of this lifecycle, ML assets emerge as the foundational
elements employed throughout each stage. These assets, which encompass
datasets, models, hyperparameters, configurations, and more, are significant
in ensuring the success of ML operations (MLOps). This pivotal role
necessitates the adoption of specialized machine learning asset management
tools, such as MLflow. Whether accessed through intuitive web dashboards or
command-line interfaces, these tools support the automation of operational
tasks such as tracking, exploring, retrieving, collaborating on, archiving,
and discarding ML assets idowu2021asset .

However, managing these assets is fraught with a series of challenges.
Schelter et al. schelter2015challenges  categorize the challenges of managing
ML models into three core categories: conceptual, data management, and
engineering. They highlight issues such as model definition, multiple input
handling, metric selection, data quality, and system compatibility in
programming languages and hardware. Furthermore, Polyzotis et al.
polyzotis2018data  identify three main areas of focus in data lifecycle
management for machine learning: understanding, validation/cleaning, and
preparation. Some of the most important challenges include dealing with
missing or inconsistent data, managing data from heterogeneous sources,
ensuring data privacy and security, and keeping up with the latest
developments in the ML community. Further expanding on this, Munappy et al.
munappy2022data  identify $20$ data management challenges encountered by deep
learning practitioners, categorizing them into four groups based on the
presence of solutions. These challenges cover issues ranging from data search,
quality, and privacy to data versioning, labeling, and scalability.

In response to these intricate challenges, both research communities and
companies have introduced specialized tools for asset management.
Academically, tools such as ModelHub miao2017modelhub , DLHub chard2019dlhub ,
and Pdmdims peili2018deep  have been introduced for model management. For
experiment management, prototypes include Codalab pavao2022codalab , MLflow
zaharia2018accelerating , Runway tsay2018runway , and ModelDB
vartak2018modeldb . Furthermore, for comprehensive lifecycle management,
solutions such as CANDLE wozniak2018candle , Ease.ML aguilar2021ease , VeML
le2023veml , Scanflow bravo2022scanflow , TITAN benitez2021titan , Stratum
bhattacharjee2019stratum , ProvDB miao2017provdb , and ModelOps
hummer2019modelops  have been proposed. In the industrial domain, commercial
tools such as Aim777https://aimstack.io, ClearML888https://clear.ml,
cnvrg.io999https://cnvrg.io, Comet101010https://comet.com,
Domino111111https://domino.ai, DVC121212https://dvc.org,
Kedro131313https://kedro.org, Polyaxon141414https://polyaxon.com, and
Weights&Biases151515https://wandb.ai have been developed to address these
challenges.

Despite these advances in tool support, managing ML assets using these tools
still faces important challenges. Kumar et al. kumar2017data  provide a
comprehensive review of the data management-related challenges that arise in
ML workloads and analyze the key techniques and systems that have been
developed to address these challenges. Moreover, Schlegel et al.
schlegel2023management  highlight the difficulties in ensuring the
comparability, traceability, and reproducibility of artifacts, such as models,
data, logs, and metadata, throughout the ML development lifecycle, even with
specialized tools. Furthermore, Melin melin2023tackling  underscores the need
for end-to-end versioning in the ML development lifecycle, highlights the
complexities of lineage tracing due to framework integration issues, and
advocates for clear guidelines in managing ML artifacts. In addition, Idowu et
al. idowu2021asset  highlight challenges such as limited interoperability
between tools, tight library coupling, and the overhead of the necessary code
instrumentation. They also point out the absence of standardized management
methods for ML assets, forcing users into ad hoc solutions that hinder
efficient model development.

Compared to previous studies, our research explores the experience and
feedback of tool users related to machine learning asset management, shedding
light on the unexplored challenges and their practical solutions.

##  3 Methodology

In this section, we first outline our research goals and questions, followed
by a detailed explanation of our study design.

###  3.1 Goal and Research Questions

Our study aims to investigate the challenges associated with the management of
ML assets. We aim to understand their prevalence and proposed solutions across
developer discussion forums. Our research is specifically targeted at a
diverse group of stakeholders, including data scientists, ML engineers,
infrastructure architects, and other roles in the ML development lifecycle,
such as operations managers. Collectively, we describe this group as “ML
practitioners”. The rationale behind focusing on these professionals is rooted
in their profound engagement and proficiency in the ML development lifecycle.
By understanding their perspectives and experiences, our research seeks to
illuminate the pathways for more efficient and streamlined management of ML
assets.

To achieve our stated goal, we investigate three research questions.

  * $\bullet$

RQ1: What topics are most frequently discussed related to machine learning
asset management? In RQ1, we identify common (macro-)topics in Q&A posts on ML
asset management. We then analyze and compare the frequency of different types
of inquiry and macro-topics to determine the most common ones. Our goal is to
gain a clear understanding of the key challenges that ML practitioners face in
asset management.

  * $\bullet$

RQ2: What topics of solutions exist for the challenges related to machine
learning asset management? In RQ2, we identify common (macro-)topics found in
solutions that address the challenges of ML asset management. We explore the
interconnections between challenges and their solutions to uncover discernible
trends or correlations in different domains. Our goal is to construct a
comprehensive guide for ML practitioners, helping them navigate and overcome
these challenges effectively by aligning them with proven solutions.

  * $\bullet$

RQ3: What are the commonalities and differences between developer forums in
their discussion related to machine learning asset management? In RQ3, we
investigate what is consistent and what varies between the different forums in
terms of asset management discussion. Our goal is to uncover the unique
strengths and patterns of each forum, helping ML practitioners navigate and
utilize these forums effectively for their specific needs and interests.

###  3.2 Study Design

We present the workflow of our study in Figure 2. Our study begins with the
collection of Q&A posts from developer discussion forums. Once collected, we
classify these posts into two types: problem and knowledge. Following this
classification, we preprocess the posts and employ the BERTopic technique to
extract their topics. We then manually group them into broader categories,
termed “macro-topics”. Furthermore, we evaluate and compare the frequency of
these macro-topics based on the type of inquiry and the specific discussion
forums from which they are sourced. This evaluation helps to address our
research questions, specifically RQ1 and RQ3. To wrap up our study, we
manually label the solutions found in the posts and extract the topics
associated with these solutions. This step sets the stage for the macro-topic
mapping between challenges and solutions in RQ2.

Figure 2: Study workflow to analyze Q&A posts related to ML asset management
from developer discussion forums.

####  3.2.1 Phase 1: Data collection

In this phase, our goal is to collect Q&A posts related to ML asset management
from multiple developer discussion forums. To achieve this, we first start a
multivocal literature review, aiming to compile an exhaustive list of tools
that are specifically designed to manage ML assets. From this compilation, we
derive a curated set of tool-specific keywords. Using these identified
keywords, we then extract relevant Q&A posts from multiple developer
discussion forums.

Step 1.1: Collect ML asset management-related tools.

We begin by reviewing a range of literature, focusing on references to ML
asset management tools. We formulate our initial search queries by refining
the strategies used for literature retrieval, as introduced in the
foundational work on asset management by Idowu et al. idowu2021asset . To
expand our search, we include terms such as “MLOps tools” and “machine
learning experiment management tools”, among others related to asset
management. Using these terms, we search Github Awesome
Lists161616https://github.com/topics/awesome and Google
Scholar171717https://scholar.google.com to find relevant literature, with the
filtered results listed in Table 9 (Appendix B). We follow a progressive stop
method: If we do not find any relevant articles on three consecutive search
pages, we conclude our search. Note that we ignore the literature that is
similar to any work identified in the previous search. Furthermore, we employ
a backward snowballing process jalali2012systematic  to examine the references
cited in the literature we review, identifying additional relevant works. This
approach ensures that we consider all pertinent information and insights
available in the existing body of knowledge.

Then we apply the following exclusion criteria to the tools that are
referenced in the collected literature.

  * $\bullet$

The tool is either restricted for in-house use, such as FBLearner
Flow181818https://venturebeat.com/dev/facebook-details-its-company-wide-
machine-learning-platform-fblearner-flow, or no longer available, such as Dot
Science191919https://www.datanami.com/this-just-in/dotscience-is-shutting-
down.

  * $\bullet$

The tool is hosted in an open-source repository, but it has earned less than
$100$ stars. Although the number of stars in a repository may not be the only
indicator of its quality or utility, it is often seen as a measure of trust
and usage in the community borges2018s .

  * $\bullet$

The tool appears to be poorly maintained or not adequately documented. For
example, its open-source repository has not received updates for more than a
year. This metric has been used in the previous literature as an indicator to
identify dormant open-source projects coelho2017modern ;
izquierdo2017empirical ; mens2014survivability ; khondhu2013all .

  * $\bullet$

The tool is specifically designed for a particular type of ML asset, such as a
dataset or model, such as Dolt202020https://github.com/dolthub/dolt.

  * $\bullet$

The tool is a multipurpose asset management tool that is not specifically
tailored for ML applications, such as
Pachyderm212121https://github.com/pachyderm/pachyderm.

  * $\bullet$

The tool is in the research phase and has not yet been discussed in any
developer forum or used in open-source repositories, such as VeML le2023veml .

We review more than $500$ tools and narrow them down to $25$ tools after
applying the above exclusion criteria. We present the status (open source or
proprietary), launch date, and website address of each tool in Table 10
(Appendix B). In this table, the status indicates whether they are open-source
or proprietary tools; the launch date indicates the date when the tool is
accessible to the public; and the website shows the links to these tools. We
find that 52% of the tools (13 of 25) feature open source repositories,
indicating a significant presence of open source options for machine learning
asset management.

Step 1.2: Scrape the tool-related posts from the developer discussion forums

Having the list of tools, we scrape Q&A posts from various discussion forums
that mention the names of tools. Our focus is on forums predominantly used by
developers, specifically those that cater to ML practitioners. In our study,
we have chosen to analyze posts from three distinct types of forum: general
forums including Stack Overflow, tool-specific forums (officially hosted by
the tool providers for discussions related to their tools), such as the MLflow
Google Group, and repository-specific forums (dedicated to discussions around
specific repositories), such as GitHub Issues. In the following sections, we
detail our methodology for collecting posts from each of these forums.

General forums

For general developer discussion forums, we exclusively concentrate on Stack
Overflow. Stack Overflow222222https://stackoverflow.com is a renowned platform
where people, ranging from beginners to experienced professionals, engage in
asking and answering questions about programming and software development.
Building on this premise, it is worth noting that, instead of explicitly
discussing ML asset management concepts, ML practitioners often allude to
these concepts by referencing ML asset management tools in their posts. This
can be either through direct mentions of the tool’s name or, more commonly,
via tags. Specifically, tags play a pivotal role in providing context to
discussions about ML asset management tools.

Given their significance, we prioritize tags as our primary filtering
criterion for the tools under investigation. Thus, we employ an iterative and
heuristic process to curate a relevant set of tags. For example, entering the
keyword “[*sagemaker*]” into the Stack Overflow search bar retrieves several
tags in the search results, such as amazon-sagemaker, amazon-sagemaker-studio,
and amazon-sagemaker-neo, among others. In particular, some retrieved tags may
not be directly related to asset management. Therefore, we meticulously review
each tag in conjunction with its associated discussions to determine its
relevance. Based on this evaluation, we compile a comprehensive list of tags
that accurately represent the tools under study, as illustrated in Table 11
(Appendix B). Using these tags, we formulate an SQL query, detailed in our
replication package, to extract data from the Stack Exchange Data
Explorer232323https://data.stackexchange.com. This query enables us to
download $6,755$ unique Stack Overflow posts (up to $26^{th}ofJuly,2023$),
ensuring a comprehensive and relevant dataset for our analysis.

Tool-specific forums

In addition to general developer discussion forums, we check the official
website of each curated tool for information about any public discussion forum
hosted by the tool developers. We provide the website addresses of the tool-
specific discussion forums in Table 12 (Appendix B). It is important to note
that we have chosen to exclude official chat platforms such as Discord and
Slack. This decision is based on the fact that chat data is typically held
privately by tool organizations, and their policies prevent us from collecting
this data for any public use. Since our primary focus is on Q&A posts in these
discussion forums, we have opted to exclude content, such as event
announcements242424https://discuss.dvc.org/c/blog-discussions/5, which do not
align with the objectives of our study. On the other hand, forums such as the
MLflow Google Group do not differentiate between Q&A posts and other types of
content, such as product announcements. In these scenarios, we employ a manual
filtering mechanism during our closed card sorting process, which is
elaborated on in Step 2.1. Following this methodology, we have identified
$4,962$ unique posts on tool-specific forums (up to $26^{th}July,2023$).

Repository-specific forums

For repository-specific forums, we consider code collaboration platforms, such
as GitHub252525https://github.com/features/issues,
GitLab262626https://docs.gitlab.com/ee/user/project/issues and
Bitbucket272727https://support.atlassian.com/bitbucket-cloud/docs/understand-
bitbucket-issues. These platforms offer developers the opportunity to raise
inquiries through repository issue channels. Although the discussion within
these channels is tailored toward specific projects, they frequently
underscore recurring challenges and community-proposed solutions related to ML
asset management. Furthermore, GitHub provides an optional feature called
“GitHub Discussions” for repositories, specifically catering to broader
communication needs, such as Q&A
sessions282828https://github.com/features/discussions. We leverage these
channels to extract insights relevant to our study.

First, we collect all repositories that have a dependency on the curated tools
in their codebase. We observe that certain tools, such as DVC, possess GitHub
dependency graphs. These graphs are essentially manifests detailing the
ecosystems and packages upon which the tool relies. For such tools, we extract
the necessary information from dependency repositories using an open source
scraping tool292929https://pypi.org/project/github-dependents-info. For tools
that lack dependency graphs, we rely on “usage patterns” to identify their
presence in the repository codebase. A usage pattern is defined as a regular
expression that represents a specific code snippet that indicates the use of a
curated tool, such as importing a specific library. Our curated usage patterns
are found in Table 13 (Appendix B). We derive these patterns by thoroughly
examining the official documentation of each curated tool.

Then we resort to Sourcegraph to collect repository information that matches
the usage pattern. Sourcegraph303030https://sourcegraph.com is a code search
and navigation tool, facilitating developers to search, navigate, and
understand their codebases on major code collaboration platforms such as
GitHub, GitLab, and BitBucket. From this search, we have identified $37,782$
repositories on GitHub and $10$ on GitLab. Subsequently, we extracted issue
posts from these repositories using the respective platform APIs. During this
process, we have encountered some inaccessible repositories that return HTTP
error status codes. These errors signify either a policy violation, as
indicated by a $451$ error, or the nonexistence of the repository, represented
by a $404$ error. Of the repositories that we can access, we download
$437,652$ unique issues from GitHub and $7,675$ from GitLab (up to
$26^{th}ofJuly,2023$).

To collect posts from the GitHub Discussions, we reuse the GitHub repositories
during the aforementioned repository collection. We then narrow our focus to
the “Q&A” channels within each discussion forum, such as the one in Apache
Airflow313131https://github.com/apache/airflow/discussions/categories/q-a,
since only these channels contain Q&A posts, distinguishing them from feature
requests or polls. As a result, our dataset ends up with $6,912$ unique GitHub
Discussions posts in repository-specific forums.

Step 1.3: Filter posts that are irrelevant to our study

In this section, we filter out posts that are not relevant to our research
objectives. For posts originating from Stack Overflow and tool-specific
forums, we would manually filter them during closed card sorting detailed in
Step 2.1.

For GitHub Discussions posts, we systematically curate a list of keywords
pertinent to the tools being examined. First, we extract tool-specific
keywords from the discussions heuristically. Subsequently, we assess the
relevance of these keywords by searching for them within the posts to
determine their effectiveness in identifying discussions pertinent to each
tool. For example, our analysis of discussions related to Azure Machine
Learning revealed that practitioners frequently use terms, such as “aml”,
“azure machine learning”, “azure ml”, “azure-ml”, and “azureml”. Table 14
(Appendix B) provides a comprehensive list of keywords identified for each
tool evaluated. We then use these keywords to filter and retain tool-relevant
posts, selecting any post whose title contained at least one of these
keywords. Using this methodology, we identify $43$ relevant posts for further
analysis from a total of $6,912$.

When analyzing issues on GitHub or GitLab, our initial step involves
scrutinizing the 644 unique labels of all collected repositories. From our
examination, we find that only four keywords – “bug”, “crash”, “error” and
“invalid” – indicate a user inquiry-related issue. As a result, we exclude
issues that do not contain these keywords in their labels, while keeping those
with empty labels for additional review. Next, we further narrow our selection
to issues whose titles contain any of the keywords outlined in Table 14
(Appendix B). After applying these filters, we are left with $3,332$ issues
from the initial $437,652$ on GitHub and $3$ issues from the $7,675$ on
GitLab. Following this, we select a random sample of $385$ posts from the
refined post collection to achieve a confidence level of $95\%$ with a margin
of error of $5\%$. Upon reviewing the content of these posts, we confirm that
they are indeed discussing the specified tools, which validates our choice of
keywords.

####  3.2.2 Phase 2: Inquiry categorization

In this phase, our goal is two-fold in terms of inquiry categorization. First,
we analyze the types of inquiry to understand their inherent characteristics.
Second, we identify the common topics that emerge from these inquiries. To
achieve this, we begin with closed card sorting wood2008card  to categorize
the types of inquiry from the collected posts. After categorizing, we
preprocess the content of these posts and employ an appropriate topic modeling
method. We then fine-tune the models to determine the optimal hyperparameters.
Once the topics are identified, we manually group them into macro-topics. To
address RQ1, we have evaluated the prevalence of each (macro-)topic, along
with the type of inquiry. For RQ3, we similarly analyze prevalence patterns
across discussion forums, drawing parallels with the methods used for RQ1.

Step 2.1: Closed card sorting on inquiries

In the Q&A context, inquiries primarily fall into two categories based on user
intent: “knowledge inquiry” and “problem inquiry”, as illustrated in Figures 3
and Figure 4, respectively. Knowledge inquiry aims to understand concepts such
as tool usage, technology comparison, and best practices, while problem
inquiry aims to address specific issues such as bugs, errors, and performance
degradation. This classification consolidates previous categorizations of
“conceptual”, “knowledge-based”, and “howto” inquiries, as proposed by Chen et
al. chen2023practitioners  and Treude et al. treude2011programmers , under the
umbrella of knowledge inquiries. For instance, a question such as “What is the
correct way to fine-tune a model?” is labeled as a “conceptual” inquiry by
Treude et al. treude2011programmers , whereas “How to fine-tune a model” is
considered a “howto” inquiry by Chen et al. chen2023practitioners . Given the
minor difference in verbal semantics, distinguishing between them in our
research proves challenging and leads us to combine them into the knowledge
inquiry category for clarity. This grouping reduces ambiguity among these
types of inquiry, offering a clearer understanding of the common types of
inquiry in discussion forums.

Figure 3: A solved knowledge inquiry from Weights&Biases discussion forum.

Figure 4: A closed problem inquiry in GitHub Issues.

In addition to the problem and knowledge inquiries, posts that are non-
inquiries323232https://github.com/awslabs/amazon-neptune-tools/issues/38, are
mistagged333333https://stackoverflow.com/questions/63844663, or are
irrelevant343434https://www.googlecloudcommunity.com/gc/AI-ML/What-you-think-
about-CHATGPT/m-p/506958 are labeled as “not applicable” (NA). Examples of
non-inquiry posts encompass feature requests, event announcements, and user
feedback surveys, to name a few.

The first and second authors opt to perform closed card sorting
collaboratively, focusing on the three aforementioned types of inquiry. Closed
card sorting is a method in which participants sort items into pre-defined
categories on the collected posts wood2008card . From the total of $15,095$
posts, we randomly select $385$, providing a confidence level of $95\%$ and a
margin of error of $5\%$. This sample size ensures an adequate representation
of the general data hastie2009elements . The inter-rater reliability of this
process, measured by Cohen’s kappa mchugh2012interrater , is approximately
$0.932$. This high value reflects a strong agreement between the authors,
underscoring the consistency and reliability of our categorization process. In
cases of disagreement, the authors engage in discussions to reconcile their
perspectives. After filtering out posts with “NA” labels, the resulting
dataset comprises $6,749$ Stack Overflow posts, $4,774$ tool-specific posts,
and $2,403$ repository-specific posts.

Step 2.2:Extract post content for topic modeling

After categorizing the types of inquiry, we explore the topics of these
inquiries using topic modeling, as described in detail in Step 2.4. To train a
robust topic model, we opt for three distinct ways to extract content from the
posts to achieve comprehensive coverage and the most accurate and relevant
results.

  1. 1.

We extract the titles of the posts.

  2. 2.

We extract the titles of the posts along with their bodies.

  3. 3.

We generate a refined title for each post using
GPT-4353535https://platform.openai.com/docs/models/gpt-4, a large multimodal
language model created by OpenAI. The parameter settings of GPT-4 are detailed
in Table 15 (Appendix B). Our methodology involves a rigorous and iterative
fine-tuning process of the input prompts, which includes approximately $20$
iterations. In every iteration, we meticulously adjust the input prompts to
explore possible improvements in the coherence score associated with the
identified topics. Subsequently, we conduct a manual assessment of the
contextual relevance of each generated title. This iterative fine-tuning
continues until we observe no further improvement in coherence scores,
signifying the achievement of optimal prompt refinement. Consequently, the
final optimized prompt, including the original content of the post, is sent to
GPT-4 to generate the most coherent and contextually relevant refinement of
the title of a given post.

> Refine the title of the following post to make it short and clear in simple
> English. \n Title: [title text here] \n Body: [body text here] \n Refined
> Title:

We simplify and edit the content of $153$ posts that are too long to fit in
GPT-4’s $8,000$ token limit before feeding it to the model. This process
involves removing verbose and non-essential sections, such as execution
traces, to ensure that the text remains within the token constraints.

We opt to summarize the posts using GPT-4 due to observed inconsistencies in
the quality of the titles and content of the posts. On the one hand, this
decision comes from two main observations. First, the titles of the
posts363636https://github.com/fastai/fastai/issues/3085 can be vague, reducing
the efficiency of our topic modeling efforts. This is a significant issue, as
unclear titles can obscure the post’s main subject, leading to less accurate
topic classification. Second, the performance of topic modeling algorithms can
suffer from repetition of words within posts barde2017overview . Specifically,
excessive use of non-essential repeated words can compromise the algorithm’s
ability to accurately identify the core topics of discussion. Together, these
factors necessitate a more sophisticated approach to post summarization to
enhance our topic modeling process. On the other hand, recent research
suggests that GPT-4 serves as a reliable and consistent substitute for human
labeling in summarization tasks gilardi2023chatgpt ; cheng2023gpt . In
particular, a recent study shows that GPT-4 achieves a Spearman correlation of
$0.514$ with human labelers in summarization tasks, surpassing all previous
methods by a significant margin liu2023gpteval .

For evaluation purposes, we randomly sample $385$ posts from a total of
$15,065$, with a confidence level of $95\%$ and a margin of error of $5\%$.
After comparing these refined titles with the originals, we agree that the
GPT-4 generated versions match or surpass the original titles in brevity,
readability, and precision. To illustrate, consider the previous Fastai issue
#3085: the refined title, “Broken Documentation Page for Neptune Callback”, is
notably more descriptive than its original, _i.e._ ,
“https://docs.fast.ai/callback.neptune.html broken”.

Step 2.3: Preprocess post content with NLP techniques

After extracting or generating content using the three aforementioned methods,
we preprocess it using basic natural language processing (NLP) techniques.
This preprocessing is crucial for topic modeling, as it ensures that the data
are clean and ready for accurate and meaningful results. The NLP preprocessing
encompasses the following steps:

  1. 1.

Remove any escape character, punctuation mark, or numeric value.

  2. 2.

Remove any tool-specific keyword based on Table 14 (Appendix B).

  3. 3.

Keep nouns and verbs only.

  4. 4.

Remove any stop word. In our study, we categorize stop words into three
distinct types.

    1. (a)

Everyday words in English, _e.g._ , “you”, “like”.

    2. (b)

Discussion-specific words. Discussion-specific words are words that frequently
occur in Q&A posts, but have minimal relevance to the setting of asset
management topics, _e.g._ , “discuss”, “post”, “difference”. In particular, we
exclude error-related terms, such as “error”, “bug”, and “crash”, which occur
significantly within problem inquiries in our research, to enhance the clarity
and interpretability of generated topics.

    3. (c)

Task-specific words. Task-specific words are words that are closely associated
with ML tasks, but have limited relevance when it comes to the discussion of
asset management. Such terms include “regression”, “forecasting”,
“segmentation” and more. By filtering out task-specific stop words, we improve
the relevance and clarity of our topics related to asset management.

We evaluate each stop word by checking the coherence score and reviewing the
top 10 keywords for each topic. For a detailed list of each type of stop
words, we refer to our replication packages replication_package .

Following these steps, the preprocessed text is ready for input into BERTopic
for hyperparameter tuning.

Step 2.4:Perform topic modeling

We use BERTopic373737https://github.com/MaartenGr/BERTopic to cluster topics
from preprocessed content. Previous literature indicates that BERTopic
consistently outperforms its competitors in various topic modeling tasks
grootendorst2022bertopic  and is frequently used in empirical software
engineering research diamantopoulos2023semantically ; gu2023self ; tao2023code
. BERT embeddings are adept at capturing semantic nuances, often outperforming
sparse, high-dimensional bag-of-words, or n-gram models. Unlike LDA, which
relies on word count, BERTopic captures the semantics of words beyond their
frequency. For example, the function “sort” might be referred to as “arrange”
or “classify” depending on the programming language or the preference of the
developers. Thus, BERTopic has the unique capability to group words that may
convey similar meanings but are phrased differently.

Since the default hyperparameters provided by BERTopic might not guarantee
optimal performance, our goal is to define a comprehensive hyperparameter
space for fine-tuning. This search space is initialized by a review of the
official BERTopic documentation and insights from its GitHub
issues383838https://github.com/MaartenGr/BERTopic/issues. As we navigate the
search, we pay particular attention to variations in the C_V score, a metric
commonly used to measure the quality of topics produced by topic models
syed2017full . If we observe that the most significant improvements occur at
the boundaries of the current search range, we expand the hyperparameter
search space heuristically. Conversely, if the model’s performance begins to
deteriorate, we cease expansion to ensure its stability and reliability. When
certain hyperparameters do not lead to noticeable changes in performance, we
regard them as irrelevant and exclude them from our search.

We show the range of hyperparameters used to fine-tune BERTopic for modeling
challenges and solutions, respectively, in Table 16 (Appendix B). The optimal
hyperparameters are marked in green. Although coherence C_V is our primary
metric, we also consider other metrics when the coherence score of generated
topic models is comparable. For instance, we evaluate the number of topics and
outliers produced. Outliers refer to posts that do not fall into any of the
created topics, suggesting weak semantic connections. Furthermore, we manually
assess $5\%$ of the clustered posts for each topic to ensure their relevance
and consistency. If the generated topics are difficult to interpret or have
significant overlap, we would review and adjust the hyperparameters or go back
to earlier steps. This could involve changing our preprocessing methods,
including reviewing the list of stop words.

Upon identifying the most suitable topic model, we assign the most probable
topic to each post. For outliers, we integrate them with their closest topic
using the HDBSCAN technique. HDBSCAN, as described by McInnes et al.
mcinnes2017hdbscan , is a soft clustering algorithm that identifies clusters
of varying densities in the data without the need to predefine the number of
clusters.

Step 2.5:Aggregate into macro-topics

Upon reviewing the initial set of $133$ topics, we find them too numerous for
an effective comparison. In addition, many of these topics exhibit strong
interrelations. To streamline our analysis, we aggregate these topics into
macro-topics. These macro-topics, which can be thought of as “categories of
topics”, represent high-level groupings of multiple interconnected topics with
similar concerns. Previous studies chen2020comprehensive ;
bagherzadeh2019going ; Ahmed2018WhatDC  have employed the term “category” to
denote such clusters, but we choose to use the term “macro-topics” for this
aggregation. In the qualitative coding phase, we apply the “negotiated
agreement” campbell2013coding  technique to improve the reliability of the
coding. This technique involves collaborative coding by the authors to reach a
consensus and refine the definitions of the codes, as seen in previous studies
such as those by Chen et al. chen2023practitioners . Our consensus on coding
is inspired by key ideas from seminal studies within the MLOps domain. We have
been particularly influenced by the works of Idowu et al. idowu2021asset ,
Amershi et al. amershi2019software , and Schlegel et al.
schlegel2023management .

Step 2.6:Collect prevalence metrics

Next, we evaluate the macro-topics based on the “prevalence” metrics, a common
practice in previous studies openja2020analysis ; Ahmed2018WhatDC ;
rosen2016mobile ; venkatesh2016client ; Yang2016WhatSQ . Prevalence measures
the frequency with which a topic emerges in our analyzed posts, expressed as a
percentage of the entire dataset.

####  3.2.3 Phase 3: Solution categorization

In this phase, our objective is to understand and categorize the strategies
that ML practitioners use to address inquiries, aiming to find prevalent
solutions and explore their interconnection. We start by applying open card
sorting to solved posts to summarize the solutions presented in them. With the
labels generated from this process, we then use BERTopic for topic modeling to
further categorize these solutions. After extracting the topics, we group them
into macro-topics for a more holistic view. Then we investigate how frequently
each solution macro-topic appears. Lastly, we analyze the correlation between
the identified challenges and their corresponding solutions.

Step 3.1: Open card sorting on solutions

It is crucial to understand how inquiries are addressed and concluded in
different developer discussion forums. Platforms such as Stack Overflow,
GitHub Discussions, and many tool-specific forums, such as Microsoft
Q&A393939https://learn.microsoft.com/en-us/answers/, allow users to mark any
answer to an inquiry as “accepted”. This acceptance is a clear indication that
an inquiry is “solved”, which generally encapsulates a “solution” that can
effectively address the initial concerns. Figure 3 shows a knowledge
inquiry404040https://community.wandb.ai/t/axis-scales/2892 in the
Weights&Biases discussion forum looking for best practices to visualize
datasets. This inquiry is addressed by a suggestion to create reports using
different panels.

Unlike traditional Q&A forums, some repository-specific discussion forums,
such as GitHub Issues, do not employ an answer acceptance mechanism. Instead,
they adopt a “closure” mechanism, indicating that the inquiry is solved, no
longer considered active, or requires attention. Figure 4 shows a closed
problem inquiry414141https://github.com/DagsHub/fds/issues/39 following a
merge request.

Following the above criteria, the first and second authors opt to perform open
card sorting on the solved posts to categorize the solutions provided for user
inquiries. Open card sorting is a method in which participants receive a set
of cards, each containing a single piece of content or data, and are asked to
group these cards into categories in a way that makes sense for them, without
any predefined categories wood2008card . We choose open card sorting due to
its ability to handle the versatility of solutions related to asset
management. We randomly sample $385$ posts from $4,684$ solved posts, aiming
for a $95\%$ confidence level and a $5\%$ margin of error. Our labels show a
high level of agreement, confirmed by a Cohen’s kappa of approximately
$0.852$, indicating strong consistency in categorization. In cases of
disagreement, the authors engage in discussions to reconcile their
perspectives.

To label posts, we use $2-5$ words that start with a verb and end with a noun.
Although adjectives are selectively added for clarity, adverbs are used
sparingly. This length is chosen to meet both the contextual demands of
BERTopic for informative context and to resonate with human labeling practices
for consistency. Stop words, punctuation, and non-UTF8 characters are excluded
to ensure that only essential information is retained, avoiding any
preprocessing that could result in the loss of important data. Following these
criteria, the first author independently labels the remaining solved posts.
Figure 3 demonstrates a solution labeled “create report”.

During open card sorting, we notice that solved posts do not always have a
satisfactory solution. We identify three types of abnormally solved inquiries,
which we subsequently exclude from our analysis. We give the definition for
each type of inquiry below.

  * $\bullet$

Non-inquiry: This type refers to posts that seem to raise concerns, but upon
closer inspection do not contain actual
problems424242https://stackoverflow.com/questions/56046428.

  * $\bullet$

Intermittent inquiry: In this category, the issues mentioned are inconsistent
and cannot be replicated
regularly434343https://stackoverflow.com/questions/72641789.

  * $\bullet$

Unsolved inquiry: These are inquiries that are marked as accepted or closed
but still have unsolved
issues444444https://stackoverflow.com/questions/65884046.

Step 3.2:Perform topic modeling

After the open card sorting process, we input our labels into BERTopic for
topic modeling. Following a procedure similar to Step 2.4, we identify the
optimal hyperparameters by optimizing the coherence score, minimizing the
number of outliers, and reducing the overlap of topics.

If the BERTopic performance falls short of our expectations, we undertake a
manual adjustment of our labels. For example, in our initial setup, we labeled
solutions related to Docker images with the term “image”. However, we later
realized that this term also encompassed solutions for “image editing”,
leading to incorrect grouping. To address this, we revised the label for
Docker image solutions to “docker-image”, while keeping the label “image” for
solutions concerning image editing. Such refinements are essential to prevent
future misclustering and ensure accurate categorization.

Similar to Step 2.4, once we identify the optimal topic model, we assign the
most probable topic to each solved post using the HDBSCAN technique.

Step 3.3:Aggregate into macro-topics

Given the observation that many of these topics ($79$ in total) share close
relationships, the first three authors manually aggregate them into macro-
topics using the negotiated agreement technique, similar to Step 2.5. If the
generated topics are difficult to interpret or have significant overlap, we
would review and adjust the hyperparameters or go back to the earlier steps.
This could involve adjusting the open card sorting standard.

Step 3.4:Collect prevalence metrics

In our study, we first collect metrics on the prevalence of each macro-topics.
Using this data, we then create a heatmap that illustrates the number of posts
between challenge and solution macro-topics. This approach not only helps us
understand which solutions are commonly associated with specific challenges,
but also sheds light on the potential causes leading to those challenges.

##  4 RQ1 Results: Prevalence of Challenge Topics

In RQ1, we investigate the challenges that ML practitioners face in asset
management. To begin with, we categorize the collected posts into two primary
types: knowledge and problem inquiries. Subsequently, we extract the topics of
these inquiries, aggregate them into macro-topics, and clearly define each of
these macro-topics along with illustrative examples. We also evaluate the
prevalence of these macro-topics by measuring their frequency in the collected
posts. Furthermore, we explore the intricate relationship between macro-topics
and curated tools, emphasizing their correlation in the inquiries.

###  4.1 Prevalence of posts based on type of inquiry

We find that problem inquiries make up $59.42\%$ of the Q&A posts, whereas
knowledge inquiries account for $40.58\%$. This shows a higher prevalence of
problem inquiries compared to knowledge inquiries. This trend aligns with the
findings of other studies in different domains, including those related to
service mesh systems chen2023practitioners . However, this trend does not
indicate that software engineers are typically more inclined to encounter
technical issues than theoretical ones. For example, a study by Treude et al.
treude2011programmers  reports a higher prevalence of knowledge inquiries
compared to other types of inquiry on Stack Overflow.

###  4.2 Macro-topics of the posts

We identify $133$ distinct topics from the collected posts, using the topic-
modeling approach outlined in Step 2.5. Our optimal topic model achieves a
coherence score of $0.7478$ and identifies $3,584$ posts as outliers. We have
listed the names of all identified topics, each accompanied by an index, in
Table LABEL:rq1:topic_list (Appendix B). For a description of each topic, we
refer to our replication package replication_package . Then we aggregate the
topics into 16 macro-topics, which offer in-depth insights into the challenges
of ML asset management, independent of specific frameworks and technologies.
This study denotes macro-topics with the prefix $\hat{C}$ and topics with the
prefix $C$. The subscript that follows each prefix indicates the
(macro-)topic’s index. In Table 1, we illustrate each macro-topic with its
index, name, prevalence, and underlying topics.

Table 1: Macro-topic list of the identified challenges. Index | Name | Prevalence (%) |  Topic List
---|---|---|---
$\hat{C}_{01}$ | Code Development | 2.42 |  [27, 28, 57]
$\hat{C}_{02}$ | Code Management | 0.74 |  [30]
$\hat{C}_{03}$ | Computation Management | 7.82 |  [19, 24, 34, 37, 42, 44, 61, 102]
$\hat{C}_{04}$ | Data Development | 4.12 |  [10, 13, 22, 29]
$\hat{C}_{05}$ | Data Management | 8.11 |  [33, 35, 38, 54, 67, 75, 76, 78, 79, 89, 117, 125, 131]
$\hat{C}_{06}$ | Environment Management | 18.89 |  [2, 3, 4, 7, 14, 15, 39, 40, 48, 52, 60, 64, 87, 88, 96, 97, 99, 101, 105, 107, 110, 115, 123]
$\hat{C}_{07}$ | Experiment Management | 2.52 |  [18, 116]
$\hat{C}_{08}$ | File Management | 6.79 |  [26, 36, 47, 55, 59, 70, 72, 121, 129]
$\hat{C}_{09}$ | Model Deployment | 10.59 |  [9, 17, 21, 23, 43, 50, 53, 56, 66, 71, 80, 84, 94, 111, 118, 119, 130, 132]
$\hat{C}_{10}$ | Model Development | 9.00 |  [8, 11, 12, 20, 31, 46, 83, 100, 114, 126, 133]
$\hat{C}_{11}$ | Model Management | 6.13 |  [32, 62, 69, 77, 82, 103, 106, 108, 109, 112, 127]
$\hat{C}_{12}$ | Network Management | 3.49 |  [73, 81, 85, 104, 113, 124, 128]
$\hat{C}_{13}$ | Observability Management | 6.29 |  [5, 41, 49, 63, 68, 91, 92, 93, 98]
$\hat{C}_{14}$ | Pipeline Management | 8.51 |  [1, 16, 45, 65, 86, 95, 120, 122]
$\hat{C}_{15}$ | Security Management | 3.20 |  [25, 51, 58, 74, 90]
$\hat{C}_{16}$ | User Interface Management | 1.36 |  [6]

We compare the distribution of the macro-topics according to the type of
inquiry and display the results in Figure 5. In the subsequent sections, we
explain each macro-topic by providing its definition, discussing underlying
topics, and offering a snapshot of typical content through a sample post. For
the sake of clarity in these samples, we abbreviate the verbose content in the
posts: long descriptions are denoted by [TEXT], code and logs by [CODE],
figures by [IMAGE], and hyperlinks by [URL].

Figure 5: A histogram representing the prevalence of each macro-topic in the
knowledge and problem inquiries. $\hat{C}_{01}$ Code Development
---
Definition: This macro-topic represents the posts related to creating and
optimizing the scripts and code that govern the behavior of a machine learning
system. It involves defining the attributes of an object ($C_{27}$), setting
the parameters ($C_{28}$), and using APIs ($C_{57}$) to facilitate
interoperations between different components of the software. Code development
ensures a cohesive and functional codebase that can successfully drive machine
learning operations and workflows.
Example: The following example454545https://github.com/wandb/edu/issues/103 is
a problem inquiry from GitHub Issues. The user faces an attribute error when
trying to access the WANDB_PROJECT attribute from the params module in a
Weights&Biases run.
$E_{01}$ When starting a new W&B run, project=params.WANDB_PROJECT returns
AttributeError: module ‘params’ has no attribute WANDB_PROJECT
$\hat{C}_{02}$ Code Management
Definition: This macro-topic represents the posts related to tracking,
documenting, and monitoring changes and versions of codebases
rochkind1975source ; loeliger2012version . It leverages tools such as Git
($C_{30}$), which serve as version control systems, to ensure consistency,
collaboration, and traceability throughout the machine learning development
lifecycle.
Example: The following
example464646https://stackoverflow.com/questions/75047065 is a problem inquiry
from Stack Overflow. The user encounters an error with DVC Git hooks when
running git push.
$E_{02}$: DVC has Git hooks that are installed with the dvc install. The hooks
were working fine, but after an error with dvc push and the DVC remote, I
cannot git push because before git push gets executed, dvc push runs and
generates an error. This means that I cannot push. How can I disable the DVC
Git hooks so that I will no longer face the issue?
$\hat{C}_{03}$ Computation Management
Definition: This macro-topic represents the posts related to orchestration,
allocation, and oversight of computational resources and services to ensure
optimal and efficient operations in machine learning workflows
mustafa2015resource ; manvi2014resource . This includes monitoring instances
($C_{19}$), compute infrastructure ($C_{24}$), servers ($C_{37}$), compute
clusters ($C_{42}$), system memory ($C_{44}$), specialized CUDA memory
($C_{61}$), handling situations where resource limits ($C_{34}$) or quota
requests ($C_{102}$) are exceeded. Computation management ensures that
computational resources are optimally allocated, scalable, and responsive to
the demands of machine learning workflows, while maintaining system stability
and efficiency.
Example: The following example474747https://github.com/Azure/azureml-
examples/issues/242 is a problem inquiry from GitHub Issues. The user faces an
error when setting up a Dask cluster following the tutorial.
$E_{03}$: When I follow the using-dask tutorial to set up a Dask cluster, I
fail to create a new cluster with an odd error that comes up: [Errno 113] No
route to the host. I tried searching for it, but the results on Google will
come with port and routing issues, which I don’t know if are directly related
to the problem here.
$\hat{C}_{04}$ Data Development
Definition: This macro-topic represents the posts related to preparing,
processing, and managing data for machine learning operations
sorokin2008utility ; mckinney2011pandas ; isah2019survey . It encompasses
tasks such as data annotation for training models ($C_{10}$), leveraging
distributed computing systems such as Spark for large-scale data processing
and analytics ($C_{13}$), manipulating the structure or values of dataframes
($C_{22}$), and using libraries such as pandas for versatile data manipulation
and analysis ($C_{29}$). Data development ensures that data are properly
curated, transformed and ready for subsequent machine learning tasks.
Example: The following example484848https://github.com/aws/amazon-sagemaker-
examples/issues/698 is a problem inquiry from GitHub Issues. The user faces an
error related to a mismatch between the number of labels in the multi-label
data and the num_classes parameter.
$E_{04}$: I am getting an error while running the sagemaker image
classification multilabel example: Failed Reason: ClientError: Mismatch
between the number of labels in the multilabel data and the num_classes
parameter.
$\hat{C}_{05}$ Data Management
Definition: This macro-topic represents the posts related to storing,
orchestrating, retrieving, sharing, and connecting various data resources and
artifacts throughout the machine learning lifecycle rigby2009collaboration ;
barrak2021co . It encompasses the use of buckets for storage ($C_{33}$), data
factories ($C_{35}$), managing ML features in feature stores ($C_{38}$),
downloading artifacts ($C_{54}$), using blob storage for unstructured data
($C_{67}$), exporting ($C_{75}$) and importing ($C_{117}$) data, organizing
data in datasets ($C_{76}$), reading image formats ($C_{78}$), managing
various artifacts produced during ML workflows ($C_{79}$), utilizing data
stores for ML data ($C_{89}$) and connecting to relational database systems
($C_{131}$) while benefiting from shared cache storage ($C_{125}$). Data
management ensures efficient storage, retrieval, and flow of data and
artifacts, facilitating seamless machine learning operations and workflows.
Example: The following example494949https://github.com/aws-samples/sagemaker-
ssh-helper/issues/28 is a problem inquiry from GitHub Issues. The user faces
an invalid bucket name error when running sm-local scripts and using the
SageMaker defaults from a configuration file.
$E_{05}$: Issue: Invalid bucket name “sagemaker.config INFO - Fetched default
config from location”: The name of the bucket must match the regex. This error
message appears when a user executes ‘sm-local’ scripts and uses SageMaker
defaults to set values from the configuration file.
$\hat{C}_{06}$ Environment Management
Definition: This macro-topic represents the posts related to creating,
configuring, deploying, and maintaining the various software and
infrastructure elements necessary for the development, training, and
deployment of machine learning models belguidoum2007dependency ;
esparrachiari2018tracking . It encompasses the management of notebook
instances ($C_{02}$), Docker images ($C_{03}$), software versions ($C_{04}$),
studios for ML development ($C_{07}$), RStudio IDE ($C_{14}$), workspaces
($C_{15}$), software configurations ($C_{39}$), initialization processes
($C_{40}$), Python interpreter ($C_{48}$), library imports ($C_{52}$),
Terraform infrastructure creation ($C_{60}$), software configurations
($C_{64}$), module usage ($C_{87}$), deployments via Kubernetes ($C_{88}$),
handling missing modules ($C_{96}$), package installations ($C_{97}$),
notebook imports ($C_{99}$), software installations ($C_{101}$), training
container configurations ($C_{105}$), container registries ($C_{107}$), global
YAML configurations ($C_{110}$), software dependencies ($C_{115}$), and
interactive Jupyter Notebook environments ($C_{123}$). Environment management
ensures the integration and operation of tools, platforms, and configurations
in the ML development lifecycle.
Example: The following
example505050https://stackoverflow.com/questions/50441181 is a knowledge
inquiry from Stack Overflow. The user inquires about the duties of ensuring
consistency and updating software packages in a serverless compute environment
such as SageMaker.
$E_{06}$: I am learning AWS SageMaker which is supposed to be a serverless
computing environment for machine learning. In this type of serverless compute
environment, who is supposed to ensure the consistency of the software package
and update the versions?
$\hat{C}_{07}$ Experiment Management
Definition: This macro-topic represents the posts related to designing and
executing controlled tests to collect data, evaluate models or hypotheses, and
efficiently handle situations where multiple runs or executions overlap or
nest in each other idowu2021asset . It encompasses the structured process of
conducting controlled tests or investigations ($C_{18}$) and managing
conditions where multiple runs or executions are intertwined ($C_{116}$).
Experiment management streamlines the execution and evaluation of controlled
tests while handling overlapping or nested runs in the ML development
lifecycle.
Example: The following
example515151https://stackoverflow.com/questions/60088889 is a problem inquiry
from Stack Overflow. The user faces an error when creating a new experiment
after deleting one with the same name.
$E_{07}$: [TEXT] I am using Mlflow w/ backend Postgres db. Here is what I have
run: [CODE] This deletes the experiment, but when I run a new experiment with
the same name as the one I just deleted, it will return this error: [CODE]
$\hat{C}_{08}$ File Management
Definition: This macro-topic represents the posts related to the handling,
storage, conversion, and manipulation of files agrawal2007five ; lu2013study .
It encompasses the organization of file directories ($C_{26}$), the conversion
of file formats ($C_{36}$), the uploading ($C_{47}$) and downloading
($C_{72}$) files, the reading of files ($C_{55}$), the storage solutions
($C_{59}$), the serialization and deserialization of files ($C_{70}$), the
management of input and output results ($C_{121}$) and the handling of
training files ($C_{129}$). File management ensures efficient storage,
retrieval, and manipulation of data and models throughout the machine learning
lifecycle.
Example: The following
example525252https://stackoverflow.com/questions/56269391 is a problem inquiry
from Stack Overflow. The user cannot read the files into the DVC control when
they are missing from the remote.
$E_{08}$: I ran into a problem with DVC when some files are missing on the
remote. [TEXT] How can I remove files from the DVC control and add them again?
$\hat{C}_{09}$ Model Deployment
Definition: This macro-topic represents the posts related to integrating a
trained machine learning model into an application or system to make real-time
predictions or decisions paleyes2022challenges ; baier2019challenges . It
encompasses prediction ($C_{09}$), deploying a model ($C_{21}$, $C_{53}$,
$C_{80}$, $C_{94}$), offering it through web services ($C_{17}$), setting up
inference ($C_{23}$, $C_{130}$) and model endpoints ($C_{43}$), deploying such
endpoints ($C_{50}$, $C_{118}$), batch prediction ($C_{56}$), invoking
endpoints ($C_{66}$, $C_{119}$), leveraging serverless platforms such as
Endpoint Lambda ($C_{71}$) and ACI ($C_{111}$), and scoring ($C_{84}$) or
serving the model ($C_{132}$) for production use. Model deployment ensures
that trained machine learning models are readily accessible and functional in
production environments for real-time or batch predictions.
Example: The following
example535353https://github.com/huggingface/transformers/issues/13111 is a
problem inquiry from GitHub Issues. The user faces an error when trying to
deploy it to SageMaker endpoints.
$E_{09}$: I have trained and saved a BertForSequenceClassification model to
S3. I then used this notebook [URL] to deploy the model to SageMaker
Endpoints. I ran: [CODE] I get the following error: [CODE]
$\hat{C}_{10}$ Model Development
Definition: This macro-topic represents the posts related to training, tuning,
and optimizing machine learning models nguyen2019machine ; wang2019various ;
bahrampour2015comparative . It comprises the use of ML frameworks such as
scikit-learn ($C_{114}$), PyTorch ($C_{08}$), and TensorFlow ($C_{12}$),
paired with optimization methods including fine-tuning ($C_{133}$),
hyperparameter adjustments ($C_{11}$), and sweeps ($C_{20}$), while
undertaking tasks such as training jobs ($C_{31}$) and distributed training
($C_{46}$), and focusing on both general training ($C_{83}$, $C_{100}$) and
specialized training of YOLO models ($C_{126}$). Model development ensures the
creation of robust and efficient machine learning models that are trained,
optimized, and validated using best practices and tools.
Example: The following example545454https://github.com/aws/amazon-sagemaker-
examples/issues/670 is a problem inquiry from GitHub Issues. The user faces a
No such file or directory error when running a training job for a custom
algorithm following the tutorial.
$E_{10}$: I am trying to create a custom algorithm following the instructions
given in this tutorial [URL]. When I am running the training job, it is
failing with the error No such file or directory. As per the documentation,
SageMaker should create these documents and copy the data and artifacts during
the runtime. But this is not happening.
$\hat{C}_{11}$ Model Management
Definition: This macro-topic represents the posts related to creating, saving,
registering, loading, storing, compiling, and exchanging machine learning
models, including their different formats, states, and associated metadata
jiang2023empirical ; xiu2020exploratory . It encompasses creation ($C_{82}$),
storage ($C_{32}$), registration ($C_{62}$), loading ($C_{69}$), centralized
storage in registries ($C_{77}$), compilation in NEO format ($C_{103}$),
utilization of specialized frameworks such as Huggingface ($C_{106}$),
exchange through formats such as ONNX ($C_{108}$), general use of trained or
pre-trained models ($C_{109}$), distinction of trained models ($C_{112}$) and
checkpointing of models during training ($C_{127}$). Model management ensures
the systematic development, storage, optimization, and tracking of machine
learning models throughout their lifecycle.
Example: The following
example555555https://stackoverflow.com/questions/74257398 is a knowledge
inquiry from Stack Overflow. The user asks about retrieving weights from the
best performing Keras model in Optuna.
$E_{11}$: Is there a way to pull the weights from the Keras model that
performs the best and was created using an Optuna study? [TEXT]
$\hat{C}_{12}$ Network Management
Definition: This macro-topic represents the posts related to the
establishment, monitoring, and optimization of communication links and
interactions related to the endpoints of the machine learning model
xia2014survey ; kreutz2014software . It involves addressing broken links
($C_{73}$), managing endpoint content ($C_{81}$), conducting endpoint ping
tests ($C_{85}$), establishing connections ($C_{104}$), utilizing web
interfaces ($C_{113}$), receiving response endpoints ($C_{124}$), and
overseeing endpoint transition ($C_{128}$). Network management ensures
seamless integration, availability, and responsive interactions of the machine
learning model endpoints in the ML development lifecycle.
Example: The following example565656https://github.com/allegroai/clearml-
server/issues/201 is a knowledge inquiry from GitHub Issues. The user inquires
about changing the default API server port in ClearML and updating the system
with new credentials.
$E_{12}$: I would like to change the default apiserver port from 8008 to
18008. I have changed it to docker-compose.yml. The ClearML app runs without
problems. But when I create new credentials, it gives me an old port there. It
would be great if I could change this port so it would show up when creating
credentials. How to achieve it?
$\hat{C}_{13}$ Observability Management
Definition: This macro-topic represents the posts related to tools, processes,
and practices designed to record, monitor, compare, and visualize events,
metrics, and performance related to machine learning models throughout their
lifecycle goniwada2022observability ; klaise2020monitoring . It encompasses
the recording of events through logs ($C_{05}$), the usage of custom metrics
($C_{41}$), visualization with TensorBoard ($C_{49}$), logging performance
metrics ($C_{63}$), benchmarking model performance ($C_{68}$), generating
training reports ($C_{91}$), logging model specifics ($C_{92}$), continuous
model monitoring ($C_{93}$) and integration of logs in Amazon CloudWatch
($C_{98}$). Observability management ensures continuous, transparent, and
efficient monitoring and analysis of machine learning models, fostering
improved performance, reliability, and insights.
Example: The following
example575757https://stackoverflow.com/questions/71505796 is a knowledge
inquiry from Stack Overflow. The user faces an issue with a binary
classification metric evaluation function that leads to inaccurate recall
metrics.
$E_{13}$: I was trying to assess the metrics for binary classification and got
a very strange result. It appears that eval_and_log_metrics will not be able
to tell which one is the true positive in the y_test. So it just averages out
true positive and true negative, which gave inflated recall metrics. How to
pass the true positive value to the eval_and_log_metrics?
$\hat{C}_{14}$ Pipeline Management
Definition: This macro-topic represents the posts related to the orchestration
and execution of tasks or steps in a machine learning process
schlegel2023management ; xin2021production . It orchestrates the comprehensive
progression of a project through defined phases such as individual pipeline
steps ($C_{01}$), handling large data through batch transform ($C_{16}$),
mitigating issues such as Stuck Queue, incorporating inference pipeline
strategies ($C_{65}$), improving efficiency through parallelization jobs
($C_{86}$), defining pipeline training steps ($C_{95}$), establishing
lifecycle configuration ($C_{120}$), and integrating custom job tasks
($C_{122}$) to foster streamlined, effective and robust machine learning
operations. Pipeline management ensures a structured and efficient approach to
handling machine learning projects from start to finish. Example: The
following
example585858https://github.com/Azure/MachineLearningNotebooks/issues/1927 is
a knowledge inquiry from GitHub Issues. The user asks about retrieving
information from existing pipelines, including their scheduling and metadata,
using the Azure Machine Learning Python SDK.
$E_{14}$: I use Azure machine learning services to run an ml-model. When I go
to the GUI of AML I can see all the existing pipelines, but I cannot see how
they are scheduled. I need to obtain all published pipelines and the
associated metadata. How can I get information about an existing pipeline with
the Python SDK?
$\hat{C}_{15}$ Security Management
Definition: This macro-topic represents the posts related to the control and
protection of machine learning resources and services soomro2016information ;
werlinger2009integrated . It encompasses the structured utilization and
control over user accounts ($C_{25}$), leveraging authentication processes
($C_{51}$) for verified access, ensuring secure API interactions through API
keys ($C_{58}$), controlling access through permission authorizations
($C_{74}$) and handling instances of permission denial ($C_{90}$) to safeguard
resources and services. Security management serves as the guardian in MLOps,
orchestrating user account setups, verifying identities, and facilitating
secure API accesses while meticulously granting permissions to prevent
unauthorized operations and access.
Example: The following
example595959https://stackoverflow.com/questions/64039980 is a problem inquiry
from Stack Overflow. The user faces a permission error when deploying a
TensorFlow model using Sagemaker.
$E_{15}$: I can train my model using the Sagemaker TensorFlow container. Below
is the [CODE]. But when I try to deploy the model, I get the following error:
UnexpectedStatusException: Error hosting endpoint: Failed. Reason: Make sure
that all images included in the model for the production variant AllTraffic
exist and that the execution role used to create the model has permission to
access them.
$\hat{C}_{16}$ User Interface Management
Definition: This macro-topic represents the posts related to design,
implementation, and maintenance of visual tools and dashboards that enable
users to interact with and interpret machine learning systems, models, and
data. It encompasses the creation and management of visual representations of
data or information, such as plots ($C_{06}$). User interface management
enhances user experience by leveraging visual tools to facilitate effective
and intuitive data representation and analysis.
Example: The following example606060https://community.wandb.ai/t/vega-
code/4605 illustrates a knowledge inquiry from Weights&Biases discussion
forum. The user inquires about the accessibility of the underlying source code
for a Vega or Vega-lite visualization in a Weights&Biases panel.
$E_{16}$: Is it possible to see the underlying source code of Vega or Vega-
lite for a particular panel on wandb?

###  4.3 Prevalence of posts based on macro-topics

We find that Environment Management ($\hat{C}_{06}$) is the most prevalent
macro-topic among the collected posts, constituting $18.89\%$ of the total
posts. The high prevalence indicates that ensuring a consistent and controlled
environment is challenging for ML practitioners throughout ML asset
management. ML models are particularly sensitive to their design, training,
and deployment environments616161https://neptune.ai/blog/ml-model-packaging.
Even minor differences in framework versions, library dependencies, or
hardware configurations can result in performance inconsistencies or failures
enck2022top .

Moreover, we find that Model Deployment ($\hat{C}_{11}$) ranks as the second
most prevalent macro-topic at $10.59\%$, with Model Development
($\hat{C}_{09}$) closely following as the third most prevalent macro-topic at
$9\%$. The prominence of Model Deployment ($\hat{C}_{11}$) indicates that a
substantial part of the challenges and focus in ML asset management is on
transitioning models from development or testing phases to production. The
absence of well-defined routes from development to production exacerbates this
issue, and is connected to an alarming AI project failure rate of nearly
50%626262https://aws.amazon.com/blogs/startups/scaling-ai-ml-and-accelerating-
ai-development-with-anyscale-and-aws/. Similarly, the notable prevalence of
model development emphasizes the importance of core activities in the building
of ML models. Activities such as selecting appropriate algorithms, feature
engineering, and model training remain pivotal areas of discussion in the ML
development lifecycle giray2021software ; baier2019challenges .

Lastly, we observe that Code Management ($\hat{C}_{02}$) remains the least
prevalent macro-topic, accounting for only $0.74\%$ of the posts. This may be
due to the perception of source code management as well-established, along
with the specialized nature of the forums related to ML asset management,
_i.e._ , discussions about code management are typically found on other
platforms. For instance, the tag “Git” on Stack Overflow serves as a lively
venue for discussions on this topic, underlining the trend of these dialogues
migrating to spaces specifically dedicated to code management.

###  4.4 Mapping between macro-topics and tools

In Figure 6, we present a heatmap showing the relationship between various
tools and challenge macro-topics. Each block in the heatmap represents the
percentage of posts associated with a specific tool to each of the challenge
macro-topics. One key finding from the heatmap is that the $55\%$ (11 out of
20) tools are primarily associated with inquiries related to Environment
Management ($\hat{C}_{06}$). This list includes Aim ($50.00\%$), AzureML
($22.37\%$), Comet ($30.77\%$), Guild AI ($28.69\%$), H20 AI Cloud
($24.56\%$), Kedro ($33.44\%$), MLflow ($18.77\%$), Neptune ($20.47\%$),
Sacred ($43.48\%$), SageMaker ($18.18\%$) and Vertex AI ($16.93\%$). This
trend suggests a strong focus of posts on managing the software environment
and dependencies. The diverse ways in which different tools handle environment
and dependency management underline the unique integration challenges each
encounters.

In contrast, Domino stands out with a significant $38.46\%$ of user inquiries
related to Code Development ($\hat{C}_{01}$). This emphasizes the challenges
Domino users face when creating ML-related components or features. Similarly,
Optuna ($32.61\%$) and Weights&Biases ($20.27\%$) predominantly receive
inquiries about Model Development ($\hat{C}_{10}$), pointing to the
difficulties users face when developing ML models using these tools. An
outlier is Determined, which is exclusively related to Computation Management
($\hat{C}_{03}$). However, this association might not be indicative of a
broader trend, as only a single post corresponds to this tool in our collected
posts. This suggests that challenges related to Determined might be
underrepresented in our survey.

Figure 6: Mapping between tools and challenge macro-topics based on the number
of posts (Normalized across challenge macro-topics).

Summary of RQ1 $\bullet$ Problem inquiries (59.42%) appear more frequently
than knowledge inquiries (40.58%) in Q&A posts related to machine learning
asset management. $\bullet$ Environment Management ($18.89\%$), Model
Deployment ($10.59\%$) and Model Development ($9\%$) are the most discussed
topics in related posts to machine learning asset management, indicating a
strong emphasis on the integration, training, and serving of machine learning
models in practical applications. $\bullet$ $55\%$ of the tools receive the
majority of their inquiries related to Environment Management. This pattern
indicates a prevalent challenge in the ability of these tools to seamlessly
integrate with other software platforms, highlighting an area for improvement
and development.

##  5 RQ2 Results: Prevalence of Solution Topics

In RQ2, we examine the solutions proposed by ML practitioners to address the
challenges in ML asset management. We define these topics, offer illustrative
examples, and compare their prevalence. In addition, we analyze the
distribution of solutions by topic and type of inquiry. Lastly, linking
challenges to their respective solution topics, we seek to find the prevailing
patterns of solutions in ML asset management.

###  5.1 Macro-topics of the solutions

We use open card sorting to categorize solved inquiries, as detailed in Step
3.1. The frequency of each type of inquiry is presented in Table 3. In
particular, $9.86\%$ ($159$ in total) GitHub issues are marked as “closed”
without any solution found in their posts. This might indicate that these
issues may be of low priority or may not have a significant impact on the
software project.

Table 3: Types of solved inquiries across discussion forums. Discussion forum |  Number of
normal
inquiries |  Number of
non-inquiries |  Number of intermittent inquiries |  Number of non-solved inquiries
---|---|---|---|---
Stack Overflow |  2187 |  2 |  4 |  5
GitHub Issues |  1432 |  6 |  15 |  159
Tool-specific forums |  849 |  1 |  1 |  8
GitHub Discussions |  17 |  0 |  0 |  0
GitLab Issues |  1 |  0 |  0 |  0

We identify $79$ distinct topics from the solutions of normal inquiries using
the topic-modeling approach outlined in Step 3.3. Our optimal topic model
achieves a coherence score of $0.9085$ and identifies $890$ posts as outliers.
In our study, we use the prefix $\hat{R}$ to represent macro-topics and $R$ to
represent regular topics. A detailed list of these topics, along with their
indices and names, is provided in Table LABEL:rq2:topic_list (Appendix B). For
further descriptions of each topic, please refer to our replication packages
replication_package .

Following the practice of RQ1, we formulate $18$ macro-topics to encapsulate
the original topics. Table 4 shows the macro-topics, their prevalence, and
underlying topics. A notable observation is the naming overlap between certain
macro-topics associated with solutions and those linked to challenges. This
overlap signifies their shared relevance to specific aspects of asset
management, underscoring the interconnectedness of challenges and solutions
within the same thematic groups.

Table 4: Macro-topic list of the identified solutions. Index | Name | Prevalence (%) |  Topic List
---|---|---|---
$\hat{R}_{01}$ | Code Development | 15.35 |  [12, 13, 27, 28, 47, 56, 57, 58, 59, 61, 63, 84]
$\hat{R}_{02}$ | Code Management | 0.80 |  [43]
$\hat{R}_{03}$ | Computation Management | 5.25 |  [19, 30, 35, 53, 60]
$\hat{R}_{04}$ | Data Development | 3.66 |  [31, 33, 54]
$\hat{R}_{05}$ | Data Management | 4.81 |  [40, 42, 70, 75, 77]
$\hat{R}_{06}$ | Environment Management | 23.31 |  [1, 5, 7, 15, 21, 22, 23, 25, 34, 37, 41, 44, 62, 66, 67, 69, 72, 73, 76, 78, 85]
$\hat{R}_{07}$ | Experiment Management | 4.12 |  [11, 39, 71, 83]
$\hat{R}_{08}$ | File Management | 9.64 |  [16, 20, 29, 36, 49, 55, 64, 65, 68, 86]
$\hat{R}_{09}$ | Model Deployment | 5.01 |  [24, 26, 38, 79, 80]
$\hat{R}_{10}$ | Model Development | 2.10 |  [3]
$\hat{R}_{11}$ | Model Management | 4.32 |  [10, 32, 52]
$\hat{R}_{12}$ | Network Management | 2.92 |  [9, 51]
$\hat{R}_{13}$ | Observability Management | 3.41 |  [6, 50, 74]
$\hat{R}_{14}$ | Pipeline Management | 4.08 |  [14, 18, 45]
$\hat{R}_{15}$ | Security Management | 4.54 |  [4, 17, 81]
$\hat{R}_{16}$ | User Interface Management | 1.04 |  [48]
$\hat{R}_{17}$ | Comparison & Recommendation | 2.46 |  [2, 82]
$\hat{R}_{18}$ | Maintenance & Support | 3.19 |  [8, 46]

In this section, we define each solution macro-topic and provide corresponding
examples from our dataset of collected posts. Since the first $16$ solution
macro-topics share identical names with their challenge counterparts, we focus
exclusively on the two uniquely named solution macro-topics in this section,
directing readers to the Appendix A for the illustrations of the others.

$\hat{R}_{17}$ Comparison & Recommendation
---
Definition: Comparison & Recommendation represents the posts related to the
analytical process in which the efficacy and suitability of various tools,
such as SDKs, APIs, databases, platforms, runtimes, versions, tasks, data or
pipelines, are examined ($R_{82}$) and suggested ($R_{02}$) during the ML
development lifecycle. Comparison and recommendation facilitate the selection
of the most appropriate tools and resources regarding ML assets, thus
promoting efficiency and superior outcomes through well-informed decisions in
the production environment.
Example: The following
example636363https://stackoverflow.com/questions/72408785 provides supported
runtimes to help the user make an informed decision.
$E_{17}$: Accepted Answer: MLRun has several different ways to run a piece of
code. At this time, the following runtimes are supported: [TEXT]
$\hat{R}_{18}$ Maintenance & Support
Definition: Maintenance & Support represents the posts related to the
continuous process of updating and improving machine learning systems by
implementing patches or fixes ($R_{08}$), and managing operating issues
through support tickets ($R_{46}$) grubb2003software ; kitchenham1999towards .
It ensures the stability, performance, and effective communication of machine
learning-based systems.
Example: The following
example646464https://stackoverflow.com/questions/73435172 provides the
development plan for the DVC-Hydra integration support.
$E_{17}$: Accepted Answer: A DVC-Hydra integration is in development. You can
see the proposal in [URL] and the development progress in [URL]. [TEXT]

###  5.2 Prevalence of solutions based on macro-topics

Environment Management ($\hat{R}_{06}$) stands out as the most prevalent
($23.31\%$) solution macro-topic related to ML asset management, as
illustrated in Table 4. This prominent prevalence underscores the intricacies
and significance of overseeing environments and dependencies throughout ML
asset management. Ensuring adept management of these ML environments and
dependencies is paramount for achieving consistent and reproducible model
training and deployment across diverse infrastructures.

Following this, Code Development ($\hat{R}_{01}$) constitutes $15.35\%$ and is
the second most prevalent solution macro-topic. The prominence of this macro-
topic reflects the evolving complexities and nuances of crafting code
specifically for ML applications, as opposed to traditional software
development. This emphasizes the need for robust development practices,
continuous integration, and testing methodologies tailored for ML. File
Management ($\hat{R}_{08}$) ranks as the third most prevalent macro-topic with
a proportion of $9.64\%$ of Q&A posts. This prominence underscores the
essential role of structured file storage, retrieval, and manipulation in the
ML development lifecycle.

###  5.3 Mapping between challenge and solution macro-topics

Knowledge inquiries are commonly addressed with solutions specific to their
respective topics. This correlation can be observed in Figure 7, which shows
the mapping between challenges and solution macro-topics in knowledge
inquiries. Each cell in the heatmap represents the normalized number of posts
corresponding to a specific challenge macro-topic across various solutions.
The prominent dark blocks along the diagonal suggest that challenges in a
particular domain are frequently solved in the same domain. In our study, we
term this phenomenon as “self-resolution”. For instance, inquiries related to
Environment Management ($\hat{C}_{06}$) are predominantly addressed using
environment-specific solutions ($\hat{R}_{06}$). To highlight the prevalence
of self-resolution, we set a threshold of $25\%$ to indicate a significant
rate of self-resolution. As a result, $56.25\%$ (9 out of 16) macro-topics
meet this criterion, implying that the majority of knowledge inquiries rely on
domain-specific knowledge from their corresponding macro-topic for solution.

Figure 7: Mapping of challenge marco-topics (knowledge inquiry) to solution
macro-topics, showing the normalized number of posts based on post
interactions.

Conversely, we notice a pattern in which specific types of challenge are often
addressed by solutions from different areas, a phenomenon we term “counter-
self-resolution” in our study. For example, challenges in Code Management
($\hat{C}_{02}$) are primarily solved by methods in Code Development
($\hat{R}_{01}$). This connection highlights that effective code management is
closely related to the processes and practices in code development. Similarly,
challenges related to Experiment Management ($\hat{C}_{07}$) are primarily
tackled with solutions in Environment Management ($\hat{R}_{06}$). This
reveals that the effectiveness of experiment management depends on robust
environment management practices.

Lastly, since Environment Management ($\hat{R}{06}$) emerges as the most
prevalent macro-topic in knowledge inquiries, we take a closer examination at
the mapping of the underlying topics in this macro-topic. Figure 8 illustrates
the relationship between challenge macro-topics and the underlying solution
topics in Environment Management ($\hat{R}{06}$). We observe that Container
Customization ($R_{21}$) is the most prevalent solution topic ($13.46\%$) for
Environment Management ($\hat{R}_{06}$) solutions. Specifically, it accounts
for $50\%$ of the environment-oriented solutions to Network Management
($\hat{C}_{12}$) challenges. Furthermore, this figure highlights that Package
Installation ($R_{05}$) emerges as the most prevalent solution ($100\%$) for
User Interface Management ($\hat{C}_{16}$). The figure also points out that
for Code Management ($\hat{C}{02}$) challenges, Package Addition ($R_{23}$)
and Environment Variable Management ($R_{34}$) are the most prevalent
solutions, each accounting for $50\%$.

Figure 8: Zooming down to the level of individual solution topics for
Environment Management ($\hat{R}_{06}$) in Figure 7.

Problem inquiries are less commonly addressed through self-resolution. This
observation is evident in Figure 9, where we illustrate the normalized number
of posts between challenge and solution macro-topics in problem inquiries.
Herein, the diagonal cells in the heatmap are lighter compared to those
related to knowledge inquiries. Using a $25\%$ threshold, we observe that only
$25\%$ (4 out of 16) of the macro-topics display a high level of self-
resolution. This suggests a lower trend for self-resolution in problem
inquiries, likely due to their inherent complexity or external, cross-domain
factors influencing them.

Figure 9: Mapping of challenge marco-topics (problem inquiry) to solution
macro-topics, showing the normalized number of posts based on post
interactions.

Conversely, we note that the counter-self-resolution trend is more pronounced
in problem inquiries compared to knowledge ones. A case in point is again the
resolution of Code Management ($\hat{C}_{02}$) challenges with Code
Development ($\hat{R}_{01}$) methods. Furthermore, Environment Management
($\hat{R}_{06}$) emerges as the most prevalent solution macro-topic for
$62.5\%$ (10 out of 16) macro-topics, including Computation Management
($\hat{C}_{03}$), Data Development ($\hat{C}_{04}$), Data Management
($\hat{C}_{05}$), Experiment Management ($\hat{C}_{07}$), Model Deployment
($\hat{C}_{09}$), Model Development ($\hat{C}_{10}$), Model Management
($\hat{C}_{11}$), Network Management ($\hat{C}_{12}$), Observability
Management ($\hat{C}_{13}$) and User Interface Management ($\hat{C}_{16}$).
This pattern implies that software environment and dependency management are a
foundational and versatile solution, addressing a wide range of challenges on
different topics. Furthermore, challenges in Observability Management
($\hat{C}_{13}$) are commonly addressed with Code Development ($\hat{R}_{01}$)
solutions. This implies that the quality and effectiveness of evaluation and
monitoring is significantly influenced by the methodologies and practices used
in code development.

Lastly, since Environment Management ($\hat{R}{06}$) emerges as the most
prevalent macro-topic in problem inquiries, we also examine the mapping of its
underlying topics in Figure 10. In particular, Package Upgrade ($R_{01}$),
accounting for $20.45\%$ of the cases, is the most frequently encountered
solution topic in the broader category of software dependency and environment,
for the majority of challenge macro-topics, with the exception of Code
Management ($\hat{C}_{02}$) and User Interface Management ($\hat{C}_{16}$).
This pattern indicates a significant dependence on external packages and
libraries in machine learning projects. As these packages undergo updates,
they may present new features, optimizations, or necessary bug fixes.
Furthermore, Package Removal ($R_{62}$) stands out as the most prevalent
environment-oriented solution ($100\%$) to address the challenges related to
Code Management ($\hat{C}_{02}$). It also reveals that Package Installation
($R_{05}$) is the most prevalent environment-oriented solution ($42.86\%$) for
User Interface Management ($\hat{C}_{16}$). This trend further underscores the
point that package management is at the heart of environment management,
maintenance, and deployment enck2022top . It allows developers and system
administrators to define, reproduce, and share their software environments
consistently and predictably.

Figure 10: Zooming down to the level of individual solution topics for
Environment Management ($\hat{R}_{06}$) in Figure 9.

Summary of RQ2 $\bullet$ Environment Management ($23.31\%$), Code Development
($15.35\%$), and File Management ($9.64\%$) are the most discussed solution
topics related to machine learning asset management, indicating that these
areas are the most common pain points or areas of interest for machine
learning practitioners. $\bullet$ $56.25\%$ (9 out of 16) of the challenge
macro-topics in knowledge inquiries show a high-level self-resolution. In
contrast, only $25\%$ (4 out of 16) of the challenge macro-topics in problem
inquiries demonstrate a high self-resolution rate. This suggests that a more
interconnected and cross-disciplinary approach is needed when addressing
specific problems, as opposed to seeking general knowledge. $\bullet$ The
counter-self-resolution trend is more pronounced in problem inquiries than in
knowledge inquiries. This may imply that problem inquiries necessitate a
broader spectrum of insights, often derived from external or cross-
disciplinary domains, to devise viable solutions.

##  6 RQ3 Results: Comparison of Discussion Forums

In RQ3, we explore commonalities and differences in developer discussion
forums about ML asset management. Specifically, we compare Stack Overflow,
repository-specific forums, and tool-specific forums. We start our analysis by
examining the general prevalence metrics on various forums. This initial step
gives us a broad overview of the landscape. Next, we narrow our focus to
compare these metrics based on different types of inquiry in each forum. This
allows us to understand how each forum performs in addressing specific types
of inquiry. In the final step, we shift our attention to macro-topics in these
forums, comparing the prevalence metrics once more.

###  6.1 Comparison of discussion forums in general

We find that Stack Overflow is the main source of inquiries, accounting for
$48.65\%$ of the total posts. Tool-specific forums are next, contributing
$34.19\%$, while repository-specific forums have the fewest, at only
$17.16\%$. The prevalence of Stack Overflow posts is expected due to its
popularity for developers worldwide vasilescu2013stackoverflow . Its highest
number of posts reflects its popularity and wide reach among ML practitioners.
However, the low number of posts on repository-specific forums, such as
GitHub, could indicate a narrower focus of discussions, often centered on
project-specific issues, bugs, or feature requests, rather than the more
general questions and broad topics typically found on Stack Overflow
vadlamani2020studying ; squire2015should .

###  6.2 Comparison of discussion forums based on tools

A distinct distribution emerges in the preferences of the types of discussion
forum among users of different tools, as illustrated in Figure 11. The
intensity of the color in the heatmap indicates the frequency of posts for
each forum-tool pairing. In particular, $25\%$ (5 out of 20) tools, namely DVC
($65.77\%$), Domino ($100\%$), Guild AI ($100\%$), Polyaxon ($94.44\%$), and
Weights & Biases ($69.36\%$), have the majority of their discussions
originating from tool-specific forums. This trend could imply a positive user
experience on these dedicated forums, which deserves further investigation
into the factors contributing to this preference. Conversely, we observe that
$20\%$ (4 out of 20) tools, including Aim ($100\%$), Comet ($80.77\%$),
Determined ($100\%$), and Neptune ($90.55\%$), predominantly have discussions
in repository-specific forums. This could reflect a tendency among
practitioners to discuss the challenges faced in open-source projects where
these tools are utilized as dependencies. Furthermore, $35\%$ of the tools (7
out of 20), including Amazon SageMaker ($64.89\%$), H2O AI Cloud ($85.96\%$),
Kedro ($61.20\%$), MLRun ($100\%$), MLflow ($54.07\%$), Optuna ($67.03\%$),
and Vertex AI ($68.48\%$), have the most of their discussions on Stack
Overflow. This behavior could highlight a preference to engage with a wider
community or a higher popularity of Stack Overflow among developers in
general.

Figure 11: Normalized distribution of Q&A posts across different types of
forum.

###  6.3 Comparison of discussion forums based on macro-topics

Environmental management ($\hat{C}_{06}$) is the most prevalent macro-topic in
all forums, comprising $25.53\%$ of repository-specific issues, $18.34\%$ of
Stack Overflow posts, and $16.34\%$ of discussions in tool-specific forums.
This trend reinforces the RQ1 finding that software environment and dependency
management are the primary macro-topics in ML asset management. In contrast,
while Model Deployment ($\hat{C}{09}$) is the second most frequent topic on
Stack Overflow ($12.49\%$) and tool-specific forums ($9.98\%$), it is ranked
sixth on repository-specific forums at $6.45\%$. Conversely, Model Development
($\hat{C}{10}$) is second on repository-specific forums ($9.71\%$) but falls
to fourth on Stack Overflow ($9.11\%$) and tool-specific forums ($8.49\%$).

Differences in the distribution of posts across macro-topics between any two
discussion forums are significant, as evidenced by three pairwise Chi-square
goodness of fit tests, conducted at a significance level of $0.05$ (Table 6).
Chi-square tests are commonly used to evaluate the alignment between observed
data and a hypothesized distribution pearson1900x . Given the multiple tests
conducted, we adjust the p-values to the so-called “q-values” storey2002direct
using the Benjamini-Hochberg procedure to account for the False Discovery Rate
(FDR) benjamini1995controlling . This adjustment helps mitigate overestimation
of significant findings that can arise by chance and is an alternative to
related approaches, such as Bonferroni correction dunn1961multiple , which
modify the alpha value instead of the p-value.

Table 6: Chi-squared test results between any two different discussion forums across macro-topics. Forum names | q-value | Cramér’s V
---|---|---
Repository-specific forums vs. Stack Overflow | 0.000 | 0.177
Repository-specific forums vs. Tool-specific forums | 0.000 | 0.169
Stack Overflow vs. Tool-specific forums | 0.000 | 0.132
Table 7: Intepretation of Cramér’s V estimator Negligible | Weak | Moderate | Relatively strong | Strong | Very strong
---|---|---|---|---|---
$0\sim 0.1$ | $0.1\sim 0.2$ | $0.2\sim 0.4$ | $0.4\sim 0.6$ | $0.6\sim 0.8$ | $0.8\sim 1$

Following the Chi-square tests, we also calculate the Cramér’s V values, which
range from $0.132$ to $0.177$. Cramér’s V is a metric that indicates the
strength of association between two variables cramer1999mathematical . As
depicted in Table 7, these values signify weak associations between the
distribution of Q&A posts and the associated forums. Therefore, our
statistical analysis implies a significant difference, but its practical
implications are not pronounced. Despite this, each forum exhibits a unique
pattern of macro-topics discussed. Hence, ML practitioners are still
recommended to adapt their discussions to align with the distinct
characteristics and preferences of each forum’s audience.

Summary of RQ3 $\bullet$ Stack Overflow is the main source of inquiries,
accounting for $48.65\%$ of the total posts. Tool-specific forums are next
($34.19\%$), while repository-specific forums have the fewest ($17.16\%$).
This pattern might imply that the extensive user base and broad discussion
topics make Stack Overflow the go-to platform for inquiries, while repository-
specific forums, such as GitHub, are favored for project-oriented discussions.
$\bullet$ A total of $25\%$ of the analyzed tools have majority of user
inquiries on tool-specific forums, $20\%$ on repository-specific forums, and
$35\%$ on Stack Overflow. This pattern suggests a potential area of
exploration for tool maintainers to improve user engagement on their
respective forums, fostering a better collaborative community. $\bullet$ In
all types of forums, Environment Management stands out as the most prevalent
topic, indicating a critical need for improved tooling, documentation, and
best practices in environment setup and dependency management.

##  7 Implications

Our research provides invaluable insights for a diverse group of ML
practitioners, including researchers, educators, and tool/application
developers in the software and machine learning domain. Through our findings,
we aim to guide their future efforts toward more efficient and cohesive
management of ML assets.

###  7.1 Implications for Researchers

##### Prevalent Discussion Topics:

Our findings show that environment configuration, model training, and
deployment are highly prevalent in asset management. Because of this, we see a
great opportunity for more research on environment management systems that
work well with all stages of ML development and should be flexible enough to
work with different ML tools and platforms.

##### Emerging Foundation Models:

To evaluate the prevalence of discussions on foundation models
bommasani2021opportunities  (FM), we perform an analysis based on the
frequency of related posts within our dataset. First, we compile a list of
keywords associated with FMs from leading GitHub repositories including
Awesome-LLM656565https://github.com/Hannibal046/Awesome-LLM, Open
LLMs666666https://github.com/eugeneyan/open-llms, and Awesome-Multimodal-
Large-Language-Models676767https://github.com/BradyFU/Awesome-Multimodal-
Large-Language-Models. Our systematic search through titles and post bodies,
after preprocessing and applying conditional checks to filter out irrelevant
content, reveals a modest presence ($0.76\%$) of FM discussions in our
dataset. Although insightful, this figure does not suffice to definitively
assess the state of asset management in foundation models. The observed low
frequency can be attributed to the nascent stage of local FM system deployment
and the associated demand for asset management independent of FM providers,
such as OpenAI686868https://openai.com. This trend began to materialize in the
last half of 2023, predominantly after the cut-off date of our data
collection, which ended on $26^{th}July,2023$. In particular, the increase in
local FM deployments was significantly influenced by the public release of
LLaMA-2 touvron2023llama  on $18^{th}July,2023$ and Mistral-7B
jiang2023mistral  on $27^{th}Sep,2023$. Although there may be overlaps in
asset management needs between pre-FM and FM models, future research should
focus on analyzing FM-related discussion posts that have emerged since the
late summer of 2023.

###  7.2 Implications for Educators

##### MLOps Curriculum Design:

Our research reveals that ML practitioners often face challenges in the
software environment and dependency management, model deployment, and model
development during asset management. These findings highlight the need to
adapt the curriculum design. Educational institutions should prioritize course
materials, hands-on experiments, and real-world case studies in the
aforementioned areas. MLOps enthusiasts, not limited to computer science
students, would greatly benefit from these custom curricula.

###  7.3 Implications for Application Developers

##### Holistic ML Development:

Our analysis identifies 133 topics, grouped into 16 overarching macro-topics
across various domains, suggesting that application developers should adopt a
holistic approach to asset management. This approach includes essential steps,
such as data preparation, environment setup, and model training. Much like how
full-stack developers benefit from understanding every stage of development,
ML developers too gain from a similar wide-ranging understanding. Adopting
this holistic approach could help application developers uncover and address
hidden technical debts sculley2015hidden  in the ML development lifecycle. As
a result, they face fewer obstacles and enjoy a smoother development journey.

##### Tool Selection Bootstrapping:

We find a clear pattern of association between certain tools and specific
challenge macro-topics. This information is invaluable for application
developers, offering a roadmap to anticipate and mitigate potential challenges
during the tool selection process. AzureML and SageMaker, for instance, are
associated with a broad range of macro-topics, underscoring their versatility.
However, developers should also be prepared for the diverse challenges that
come with such comprehensive platforms. Furthermore, the strong association of
a significant number of tools with software dependency and environment
management underscores a common area of challenge. Developers are recommended
to pay special attention to this aspect, evaluating how the features and
support of each tool can help navigate environment-related issues effectively.

###  7.4 Implications for Tool Developers

##### Feature Enhancement Prioritization:

Our study shows that ML practitioners face great challenges in solving issues
related to user interface, data development, experiment management, and code
management. Therefore, we recommend that tool developers prioritize the design
of more intuitive interfaces, enhanced data management features, and robust
tools to manage experiments and code. For emerging tool developers, tackling
these identified challenges not only addresses the core issues faced by ML
practitioners but also opens the door to establishing a niche market by
providing tailored solutions to these distinct pain points.

##### User Engagement Analysis:

Tool developers, especially those who manage discussion forums, play a crucial
role in shaping the experience of users who post their inquiries on these
platforms parra2022comparative . Tools, such as Amazon SageMaker, H2O AI
Cloud, Kedro, MLRun, MLflow, Optuna, and Vertex AI primarily have their
discussions hosted on Stack Overflow. This trend may signify a preference
among users to engage with a broader community or certain tool-specific
forums. This suggests that maintainers of other tools could adopt certain
practices from these successful forums to foster more interactions.

##  8 Threats to Validity

We address the validity threats in our research by examining them from four
perspectives: conclusion validity, construct validity, external validity, and
internal validity. For each type of threat, we present the specific actions we
take to mitigate them.

###  8.1 Conclusion validity

##### Macro-topic aggregation

: To mitigate the inherent risk of individual biases that might lead
contributors to different conclusions about topic groupings, we have adopted a
rigorous standard to aggregate topics. Initially, the first three authors take
on the task independently. Following this, they conduct a collaborative review
in which they cross-examine their individual categorizations. This collective
approach aimes not only to harmonize their perspectives, but also to minimize
any cognitive biases that might skew the results. We refer to established
literature if disagreements about the categorization of a specific topic
persist. By anchoring our decisions in prior research, we aim to ensure that
our categorizations are not only consistent but also justifiably grounded.

##### Open card sorting

: We have manually labeled the entire dataset consisting of $4,687$ solved
posts, ensuring that we incorporate every available data point for a thorough
analysis. By leveraging this comprehensive sample size, we significantly
enhance the statistical power of our study. Such an increase not only
strengthens the reliability of our findings, but also substantially mitigates
the risk of Type II errors. In the context of our study, a Type II error might
lead us to inadvertently neglect a recurring category frequently used by
practitioners - a misstep that could be attributed to the limitations inherent
in analyzing a restricted dataset.

###  8.2 Construct validity

##### Closed card sorting

: A primary threat to the construct validity of our study stems from the
subjectivity inherent in the manual filtering process of tool-specific forum
data. Despite implementing a multi-stage review protocol and involving several
authors to minimize personal biases, the possibility of subjective
interpretation in selecting and categorizing data cannot be eliminated. This
subjectivity could impact the reliability and generalizability of our
findings.

##### Macro-topic aggregation

: A specific topic may relate to multiple macro-topics, leading to some
uncertainty in our macro-topic aggregation choices. To address this, we
conduct a manual review of a 5% sample of posts on each topic. Through this
careful examination, we identify the macro-topic that the majority of these
topics align with most closely.

##### Open card sorting

: The first and second authors possess over three years of experience in
software development. This expertise plays a crucial role in minimizing
threats to construct validity during the open card sorting process. Before the
categorization process, these contributors engage in extensive discussion to
establish a clear consensus on the labeling guideline, as outlined in Step
3.1. This step guarantees that the labels accurately capture the content of
the solutions.

##### Post title refinement

: Subtle nuances and the core essence of original posts may not always be
accurately manifested in GPT-4 refined titles. To mitigate this, the first two
authors work together to review the refined titles by examining a sample of 5%
posts. When they find discrepancies, they tweak the prompts to better align
the refined titles with the content of the posts. Through this adjustment
process, they observe that tweaking the prompts results in topic models with
higher coherence (C_V). Our goal is to ensure that the GPT-4 refined titles
authentically encapsulate the intent of the original posts. This approach is
in line with the recommendations by Sallou et al. sallou2024breaking  that
stress the importance of reviewing an LLM’s output to ensure its applicability
and validity in software engineering research.

##### Topic model selection

: In topic modeling, it is possible to encounter topics with overlapping key
terms that may blur the distinctiveness of each topic. To address this, we
examine the top-10 topic models, focusing on those with the highest coherence
C_V value, to better understand both challenges and solutions. We manually
review a sample of 5% of posts for each topic to assess their thematic
representation and degree of overlap. Through this comparative approach, we
aim to identify the topic model that best fits our study.

###  8.3 External validity

##### Forum evolution

: We have collected Q&A posts from multiple developer discussion forums up to
$26^{th}ofJuly,2023$. We acknowledge that the content on these forums is
dynamic and subject to constant updates and modifications. Consequently, there
is a possibility that the information, challenges, and solutions we analyze
may become obsolete over time. To mitigate this concern, we employ rigorous
analytical methods to validate the relevance and applicability of our
findings. We aim to ensure that the insights derived are not constrained by
the collection period, but remain valuable and applicable in a longer
timeframe.

##### Forum selection

: To ensure a comprehensive and representative dataset on ML asset management
from the users’ perspective, we source data from a diverse spectrum of
discussion forums and tools. This strategy addresses potential threats to
external validity arising from user preferences. Our dataset incorporates
posts from general forums, such as Stack Overflow, repository-specific forums,
such as GitHub Issues, and tool-specific forums, such as the MLflow Google
Group.

##### Post collection

: The absence of posts on certain topics does not necessarily imply the
absence of such challenges for a given tool. Instead, this might reflect a
potential lack of community support, which is a critical factor in tool
adoption and usability. To mitigate this threat, we have broadened our scope
of data collection by including data from various discussion channels, such as
Stack Overflow, GitHub, and tool-specific websites. This approach aims to
provide a more comprehensive understanding of the challenges within the
domain.

##### Post translation

: In the open card sorting process, we utilize the built-in translator of
Microsoft Edge browser696969https://www.microsoft.com/en-us/edge to convert
non-English text to English text. Although such translations might not
encapsulate every nuance or cultural reference inherent in the original
language, software engineering inquiry terminologies generally present
consistent expressions across languages. Furthermore, considering that non-
English posts constitute less than $5\%$ of all entries, any potential
influence on external validity remains minimal.

##### Tool selection

: The operationalization of ML encompasses numerous tools, each relevant to
specific aspects of asset management. Overlooking any of these tools
potentially compromises the external validity of our findings. To address
this, we actively collect information on the most representative tools for ML
asset management from various sources, including Google search engine, Google
Scholar, GitHub, and insights provided by domain experts. Our tool selection
follows the filtering criteria set by previous research idowu2021asset , as
outlined in Step 1.1. This approach ensures that our choices remain systematic
and grounded in expert knowledge.

###  8.4 Internal validity

##### Closed card sorting

: Including unrelated posts can introduce noise that can distort the findings
of our study. To mitigate this risk, we employ the closed-open carding
approach to exclude any inquiry labeled “NA”. This step helps us filter out
irrelevant data, allowing for a more precise analysis and, ultimately, more
reliable results in our study.

##### Prompt tuning

: We notice that changing the way we phrase the GPT-4 prompt gives us
different results. This shows that it is not always easy to figure out the
best way to phrase the prompts. To tackle this issue, we use a heuristic
approach to tweak the prompts. Our goal is to find the one that works best for
creating refined titles. As we make these adjustments, we see a clear
improvement in the coherence (C_V) of the topic models. Most of our new
prompts even perform better than our original ones. We then have two team
members take a closer look at a random sample of 5% of the posts to make sure
that our new titles still match what the original poster meant. We find that
for the most part, the improved titles are in line with the original intent
and are similar to labels from SE experts. This step guarantees that our
analysis and results are solid and takes into account the different ways that
prompts can be phrased.

##  9 Conclusion

In this study, we explore the challenges and their solutions associated with
ML asset management tools through an analysis of Q&A posts from various
developer discussion forums. We employ a mixed-method approach, using BERTopic
to extract and categorize the primary topics of these challenges and the
corresponding solutions. Our findings reveal a comprehensive landscape of
prevalent challenges and their solutions, offering a granular view of the
complexities and needs in managing ML assets. We identify key areas of
concern, such as the management of software environments and dependencies, as
well as the training and deployment of models. It is important to note that
the tools explored in our study accommodate a wide spectrum of ML models,
ranging from statistical models to deep neural networks. As a result, our
insights are of significant value to a diverse audience in various contexts.
These insights facilitate the judicious selection of tools, the development of
effective educational resources, and the guidance of future research efforts
towards bridging existing gaps.

Our study suggests four promising research directions for further research on
ML asset management. Firstly, we underscore the importance of focusing on the
specific issues associated with the most prevalent macro-topics identified in
our study. These areas, marked by the high frequency of occurrence, represent
critical pain points for ML practitioners. Detailed investigations into these
macro-topics can reveal underlying issues, potential solutions, and tool and
resource development opportunities.

Secondly, a more thorough analysis of the dynamics inherent in problem and
knowledge inquiries across various software engineering domains is warranted.
Our observations indicate a varied prevalence of inquiry types related to ML
asset management. Specifically, problem inquiries are more prevalent than
knowledge inquiries regarding ML asset management. This pattern aligns with
previous findings in the service mesh domain chen2023practitioners , but
diverges from general domain trends, as highlighted by earlier studies
treude2011programmers . By exploring these patterns, we can gain a clearer
understanding of the intricacies surrounding the users’ intentions.

Thirdly, our analysis shows an interesting pattern: $16$ macro-topics have the
same names for both challenges and solutions. This overlap is not mere
coincidence, but indicates a significant link between the challenges
encountered and the solutions applied in asset management. It reveals the
potential for a deeper understanding of how issues are addressed in this area.
Future research should incorporate more data points and monitor the evolution
of these challenges and solutions over time. Such efforts aim to establish a
more comprehensive theory that accurately captures these relationships.

Lastly, exploring the association between various types of FM (_e.g._ , pre-
trained, fine-tuned, instruction-tuned zhang2023instruction , proxy-tuned
liu2024tuning , knowledge-distilled gou2021knowledge ) and identifying
challenge topics related to these types presents a promising avenue. In
particular, the growing trend toward local deployment, driven by decreased
privacy concerns yao2023survey , more cost-effective model parameters
schick2020s , and the advent of open language models groeneveld2024olmo , may
signal a shift in discussions toward more sophisticated asset management in
FMOps. Such research could provide essential insights, fostering innovation in
FM deployment strategies.

Given the pivotal role that asset management occupies in the burgeoning field
of MLOps, it is essential for ongoing research, dialogue, and collaboration to
shape its trajectory, ensuring that it effectively supports the diverse and
dynamic needs of ML practitioners.

##  10 Conflict of Interest

The authors declare that they have no conflict of interest.

##  11 Data Availability Statement

The datasets generated and analyzed during this study are available in the
replication package replication_package .

## References

  * (1) URL https://github.com/zhimin-z/Asset-Management-Topic-Modeling.  https://github.com/zhimin-z/MSR-Asset-Management, https://github.com/zhimin-z/QA-Asset-Management
  * (2) URL https://github.com/topics/mlops
  * (3) URL https://github.com/topics/machine-learning-engineering
  * (4) URL https://github.com/topics/artifact-management
  * (5) URL https://github.com/topics/data-management
  * (6) URL https://github.com/topics/experiments
  * (7) URL https://github.com/topics/experiment-management
  * (8) URL https://github.com/awesome-mlops/awesome-ml-experiment-management
  * (9) URL https://github.com/topics/experiment-tracking
  * (10) URL https://github.com/topics/lifecycle-management
  * (11) URL https://github.com/topics/data-management
  * (12) URL https://github.com/topics/project-management
  * (13) URL https://github.com/topics/workflow-management
  * (14) Agrawal, N., Bolosky, W.J., Douceur, J.R., Lorch, J.R.: A five-year study of file-system metadata.  ACM Transactions on Storage (TOS) 3(3), 9–es (2007)
  * (15) Aguilar Melgar, L., Dao, D., Gan, S., Gürel, N.M., Hollenstein, N., Jiang, J., Karlaš, B., Lemmin, T., Li, T., Li, Y., et al.: Ease. ml: a lifecycle management system for machine learning.  In: Proceedings of the Annual Conference on Innovative Data Systems Research (CIDR), 2021. CIDR (2021)
  * (16) Ahmed, S., Bagherzadeh, M.: What do concurrency developers ask about?: a large-scale study using stack overflow.  Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (2018)
  * (17) Alberti, M., Pondenkandath, V., Würsch, M., Ingold, R., Liwicki, M.: Deepdiva: a highly-functional python framework for reproducible experiments.  In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 423–428. IEEE (2018)