# Startup success prediction and VC portfolio simulation using CrunchBase data

Mark Potanin
<EMAIL_ADDRESS>
Andrey Chertok
<EMAIL_ADDRESS>
Konstantin Zorin
<EMAIL_ADDRESS>
Cyril Shtabtsovsky
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Predicting startup success presents a formidable challenge due to the
inherently volatile landscape of the entrepreneurial ecosystem. The advent of
extensive databases like Crunchbase jointly with available open data enables
the application of machine learning and artificial intelligence for more
accurate predictive analytics. This paper focuses on startups at their Series
B and Series C investment stages, aiming to predict key success milestones
such as achieving an Initial Public Offering (IPO), attaining unicorn status,
or executing a successful Merger and Acquisition (M&A). We introduce novel
deep learning model for predicting startup success, integrating a variety of
factors such as funding metrics, founder features, industry category. A
distinctive feature of our research is the use of a comprehensive backtesting
algorithm designed to simulate the venture capital investment process. This
simulation allows for a robust evaluation of our model’s performance against
historical data, providing actionable insights into its practical utility in
real-world investment contexts. Evaluating our model on Crunchbase’s, we
achieved a 14 times capital growth and successfully identified on B round
high-potential startups including Revolut, DigitalOcean, Klarna, Github and
others. Our empirical findings illuminate the importance of incorporating
diverse feature sets in enhancing the model’s predictive accuracy. In summary,
our work demonstrates the considerable promise of deep learning models and
alternative unstructured data in predicting startup success and sets the stage
for future advancements in this research area.

##  1 Introduction

The prediction of startup success is a crucial task for various stakeholders,
including investors, entrepreneurs, and policymakers, as it has significant
implications for resource allocation and decision-making. It is estimated that
approximately 90% of startups fail within their first five years, a failure
rate that has remained relatively constant over the past few decades, despite
considerable advancements in technology and business practices. Consequently,
the accurate prediction of startup success can assist investors in more
effectively allocating their resources and enable entrepreneurs to make
better-informed decisions.

Recently, the proliferation of data from sources such as Crunchbase has
intensified interest in the application of machine learning techniques for the
prediction of startup success. Machine learning models can harness various
types of data, encompassing funding history, market trends, team composition,
and social media activity, to identify patterns and generate predictions.

This study presents two distinct methodologies for predicting startup success:
a supervised deep learning approach leveraging multiple data sources, and a
ranking-based approach focusing on the identification of characteristics
common to successful startups and investors. The supervised approach entails
collecting and labeling data, constructing a prediction model, and evaluating
its performance. In contrast, the ranking-based approach centers on
identifying startups and investors that exhibit shared characteristics with
successful ones.

Our train dataset consists of 34,470 companies The primary novelty of this
research lies in the application of deep learning techniques and the
integration of heterogeneous input data types. A crucial feature of our
research is the simulation of fund operations based on historical data,
resulting in a projected 14x capital growth of the fund’s portfolio. As per
machine learning metrics, our model exhibits a robust 86% ROC_AUC.

The remainder of this paper is organized as follows: Section 2 reviews the
related works in the area of startup success prediction and machine learning.
Section 3 describes dataset collection, preprocessing, and feature selection.
Section 4 presents the experimental results of the supervised approach.
Section 5 describes some other ideas about company and investor scoring.
Finally, Sections 6 and 7 provide the conclusion of the study and discuss
prospective research avenues in this domain.

##  2 Related works

The application of AI in fintech has substantially transformed the financial
services industry over the past decades [1]. For example, one of the most
well-known applications is credit risk assessment [2]. Another challenging
task could be stock market prediction [3]. This paper focuses on startup
prediction and the VC market, and there is a growing literature on analyzing
investments using machine learning.

In the paper [4], authors present a machine learning model, CapitalVX, trained
on a large dataset obtained from Crunchbase, to predict the outcomes for
startups, i.e., whether they will exit successfully through an IPO or
acquisition, fail, or remain private. They investigated MLP, Random Forest,
XGBoost and used mostly numerical features from the dataset. In [5] paper,
authors conducted a review on existing machine learning techniques that are
recently contributed to understanding the need of start-ups, trends of
business and can provide recommendations to plan their future strategies to
deal with the business problems. The study conducted by [6] underscores the
potential of machine learning applications in the venture capital industry,
demonstrating its ability to predict various outcomes for early-stage
companies including subsequent funding rounds or closure.

In another study [7], authors use behavioral decision theory to compare the
investment returns of an algorithm with those of 255 business angels (BAs)
investing via an angel investment platform. The study found that, on average,
the algorithm achieved higher investment performance than the BAs. However,
experienced BAs who were able to suppress their cognitive biases could still
achieve best-in-class investment returns. This research presents novel
insights into the interplay of cognitive limitations, experience, and the use
of algorithms in early-stage investing. This study [8] proposes a data-driven
framework, wherein the model was trained on 600,000 companies across two
decades and 21 significant features.

This review [9] provides a thorough analysis of AI applications in Venture
Capital, categorizing influential factors on a company’s probability of
success or fund-raising into three clusters: team/personal characteristics,
financial considerations, and business features. In another study [10],
authors leveraged Crunchbase data from 213,171 companies to develop a machine
learning model to predict a company’s success. Despite limiting the number of
predictors, it achieved promising results in precision, recall, and F1 scores,
with the best outcomes from the gradient boosting classifier.

This study [11] explores the untapped potential of web-based open sources in
contrast to just structured data from the startup ecosystem. A significant
performance enhancement is demonstrated by incorporating web mentions of the
companies into a robust machine learning pipeline using gradient boosting.

This study [12] aims to assist VC firms and Angel investors in identifying
promising startups through rigorous evaluations, emphasizing the impact of
founder backgrounds and capital collected in seed and series stages. This very
recent paper published in 2023 [13] introduces a novel model for predicting
startup success that incorporates both internal conditions and industry
characteristics, addressing a gap in previous research that focused primarily
on internal factors. Using data from over 218,000 companies from Crunchbase
and six machine learning models, the authors found media exposure, monetary
funding, the level of industry convergence, and the level of industry
association to be key determinants of startup success.

In this study [14], authors analyze more than 187,000 tweets from 253 new
ventures’ Twitter accounts achieving up to 76% accuracy in discriminating
between failed and successful businesses . The research outlined in [15]
investigates the methodologies used by venture capitalists when evaluating
technology-based startups, using the influence of weak (Twitter sentiment) and
strong (patents) signals on venture valuations. Findings reveal that while
both signals positively associate with venture valuations, Twitter sentiment
fails to correlate with long-term investment success, unlike patents.
Furthermore, startup age and VC firm experience act as boundary conditions for
these signal-valuation relationships.

##  3 Dataset Overview, Preprocessing, and Features

We used daily Crunchbase database export (Daily CSV Export) as the primary
data source, which is also supported by a well-documented API. The main goal
of this research was to collect a labeled dataset for training a deep learning
model to classify companies as either successful or unsuccessful.

The analysis was based on the Daily CSV Export from 2022-06-14, and only
companies established on or after 2000-01-01 were taken into account. To
refine the focus of the research, only companies within specific categories
were included, such as Software, Internet Services, Hardware, Information
Technology, Media and Entertainment, Commerce and Shopping, Mobile, Data and
Analytics, Financial Services, Sales and Marketing, Apps, Advertising,
Artificial Intelligence, Professional Services, Privacy and Security, Video,
Content and Publishing, Design, Payments, Gaming, Messaging and
Telecommunications, Music and Audio, Platforms, Education, and Lending and
Investments.

This research is focused on investment rounds occurring after round B.
However, in the Crunchbase data glossary, rounds such as series_unknown,
private_equity, and undisclosed, possess unclear characteristics. To
incorporate them into the company’s funding round history, we only included
these ambiguous rounds if they occurred after round B.

###  3.1 Successful Companies Dataset

In this research, a company is deemed successful if it achieves one of three
outcomes: Initial Public Offering (IPO), Acquisition (ACQ), or Unicorn status
(UNIC), the latter being defined as a valuation exceeding $1 billion. To
assemble a list of successful companies, we initially filtered for IPOs with
valuations above $500M or funds raised over $100M, yielding 363 companies. For
acquisitions, we applied filters to eliminate companies with a purchase price
below the maximum amount of funds raised or under $100M, resulting in 833
companies. To select unicorns, we searched for companies with a valuation
above $1 billion, utilizing both Crunchbase data and an additional table of
verified unicorns, which led to a total of 1074 unicorns.

The final dataset contains a timeline of all crucial investment rounds leading
to the success event (i.e., achieving unicorn status, IPO, or ACQ), with the
index of this event specified in the success_round column. This approach
ensures that the dataset accurately represents the history and progress of
each successful company, facilitating effective analysis.

###  3.2 Unsuccessful Companies Dataset

To supply the model with examples of ’unsuccessful’ companies, we collected a
separate dataset. We excluded companies already present in the successful
companies dataset by removing those that had IPO, ACQ, or UNIC flags. We also
eliminated a considerable number of actual unicorns from the CrunchBase
website [16] to avoid overlap. We excluded companies that have not attracted
any rounds since 2016. Additionally, we excluded companies that are
subsidiaries or parent companies of other entities. Furthermore, we used the
jobs dataset to exclude companies that have hired employees since 2017.

Additionally, we applied extra filters to exclude companies with valuation
above $100 million, as they reside in the "gray area" of companies that may
not be clearly categorized as successful or unsuccessful. By applying these
filters, we constructed a dataset comprising 32,760 companies, denoted by the
label ’0’ for unsuccessful, and 1,989 companies, denoted by the label ’1’ for
successful.

###  3.3 Features

The feature space of the model includes:

#### Founders Features

Categorical: country_code, region, city, institution_name, degree_type,
subject.
Numerical: twitter_url, linkedin_url, facebook_url, gender, is_completed,
num_degrees, num_last_startups, num_last_jobs, number_of_founders.

We incorporated three binary flags into our model to represent the presence of
founders’ social media links. Since a company can have multiple founders, it
was essential to aggregate information on all the founders for each company.
For categorical variables, the most frequent value from the list was used, and
the median for numerical variables.

#### Investors Features

Categorical: type, country_code, region, city, investor_types.
Numerical: investment_count, total_funding_usd, twitter_url, linkedin_url,
facebook_url, raised_amount_usd, investor_count, num_full.

Functions that generate features based on the founders’ and investors’ data
incorporate a date parameter as input. This approach is necessary to the model
from using future information. For example, details about the number of
companies founded or the founder’s previous job experience that took place
after the date of interest should not be incorporated into the feature set to
avoid information leakage from the future.

#### Rounds features

Categorical: country_code, investment_type, region, city, investor_name.
Numerical: sum, mean, max of raised_amount_usd, investor_count,
post_money_valuation_usd.

It is crucial to emphasize that all features related to a company’s investment
rounds are gathered at a time point prior to the beginning of the time window
of interest.

#### Categories

There are two additional types of text data - text tags representing the
companies’ field of work. For example:

  * •

category_list: Internet, Social Media, Social Network

  * •

category_groups_list: Data and Analytics, Information Technology, Software

The set of tags used in our study consists of a list of keywords separated by
commas. We used the NMF (Non-Negative Matrix Factorization) matrix
factorization method to generate features from these tags. This process
involves creating a binary table with companies represented as rows and tags
as columns, where each value in the table indicates whether a given company is
associated with a specific tag (1) or not (0). The trained matrix
factorization then converts each binary vector into a smaller dimension vector
(in our case, 30).

All categorical features are encoded using the OrdinalEncoder, while numerical
features are normalized.

##  4 Model Training, Evaluation, and Portfolio Simulation

A representation of the model’s architecture is visualized in Figure 1.

Figure 1: Model architecture

###  4.1 Backtest

The backtest period during which we tested the model spans from 2016-01-01 to
2022-01-01, and the model was retrained every 3 months. Actually, it is a
hyperparameter that could be tuned depending on the time/accuracy trade-off.
In each iteration of the backtest, the time window under consideration is
defined by the start and end dates. For example, the first iteration considers
the window with a start date of 2016-01-01 and an end date of 2016-04-01.
Companies that attracted Round B or C are selected as the "test" set for this
window.

The model is trained on the dataset described in Section 3. However, the
entire dataset cannot be used for training since it would be incorrect to
train on companies founded in the future to predict the success of companies
in the past. Therefore, only companies founded before the start of the current
time window (i.e., before 2016-01-01 in the first iteration) are considered
for training. Additionally, the success of a company (IPO/ACQ/UNICORN) may
occur in the future relative to the current window. To train the model, only
companies with success event occurred before the start of the current time
window are considered.

This approach is designed to ensure the integrity of the backtesting process,
avoiding any influence from future events. However, the drawback of this
approach is the limited number of training examples at the beginning of the
backtest (i.e., in the first iterations in 2016-2017). Consequently, the
predictive power of the model is lower at the beginning of the backtest
compared to the end. The backtest yields an array of test companies with a
score assigned to them, indicating the level of success predicted by the
model.

The model is retrained every 3 months during the backtest, resulting in a
total of 25 prediction windows. A sorted list of predictions is generated for
each window. Finally, all predictions from all windows are compiled into one
table, representing the complete backtest of predictions for the period from
2016-01-01 to 2022-01-01. This table passes to the optimization algorithm.

A decision has been made to construct a monthly portfolio based on the
backtest results. Therefore, we can conduct the backtest with a window of 1
month, covering the periods from 2016-01-01 to 2016-02-01, from 2016-02-01 to
2016-03-01, and so on, by adding or removing companies in our portfolio every
month. Again, using the example of one period, for example from 2018-01-01 to
2018-02-01, let us describe the process of selecting companies to be included
in the portfolio. We take a slice of the backtest predictions in this period
and sort them by score, which represents the model’s assessment of the success
of each company. As the size of our portfolio is limited, for instance, to 30
companies, there is no need to fill it entirely in the first months. Thus, the
logic for adding companies is as follows:

  * •

In each month, we select the top 3 companies from the sorted list of
predictions. But, a cut-off threshold for the predicted score has also been
established. However, the choice of the optimal threshold is an empirical task
and requires careful consideration. With the augmentation of the training
dataset over time, the model becomes more confident in its predictions.
Therefore, it makes sense to increase the threshold when moving along the
backtest time. One way to do this is to set the threshold as a function that
takes into account the size of the train dataset and other relevant factors.

  * •

Every month we verify the current portfolio:

    * –

success: if the company has achieved a success event (IPO/ACQ/unicorn) during
the month, it is removed from the active portfolio and marked with this flag.

    * –

longtime: if the company has not attracted any rounds within the last 730 days
(2 years, a configurable parameter), it is removed from the portfolio and
marked with this flag.

    * –

still_in: if the company is still in the portfolio at the end of the backtest,
it is marked with this flag. These are companies that were recently added to
the portfolio (in 2021-2022) and for which we cannot yet make a decision on
their success.

The result is a dataset that simulates our venture fund during the period
2016-2022, and we collected (as well as filtered) companies in it every month.
The resulting dataset contains the following fields:

  * •

uuid - a unique company identifier

  * •

name - the name of the company

  * •

enter_series_date - the date of the round in which the fund entered the
company

  * •

enter_series_value - the valuation of the company at the time of entry (if
available)

  * •

score - the company score at the time of entry (if available)

  * •

added - the date when the company was added to the portfolio

  * •

last_series_date - the date of the last round of funding, which could be an
IPO, acquisition, or the round in which the company became a unicorn

  * •

last_series_value - the valuation of the company at the time of the last round
of funding, if available

  * •

exit_reason - the reason for the fund’s exit from the company (if applicable)

  * •

expired - the date when the fund exited the company (due to success or
expiration of the holding period)

The reader may wonder why we retrain the model every 3 months while building
the portfolio with a one-month interval. Essentially, at the beginning of the
training set, we include all companies until 2016-01-01. The test set consists
of companies that received rounds B or C funding during the period from
2016-01-01 to 2016-04-01. We make predictions and add them to the overall
table. Then, we expand the training data until 2016-04-01, and the test period
becomes from 2016-04-01 to 2016-07-01, and so on. In the end, we have a
complete test table covering the period from 2016-01-01 to 2022-01-01.

After that, we go through this table with a one-month step, simulating our
venture fund’s behavior and assembling the portfolio. The fact that we first
collect all predictions and then go through them to construct the portfolio is
simply a matter of optimization. We do not look into the future in any way.

###  4.2 Backtest settings

In this study, several experiments were conducted with different backtest
configurations, we called them earlybird and any. The earlybird configuration
exclusively permits entry for companies only in rounds B or C, while the any
configuration broadens the entry criteria to any round within the list of
series_b,series_c,series_d,series_e,series_f,
series_g,series_h,series_j,series_i, as long as they are within the considered
backtest window.

The choice of entry configuration depends on the stage at which we enter the
company. Similarly, the choice of exit configuration depends on when we decide
to exit the company based on its success event (IPO/ACQ/unicorn), as discussed
in Section 2.2. However, since the "unicorn" status can occur in the early
rounds, there is a question of which round to exit. Two approaches were
considered: using first approach we exit the company when the first success
event occurs, while using last approach we exit on the last success event,
analogous to "we sit until the end."

The main approach used in this study is the earlybird_last due to business
requirements. However, this approach has its drawbacks, such as the fact that
the company success flag becomes known later in time, resulting in a smaller
dataset size for training at the beginning of the backtest and a slightly
lower quality of the backtest compared to the earlybird_first approach.

###  4.3 Results

The primary output of the algorithm is the backtest Table 2, sorted by the
time the company was added to the portfolio. The table includes an exit_reason
column, which serves as the main metric for evaluating model quality on the
backtest. This column can take on the following values:

  * •

success: the company had a successful round (unicorn/acquisition/IPO), and we
exited

  * •

longtime: a negative case where we left the company because it didn’t have a
successful event and had no rounds for two years

  * •

STILL_IN: a gray area, mainly consisting of companies that were recently added
to the backtest

Hence, an optimal backtest is characterized by the maximum quantity of
successful companies and a minimal number of companies categorized as
longtime. Table 2 (earlybird_last) is the basic configuration based on
business requirements. We enter in the first rounds (B/C) and exit in the last
round. However, the model may not work very well at the beginning of the
backtest due to limited data for training. In the Table 3 (any_last)
configuration, we can observe a large number of known unicorns simply because
we allow the model to enter in later rounds.

###  4.4 Capital Growth

Traditional metrics utilized in machine learning may not be directly
transferable to the AI investor due to changes in data availability over time
and class imbalance in the dataset. Therefore, we assess the model’s
performance based on the presence of well-known companies in the resulting
portfolio and the financial growth of the companies. In this subsection, we
focus on the latter assessment.

To calculate the PnL of the success of companies, we need the company
valuation during entry and exit rounds. The valuation of companies that exited
due to longtime is set to zero. For companies marked as STILL_IN, we use their
last known valuation since they are the youngest companies in the portfolio.
The PnL is divided into realized and unrealized components. The unrealized PnL
illustrates the current cumulative valuation of the portfolio, incorporating
the presently known rounds, in contrast, the realized PnL denotes the
cumulative sum garnered by exiting thriving companies and consequent capital
growth. Results with exit reasons and valuations are presented in Table 2.
Unfortunately, we didn’t have valuation data for all companies. There is a
column ’Used in Capital Growth’ that shows whether the company was used to
calculate the PnL.

We present cumulative PnL and the current portfolio size over time in Figure
2, with a step size of 1 month. The sharp rise in the middle of 2021
corresponds to the exit from Revolut. The companies that remained in the
portfolio at the end of 2021 are all marked as STILL_IN. Overall, the PnL
graph shows a positive trend, indicating the financial growth of the portfolio
over time.

Figure 2: Capital growth

To evaluate the algorithm via conventional machine learning metrics, we employ
cross-validation for time-series analysis with a 1-year test window, spanning
the years from 2016 to 2022. Within this test window, we focus on companies
that secured B or C funding rounds during a given year and subsequently
achieved success. Furthermore, to ensure the integrity of our analysis, the
training dataset for each fold exclusively comprises companies whose success
or failure status was known prior to the commencement of the test window.
Standard binary classification metrics can be used to evaluate the performance
of the model, and Recall is of particular interest to us. The minimization of
False Negatives (FN) holds greater significance than that of False Positives
(FP) in order to circumvent the omission of successful companies. Finally, in
Table 1 we present metrics that have been averaged across 6 folds for a
comprehensive evaluation of our predictive model’s performance:

Table 1: Metrics Precision | Recall | ROC AUC | PR AUC
---|---|---|---
0.92 | 0.64 | 0.86 | 0.65

##  5 Other approaches

###  5.1 Investors ranking model

All investors could be scored in terms of frequency, amount, and field of
investments. Also, an investor could be an indicator of a company’s potential
failure or success. This scoring was carried out in three stages:

  1. 1.

Through an autoencoder model with several modalities, we created vector
representations for each investor

  2. 2.

According to experts’ estimates, we select a group of top investors, and
further create the centroid of this group in the vector space

  3. 3.

We rank investors according to distance from the centroid

An elevated score corresponds to a proximate alignment with top investors.
Results are presented in Table 5. If the lead investor of a company has a low
score, it could be an indicator that such a company should be excluded from
consideration.

Example: Company 14W has a score of 0.9 and invests in IT companies, incl.
unicorns (for example, European travel management startup TravelPerk).

###  5.2 Founders ranking model

According to some characteristics - the number of previous startups (founder,
co-founder), their area, success, etc. - we can also score founders. An
escalated score is indicative of a company’s enhanced credibility. The results
of these models can be used both for preliminary scoring of companies and as
independent features in other models. An example is presented in Table 5.

###  5.3 Unicorn recommendation model

It was revealed that the median time for a company to achieve the status of a
"unicorn" is 4-5 years. Thus, in this period of time, about half of the
unicorns have reached this status, moreover, the second half is waiting in the
wings in the near future. This model identifies nascent companies established
within this 4-5 year time frame, isolates ’unicorns’ within this subset,
scores entities bearing the greatest resemblance and subsequently generates a
list of the top 30 recommendations.

For 2016-2021 simulation run:

  * •

On Jan 1st of each year, a list of recommendations of potential unicorns is
formed.

  * •

Every month, in case of the announcement of a round (series_X), a company is
added to the portfolio if its valuation is below 1 billion and the round is
not too high.

  * •

Companies that have reached 2.5 billion or have not had rounds for 3 years are
removed from the portfolio.

As a result, at the end of the period, a portfolio of companies was formed.
The limitation in this context is the scarcity of information related to
post_money_valuation information. Further development: as new data become
available, building a more complex recommendation system. The results are
presented in Table 6.

##  6 Conclusion

Traditionally, venture capital investment decisions have largely been guided
by the investors’ intuition, experience, and market understanding. While these
elements remain significant, there’s a growing recognition that these
traditional approaches can be greatly enhanced by integrating data-driven
insights into the investment decision-making process.

Our paper comprehensively examines a predictive model for startups based on an
extensive dataset from CrunchBase. A meticulous review and analysis of the
available data were conducted, followed by the preparing of a dataset for
model training. Special attention was given to the selection of features which
include information about founders, investors, and funding rounds.

The article also underlines a thoughtfully designed backtest algorithm,
enabling a fair evaluation of the model’s behavior (and the simulation of a VC
fund based on it) from a historical perspective. Rigorous efforts were made to
avoid data leakage, ensuring training at any given point only utilized data
that would have been known at that time. Several configurations were explored
regarding the funding rounds at which the fund could invest in a company and
the timing of exits. The primary evaluative metrics were derived from a
backtest table (Table 2), which chronicles instances of company entries,
exits, and the corresponding success statuses. Utilizing additional data on
company valuations, we calculated the Capital Growth, illustrating the fund’s
impressive economic impact over time. To sum up, this work primarily focused
on the variety of input features, the integrity of the backtest, and the
realistic simulation of the portfolio from a historical perspective.
Additionally, we proffer a series of propositions aimed at enhancing the
existing model, primarily revolving around the access to supplementary data
repositories.

Within the highly competitive and dynamic investment environment, the
assimilation of data-driven decision-making practices transitions from being
an option to becoming a necessity. As such, venture capitalists that
effectively harness the potential of AI and machine learning will likely
secure a significant competitive advantage, positioning themselves for success
in the new era of venture capitalism.

##  7 Further Research

In terms of further work, a promising direction is the usage of different
sources of text data about companies, founders, and investors. This could
involve leveraging social media platforms such as Twitter and LinkedIn, as
well as parsing the websites of the companies themselves.

Additionally, it may be worth adjusting the foundation date filter to include
companies founded in 1995, rather than the current start date of 2000-01-01.
However, this could potentially result in an influx of companies from the
"dotcom bubble" period.

The current strict filters used to determine successful companies
(IPO/ACQ/UNICORN) could also be loosened to potentially capture more companies
in the "gray area" between success and failure.

Finally, it may be worth conducting experiments to determine the optimal
threshold value for adding companies to the portfolio, taking into account the
size of the portfolio.

These additional tasks can provide valuable insights and enhance the
effectiveness of the AI investor backtest model. Analyzing the presentation
materials, video interviews, and source code of software companies can provide
a better understanding of the company’s strategy, goals, and potential.
Developing information collection systems to automate this process can save
time and improve accuracy.

Evaluating the influence of macroeconomic elements and technological
trajectories on startups may facilitate the identification of potential risks
and opportunities. It can also aid in the development of exit strategies.
Additionally, analyzing competing studies can provide insights into the market
and competition, which can inform investment decisions.

## References

  * [1] Shamima Ahmed, Muneer M Alshater, Anis El Ammari, and Helmi Hammami.  Artificial intelligence and machine learning in finance: A bibliometric review.  Research in International Business and Finance, 61:101646, 2022\.
  * [2] Si Shi, Rita Tse, Wuman Luo, Stefano D’Addona, and Giovanni Pau.  Machine learning-driven credit risk: a systemic review.  Neural Computing and Applications, 34(17):14327–14339, 2022.
  * [3] Dale W Jorgenson, Martin L Weitzman, Yoshua X ZXhang, Yann M Haxo, and Ying X Mat.  Can neural networks predict stock market?  AC Investment Research Journal, 220(44), 2023.
  * [4] Greg Ross, Sanjiv Das, Daniel Sciro, and Hussain Raza.  Capitalvx: A machine learning model for startup selection and exit prediction.  The Journal of Finance and Data Science, 7:94–114, 2021.
  * [5] Ajai Mishra, Dharm Singh Jat, and Durgesh Kumar Mishra.  Machine intelligence for predicting new start-ups success: A survey.  In Proceedings of the International Conference on Data Science, Machine Learning and Artificial Intelligence, DSMLAI ’21’, page 99–105, New York, NY, USA, 2022. Association for Computing Machinery.
  * [6] Javier Arroyo, Francesco Corea, Guillermo Jimenez-Diaz, and Juan A Recio-Garcia.  Assessment of machine learning performance for decision support in venture capital investments.  Ieee Access, 7:124233–124243, 2019.
  * [7] Ivo Blohm, Torben Antretter, Charlotta Sirén, Dietmar Grichnik, and Joakim Wincent.  It’sa peoples game, isn’t it?! a comparison between the investment returns of business angels and machine learning algorithms.  Entrepreneurship Theory and Practice, 46(4):1054–1091, 2022.
  * [8] Francesco Corea, Giorgio Bertinetti, and Enrico Maria Cervellati.  Hacking the venture industry: An early-stage startups investment framework for data-driven investors.  Machine Learning with Applications, 5:100062, 2021.
  * [9] Francesco Corea and Francesco Corea.  Ai and venture capital.  An Introduction to Data: Everything You Need to Know About AI, Big Data and Data Science, pages 101–110, 2019.
  * [10] Kamil Żbikowski and Piotr Antosiuk.  A machine learning, bias-free approach for predicting business success using crunchbase data.  Information Processing & Management, 58(4):102555, 2021.
  * [11] Boris Sharchilev, Michael Roizner, Andrey Rumyantsev, Denis Ozornin, Pavel Serdyukov, and Maarten de Rijke.  Web-based startup success prediction.  In Proceedings of the 27th ACM international conference on information and knowledge management, pages 2283–2291, 2018.
  * [12] Jaiesh Singhal, Chinmayi Rane, Yash Wadalkar, Mohit Joshi, and Amol Deshpande.  Data driven analysis for startup investments for venture capitalists.  In 2022 International Conference for Advancement in Technology (ICONAT), pages 1–6. IEEE, 2022.
  * [13] Jongwoo Kim, Hongil Kim, and Youngjung Geum.  How to succeed in the market? predicting startup success using a machine learning approach.  Technological Forecasting and Social Change, 193:122614, 2023.
  * [14] Torben Antretter, Ivo Blohm, Dietmar Grichnik, and Joakim Wincent.  Predicting new venture survival: A twitter-based machine learning approach to measuring online legitimacy.  Journal of Business Venturing Insights, 11:e00109, 2019.
  * [15] Andranik Tumasjan, Reiner Braun, and Barbara Stolz.  Twitter sentiment as a weak signal in venture capital financing.  Journal of Business Venturing, 36(2):106062, 2021.
  * [16] Crunchbase.  Crunchbase Unicorn Company List.  https://news.crunchbase.com/unicorn-company-list/.  Accessed: <access date>.
  * [17] PitchBook.  https://pitchbook.com.

##  Appendix A Appendix A

Table 2: Earlybird Last Name | Added | Exit Reason | Expelled | Used in Capital Growth
---|---|---|---|---
Flatiron | 2016-02-01 | longtime | 2018-02-01 | TRUE
Klarna | 2016-02-01 | success | 2021-07-01 | TRUE
Looker | 2016-02-01 | success | 2019-07-01 | FALSE
Magic Leap | 2016-03-01 | success | 2021-11-01 | TRUE
Jumia Group | 2016-04-01 | longtime | 2018-04-01 | TRUE
Monedo | 2016-04-01 | longtime | 2019-01-01 | TRUE
Vectra | 2016-04-01 | success | 2021-05-01 | FALSE
Pivotal | 2016-06-01 | longtime | 2018-06-01 | TRUE
Barefoot Networks | 2016-07-01 | longtime | 2020-09-01 | FALSE
CommonBond | 2016-08-01 | longtime | 2020-04-01 | FALSE
GitHub | 2016-08-01 | success | 2018-07-01 | TRUE
Coinbase | 2016-08-01 | success | 2021-05-01 | TRUE
Auris Health | 2016-09-01 | success | 2019-03-01 | TRUE
Distil Networks | 2016-09-01 | longtime | 2018-09-01 | FALSE
Behalf | 2016-09-01 | STILL IN |  | FALSE
PillPack | 2017-01-01 | success | 2018-07-01 | FALSE
Pluralsight | 2017-01-01 | longtime | 2019-01-01 | TRUE
Exabeam | 2017-03-01 | success | 2021-06-01 | FALSE
Learner-Centered Collaborative | 2017-04-01 | longtime | 2020-05-01 | FALSE
Earnest Operations | 2017-04-01 | success | 2017-11-01 | FALSE
Aledade | 2017-04-01 | longtime | 2020-01-01 | TRUE
Qualtrics | 2017-05-01 | longtime | 2019-05-01 | TRUE
Cohesity | 2017-05-01 | success | 2020-05-01 | TRUE
Robinhood Markets | 2017-05-01 | success | 2021-08-01 | TRUE
Wandera | 2017-06-01 | longtime | 2019-06-01 | FALSE
OpenGov | 2017-06-01 | longtime | 2021-10-01 | FALSE
Symphony Communication Services | 2017-06-01 | success | 2019-07-01 | TRUE
Casper | 2017-07-01 | longtime | 2021-04-01 | TRUE
Segment | 2017-08-01 | success | 2020-11-01 | FALSE
Revolut | 2017-08-01 | success | 2021-08-01 | TRUE
Spire Global | 2017-12-01 | success | 2021-09-01 | FALSE
ClearStory Data | 2018-04-01 | longtime | 2020-04-01 | TRUE
Helix | 2018-04-01 | longtime | 2020-03-01 | FALSE
Aircall | 2018-06-01 | success | 2021-07-01 | FALSE
Zoox | 2018-08-01 | success | 2020-07-01 | TRUE
Yesware | 2018-09-01 | longtime | 2020-09-01 | FALSE
import.io | 2019-01-01 | longtime | 2021-01-01 | FALSE
Fair | 2019-01-01 | longtime | 2021-01-01 | TRUE
Petal | 2019-02-01 | STILL IN |  | TRUE
Bolt | 2019-08-01 | STILL IN |  | FALSE
Ginger | 2019-10-01 | success | 2021-09-01 | TRUE
Blue Planet-works Japan | 2019-10-01 | longtime | 2021-10-01 | FALSE
Porch Group | 2020-02-01 | success | 2021-01-01 | TRUE
OneTrust | 2020-03-01 | success | 2021-05-01 | TRUE
Pony.ai | 2020-03-01 | STILL IN |  | TRUE
Bakkt | 2020-04-01 | success | 2021-11-01 | TRUE
DigitalOcean | 2020-06-01 | success | 2021-04-01 | TRUE
Wandelbots | 2020-07-01 | STILL IN |  | FALSE
Withings | 2020-08-01 | STILL IN |  | FALSE
Northvolt | 2020-10-01 | success | 2021-07-01 | TRUE
WireWheel | 2021-03-01 | STILL IN |  | TRUE
Skycatch | 2021-04-01 | STILL IN |  | FALSE
Bitstocks Trading Limited | 2021-05-01 | STILL IN |  | TRUE
Placer.ai | 2021-05-01 | STILL IN |  | TRUE
Grover | 2021-05-01 | STILL IN |  | TRUE
Genies | 2021-06-01 | STILL IN |  | TRUE
Heyday | 2021-06-01 | STILL IN |  | TRUE
Pipe | 2021-06-01 | success | 2022-01-01 | TRUE
Lendbuzz | 2021-07-01 | STILL IN |  | FALSE
ShipHawk | 2021-07-01 | STILL IN |  | FALSE
DuckDuckGo | 2021-07-01 | STILL IN |  | TRUE
Railsbank | 2021-08-01 | STILL IN |  | TRUE
Ripio | 2021-10-01 | STILL IN |  | TRUE
Jeeves | 2021-10-01 | STILL IN |  | TRUE
slice | 2021-12-01 | STILL IN |  | TRUE
Terra | 2022-01-01 | STILL IN |  | FALSE
Table 3: Any Last Name | Enter Series | Added | Exit Reason | Expelled
---|---|---|---|---
Klarna | series_c | 2016-02-01 | success | 2021-07-01
Snapdeal | series_j | 2016-02-01 | longtime | 2019-06-01
Magic Leap | series_c | 2016-03-01 | success | 2021-11-01
Carbon Black | series_f | 2016-03-01 | longtime | 2018-03-01
Domo | series_d | 2016-04-01 | success | 2018-07-01
Monedo | series_c | 2016-04-01 | longtime | 2019-01-01
Airbnb | series_e | 2016-05-01 | success | 2021-01-01
Slack | series_f | 2016-05-01 | longtime | 2020-09-01
Snap | series_f | 2016-06-01 | success | 2017-04-01
Pivotal | series_c | 2016-06-01 | longtime | 2018-06-01
Anki | series_d | 2016-07-01 | longtime | 2019-04-01
Uber | series_g | 2016-07-01 | success | 2019-06-01
GitHub | series_b | 2016-08-01 | success | 2018-07-01
Behalf | series_c | 2016-09-01 | STILL IN |
Kaltura | series_f | 2016-09-01 | longtime | 2018-09-01
Sckipio Technologies | series_c | 2016-10-01 | longtime | 2020-06-01
DraftKings | series_e | 2016-10-01 | success | 2020-05-01
Paxata | series_d | 2016-11-01 | longtime | 2018-11-01
BuzzFeed | series_g | 2016-12-01 | longtime | 2018-12-01
Stripe | series_d | 2016-12-01 | success | 2021-04-01
Pluralsight | series_c | 2017-01-01 | longtime | 2019-01-01
Nexenta Systems | series_g | 2017-01-01 | longtime | 2019-01-01
Funding Circle | series_f | 2017-02-01 | success | 2018-10-01
InsideSales | series_d | 2017-02-01 | longtime | 2019-02-01
SoFi | series_f | 2017-03-01 | success | 2021-06-01
Comprehend Systems | series_c | 2017-03-01 | longtime | 2021-03-01
Earnest | series_c | 2017-04-01 | success | 2017-11-01
Instacart | series_d | 2017-04-01 | success | 2021-04-01
Flipkart | series_j | 2017-05-01 | longtime | 2019-09-01
Lyft | series_g | 2017-05-01 | success | 2019-04-01
Wish | series_f | 2017-06-01 | success | 2021-01-01
The Honest Company | series_e | 2017-11-01 | longtime | 2020-07-01
Warby Parker | series_e | 2018-04-01 | success | 2021-10-01
Nextdoor | series_g | 2018-06-01 | longtime | 2021-10-01
CloudBees | series_e | 2018-07-01 | longtime | 2020-07-01
Dataminr | series_e | 2018-07-01 | longtime | 2020-07-01
Numerify | series_d | 2018-11-01 | longtime | 2020-11-01
import.io | series_b | 2019-01-01 | longtime | 2021-01-01
Fair | series_b | 2019-01-01 | longtime | 2021-01-01
Petal | series_b | 2019-02-01 | STILL IN |
Ola | series_j | 2019-02-01 | STILL IN |
Percolate | series_d | 2019-04-01 | longtime | 2021-04-01
Cloudflare | series_e | 2019-04-01 | success | 2019-10-01
Optimizely | series_d | 2019-07-01 | success | 2020-10-01
Dstillery | series_d | 2019-07-01 | longtime | 2021-07-01
Thumbtack | series_f | 2019-08-01 | success | 2021-07-01
Zocdoc | series_d | 2019-08-01 | STILL IN |
One97 | series_g | 2019-12-01 | success | 2021-12-01
ATLAS CAPITAL GROUP | series_b | 2020-02-01 | STILL IN |
OneTrust | series_b | 2020-03-01 | success | 2021-05-01
DigitalOcean | series_c | 2020-06-01 | success | 2021-04-01
Wandelbots | series_b | 2020-07-01 | STILL IN |
Northvolt | series_b | 2020-10-01 | success | 2021-07-01
Splashtop | series_e | 2021-02-01 | STILL IN |
Bitstocks Trading Limited | series_c | 2021-05-01 | STILL IN |
Pipe | series_b | 2021-06-01 | success | 2022-01-01
Ripio | series_b | 2021-10-01 | STILL IN |
Delphix | series_d | 2021-11-01 | STILL IN |

Table 4: Investors scoring Name | Score
---|---
14W | 0.900773
Mubadala Capital Ventures | 0.898600
CITIC Capital Holdings | 0.898527
S3 Ventures | 0.898228
Chrysalix Venture Capital | 0.897628
BlueCross BlueShield Venture Partners | 0.897500
Foxconn Technology Group | 0.897093
VNV Global | 0.896948
Camden Partners | 0.896848
GrandBanks Capital | 0.896780
Rakuten | 0.896531
Lumira Ventures | 0.894433
Openspace | 0.894398
BRM Capital | 0.894394
VenturEast | 0.894311
Harmony Partners | 0.893963
Capital Today | 0.893536
Eastward Capital Partners | 0.893342
Takeda Ventures | 0.893278
Clarus Ventures | 0.893153

Table 5: Founders scoring Name | Score
---|---
Adam Neumann | 1.000000
David Fano | 0.997524
Abhi Maruvada | 0.985468
Emanuele Pagani | 0.981734
Ryan Norton | 0.979890
… | …
Dr. Jonas Sundqvist | 0.089640
Onno Huyser | 0.084219
Stéphane Akaya | 0.084219
Mark Van Wyk | 0.084219
Pit Zens | 0.000000

Table 6: Unicorns recommendation model Name | Enter Series | Last Series Value
---|---|---
Coursera | series_c | 2 500 000 000
H2O.ai | series_b | 1 700 000 000
Freee | series_c |
Cybereason | series_c | 3 100 000 000
Honest Buildings | series_b |
Reserve | series_unknown |
Qumulo | series_c | 1 200 000 000
Silent Circle | series_c |
Vlocity | series_b | 1 000 000 000
Aura Financial | series_c |
MoneyLion | series_a |
Databricks | series_c | 2 750 000 000
OakNorth | series_b | 2 800 000 000
ProtectWise | series_b |
Earnest | series_c |
Cohesity | secondary_market | 2 500 000 000
Spring | series_c |
Darktrace | series_d | 1 650 000 000
Bread | series_unknown |
Guideline | series_c | 1 150 000 000
Nexxiot | series_b |
Cato Networks | series_c | 2 500 000 000
CloudMinds | series_b |
Stash Financial | series_e | 1 400 000 000
Branch International | series_c |
Fungible | series_c | 700 000 000
BlueVoyant | series_b |
Reali | series_b |
AMD Pensando | series_c |
IntSights | series_d |
BigID | series_c | 1 250 000 000
Flutterwave | series_b | 1 000 000 000
K Health | series_c | 1 400 000 000
Doma | series_c |
BlockFi | series_a |
Tomorrow.io | series_c |
Kin Insurance | series_b |
TrueLayer | series_c | 1 000 000 000
bitkey Japan | series_unknown |
MegazoneCloud | series_b |


# Measurement of charged-particle jet properties in p–Pb collisions at
$\sqrt{s_{\rm NN}}=5.02$ TeV with ALICE

Prottoy Das on behalf of ALICE Collaboration  Department of Physics, Bose
Institute, Kolkata 700091, West Bengal, India
<EMAIL_ADDRESS>
###### Abstract

We present the measurement of charged-particle jet properties in minimum bias
p–Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV in ALICE. Jets are
reconstructed from charged particles at midrapidity using the anti-$k_{\rm T}$
jet finding algorithm with jet resolution parameter $R$ = 0.4. The mean
charged particle multiplicity and jet fragmentation function for leading
charged-particle jets in the $p_{\rm T}$ interval 10 $<p_{\rm T,jet}^{\rm
ch}<$ 100 GeV/c are measured and compared with theoretical model predictions.

###### keywords:

Charged-particle jet; mean jet-constituent multiplicity; jet fragmentation
function.

##  1 Introduction

Jets are collimated sprays of particles produced from the fragmentation and
hadronization of hard-scattered partons in high-energy hadronic and nuclear
collisions. Jet properties are sensitive to the details of parton showering
process and are expected to be modified in the presence of a dense partonic
medium. Measurements of intra-jet properties in p–Pb collisions are useful to
investigate cold nuclear matter effects [1] and enrich our current
understanding of particle production in such collision systems. In this work,
we present the measurement of charged-particle jet properties, such as the
mean charged particle multiplicity and fragmentation functions, for leading
jets in the $p_{\rm T}$ interval 10 $<p_{\rm T,jet}^{\rm ch}<$ 100 GeV/c at
midrapidity in p–Pb collisions at a center of mass energy per nucleon-nucleon
pair $\sqrt{s_{\rm NN}}$ = 5.02 TeV with ALICE. Results are compared with
theoretical model predictions.

##  2 Analysis details

The data presented here were collected with the ALICE apparatus in 2016.
Detail information about the ALICE detector can be found in Ref. [2]. Events
are selected for this analysis using a minimum bias trigger condition which
requires the coincidence of signals in the V0A and V0C forward scintillator
arrays [3]. Only events with a primary vertex within 10 cm from the nominal
interaction point along the beam direction ($|z_{\rm vertex}|$=0) are
considered resulting into 5.15$\times$108 events. Charged particles
reconstructed with the Inner Tracking System (ITS) [4] and the Time Projection
Chamber (TPC) [5] are used for the reconstruction of the primary vertex and
jets. These detectors are placed inside a large solenoidal magnet that
provides a homogeneous magnetic field $B$ = 0.5 T.

Charged tracks with $p_{\rm T}>$ 0.15 GeV/$c$ within a pseudorapidity range
$|\eta|<$ 0.9 over the full azimuth are accepted. Charged-particle jets are
reconstructed from the selected tracks using anti-$k_{\rm T}$ jet finding
algorithm [6] with the $p_{\rm T}$-recombination scheme of FastJet package [7]
with jet resolution parameter, $R$ = 0.4. Only leading jets (jet of highest
$p_{\rm T}$ in an event) with 10 $<p_{\rm T,jet}^{\rm ch}<$ 100 GeV/c are
considered for this analysis. The mean charged particle multiplicity in
leading jet ($\langle N_{\rm ch}\rangle$) and leading-jet fragmentation
function ($z^{\rm ch}=p_{\rm T,track}/p_{\rm T,jet}^{\rm ch}$, where $p_{\rm
T,track}$ is the $p_{\rm T}$ of jet constituent inside the leading-jet cone)
are studied as a function of jet $p_{\rm T}$.

Contribution from the underlying event (UE; coming from sources other than the
hard scattered partons) is estimated using the perpendicular-cone method and
subtracted on a statistical basis after unfolding as reported in Fig. 1. In
the perpendicular-cone method, circular cones of radius $R$ = 0.4 at the same
$\eta$ as the leading jet and perpendicular to the leading jet axis are used
for the estimation of the contribution from the UE.

Figure 1: Correction procedure to subtract UE contributions for $\langle
N_{\rm ch}\rangle$ and $z^{\rm ch}$ (40 $<p_{\rm T,jet}^{\rm ch}<$ 60 GeV/c):
Unfolded distributions without UE subtraction (left), UE contribution (middle)
and after UE subtraction (right).

A 2-dimensional Bayesian unfolding technique [8] (implemented in RooUnfold [9]
package) is applied to correct for instrumental effects such as track-
reconstruction efficiency and momentum resolution. For each of the jet
observables, a 4D response matrix is constructed from a Monte Carlo (MC)
simulation performed with the DPMJET [10] event generator and the generated
particles are transported through the GEANT detector simulation. DPMJET is a
multipurpose generator based on the Dual Parton Model (DPM) and is able to
simulate a wide variety of collision systems for energies ranging from a few
GeV up to the highest cosmic-ray energies. To construct the response matrix,
the detector- and particle-level jets are matched geometrically and only the
leading detector-level jet and the corresponding matched particle-level jet in
an event are considered.

Systematic uncertainties from various sources such as tracking efficiency,
modelling of the jet properties and the detector response in the MC
simulation, choice of regularization parameter or number of iterations in
Bayesian unfolding, change in prior distribution, and underlying event
contribution are estimated and added in quadrature to calculate the total
systematic uncertainty. The total systematic uncertainty is found to be 5% -
12% in the $\langle N_{\rm ch}\rangle$ analysis whereas in the $z^{\rm ch}$
analysis, it is estimated to be $\sim$20% for 20 $<p_{\rm T,jet}^{\rm ch}<$ 30
GeV/c and 25% - 45% for 40 $<p_{\rm T,jet}^{\rm ch}<$ 60 GeV/c.

##  3 Results and discussion

Figure 2(a) (top panel) shows the $\langle N_{\rm ch}\rangle$ distribution as
a function of leading jet $p_{\rm T,jet}^{\rm ch}$. The blue markers represent
the data and the green curve shows the DPMJET (GRV94 [11]) prediction. The
ratio between the data and DPMJET is depicted in the bottom panel of Figure
2(a). It is observed that $\langle N_{\rm ch}\rangle$ increases with $p_{\rm
T,jet}^{\rm ch}$ and DPMJET reproduces the data for $p_{\rm T,jet}^{\rm ch}>$
30 GeV/$c$ within uncertainties.

(a) Top panel: Unfolded and UE subtracted $\langle N_{\rm ch}\rangle$
distribution as a function of leading jet $p_{\rm T,jet}^{\rm ch}$. Bottom
panel: ratio between the data and MC results.

(b) Top panel: Unfolded and UE subtracted $z^{\rm ch}$ distribution for
leading jets with 20 $<p_{\rm T,jet}^{\rm ch}<$ 30 GeV/c. Bottom panel: ratio
between the data and MC results.

(c) Top panel: Unfolded and UE subtracted $z^{\rm ch}$ distribution for
leading jets with 40 $<p_{\rm T,jet}^{\rm ch}<$ 60 GeV/c. Bottom panel: ratio
between the data and MC results.

(d) Unfolded and UE subtracted $z^{\rm ch}$ distributions for leading jets
with 20 $<p_{\rm T,jet}^{\rm ch}<$ 30 GeV/c and 40 $<p_{\rm T,jet}^{\rm ch}<$
60 GeV/c. Scaling of jet fragmentation with $p_{\rm T,jet}^{\rm ch}$ is
observed within systematic uncertainties.

Figure 2:

Figures 2(b) and 2(c) (top panels) show the $z^{\rm ch}$ distributions for
leading jets with 20 $<p_{\rm T,jet}^{\rm ch}<$ 30 GeV/c and 40 $<p_{\rm
T,jet}^{\rm ch}<$ 60 GeV/c respectively. Results are compared with the DPMJET
(GRV94) predictions. The ratios between the data and DPMJET predictions are
shown in the bottom panels. It is found that DPMJET reproduces the $z^{\rm
ch}$ distributions in both $p_{\rm T,jet}^{\rm ch}$ ranges within systematic
uncertainties. The comparison of the $z^{\rm ch}$ distributions in the
intervals 20 $<p_{\rm T,jet}^{\rm ch}<$ 30 GeV/c and 40 $<p_{\rm T,jet}^{\rm
ch}<$ 60 GeV/c shown in Figure 2(d) indicate that the $z^{\rm ch}$
distribution follows a scaling behaviour with $p_{\rm T,jet}^{\rm ch}$ within
systematic uncertainties.

##  4 Summary

We have presented the measurement of charged-particle jet properties in
minimum bias p–Pb collisions at 5.02 TeV in ALICE. Results are compared with
DPMJET (GRV94) predictions, which reproduces both the measured distributions
($\langle N_{\rm ch}\rangle$ and $z^{\rm ch}$) within uncertainties except for
$\langle N_{\rm ch}\rangle$ at very low $p_{\rm T,jet}^{\rm ch}$ ($<30$
GeV/$c$). A scaling of jet fragmentation with leading charged-particle jet
$p_{\rm T}$ is observed within systematic uncertainties.

## Acknowledgements

Prottoy Das would like to acknowledge the Institutional Fellowship of Bose
Institute, the ALICE project grant [SR/MF/PS-02/2021-BI (E-37125)] and the
computing server facility at Bose Institute.

## References

  * [1] Armesto, Néstor, EPJ Web Conf. 171 11001 (2018).
  * [2] ALICE (B. B. Abelev et al.), Int. J. Mod. Phys. A 29, 1430044 (2014).
  * [3] ALICE (P. Cortese et al.), tech. rep., 9, 2004.
  * [4] ALICE (K. Aamodt et al.), JINST 5 (2010) P03003.
  * [5] ALICE (J. Alme et al.), Nucl. Instrum. Meth. A622 (2010) 316–367.
  * [6] Matteo Cacciari et. al., JHEP04(2008)063.
  * [7] Matteo Cacciari et. al., Eur.Phys.J.C 72 (2012) 1896.
  * [8] G. D’Agostini, Nucl. Instrum. Meth. A 362, 487-498 (1995).
  * [9] RooUnfold, https://gitlab.cern.ch/RooUnfold/RooUnfold.
  * [10] Roesler et al., https://doi.org/10.1007/978-3-642-18211-2_166.
  * [11] A. Vogt (DESY), https://doi.org/10.48550/arXiv.hep-ph/9507241.


# An Overview of Challenges in Egocentric Text-Video Retrieval

Burak Satar1,2
Hongyuan Zhu1
Hanwang Zhang2
Joo Hwee Lim1,2 1Institute for Infocomm Research, A*STAR, Singapore  2School
of Computer Science and Engineering, NTU, Singapore
{burak_satar, zhuh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Text-video retrieval contains various challenges, including biases coming from
diverse sources. We highlight some of them supported by illustrations to open
a discussion. Besides, we address one of the biases, frame length bias, with a
simple method which brings a very incremental but promising increase. We
conclude with future directions.

##  1 Introduction

Text-video retrieval aims to retrieve relevant videos given a textual query or
vice versa. Recent studies [3, 5, 14, 15, 1, 13, 10] show significant advances
either by using exocentric datasets [11] or egocentric datasets [4, 6]. Many
works address challenges for various text-video-related tasks [18, 17, 9, 2,
7] including some biases that can affect their results. However, to the best
of our knowledge, no work has yet to be dedicated to addressing the biases in
the text-video retrieval task.

Using a baseline work, we share insights specifically for the text-video
retrieval task. The first is on biases coming from static visual features, the
object’s size or the camera’s angle, and a possible effect from the
punctuation in the textual query. The second one is ambiguity, especially in
the visual features. The last one is the frame length bias and a preliminary
method and experiment. The contribution of this extended abstract is twofold.
1) We highlight various biases to bring discussion and awareness. 2) We share
a baseline approach and the result for one of the biases.

##  2 Challenges to Report

We report possible challenges we face during our previous and ongoing works,
especially on the Epic-Kitchen-100 dataset. These include various specific
examples after examining the dataset and the possible effects of the biases on
the retrieved items. While we only report the first two challenges, we suggest
a baseline solution for the third.

###  2.1 Textual Bias or Visual Bias?

In the first example, we examine a sample in the text-to-video retrieval task
as shown in Figure 1. We find out that ’open fridge.’ query, fails severely to
retrieve the Ground Truth (GT) video, ranking it over 1,000. When we check the
first ten retrieved videos, their corresponding captions are all about ’open
cupboard’. After we examine the other queries, which include (open, fridge) as
a pair of verbs and nouns, the results are pretty good. Thus, one may say that
the dot ’.’ punctuation would bring a bias.

While the punctuation would affect the model, another possibility may arise
after checking the visual samples. For instance, only in the ’open fridge.’
example, the refrigerator is small-size based on the visual frames. In
contrast, the fridge is normal-size in all the other examples mentioned (open,
fridge). However, why does it even matter? Because there are many visual
similarities between the samples from the cupboards and the small-size
fridges, such as size similarity and spatial location similarity that stay on
the lower side of the kitchen, which also makes the angle of the head-mounted
camera similar. Basically, the visual similarity of the small-size fridge with
the cupboards and dissimilarity with the other fridges may affect the
retrieved clips.

Lastly, we may share one more observation that the variation in the verb or
noun does not affect the model’s performance. When the textual queries are
’opening the fridge’, ’open the fridge’, ’opening fridge’ and ’open fridge’,
the results are pleasing. Nevertheless, we note that the fridges are all
normal-size in these samples.

|  |
---|---|---
|  |
(a) caption: ’open fridge.’ | (b) caption: ’open cupboard’ | (c) caption: ’opening the fridge’
It fails to retrieve GT clip | Visual similarity with (a) | Visual dissimilarity with (a)
Figure 1:  Textual queries and samples from the retrieved clips for a possible
bias from the textual and visual side. While the punctuation in the first
sample could bring a bias, visual discrepancy inside the ’fridge’ noun class
may affect the model. For instance, the fridge is small-size, only in one
clip.

###  2.2 Ambiguity at Feature Set

|  |  |
---|---|---|---
(a) Sampled frame, 1st. | (b) Sampled frame, 2nd. | (c) Sampled frame, 24th. | (d) Sampled frame, 25th.
Figure 2: Various sampled frames from the caption: ’pick up rubbish’ could
bring ambiguity. For instance, the first two and last two frames may suggest
the action of ’take knife’ and ’clean hand’. By using the same query, our
baseline method retrieves a video clip regarding ’take knife’ at fourth rank.

Another example is that when we retrieve various clips based on a textual
query, we find various clips that are completely irrelevant to the text, as
shown in Figure 2. While some of them may be explained by a bias, as shown in
the next part, others happen due to the ambiguity in the visual feature. For
instance, when checking uniformly sampled frames of the action as a pair of
(verb, noun), we see that the first and last two sampled frames could be
irrelevant to the action itself. Then, it also affects the result after
examining the retrieved video clips. Considering a caption of ’pick up
rubbish’, four sampled frames out of twenty-five contain irrelevant activities
to the GT caption, which could be seen as ’take knife’, ’put knife back’ or
’clean hand’. As a result, the model fails to rank the GT video clip and ranks
it over 300. However, more importantly, a video clip with its corresponding
caption ’take knife’ rank at 4, showing that this ambiguity may affect the
retrieved clips.

###  2.3 Frame Length Discrepancy

We report another bias that comes from the train and test set discrepancy in
the dataset, as shown in Figure 3(a). The same trend can be seen after
examining the classes between the train and the test set. We refer to the
class as a semantic pair of a (verb, noun). Our observations show that the
problem of the length bias in model training could harm retrieval performance
on test sets with length discrepancy. More specifically, frame length
discrepancy between training and test sets of trimmed video clips causes non-
relevant video clips to be retrieved just because their frame length is
similar to the training class’s average frame length.

##  3 Method for Frame Length Bias

We propose a simple method to mitigate the bias by removing short and long
clips of certain classes to lower the discrepancy between the train and test
sets. A naive method would delete video clips only for one video class and
then train the model. After repeating the same approach for classes, all the
similarity matrices would be summed up.

Another simple method would be deleting long and short video clips in a class
while repeating this for every class by following two basic rules. 1) lower
the discrepancy to a pre-set margin $\alpha$ 2) the number of video clips for
a class cannot be less than a certain number $\beta$ to ensure having enough
samples for training. Once the discrepancy gets lower, we do the training.

(a) The histogram shows the discrepancy of a caption: ’pick up rubbish’. The
GT recall is at 397th rank. The same disparity occurs among many classes
(semantic pairs of verb and noun) between the train/test set.

(b) Average frame length comparison between train and test set showing that
clips in the test set are longer than the training set,

Figure 3: Bias Verification on Epic-Kitchens-100.

##  4 Experiments

Implementation details. We follow a baseline method [3, 12] that contains a
text encoder, a video encoder and a text-video matching module. The pre-set
margin $\alpha$ is chosen as 60 to reduce the discrepancy for every class in
our simple method. For $\beta$, we decide that a class, meaning a semantic
pair of verbs and nouns, should include more than ten samples. Specifically,
we delete 2,392 clips from 164 classes, equivalent to 3.6% of all the data.

Dataset. We use Epic-Kitchen-100 [4] for our experiments, which is an
egocentric dataset capturing daily kitchen actions. We utilize uniformly
sampled features shared by [8].

Evaluation metrics. We use recall and mean average precision (mAP) as usual.
Besides, nDCG (normalized discounted cumulative gain) is also reported by
following [16]. The details about the calculation of nDCG via relevancy matrix
can be found here[4, 16].

Results. Table 1 shows the effect of the first simple method on an individual
sample. Although this is impractical to apply for every class, we share the
results since the result also re-verify the bias. It indicates that if the gap
between the training and test set regarding average frame length decreases, 1)
the average frame length of the top 20 retrieved video clips increases, and 2)
the ground truth value at recall improves. On the other hand, Table 2 shows
that we slightly overpass the baseline when we follow our second simple
method.

| Avg frame length |  |  |
---|---|---|---|---
Method | Train set | Test set |  # of samples |  GT rank recall |  Avg frame length of the top 20 retrieval
| Baseline +
---
Remove long clips
36.26 | 287.93 | 57 | 342 | 167.82
Baseline | 198.06 | 287.93 | 88 | 148 | 203.59
| Baseline +
---
Remove short clips
286.05 | 287.93 | 57 | 58 | 224.22

Table 1: Caption: ‘put down mozzarella’ for the first naive method. Red colour
refers to directly proportional, while blue is inversely proportional.

Method | Epic Kitchen
---|---
nDCG (avg) | mAP (avg)
Baseline | 39.15 | 38.54
Baseline + Ours | 39.44 | 38.67

Table 2: Comparison between baseline and our second simple method. Avg refers
to the average between the text-to-video and the video-to-text score of nDCG.

##  5 Conclusion

We see tremendous progress in the field of text-video-related tasks. However,
certain biases may still stay in the loop. We report the recent challenges we
face with the motivation of bringing awareness and a discussion. Plus, we
suggest a naive approach to address one of them. Future work. 1) It may be
challenging to disentangle the biases coming from both the visual and textual
sides. We suggest doing deep-dive work on the effect of punctuations or
spatial bias, including the size of the object and the angle of the camera
would shed light. 2) To reduce ambiguity, feature purification can be applied
with the guidance of the GT captions and an off-the-shelf action recognition
tool. 3) We aim to address the frame length bias more effectively in our
ongoing work.

## References

  * [1] Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), page 1708–1718, Montreal, QC, Canada, Oct 2021. IEEE.
  * [2] Sofia Broomé, Ernest Pokropek, Boyu Li, and Hedvig Kjellström.  Recur, attend or convolve? on whether temporal modeling matters for cross-domain robustness in action recognition.  In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4188–4198, 2023.
  * [3] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10635–10644, 2020.
  * [4] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  International Journal of Computer Vision (IJCV), 130:33–55, 2022\.
  * [5] Alex Falcon, Giuseppe Serra, and Oswald Lanz.  Learning video retrieval models with relevance-aware online mining.  In International Conference on Image Analysis and Processing, pages 182–194. Springer, 2022.
  * [6] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik.  Ego4d: Around the world in 3,000 hours of egocentric video.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 18973–18990, New Orleans, LA, USA, Jun 2022. IEEE.
  * [7] Kensho Hara, Yuchi Ishikawa, and Hirokatsu Kataoka.  Rethinking training data for mitigating representation biases in action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3349–3353, June 2021.
  * [8] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  (arXiv:1908.08498), Aug 2019.  arXiv:1908.08498 [cs].
  * [9] Jie Lei, Tamara L. Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning, 2022.
  * [10] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou.  Egocentric video-language pretraining.  In NeurIPS. arXiv, Oct 2022.  arXiv:2206.01670 [cs].
  * [11] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, 2019.
  * [12] Burak Satar, Zhu Hongyuan, Xavier Bresson, and Joo Hwee Lim.  Semantic role aware correlation transformer for text to video retrieval.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 1334–1338, 2021.
  * [13] Burak Satar, Hongyuan Zhu, Hanwang Zhang, and Joo Hwee Lim.  Exploiting semantic role contextualized video features for multi-instance text-video retrieval epic-kitchens-100 multi-instance retrieval challenge 2022, 2022.
  * [14] Burak Satar, Hongyuan Zhu, Hanwang Zhang, and Joo Hwee Lim.  Rome: Role-aware mixture-of-expert transformer for text-to-video retrieval, 2022.
  * [15] Michael Wray, Gabriela Csurka, Diane Larlus, and Dima Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), page 450–459, Seoul, Korea (South), Oct 2019. IEEE.
  * [16] Michael Wray, Hazel Doughty, and Dima Damen.  On semantic similarity in video retrieval.  In CVPR, 2021.
  * [17] Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua.  Deconfounded video moment retrieval with causal intervention.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 1–10. Association for Computing Machinery, 2021.
  * [18] Sunjae Yoon, Ji Woo Hong, Eunseop Yoon, Dahyun Kim, Junyeong Kim, Hee Suk Yoon, and Chang D. Yoo.  Selective query-guided debiasing for video corpus moment retrieval.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, page 185–200, Cham, 2022. Springer Nature Switzerland.