# Improving One-Shot Transmission in NR Sidelink Resource Allocation for V2X
Communication

Hojeong Lee and Hyogon Kim

###### Abstract

The Society of Automotive Engineers (SAE) has specified a wireless channel
congestion control algorithm for cellular vehicle-to-everything (C-V2X)
communication in J3161/1. A notable aspect of J3161/1 standard is that it
addresses persistent packet collisions between neighboring vehicles. Although
the chances are slim, the persistent collisions can cause so called the
wireless blind spot once the event takes place. Then the involved vehicles
cannot inform their presence to neighboring vehicles, an extremely dangerous
condition for driving safety. J3161’s solution to the problem is stochastic
one-shot transmission, where the transmission occasionally occurs in a
resource that is not originally reserved. Through the one-shot transmission,
the worst-case packet inter-reception time (PIR) is bounded and the wireless
blind spot problem can be effectively mitigated. Interestingly, the standard
one-shot transmission scheme does not resolve the persistent collision
relation itself. This paper shows that by breaking out of the relation as soon
as the persistent collision condition is identified, vehicles can improve the
worst-case PIR by approximately 500 ms, the number of packet collisions per
persistent collision event by 10%, and the number of total collisions by 15%
to 57% over the standard one-shot transmission.

###### Index Terms:

SAE J3161/1, one-shot transmission, persistent collision, communication
outage, packet inter-reception time (PIR), packet reception ratio (PRR).

##  I Introduction

The 3rd Generation Partnership Project (3GPP) has been standardizing Cellular
V2X (C-V2X) communication technology since Release 14. One of the core
features of the C-V2X standards is the sidelink communication that enables
direct communication between vehicles. The wireless resource for the sidelink
communication can be allocated in either centralized or distributed manner.
The base station orchestrates the allocation in the former (called Mode 1),
whereas vehicles autonomously allocate the resource using a distributed
algorithm in the latter (called Mode 2) [1]. Because vehicles could be out of
network coverage, Mode 2 is considered the base mode. The distributed resource
allocation algorithm used in Mode 2 is called the Sensing-Based Semi-
Persistent Scheduling (SB-SPS) [1].

As the name implies, the SB-SPS algorithm (henceforth “SPS” for convenience)
utilizes the sensed resource use pattern of neighbor vehicles to avoid the
resource locations where the neighbors are expected to transmit. It also
utilizes each neighbor’s explicit resource reservation information for its
next transmission written in the control part of the packet [2]. Once a host
vehicle decides on a resource least likely to be used by its neighbors, it
uses the same resource for a certain number of subsequent packet transmissions
without additional signaling, hence the name Semi-Persistent Scheduling. In
this paper, we will call this series of packets using the same frequency
resource over the average one-second period by the name of a “packet run.”
Note that the same resource can be kept in the next packet run with the
resource keep probability $0\leq P_{k}\leq 0.8$ [1], so it can be a few
seconds before a vehicle re-selects a different resource [3].

An issue with SPS is that it cannot completely eliminate packet collisions.
For instance, two vehicles that came to re-select a resource almost
simultaneously can pick the same resource. If their message transmission
periods are the same, the packet collision will repeat at every subsequent
message transmission. The higher the resource keep probability $P_{k}$, the
longer the repeated packet collisions. At a given resource keep probability
$P_{k}$, the number of packet runs that retain the same frequency resource at
a vehicle is geometrically distributed. Then, the two vehicles will break out
of the consecutive packet collisions relation as the shorter of the two packet
runs expires and the vehicle with the shorter run re-selects. Here, the
average number of the shorter packet run is given by

$\mathbb{E}[\min(l_{1},l_{2})]=\frac{1}{2p-p^{2}}$ (1)

for $p=1-P_{k}$. According to Irwin-Hall distribution, the number of packet
transmissions in the packet runs tends to a normal distribution centered at
$l\overline{X}$ where $X$ is the run length of each packet run. For instance,
$\overline{X}=10$ if the messaging period is 100 ms. As Fig. 1 shows, the
average duration in Eq. (1) can easily persist for over a few seconds once the
repeated collision event takes place, especially as $P_{k}\rightarrow 0.8$.
The worst-case duration can be longer. Note that blindly reducing $P_{k}$ is
not a solution, either, because it makes re-selections more frequent in SPS.
Consequently, it causes more packet collisions and generally leads to poor
packet delivery performance [4].

Figure 1: Average time until breakout from repeated packet collisions relation

The cost of a long, repeated-collision event is potentially high. The event
renders the involved vehicles to be unrecognized by neighboring vehicles and
by each other during the persistent collision event. Onboard sensors, such as
radars, LiDARs, and cameras may still be able to maintain awareness in many
situations, but in others like the non-light-of-sight (NLoS) positions,
neighboring vehicle movements become difficult to trace under the consecutive
packet loss event. Therefore, in safety-critical situations, this
communication lull may push a vehicle into dangerous driving conditions [3],
which might even be worse off than not relying on the C-V2X safety
communication at all.

In order to address the persistent packet collision problem, the Society of
Automotive Engineers (SAE) J3161/1 stipulates stochastic One-Shot Transmission
mechanism for C-V2X communication environment [5]. Under the scheme, vehicles
occasionally transmit their packets in a resource that is not originally
reserved (see Fig. 2(a)). Through the one-shot transmission, the worst-case
packet inter-reception time (PIR) is bounded and the wireless blind spot
problem can be effectively mitigated. Interestingly, however, the standard
one-shot transmission scheme does not resolve the persistent collision
relation itself. Such passive approach neglects unnecessary packet collisions
thereby degrading packet delivery performance under SPS.

In this paper, we demonstrate that by exiting the relation as soon as the
persistent collision condition is identified, the vehicles can further improve
PIR and packet reception ratio (PRR) than in the standard one-shot
transmission. Compared with the duration of repeated collisions in SPS (Fig.
1), the proposed scheme reduces the worst-case PIR to less than 3 seconds at
the farthest required communication range of 320 meters [6] for $P_{k}=0.8$.
Note that this performance is approximately the minimum duration in SPS.
Therefore, the proposed scheme significantly improves the long-tail PIR
values. Compared with the J3161/1 One-Shot mechanism, the proposed solution
reduces the worst-case PIR by approximately 500 ms, the number of packet
collisions per persistent collision event by 10%, and the number of total
collisions by 15% to 57% over the standard one-shot transmission.

The rest of the paper is organized as follows. Section II briefly discusses
the prior work on resolving the problem of repeated packet collisions. In
particular, it includes the previous efforts to improve on the performance of
one-shot transmission. Section III provides the background on SPS resource
allocation and the standard J3161/1 one-shot transmission. It then discusses
the proposed modification to SPS. Section IV evaluates the performance of SPS,
one-shot, and the proposed scheme in terms of PRR, PIR, and the collision
statistics. Finally, Section V concludes the paper.

##  II Related work

Bazzi et al. [3] proposed to curtail semi-persistent resource use at a fixed
length, overriding repeated reselection of the same resource over a certain
limit, even when it is prescribed by a high probability of keeping the same
frequency resource. As a side-effect, any persistent collisions are limited to
a deterministic length. But the proposed limit is significantly long than that
stipulated in the one-shot transmission in J3161/1. Gholmieh et al. [7]
analyzed that continuous collisions of packets can be reduced without
noticeable loss of packet reception ratio (PRR) performance through one-shot
transmission. In Fouda et al. [8], the authors analyzed the performance
according to the range of one-shot counters. It showed that the performance of
the tail of information age (IA) and inter-packet gap (IPG) was better when
the one-shot counter range was [2..6], which is smaller than [5..15]. Saifudd
et al. [9] showed that under J3161/1 congestion control, a timer-based one-
shot transmission performs better than the standard counter-based scheme. This
is because when congestion control of J3161 intervenes, Inter-Transmit Time
(ITT) increases. This implies that as the vehicle density increases, the time
it takes for the one-shot counter to reach zero also increases. Consequently,
the duration required to escape from consecutive collisions becomes longer,
particularly compared to situations with lower vehicle density.

Unfortunately, none of the counter-based and timer-based one-shot transmission
methods proposed so far do not resolve the persistent collision condition
itself. However, vehicles can ascertain whether other vehicles are utilizing
the same resource through one-shot transmission. Therefore, it is neither
desirable nor necessary to resume transmission on the resource identified as
involved in the persistent collision condition. The enhancement discussed
below that by breaking out as soon as the persistent collision condition is
identified, packet delivery performance is indeed improved.

##  III Solution approach

###  III-A Background

Under SPS, each vehicle runs a re-selection counter that is decremented upon
each transmission. Once the frequency resource for the next packet run is re-
selected, the counter is randomly initialized to a value in

$\bigg{[}5\times\frac{100}{\max(RRP,20)}:\
15\times\frac{100}{\max(RRP,20)}\bigg{]}$

where $RRP$ is the resource reservation period in SPS [10]. In this paper, we
will assume $RRP=100$ ms for illustrative purpose because it is the most
popular messaging period for safety-related V2X applications, although the
proposed scheme also applies to other RRP configurations.

SAE J3161/1 also defines a congestion control algorithm, where the gap between
the transmitted packets called the Inter-Transmission Time (ITT) is controlled
as follows [5]:

$ITT\ {\rm[s]}=\begin{cases}0.1,&\overline{VD}\leq 25\\\
\overline{VD}/250,&25<\overline{VD}<150\\\
0.6,&150\leq\overline{VD}\end{cases}$ (2)

where the average vehicle density $\overline{VD}$ is obtained from the
instantaneous $VD$ as $\overline{VD(t)}\leftarrow 0.05\times
VD+0.95\times\overline{VD(t-1)}$. $VD$ is the number of vehicles within 100 m
of the observing vehicle.

###  III-B One-shot transmission in J3161/1

The stochastic one-shot transmission in J3161/1 effectively mitigated the
continuous collision problem by providing a mechanism to occasionally step
aside and use a resource different from the currently reserved one. It is
executed more frequently than re-selection, and is executed regardless of
whether or not a persistent collision event is ongoing. In case the persistent
collision event is underway, the one-shot transmission fragments the long
series of packet collisions into many smaller ones. Compared with the
unmodified SPS, therefore, neighbor vehicles can occasionally have a chance to
notice a formerly unheard vehicle due to consecutive collisions.

According to SAE J3161/1, each vehicle additionally runs an one-shot counter
$C_{O}$. It is randomly initialized to a value in $[2:6]$, and decremented
with each transmission under SPS. Let $R_{sps}$ denote the resource allocated
by SPS, and $R_{os}$, by one-shot. Based on the values of $C_{R}$ and $C_{O}$,
the following logic is executed:

  1. 1.

If $C_{O}=0$ and $C_{R}=0$

     * •

Transmit in $R_{os}$ instead of in $R_{sps}$; Initialize $C_{O}$

     * •

Perform re-selection conditioned on $P_{keep}$; Initialize $C_{R}$

  2. 2.

If $C_{O}=0$ and $C_{R}\neq 0$

     * •

Transmit in $R_{os}$ instead of in $R_{sps}$; Initialize $C_{O}$

     * •

Do not decrement $C_{R}$

  3. 3.

If $C_{O}\neq 0$ and $C_{R}=0$

     * •

Perform re-selection conditioned on $P_{keep}$

     * •

If moving, initialize $C_{R}$ and $C_{O}$

     * •

If not moving, initialize only $C_{R}$

In case neither counter is 0, normal SPS-reserved transmission takes place and
both counters are decremented by 1. Notice that there is no provision to exit
the persistent packet collision condition itself.

###  III-C Proposed enhancement

We propose an enhancement that curtails the persistent collision event as soon
as an involved vehicle finds it. The vehicle then performs the SPS re-
selection, exiting the collision relation. In this paper, we assume that the
one-shot resource $R_{os}$ is randomly selected among the candidate resources
recommended by SPS when $R_{sps}$ was selected, and that is within 3 slots
after $R_{sps}$. This is to minimize the interference to other vehicles’ SPS
transmissions that could happen at the chosen one-shot resource. Also,
limiting the time offset of the one-shot transmission to within 3 slots of the
originally reserved resource is to minimize the impact on the packet delay
budget (PDB). If no resource exists that is recommended by SPS within the time
limit, random selection is performed with the same time constraint. Depending
on applications, the PDB constraint may be less strict. Then, an one-shot
resource can be selected with larger offsets, reducing the potential of packet
collisions. Since the proposed scheme utilizes the information already
available after $R_{sps}$ selection, it does not incur additional processing
overhead that would be incurred if we separately invoked the complex SPS
processing. In the enhancement, in Condition (2) in the logic presented above,
we replace

  * •

Do not decrement $C_{R}$

with

  * •

If other vehicle is observed to transmit in $R_{sps}$, perform SPS re-
selection; Initialize $C_{R}$ and $C_{O}$

  * •

Otherwise, do not decrement $C_{R}$

Fig. 2(a) and (b) illustrate the difference between J3161/1’s one-shot
transmission and the enhancement. In the figure, ‘UE’ refers to a user
equipment, which is the on-board V2X communication unit on a vehicle. The
standard one-shot scheme in Fig. 2(a) detects the persistent collision
condition by intermittently performing one-shot transmission. However, even
after it finds the condition through the one-shot transmission, the involved
vehicles UE1 and UE2 do not exit the condition but keep using the collision
resource $R_{sps}$ for future transmissions until the next re-selection. In
the proposed enhancement (Fig. 2(b)), however, the persistent collision
condition itself is resolved by the vehicle that first realizes the condition
and immediately moves to the one-shot resource $R_{os}$. Namely, UE1 first
performs the one-shot transmission and realizes that its previous resource
$R_{sps}$ has another user UE2, so UE1 performs a re-reselection through SPS
and moves to a different resource. The persistent packet collision condition
is thus broken.

(a) One-shot Transmissions according to J3161/1

(b) Proposed method using One-shot Transmissions

Figure 2: Standard one-shot transmission scheme vs. proposed enhancement

Finally, when three or more vehicles are involved in a persistent packet
collision condition, the condition can be found by the one-shot performing
vehicle through high Received Signal Strength Indicator (RSSI) value even if
the transmission(s) there is not decoded. But the probability of such multi-
vehicle persistent collisions is deemed low, so we focus on the two-vehicle
case in this paper.

##  IV Performance evaluation

In this section, we evaluate the impact of the proposed enhancement in terms
of packet delivery performance. To this end, we conducted simulation
experiments comparing three schemes in the C-V2X environment, as follows:

  1. 1.

SPS without one-shot transmission (“SPS”)

  2. 2.

One-shot transmission as defined by J3161/1 (“One-shot”)

  3. 3.

Proposed solution (“Proposed”)

As to the metrics, we utilized the distribution of Packet Inter-Reception time
(PIR), number of consecutive collisions and collision events, and Packet
Reception Ratio (PRR). PIR and PRR are the most significant measures of packet
delivery performance in V2X communication and are defined as follows [6]:

  * •

PRR: ratio of successfully received packets to the total packets transmitted
within 320 meters and binned into 20-meter segments

  * •

PIR: time elapsed between two successive receptions of two different packets
transmitted from the same vehicle within the same range of 320 meters

###  IV-A Simulation setting

For simulation experiments, we employed LTEV2Vsim [11], an open-source
simulator designed for C-V2X environment. Specific simulation parameter values
are summarized in Table I. Note that the presented results are obtained with
J3161/1 congestion control enabled whereas hybrid automatic repeat and request
(HARQ) disabled because we consider periodic broadcast traffic such as Basic
Safety Message (BSM) [12]. Also, we do not use the blind retransmission
because it doubles the bandwidth use, which can cause a severe congestion and
the J3161/1 congestion control will increase the ITT. It will make information
update much less frequent, betraying the very purpose of periodic beaconing
through BSM packets.

TABLE I: Simulation parameter configuration | Parameter | Value
---|---|---
PHY | Bandwidth (MHz) | 20 [5]
No. subchannels | 10
Subchannels/transport block | 2
Antenna gain (dB) | 3
Tx power (dBm) | 20 [5]
Noise figure of receiver (dB) | 9
Pathloss model | WINNER+B1 [6]
Shadowing distribution | Log-normal
Shadowing std. dev. (dB) | 0 (LOS), 4 (NLOS)
Antenna height (m) | 1.5
MCS index | 7
MAC | re-selection counter value | [5..15]
One-shot counter value | [2..6]
Selection window width (ms) | 100
Congestion control | J3161/1
HARQ | Off
Resource keep probability | 0.8 [1]
Application | Message size (bytes) | 250
Vehicles | Traffic density (veh./km) | [100..500]
Speed of vehicles (km/h) |  $\mu$ = 50, $\sigma$ = 3
Road length (km) | 2

###  IV-B Packet inter-reception time

Fig. 3 shows how One-shot and Proposed improve the tail PIR distribution over
the unmodified SPS. For $\rho=100$ veh./km (or equivalently $VD=20$ in Eq.
(2)), One-shot and Proposed can respectively limit the PIR below 2,200 and
1,700 ms at $10^{-6}$ probability whereas SPS has high PIR with slowly
decreasing probability. In fact, SPS cannot suppress the PIR under 3,500 ms
even at $10^{-4}$ probability in any traffic density. For other vehicle
densities, we can observe similarly significant improvement enabled by the two
one-shot schemes. However, the improvement is not free. In particular, the
standard one-shot transmission trades extremely large PIR values for more
numerous intermediate PIR values by breaking long consecutive collision events
into multiple medium-length events. For example, with $\rho=100$ veh./km, PIR
values between 200 ms and 400 ms have increased in number. Note that it is an
inevitable cost to avoid the dire situations where the periodic beacons from
two close vehicles suffer from effectively becoming “ghosts” for an extended
period of time [3]. Comparing One-shot and Proposed, the latter always
outperforms former. This is because Proposed eliminates the residual
consecutive collision events after the first one-shot opportunity.

(a) $\rho=100$

(b) $\rho=200$

(c) $\rho=300$

(d) $\rho=400$

Figure 3: PIR distribution for different vehicle density environments

A further improvement could be made possible by forcing a one-shot
transmission as soon as the resource re-selection is performed. Assuming that
consecutive packet collisions between more than two vehicles are rare, we can
force each vehicle perform a one-shot transmission for the first packet
transmission after a re-selection with a probability of 1/2. In case there is
indeed another vehicle that selected the same resource with the same RRP, one
of the vehicles can identify the consecutive collision condition and move to
other resource with probability 1/2. However, it may have a complicated affect
on SPS, so we leave it for a future work.

###  IV-C Collision statistics

Fig. 4 compares the collision characteristics of the three schemes. Fig. 4(a)
shows the average number of consecutive collisions in a persistent collision
event. Although the tail distribution information is not available as in Fig.
3, the average run length can be shown to decrease with the proposed
enhancement. Although Proposed significantly reduces the number of such events
over One-shot, the run length per event only slightly differs between them.

(a) Average no. of packet collisions per persistent collision event

(b) Total number of packet collisions observed throughout simulation

Figure 4: Packet collision statistics

We can easily estimate the run length for Proposed. It is the minimum $X$ of
two (colliding) one-shot runs $X1,X2\sim U[2..6]$, i.e.,

$\mathbb{E}[X]=\mathbb{E}(\min[X1,X2])=\sum_{i=2}^{5}2\cdot\frac{1}{5}\cdot\frac{6-i}{5}=2.4$

where a one-shot run is defined by the run length of packet transmissions
before the one-shot counter ($C_{O}$) expires. In case of One-shot, it is
slightly higher than Proposed. It can be illustrated by an example. Suppose
$C_{O}(u)=2$ and $C_{O}(v)=6$ for two vehicles $u$ and $v$. With Proposed, the
number of collisions would be 2. However, with the standard one-shot, the
number of collisions before the first one-shot by $u$ is 2, but the next ones
could be longer than this minimum length. For instance, if $u$ chooses
$C_{O}(u)>2$, the next collision run will have a length larger than 2,
increasing the average.

For $\rho=200$ veh./km, both One-shot and Proposed reduce the average run
length of packet collisions compared to SPS, which is expected. But an
unexpected result is that SPS has smaller number of consecutive collisions
than at $\rho=100$ veh./km. This is because the J2945/1 congestion control
begins to kick in, changing the packet collision dynamics. The table below
shows that the average ITT is approximately 155 ms for $\rho=200$ veh./km.

$\rho$ (veh./km) | 100 | 200 | 300 | 400 | 500
---|---|---|---|---|---
Average ITT (s) | 0.101 | 0.155 | 0.232 | 0.313 | 0.387

Because only discrete RRP values $k\cdot 100\ (k\geq 1)$ ms are allowed over
100 ms in SPS [1], vehicles are transmitting at an ITT of either 100 ms or 200
ms. When the vehicles with different RRPs meet with persistent collisions, one
with 200 ms RRP will still suffer the consecutive collisions as before whereas
the one with 100 ms RRP will experience a collision every other transmission.
The latter will not qualify as consecutive collisions. However, for those
vehicles with 200 ms RRP, the collision run length will be also shortened
because the counterparts with 100 ms RRP decrement their $C_{R}$ twice as
fast. It cuts off the consecutive collisions faster than the situation of the
both have the same RC value reduction. As $\rho$ increases further, the
population ratio between the vehicles with different RRP values affect how
fast a persistent collision event is cutoff by the small RRP counterpart.

Fig. 4(b) reveals an important aspect of one-shot transmission schemes. In
particular, One-shot increases the total number of packet collisions compared
to SPS. This is natural because one-shot transmissions occur outside the
resource originally reserved ($R_{sps}$). When the persistent packet collision
does occur, each one-shot eliminates a collision. But with far higher
probability the event does not occur, and the one-shot transmissions may only
collide with SPS transmissions from other vehicles. Indeed, the figure shows
that the latter outweighs the former. This is the second cost aspect of the
one-shot transmission, from the perspective of packet collisions. Unlike the
standard one-shot, however, the proposed enhancement has even lower number of
packet collisions than SPS. This is because once an one-shot transmission that
identifies, the rest of the consecutive packet collisions retained by both SPS
and the standard one-shot scheme are avoided.

###  IV-D Packet reception ratio

The main purpose of one-shot transmission is to avoid the rare but dangerous
V2X communication outages at all costs. However, it is worthwhile to consider
how adversely average packet delivery performance is affected by it. Fig. 5
compares the PRR for the three schemes. For readability, we show for only
$\rho=100$ and $\rho=300$ veh./km. Other vehicle densities show the same
qualitative result. Although Gholmieh et al. [7] argued that PRR performance
is hardly affected, there is small PRR drop according to our simulation
results. We believe that it is unavoidable because one-shot transmission
frequently leaves the reserved resource wasted and instead uses a non-reserved
resource. Because the proposed enhancement reduces such incursions through
early re-selection, it leads to slightly better PRR over the original one-shot
transmission scheme. In essence, the proposed scheme does not degrade the PRR
performance compared to the standard one-shot transmission scheme.

Figure 5: PRR distribution for different vehicle density environments

##  V Conclusion

In safety-critical driving situations, preventing rare but dangerous
communication outages is also important. The SAE J3161/1 standard prescribes
so called the one-shot transmission that stochastically allows vehicles to
step aside from potential persistent packet collisions and reveal their
presence. However, it is too passive because it does not go as far as using
the standard re-selection mechanism readily provided by the Semi-Persistent
Scheduling (SPS) upon finding the persistent collision condition. This paper
proposes an enhancement to the one-shot scheme to perform the re-selection.
Proposed enhancement not only improves the PIR tail distribution but reduces
the number of packet collisions over both SPS and the one-shot scheme.
Moreover, it leads to slightly better PRR performance than the standard one-
shot scheme.

## Acknowledgments

This research was supported by the MSIT (Ministry of Science and ICT), Korea,
under the ICT Creative Consilience program (IITP-2023-2020-0-01819) supervised
by the IITP (Institute for Information & communications Technology Planning &
Evaluation.

## References

  * [1] 3GPP, NR; Physical layer procedures for data (Release 16), 3GPP TS 38.214, 2022.
  * [2] 3GPP, NR; Multiplexing and channel coding (Release 17), 3GPP TS 38.212, 2023.
  * [3] A. Bazzi, C. Campolo, A. Molinaro, A. O. Berthet, B. M. Masini, and A. Zanella, “On wireless blind spots in the C-V2X sidelink,” IEEE Trans. Veh. Technol., vol. 69, no. 8, pp. 9239–9243, Aug. 2020.
  * [4] Y. Jeon, S. Kuk, and H. Kim, “Reducing Message Collisions in Sensing-Based Semi-Persistent Scheduling (SPS) by Using Reselection Lookaheads in Cellular V2X,” Sensors, 18(12), 4388, 2018.
  * [5] SAE, On-Board System Requirements for LTE-V2X V2V Safety Communications, SAE J3161/1, 2022.
  * [6] 3GPP, Technical Specification Group Radio Access Network; Study on evaluation methodology of new Vehicle-to-Everything (V2X) use cases for LTE and NR, 3GPP TR 37.885 V15.3.0, June 2019.
  * [7] R. Gholmieh and H. Abbasi, “C-V2X (LTE-V2X) Performance Enhancement Through SAE J3161/1 Probabilistic One-Shot Transmissions,” in Proceedings of IEEE Global Communications Conference, 2021.
  * [8] A. Fouda, R. Berry, and I. Vukovic, “Interleaved One-shot Semi-Persistent Scheduling for BSM Transmissions in C-V2X Networks,” in Proceedings of IEEE Vehicular Networking Conference (VNC), 2021.
  * [9] M. Saifuddin, M. Zaman, Y. Fallah, and J. Rao, “Addressing Rare Outages in CV2X with Time-Controlled One-shot Resource Scheduling,” TechRxiv, 2023.
  * [10] 3GPP, NR; Medium Access Control (MAC) Protocol Specification, Release 15,v16.3.0, TS 38.321, Jan.2021.
  * [11] G. Cecchini, A. Bazzi, B. M. Masini, and A. Zanella, “LTEV2Vsim: An LTE-V2V Simulator for the Investigation of Resource Allocation for Cooperative Awareness,” in Proc. IEEE Int’l Conf. on Models and Technologies for Intelligent Transportation Systems, Naples, Italy, 2017.
  * [12] SAE, Dedicated short range communications (DSRC) message set dictionary,  SAE J2735, 2016.


utf8

# ArCorona: Analyzing Arabic Tweets in the Early Days of Coronavirus
(COVID-19) Pandemic

Hamdy Mubarak, Sabit Hassan
Qatar Computing Research Institute
HBKU, Qatar
{hmubarak<EMAIL_ADDRESS>
###### Abstract

utf8

##  1 Introduction

As the Coronavirus (COVID-19) crippled lives across the world, people turned
to social media to share their thoughts, news about vaccines or cures,
personal stories, etc. With Twitter being one of the popular social media
platforms in the Arab region, tweets became a major medium of discussion about
COVID-19. These tweets can be indicators of psychological and physical well
being, public reactions to specific actions taken by the government and also
public expectation from governments. Therefore, identifying types of tweets
and understanding their content can aid decision making by governments. It is
also important for governments to identify and prevent rumours and bad cures
since they can bring harm to society. While there have been many recent works
about tweets related to COVID-19, there are a very few targeted toward aiding
governments in their decision making in the Arab region despite Arabic being
one of the dominant languages on Twitter Alshaabi et al. (2020). Some of the
existing works use automatically collected datasets Alqurashi et al. (2020).
Manually labeled datasets are either small in size (few hundred tweets) Alam
et al. (2020) or target a different task such as sentiment analysis Haouari et
al. (2020). To fill this gap, we present and publicly share the largest (to
our best knowledge) manually annotated dataset of Arabic tweets collected from
early days of COVID-19, labeled for 13 classes. We present our data collection
and annotation scheme followed by data analysis, identifying trends, topics
and distribution across countries. Lastly, we employ machine learning and
transformer models for classification.

##  2 Related Work

Much of recent works on COVID-19 rely on queries to Twitter or distant
supervision. This allows a large number of tweets to be collected. Chen et al.
(2020) collect 123M tweets by following certain queries and accounts on
Twitter. GeoCoV19 Qazi et al. (2020) is a large-scale dataset containing 524M
tweets with their location information. Banda et al. (2020) collected 152M
tweets at the time of their writing. Li et al. (2020) identifies situational
information about COVID-19 and its propagation on Weibo. Other works include
propagation of misinformation Huang and Carley (2020); Shahi et al. (2020),
cultural, social and political impact of misinformation Leng et al. (2020) and
rumor amplification Cinelli et al. (2020). For Arabic, we see a similar trend
where few datasets are manually labeled. Alqurashi et al. (2020) provide a
large dataset of Arabic tweets containing keywords related to COVID-19.
Similarly, ArCOV-19 Haouari et al. (2020) is a dataset of 750K tweets obtained
by querying Twitter. Alam et al. (2020) annotate a small number of English
(currently 504) and Arabic tweets (currently 218) for (i) existence of claim
and worthiness of fact-checking (ii) harmfulness to society, and (iii)
relevance to governments or policy makers. Yang et al. (2020) annotate 10K
Arabic and English tweets for the task of fine-grained sentiment analysis.
Alsudias and Rayson (2020) collected 1M unique Arabic tweets related to
COVID-19 between December 2019 and April 2020. They used K-means algorithm
from Scikit-learn Python package to cluster tweets into 5 clusters, namely:
statistics, prayers, disease locations, advising, and advertising. They also
annotated random 2000 tweets for rumor detection based on the tweets posted by
the Ministry of Health in Saudi Arabia.

##  3 Data Collection

We used twarc search API111https://github.com/DocNow/twarc to collect tweets
having the Arabic word كورونا¿ (Corona) in Feb and March 2020. We collected
30M tweets in total. The reason behind selecting this word is that it’s widely
used by normal people, news
media222https://www.aljazeera.com/topics/events/coronavirus-outbreak.html and
official organizations 333https://www.who.int/ar/emergencies/diseases/novel-
coronavirus-2019 as opposed to 19-كوفيد¿ (COVID-19) which is rarely used by
normal people in different Arab countries based on our observations. We aimed
to increase diversity of tweet sources. Our collection covers the period from
Feb 21 until March 31 in which Coronavirus was reported for the first time in
All Arab countries except United Arab Emirates (AE)444We use ISO 3166-1
alpha-2 for country codes (Jan 29) and Egypt (EG) (Feb 14). The date of the
first reported Coronavirus case in Lebanon (LB) was Feb 21, and in Iraq (IQ),
Bahrain (BH), Oman (OM), and Kuwait (KW) was Feb 24, in Qatar was Feb 29, and
in Saudi Arabia (SA) was March 2. All other Arab countries came later.

##  4 Data Annotation

Class | Description | Count
---|---|---
01\. REP | Reports and announcements such as number of infections, recovery cases and deaths. | 1664
02\. ACT | Measures or actions taken by governments such as curfew, closing of country borders, shops | 1383
| and worship places. This includes discussions and consequences of these measures. |
03\. INFO | Information about the virus, symptoms, incubation period, how it spreads, mask types, etc. | 300
04\. RUMOR | Rumor or refute rumor. A rumor is a circulating story or report of uncertain or doubtful truth. | 421
05\. ADVICE | Advice or caution such as washing hands, staying at home, wearing masks and avoiding travel. | 1047
06\. SEEK_ACT | Seek actions from governments such as closing airports, and controlling prices of goods. | 587
07\. CURE | News about good and bad cure, diagnosis, ventilators, supportive medical equipment, etc. | 116
08\. VOLUNT | Volunteering efforts or donation of money, goods or services. | 133
09\. PRSNL | Personal story or opinion. | 453
10\. SUPPORT | Support or praise governments, medical staff, celebrities, etc. | 386
11\. PRAYER | Prayer. | 563
12\. UNIMP | Unrelated or unimportant such as spams or advertisements. | 786
13\. NOT_ARB | Not Arabic, e.g. Persian. | 161
Total |  | 8000
Table 1: Annotation classes and distribution: Important classes (top) and
LessImportant classes (bottom)

During the period of our study (40 days), we extracted the top retweeted 200
tweets in each day (total of 8000). We assume that the top retweeted tweets
are the most important ones which get highest attention from Twitter users.
Annotation was done manually by a native speaker who is familiar with Arabic
dialects according to class descriptions shown in Table 1. To measure quality,
we annotated 200 random tweets by a second annotator. Inter-annotator
agreement was 0.85 using Cohen’s kappa coefficient which indicates high
quality given that annotation is not trivial and some classes are close to
each other. Examples of annotation classes are shown in Figures 1 and 2.
Note: If a tweet has multimedia (image or video) or an external link (URL or
another tweet), the annotator was asked to open it and judge accordingly to
consider the full context of the tweet. For example, if a tweet has a text
about a prayer and the attached image is about number of new cases, this
should be classified as REP not PRAYER.
Data can be downloaded from this link555We share tweet id, date and class.:
https://alt.qcri.org/resources/ArCorona.tsv.

###  4.1 Limitation

We found that $\approx$10% of the tweets can take more than one class, e.g. a
tweet reports new cases and a medical advice. We plan to allow multiple labels
in future. In the current version, such tweets take the label of the first
“important” class. We consider the first 8 classes in Table 1 to be important
and the last 5 classes (PRSNL, SUPPORT, PRAYER, UNIMP and NOT_ARB) to be less
important. These classes will be merged into LessImportant class.

Figure 1: Examples for REP, ACT, INFO, RUMOR, ADVICE and SEEK_ACT classes
Figure 2: Examples for CURE, VOLUNT, PRSNL, SUPPORT, PRAYER and UNIMP classes

##  5 Analysis

Figure 3: Timeline: Number of class tweets each day

Class timeline is shown in Figure 3. We can observe the following important
notes:

  * •

Large portion of tweets can be considered as LessImportant to many people
($\approx$ 30%).

  * •

Reports (REP) and actions taken by governments (ACT) are the most retweeted
tweets.

  * •

Information about the virus (INFO) get less attention with time and there is
an increasing number of tweets about volunteering (VOLUNT).

  * •

There are continuous requests for governments to take actions (SEEK_ACT) –
especially in the beginning ($\approx$ 15%), and few tweets are about rumors
($\approx$ 5%) and cures ($\approx$ 2%).

We took a random sample of 1000 tweets and annotated them for their topics.
Figure 4 shows that, in addition to health, the virus affected many aspects of
people’s lives such as politics, economy, education, etc. We found also that
7% of tweets have hate speech, e.g. attacking China and Iran for spreading the
virus as shown in Figure 5.

Figure 4: Tweet topics Figure 5: Hate speech example

Table 2 shows country distribution and top accounts for the original authors
of tweets. Typically, people retweet tweets from ministry of health in their
countries in addition to famous news agencies and celebrities. Most of these
accounts are verified.

Country | % | Top Accounts
---|---|---
SA | 59 | SaudiNews50, SaudiMOH
OTH | 13 | MohamadAhwaze (SE), amjadt25 (UK)
OM | 7 | OmanVSCovid19, OmaniMOH
KW | 7 | Almajlliss, KUWAIT_MOH
QA | 4 | amansouraja, MOPHQatar
AE | 4 | AlHadath, AlArabiya_Brk
EG | 3 | RassdNewsN, mohpegypt
Table 2: Country distribution and top accounts

##  6 Experiments and Evaluation

We randomly split the data into sets of 6000, 1000 and 1000 tweets for train,
dev and test sets respectively. We report macro-averaged Precision (P), Recall
(R) and F1 score along with Accuracy (Acc) on test set 666Differences between
dev and test sets are $\pm 2-3\%$ (F1).. We use F1 score as primary metric for
comparison.

###  6.1 Features

#### N-gram features

We experimented with character and word n-gram features weighted by term
frequency-inverse term document frequency (tf-idf). We report results for only
the most significant ranges, namely, word [1-2] and character [2-5].

#### Mazajak Embeddings

Mazajak embeddings are word-level skip-gram embeddings trained on 250M Arabic
tweets, yielding 300-dimensional vectors Abu Farha and Magdy (2019).

###  6.2 Classification Models

#### Support Vector Machines (SVMs)

SVMs have been shown to perform decently for Arabic text classification tasks
such as spam detection Mubarak et al. (2020), adult content detection Mubarak
et al. (2021); Hassan et al. (2021), offensiveness detection Hassan et al.
(2020b, a) or dialect identification Abdelali et al. (2020); Bouamor et al.
(2019). We experimented with i) word n-gram, ii) character n-gram and iii)
Mazajak Embeddings. We used LinearSVC implementation by scikit-learn
777https://scikit-learn.org/.

#### Deep Contextualized Transformer Models (BERT)

Transformer-based pre-trained contextual embeddings, such as BERT Devlin et
al. (2019), have outperformed other classifiers in many NLP tasks. We used
AraBERT Antoun et al. (2020), a BERT-based model trained on Arabic news. We
used ktrain library Maiya (2020) that utilizes
Huggingface888https://huggingface.co/ implementation to fine-tune AraBERT. We
used learning rate of 8e-5, truncating length of 50 and fine-tuned for 5
epochs.

###  6.3 Binary Classification

First, we experiment to distinguish LessImportant tweets from others (see
Section 4). From Table 3, we can see that SVMs with character [2-5]-gram
outperformed others with F1 score of 79.8, closely followed by AraBERT with
79.6 F1.

Model | Features | Acc. | P | R | F1
---|---|---|---|---|---
Majority Class | - | 72.5 | 36.3 | 50.0 | 42.0
SVM | W[1-2] | 84.4 | 82.5 | 76.4 | 78.6
SVM | C[2-5] | 85.4 | 84.3 | 77.4 | 79.8
SVM | Mazajak | 83.9 | 80.5 | 77.7 | 78.9
AraBERT |  | 83.9 | 80.0 | 79.2 | 79.6
Table 3:  Binary classification results

###  6.4 Fine-grained Classification

Model | Features | Acc. | P | R | F1
---|---|---|---|---|---
Majority Class | - | 22.7 | 1.7 | 7.7 | 2.8
SVM | W[1-2] | 62.8 | 64.3 | 53.5 | 56.3
SVM | C[2-5] | 59.0 | 64.3 | 49.4 | 51.8
SVM | Mazajak | 60.0 | 55.1 | 51.5 | 52.4
AraBERT |  | 62.7 | 61.6 | 59.8 | 60.5
Table 4:  Fine-grained classification results

Our next set of experiments were designed for fine-grained classification for
13 classes. With F1 score of 60.5, AraBERT outperformed others (Table 4).
Error Analysis:  AraBERT confusion matrix (Figure 6) shows that PRSNL, INFO
and RUMOR are the hardest classes to identify and the most common error is
misclassifying INFO as ADVICE. We hypothesize these errors can be reduced if
larger data set is being annotated.

Figure 6: Confusion matrix normalized over true labels

##  7 Conclusion and Future Work

We present the largest publicly available manually annotated dataset of Arabic
tweets for 13 classes that includes the most retweeted tweets in the early
days of COVID-19. Followed by data analysis, we present models that can
reliably identify important tweets and can perform fine-grained
classification. In the future, we plan to compare our data to data from later
days of the pandemic.

## References

  * Abdelali et al. (2020) Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020\.  Arabic dialect identification in the wild.  _ArXiv_ , abs/2005.06557.
  * Abu Farha and Magdy (2019) Ibrahim Abu Farha and Walid Magdy. 2019.  Mazajak: An online Arabic sentiment analyser.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 192–198, Florence, Italy. Association for Computational Linguistics.
  * Alam et al. (2020) Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, and Preslav Nakov. 2020.  Fighting the covid-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society.  _arXiv:2005.00033_.
  * Alqurashi et al. (2020) Sarah Alqurashi, Ahmad Alhindi, and Eisa Alanazi. 2020.  Large arabic twitter dataset on covid-19.  _arXiv preprint arXiv:2004.04315_.
  * Alshaabi et al. (2020) Thayer Alshaabi, David R. Dewhurst, Joshua R. Minot, Michael V. Arnold, Jane L. Adams, Christopher M. Danforth, and Peter Sheridan Dodds. 2020.  The growing amplification of social media: Measuring temporal and social contagion dynamics for over 150 languages on twitter for 2009–2020.  Available online at http://arxiv.org/abs/2003.03667.
  * Alsudias and Rayson (2020) Lama Alsudias and Paul Rayson. 2020.  COVID-19 and Arabic Twitter: How can Arab world governments and public health organizations learn from social media? In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_ , Online. Association for Computational Linguistics.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020.  Arabert: Transformer-based model for arabic language understanding.  _ArXiv_ , abs/2003.00104.
  * Banda et al. (2020) Juan M Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, and Gerardo Chowell. 2020.  A large-scale covid-19 twitter chatter dataset for open scientific research–an international collaboration.  _arXiv preprint arXiv:2004.03688_.
  * Bouamor et al. (2019) Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019.  The MADAR shared task on Arabic fine-grained dialect identification.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 199–207, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2020) Emily Chen, Kristina Lerman, and Emilio Ferrara. 2020.  Tracking social media discourse about the covid-19 pandemic: Development of a public coronavirus twitter data set.  _JMIR Public Health Surveill_ , 6(2):e19273.
  * Cinelli et al. (2020) Matteo Cinelli, Walter Quattrociocchi, Alessandro Galeazzi, Carlo Michele Valensise, Emanuele Brugnoli, Ana Lucia Schmidt, Paola Zola, Fabiana Zollo, and Antonio Scala. 2020.  The covid-19 social media infodemic.  _arXiv preprint arXiv:2003.05004_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Haouari et al. (2020) Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2020.  ArCOV-19: The first Arabic COVID-19 twitter dataset with propagation networks.  _arXiv:2004.05861_.
  * Hassan et al. (2021) Sabit Hassan, Hamdy Mubarak, Ahmed Abdelali, and Kareem Darwish. 2021.  Asad: Arabic social media analytics and understanding.  In _Proceedings of the Software Demonstrations of the 16th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Hassan et al. (2020a) Sabit Hassan, Younes Samih, Hamdy Mubarak, and Ahmed Abdelali. 2020a.  ALT at SemEval-2020 task 12: Arabic and English offensive language identification in social media.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1891–1897, Barcelona (online). International Committee for Computational Linguistics.
  * Hassan et al. (2020b) Sabit Hassan, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Ammar Rashed, and Shammur Absar Chowdhury. 2020b.  ALT submission for OSACT shared task on offensive language detection.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 61–65, Marseille, France. European Language Resource Association.
  * Huang and Carley (2020) Binxuan Huang and Kathleen M Carley. 2020.  Disinformation and misinformation on twitter during the novel coronavirus outbreak.  _arXiv preprint arXiv:2006.04278_.
  * Leng et al. (2020) Yan Leng, Yujia Zhai, Shaojing Sun, Yifei Wu, Jordan Selzer, Sharon Strover, Julia Fensel, Alex Pentland, and Ying Ding. 2020.  Analysis of misinformation during the covid-19 outbreak in china: cultural, social and political entanglements.  _arXiv preprint arXiv:2005.10414_.
  * Li et al. (2020) L. Li, Q. Zhang, X. Wang, J. Zhang, T. Wang, T. Gao, W. Duan, K. K. Tsoi, and F. Wang. 2020.  Characterizing the propagation of situational information in social media during covid-19 epidemic: A case study on weibo.  _IEEE Transactions on Computational Social Systems_ , 7(2):556–562.
  * Maiya (2020) Arun S Maiya. 2020.  ktrain: A low-code library for augmented machine learning.  _arXiv preprint arXiv:2004.10703_.
  * Mubarak et al. (2020) Hamdy Mubarak, Ahmed Abdelali, Sabit Hassan, and Kareem Darwish. 2020.  Spam detection on arabic twitter.  In _Social Informatics_ , pages 237–251, Cham. Springer International Publishing.
  * Mubarak et al. (2021) Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2021.  Adult content detection on arabic twitter: Analysis and experiments.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop_.
  * Qazi et al. (2020) Umair Qazi, Muhammad Imran, and Ferda Ofli. 2020.  Geocov19.  _SIGSPATIAL Special_ , 12(1):6–15.
  * Shahi et al. (2020) Gautam Kishore Shahi, Anne Dirkson, and Tim A Majchrzak. 2020.  An exploratory study of covid-19 misinformation on twitter.  _arXiv preprint arXiv:2005.05710_.
  * Yang et al. (2020) Qiang Yang, Hind Alamro, Somayah Albaradei, Adil Salhi, Xiaoting Lv, Changsheng Ma, Manal Alshehri, Inji Jaber, Faroug Tifratene, Wei Wang, et al. 2020.  Senwave: Monitoring the global sentiments under the covid-19 pandemic.  _arXiv preprint arXiv:2006.10842_.


# Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment

Saizhuo Wang  The Hong Kong University of Science and TechnologyHong Kong SAR
<EMAIL_ADDRESS>,  Hang Yuan  The Hong Kong University of Science and
TechnologyHong Kong SAR<EMAIL_ADDRESS>,  Leon Zhou  Columbia
UniversityUSA<EMAIL_ADDRESS>,  Lionel M. Ni  The Hong Kong University
of Science and Technology (Guangzhou)China<EMAIL_ADDRESS>,  Heung-Yeung Shum  IDEA
ResearchChina<EMAIL_ADDRESS>and  Jian Guo  IDEA ResearchChina
<EMAIL_ADDRESS>
###### Abstract.

One of the most important tasks in quantitative investment research is mining
new alphas (effective trading signals or factors). Traditional alpha mining
methods, either hand-crafted factor synthesizing or algorithmic factor mining
(e.g., search with genetic programming), have inherent limitations, especially
in implementing the ideas of quants. In this work, we propose a new alpha
mining paradigm by introducing human-AI interaction, and a novel prompt
engineering algorithmic framework to implement this paradigm by leveraging the
power of large language models. Moreover, we develop Alpha-GPT, a new
interactive alpha mining system framework that provides a heuristic way to
“understand” the ideas of quant researchers and outputs creative, insightful,
and effective alphas. We demonstrate the effectiveness and advantage of Alpha-
GPT via a number of alpha mining experiments.

††copyright: none

##  1\. Introduction

A trading alpha [1] is a financial signal or a function with predictive power
over excess return or risk, and they are usually expressed via symbolic rules
or formulas (machine learning alphas are getting more popular recently but
they are not discussed in this work). Most research work in quantitative
investment focuses on how to find good alphas. See [2] for a number of such
formulaic alphas (e.g., $-\frac{close-open}{(high-low)+0.001}$ computes the
increase from open price to close price relative to the intraday volatility,
and the negative sign indicates a potential mean-reversion effect).

Figure 1. Evolution of alpha mining techniques.

Traditionally, alpha mining has two paradigms (Figure 1). The first paradigm
relies on manual modeling. Quant researchers attempt to translate their
ideas/intuitions about financial markets into formulaic alphas, test their
effectiveness and significance through backtest experiments, and analyze the
reasons of success and failure. Usually, this process is repeated for many
rounds to improve the performance of alphas. The success of this paradigm
depends heavily on the talent and expertise of individuals and suffers from
the problems of inefficiency and labor cost. On the other hand, the second
paradigm seeks alphas through search algorithms such as genetic programming
[3]. Since the search space, composed of all possible combinations for
hundreds of operators and operands (features), is incredibly large, it is
extremely compute-intensive to find satisfactory alphas during the alpha
search process.

Both of these paradigms exhibit common shortcomings. Firstly, it is a
difficult process to find a precise and concise formulaic expression that
encapsulates one’s ideas about trading signals or observed trading
opportunities and patterns. Examples include the formulaic representation of
technical analysis patterns such as Ascending Triangles [4] and Elliott Wave
Theory [5], which exist but are hard to discover. Secondly, understanding and
interpreting a large number of alphas selected by search algorithms is
especially time-consuming and labor-intensive. Lastly, it is unreasonable to
expect creative and effective alphas to come from the stroke-of-genius by
researchers or the brute-force search by algorithms, but rather, it often
comes from a repeated process of experimentation-and-analysis. However,
designing and modifying the parameters and search configurations of
algorithmic alpha mining systems is usually a menial task for researchers.

To address these challenges, we propose the third alpha mining paradigm which
enhances human-AI interaction to improve the effectiveness and efficiency of
alpha research. Based on this new paradigm, we propose the architecture of an
interactive alpha mining system, termed Alpha-GPT. This system incorporates
large language models (LLM) as a mediator between quantitative researchers and
alpha search. Alpha-GPT has three key advantages. First, it can interpret
users’ trading ideas and translate them into fitting expressions. Secondly,
Alpha-GPT can quickly summarize top-performing alphas in natural language for
ease of understanding. And finally, the user can then suggest modifications to
the alpha search which the model will automatically make to future rounds of
alpha mining. This greatly simplifies the workflow (Figure 2) and allows the
user to approach alpha mining from a high-level standpoint (in terms of
abstract ideas).

Our contributions in this work can be summarized from these standpoints:

  * •

We define a new paradigm for alpha mining utilizing human-AI interaction to
improve the effectiveness of alpha research.

  * •

We propose AlphaBot, an algorithm with domain knowledge compilation and
decompilation methods to employ the LLM as a mediator for human-AI
interaction.

  * •

We develop Alpha-GPT, a systematic framework incorporating AlphaBot to realize
our proposed paradigm and a tool for quantitative researchers.

Figure 2. Alpha-GPT internal working pipeline: After a user inputs their
ideas, the system goes into the knowledge compilation module. It uses external
memory to pull similar examples, and combines them into the system prompt. The
module passes everything to the LLM which creates valid alpha expressions and
config files. These alphas are evaluated via Alpha Search, and results are
presented to the user along with an interpretation provided by the Thoughts
Decompiler.

##  2\. User Interface of Alpha-GPT

Figure 3. User interface of Alpha-GPT. Five modules of the UI are annotated
as: (A) Dialog Box, (B) Mining Session Manager, (C) Experiment Monitor, (D)
Analytic Panel, and (E) Alpha Dashboard

In Figure 3 we introduce the user interface (UI) of Alpha-GPT, which is
composed of three main components: Session Manager, Dialog Box, and the Alpha
Mining Dashboard.

Dialog Box: Users input their trading ideas and thoughts into the dialog box.
In response, generated seed alphas, alpha searching progress, the final report
of alpha mining, as well as the performance of the generated alphas, are all
organized into system messages that provide comprehensive feedback. Users can
then analyze the results and provide further direction for alpha mining, and
this dialog continues until effective alphas are found.

Mining Session Manager: Alpha-GPT features a session-based user interface that
stores past interaction history via the session manager. These sessions also
serve to organize user-generated data, which can then be used to further
enhance the system’s performance.

Alpha Mining Dashboard: On the right half, the dashboard is used to display
and analyze alpha mining results. It provides users with more detailed
descriptions of the session. Experiment Monitor displays the alpha mining
experiment progress and current system load as well as all historical alphas
generated during a session. If a specific alpha is selected, its performances
are plotted and visualized on the Analytic Panel. The available plotting
features include a fitness curve across generations for genetic programming, a
backtest curve for a single alpha, and other peripheral analyses such as IC
distribution and Signal decay. Furthermore, Alpha Dashboard includes one-click
storage and deployment, enabling further application and analysis downstream.

Figure 4. Alpha-GPT system architecture. Part of this figure is cited from [6,
7, 8, 9].

##  3\. Architecture and Technology Challenges

Figure 4 shows the system framework we proposed for the interactive alpha
mining paradigm by distilling and abstracting from the architecture design of
Alpha-GPT. Since UI has been introduced in Section 2, we only introduce the
design idea and technical challenge of other modules behind the system.

###  3.1. AlphaBot Layer

AlphaBot is the key layer of Alpha-GPT, as it plays a mediator role in human-
AI interaction. Specifically, this layer consists of four functional modules:
1) Knowledge compiler automatically converts the intents/ideas/thoughts of
quantitative researchers into domain-specific prompts and instructions for an
LLM query; 2) LLM provides APIs or local deployment options for mainstream
large language models such as GPT-4; 3) Thought decompiler translates the
natural language output of an LLM into a configuration understandable by the
algorithmic alpha mining layer; 4) Knowledge Library incorporates extra
knowledge, information, literature and data about alpha mining to improve the
performance and accuracy of LLMs.

####  3.1.1. Knowledge compiler

On domain-specific tasks like alpha mining, a lot of user requests will
include terminology that can only be found in finance. Without further
context, a traditional LLM would be unable to understand the meaning behind
the inputs. This then introduces the need for the knowledge compiler module.
Leveraging the in-context learning capability of LLM [10], the module enhances
the original user request by providing additional context and clarifying
keywords in prompts [11]. Specific phrases like “you are a quant researcher
developing formulaic alphas” help narrow the scope of possible responses to
answers more suited for factor mining.

Since the alpha expressions must also be in a certain format to be valid,
there is a section in the prompt with each possible component of an expression
along with an explanation of how it can be utilized (e.g. “high_1D”: highest
intraday price of stocks). This addition helps the LLM correlate the user
intentions with certain functions that can fulfill that goal, and helps ensure
that the output will be valid. Because this terminology is applicable to the
wider finance domain, this portion of the prompt is also used in the thought
decompiler to prevent hallucinations. Such a module is required to allow users
to make requests in natural language without the ambiguities involved.

####  3.1.2. Large Language Model

There are currently two main ways of utilizing LLMs, each with their own
characteristics.

  * •

Online Request: A swath of companies offer access to their pre-built large
language models through APIs. These are products that charge a nominal amount
per set amount of tokens and include GPT, Claude, and Bard, to name a few. One
advantage of APIs is that it allows for ease of use, and removes the need for
computing power.

  * •

Local Deployment: LLMs can also be developed locally from the ground up. This
approach allows greater customization in how the model functions. They can be
pre-trained on specific datasets to align them towards certain purposes. This
involves domain fine-tuning by learning on relevant documents, and also
reinforcement learning from human feedback [12]. The result is an LLM that is
specifically built for a subset of tasks. However, the user then requires a
large amount of computing power [13, 14] to train and update these models.

####  3.1.3. Thought Decompiler

1

Input: Input message history $H$, Expected number of alphas $N$, expected
minimum number of alpha per round $\theta$, retry limit $\tau$

Output: List of generated alphas $l$

2 $l\leftarrow[]$;

3 while  _len(l) ¡ N_ do

4 $m\leftarrow\text{get\\_llm\\_response}(H)$;

5 $r\leftarrow\text{parse\\_exprs}(m)$;

6 if  _$\text{len}(r) <\theta$_ then

7 continue;

8

9 $v,\tilde{v}\leftarrow\text{validate}(r)$;

10 $l\leftarrow l+v$ ;

11 for  _$I$ in $\\{1..\tau\\}$_ do

12 $retry\\_message\leftarrow\text{make\\_retry\\_prompt}(H,\tilde{v})$;

13 $v,\tilde{v}\leftarrow\text{retry\\_and\\_validate}(retry\\_message)$;

14 $l\leftarrow l+v$;

15

16 $H\leftarrow$ update_prompt$(l)$

17return _l_ ;

Algorithm 1 LLM Reasoning and Thought Decompiling

There is a significant gap between LLM responses and the desired output
structure in alpha mining. Specifically, the gaps can be summarized as
challenges from the following perspectives:

  * •

Nature language to structured data: We need to convert LLM response from
natural language to structured data. For each LLM response message, we need to
extract a list of expression blocks from their raw outputs. And each
expression block follows a certain organization style such as its short name,
expression, and a paragraph of natural language description.

  * •

Token size limit: Since most state-of-the-art LLMs apply a Transformer-based
architecture with self-attention mechanisms, limits on the input sequence
length (number of tokens) is usually a very common concern. In this way, both
inputs and outputs of LLMs have an upper bound for their lengths. This may
cause two issues: 1). We cannot keep sending full conversation history to LLMs
since it will exceed the input token size limit. 2). The size of a single LLM
response is limited, meaning that we can only get a limited number of
expressions per message.

To address these issues, we propose an iterative LLM reasoning procedure
(Algorithm 1). We apply a parser based on regular expressions to parse LLM
outputs. For each LLM-generated alpha, we validate its correctness both
syntactically and semantically. We adopt an abstract syntax tree parser for
syntax checking, and for semantic correctness, we evaluate this expression
with mock data and runtime context to capture any exceptions being thrown out.

In practice, the proportion of correct alphas might be low (only 4 out of 10
alphas generated per round), making the alpha generation process inefficient.
Meanwhile, since the conversation history is appended to LLM inputs, these
incorrect expressions may also affect the generation process that follows.
Hence, we apply an iterative correcting procedure where we prompt the LLM to
re-generate incorrect alphas. Moreover, to address the problem of exceeded
token size limit, at each round we dynamically check if the token size limit
is exceeded, and if so, the input message will be truncated and reorganized to
reduce the number of tokens.

####  3.1.4. Knowledge Library

As mentioned in Section 3.1.1, few-shot in-context learning requires an
external memory that supports efficient retrieval of demonstrations relevant
to trading ideas. In Alpha-GPT, we design a protocol for organizing contents
from across multiple sources. These range from an existing collection of alpha
expressions to financial literature. When the user makes a request, the
knowledge library encodes that query, and finds similar documents that it can
incorporate into the prompt as examples.

###  3.2. Algorithmic Alpha Mining Layer

This layer serves a search enhancement function in Alpha-GPT. Specifically, it
implements an algorithmic alpha mining workflow by taking seed alphas and
improving them with the received search commands and configurations from
AlphaBot, and pushes the most appropriate alphas back to AlphaBot. It consists
of four modules: the algorithmic alpha search module generates alpha
candidates according to the commands from AlphaBot, qualified alphas are
selected from these candidates using the evaluation and backtesting module,
these alphas are further pruned according to a specific prediction target
(e.g., contribution to the return of future 10 days) in the alpha selection
module, and the final alphas are “one-click” deployed by the alpha deployment
module to guarantee the smoothness and correctness of real-time computing
during online trading.

####  3.2.1. Alpha Search Enhancement

The most popular alpha search algorithm used in industry is genetic
programming (GP), which starts from a number of alpha seeds and iteratively
selects formulaic alphas expressed by trees using random crossing and mutation
subtrees according to the fitness of a scoring function. However, GP currently
suffers from three problems: 1) Overfitting, which is extremely dangerous for
quantitative trading. This can be mitigated by strategies such as out-of-
sample evaluation incorporated in GP iterations, fit regularization to reduce
function complexity, and early stopping of iterations. These methods help
ensure alphas generalize well beyond training data, improving their
reliability. 2) Loss of diversity in alphas may result in the aggregation and
accumulation of investment risk and increase the uncertainty of returns. Alpha
diversification can be realized by enforcing more constraints in GP’s
iteration process, and it helps discover robust alpha factors that are
resilient to changing market conditions. 3) Invalid alphas are easily
generated by GP. For example, $log(0)$ and $\sqrt{-5}$, or a sum of two values
with incompatible units (e.g, volume + close). Incorporating a rule base
encompassing mathematical rules, unit consistency rule and financial domain-
specific rules could regulate alpha expression generation.

####  3.2.2. Evaluation and Backtesting

The most straightforward method of evaluating alpha is through backtesting,
which reveals the alpha’s specific performance in an investment strategy. This
process, however, introduces three significant challenges: 1) Introduction of
future information: information from a further point in time when backtesting
could have disastrous consequences on result accuracy. To mitigate this,
Timestamping is used to assign a time label to each piece of input data. This
technique allows the backtesting system to more accurately replicate market
conditions and validate test results, thereby enhancing the reliability of
alpha evaluation. 2) Estimation of transaction costs: conventional coarse-
grained backtesting cannot accurately measure transaction costs, which is
crucial for short-term alphas. To solve this, we conduct a simulation-based
[15] backtest with trade matching rules using more detailed data, such as
order book level data, for selected alphas. This enables us to model how
transaction costs and market price impact alphas at a microstructure level. 3)
The computational cost: the compute-power necessary for alpha mining is also
significant, and we address this problem in the computation acceleration
layer.

####  3.2.3. Alpha Selection

The Alpha Selection module furthers the selection process in the following
ways: 1) Deduplication and Decorrelation: new alphas need to be distinct from
existing ones, but calculating pair-wise similarity between large amounts of
alphas can be time-consuming. Algorithms like KD-Tree, Locality-Sensitive
Hashing (LSH) [16], or Approximate Nearest Neighbors (ANN) can swiftly
determine each prospective alpha’s maximum correlation with others in the
pool. 2) Importance Scoring: while an alpha’s IC score and backtest may
reflect individual performance, in real scenarios, multiple alphas are
combined together in investment strategies, and these metrics do not
accurately reflect how they perform in a larger set. A group of alphas with
low IC scores may outperform a subset composed of the highest-scoring alphas.
Thus, importance scoring techniques such as Shapley Additive exPlanations
(SHAP) [17] and Local Interpretable Model-agnostic Explanations (LIME) [18]
measure this contribution and provide a comprehensive understanding of alphas
in conjunction with one another.

####  3.2.4. Alpha Deployment

In this module, three key aspects need to be properly managed to guarantee the
smooth and correct real-time computation during online trading: 1) Dependency
Management: This involves maintaining and supervising all alpha-data inter-
dependencies to ensure sequential computation and traceability of issues. 2)
Stream-Batch Unification: Inconsistencies between live trading and historical
backtesting of alphas are unacceptable. By adopting the Kappa Architecture, a
unified real-time processing layer is maintained. Thus, all data is processed
in the same manner, eliminating inconsistencies between batch and stream
processing during alpha generation. 3) Automatic Alpha Verification: This is
employed to validate all system-maintained alphas, monitor data quality, and
identify discrepancies. This ongoing verification ensures the reliability,
timeliness, and accuracy of the deployed alphas.

###  3.3. Alpha Computation Acceleration Layer

Alpha computation requires preprocessing financial data from various sources
such as price-volume data, financial statement data, etc. Because of the
computational overhead of processing high-frequency data such as orderbook
level data, computational acceleration plays a key role. Also, in alpha
mining, billions of alphas are calculated, making the speed of alpha
calculations crucial for effectively exploring the alpha search space. Below
we outline a few key computation acceleration techniques employed. Streaming
Algorithms: particularly beneficial in performing rolling window computations
on large time-series datasets. They incrementally update calculations with
each new data point in the window, significantly optimizing computational
efficiency and memory usage for these sliding-scale operations such as
ts_corr(rolling correlation of time-series data). Vectorized Computation:
enables efficient financial data processing by eliminating explicit loops,
leveraging hardware capabilities for concurrent processing, and optimizing
memory management. SIMD (Single Instruction, Multiple Data) and SIMT (Single
Instruction, Multiple Threads) [19]: allows simultaneous computations on
multiple data points, fully exploiting hardware capabilities. Memory
Optimization: pre-allocation of memory pool, layout optimization, and zero-
copy, etc, are employed to minimize performance loss from discontinuous memory
access and unnecessary memory allocation. Data Partitioning: Divides large-
scale data into smaller, manageable chunks for independent processing.
Multithreading: enables parallel computations on each partition using multiple
threads. This approach significantly boosts computational speed when
processing large financial datasets. GPU Acceleration [20]: employs CUDA cores
for parallel processing, transforming CPU-bound operations into GPU-
accelerated tasks and improves performance for data-intensive computations.

Table 1. Comparison of average top-20 out-of-sample IC between alphas
generated by Alpha-GPT before and after search enhancement.

| Trend Discrepancy | Shape | RSI | Momentum | Mean Reversion | Flow of Funds
---|---|---|---|---|---|---
Before search enhancement | 0.01151 | 0.00995 | 0.01109 | 0.00951 | 0.01130 | 0.00952
After search enhancement | 0.02256 | 0.02190 | 0.02527 | 0.02763 | 0.02187 | 0.02160

(a) Golden-cross pattern

(b) Bollinger bands (upper breakout)

(c) Three white soldiers

Figure 5. Trading patterns and the corresponding alphas generated by Alpha-GPT
that capture them.  Figure 6. Backtest net values of alphas after different
stages of interactions. Figure 7. Alphas generated based on trading ideas and
the corresponding explanations generated by Alpha-GPT.

##  4\. Experiments

We conduct experiments on Alpha-GPT with the goal of verifying the following
research questions (RQ):

  * •

RQ1: Can Alpha-GPT generate expressions that are consistent with the input
trading ideas?

  * •

RQ2: How effective is the algorithmic alpha mining layer in enhancing the seed
alphas from LLM?

  * •

RQ3: Can users effectively interact with Alpha-GPT to guide the mining
process?

  * •

RQ4: Can Alpha-GPT successfully explain the trading ideas behind alpha
expressions?

###  4.1. Experimental Setup

The specifications of Alpha-GPT and other relevant information about our
experiments include:

Data and operators: We use inter-day volume-price data of stock markets. This
data includes the basic candlestick chart data (OHLCV), volume-weighted
average price (VWAP), and sector data. We also include 19 basic operators
implemented in [21] including time-series operations, cross-sectional
operations, group-wise operations and basic element-wise operations.

Knowledge Library: We construct the knowledge library based on the alphas
proposed in [2]. For each alpha, we first decompose it into sub-expressions
and explain them. Then we explain the combination of these sub-expressions to
form the whole trading idea. Document embeddings are indexed via
Faiss111https://github.com/facebookresearch/faiss. Note that we only employed
external memory when generating alphas for trading ideas that align well with
those in [2].

LLM and Adapter: For natural language interaction, we use OpenAI’s chat
completion API with “gpt-3.5-turbo-16k-0613” model base. For the embedding
model used in knowledge retrieval, we use OpenAI’s “text-ada-embedding-002”
API with embedding dimension of 1536. The LLM generates a batch of alphas at a
time, and will be asked to correct alphas with syntax or semantic errors.

Alpha searching and evaluation: Alphas are searched by the genetic programming
model with a fitness score defined by the Information Coefficient (IC). We
evaluate the performance of those alphas on out-of-sample criteria such as IC,
annual return, Sharpe ratio, etc.

###  4.2. Idea-Formula Consistency

We first demonstrate that Alpha-GPT can generate formulaic alphas that are
consistent with the user’s given trading idea. Figure 5 illustrates three
alpha expressions generated based on given trading ideas and their
correspondence to the patterns in the candlestick chart. The candlestick chart
is plotted from the weekly data of the S&P500 index from 2020 to 2023.

The first trading idea aims to capture golden cross patterns. The alpha value
reflects the divergence between the fast and slow moving average curves. The
second trading idea characterizes the breakout signals of Bollinger bands, and
the corresponding alpha is a binary signal that gets activated when the upper
bound is crossed. The third trading idea aims to capture three consecutive
bullish movements on the candlestick chart, and the generated alpha
successfully identified those patterns. These examples demonstrate that the
generated alphas correctly capture the trading ideas.

###  4.3. Search Enhancement

Table 1 compares the out-of-sample IC of alphas before and after search
enhancement by the algorithmic alpha mining layer on 7 different trading
ideas. We can see that search enhancement significantly improves the
performance of Alpha-GPT and is critical in the Alpha-GPT workflow.

###  4.4. Human-AI Interaction

Figure 6 illustrates the backtest curve of the alpha generated throughout the
human-AI interaction. The backtest is conducted on US stock data from 2012 to
2021. The initial trading idea is to characterize the long-short imbalance.
After several rounds of search enhancement and user interaction, the backtest
performance of the resulting alphas significantly improved. More details about
these interactions are present in the UI example in Figure 3.

###  4.5. Alpha Explanation

Figure 7 presents examples of alpha expressions generated by Alpha-GPT based
on given trading ideas, and the corresponding natural language explanations of
these alphas also generated by Alpha-GPT. From these examples we can see that
Alpha-GPT can provide appropriate explanations of the generated alphas,
relieving the burden of human researchers to interpret these expressions by
themselves.

##  5\. Related Work

A lot of algorithms have been studied for formulaic alpha mining. Examples
include Monte Carlo random search, Markov-chain Monte Carlo [22], genetic
programming [23] and their variants [3], and reinforcement learning [24].
However, these methods all require the user to directly define the algorithmic
configurations, providing limited interactivity compared with Alpha-GPT.
Meanwhile, LLMs such as GPT [25] have demonstrated emergent abilities [26] and
achieved superior performance on various tasks. Besides, LLMs have also shown
great reasoning [27, 28] and planning capabilities [29]. In this way, an LLM
can be regarded as a core thinking module and be integrated with various
peripheral tools [30] to form intelligent LLM-powered agents [31].

##  6\. Conclusion and Future Work

In this paper, we propose a new paradigm and a new system for alpha mining by
leveraging the power of large language models. Further study on prompt
engineering, LLM fine-tuning, alpha search algorithms, and knowledge library
construction can be conducted to improve the capability of this system.

## References

  * [1] Igor Tulchinsky.  Introduction to Alpha Design.  In Finding Alphas, pages 1–6. John Wiley & Sons, Ltd, 2019.
  * [2] Zura Kakushadze.  101 Formulaic Alphas.  arXiv:1601.00991 [q-fin], March 2016.  arXiv: 1601.00991.
  * [3] Tianping Zhang, Yuanqi Li, Yifei Jin, and Jian Li.  AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment, April 2020.  arXiv:2002.08245 [q-fin].
  * [4] Andrew W. Lo, Harry Mamaysky, and Jiang Wang.  Foundations of Technical Analysis: Computational Algorithms, Statistical Inference, and Empirical Implementation.  The Journal of Finance, 55(4):1705–1765, 2000.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/0022-1082.00265.
  * [5] R.N. Elliott and R.R. Prechter.  R.N. Elliott’s Masterworks: The Definitive Collection.  New Classics Library, 2005.
  * [6] Jian Guo, Saizhuo Wang, Lionel M. Ni, and Heung-Yeung Shum.  Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligence, December 2022.  arXiv:2301.04020 [cs, q-fin].
  * [7] slundberg/shap: A game theoretic approach to explain the output of any machine learning model.
  * [8] Yufei Wu, Daniele Magazzeni, and Manuela Veloso.  How Robust are Limit Order Book Representations under Data Perturbation?  In ICML Workshop on Representation Learning for Finance and E-Commerce Applications, 2021.
  * [9] Fully Sharded Data Parallel: faster AI training with fewer GPUs, July 2021\.
  * [10] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, October 2022.  arXiv:2202.12837 [cs].
  * [11] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, July 2021.  arXiv:2107.13586 [cs].
  * [12] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep Reinforcement Learning from Human Preferences.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299–4307, 2017.
  * [13] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.  arXiv:1910.02054 [cs, stat], May 2020.  arXiv: 1910.02054.
  * [14] Song Han, Huizi Mao, and William J. Dally.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [15] Selim Amrouni, Aymeric Moulin, Jared Vann, Svitlana Vyetrenko, Tucker Balch, and Manuela Veloso.  ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Markets.  In Proceedings of the Second ACM International Conference on AI in Finance, ICAIF ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [16] Aristides Gionis, Piotr Indyk, and Rajeev Motwani.  Similarity Search in High Dimensions via Hashing.  In Proceedings of the 25th International Conference on Very Large Data Bases, VLDB ’99, pages 518–529, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.
  * [17] Scott M Lundberg and Su-In Lee.  A Unified Approach to Interpreting Model Predictions.  In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [18] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery.
  * [19] John L. Hennessy and David A. Patterson.  Computer Architecture, Fifth Edition: A Quantitative Approach.  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 5th edition, 2011\.
  * [20] NVIDIA Corporation.  NVIDIA CUDA C Programming Guide, 2010.
  * [21] Jiadong Guo, Jingshu Peng, Hang Yuan, and Lionel Ming-shuan Ni.  HXPY: A High-Performance Data Processing Package for Financial Time-Series Data.  Journal of Computer Science and Technology, 38(1):3–24, February 2023.
  * [22] Ying Jin, Weilin Fu, Jian Kang, Jiadong Guo, and Jian Guo.  Bayesian Symbolic Regression, January 2020.  arXiv:1910.08892 [stat].
  * [23] Can Cui, Wei Wang, Meihui Zhang, Gang Chen, Zhaojing Luo, and Beng Chin Ooi.  AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment.  In Proceedings of the 2021 International Conference on Management of Data, pages 2208–2216, Virtual Event China, June 2021. ACM.
  * [24] Shuo Yu, Hongyan Xue, Xiang Ao, Feiyang Pan, Jia He, Dandan Tu, and Qing He.  Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning, May 2023.  arXiv:2306.12964 [cs, q-fin].
  * [25] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs], July 2020.  arXiv: 2005.14165.
  * [26] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent Abilities of Large Language Models.  Transactions on Machine Learning Research, June 2022.
  * [27] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  May 2022.
  * [28] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of Thoughts: Deliberate Problem Solving with Large Language Models, May 2023.  arXiv:2305.10601 [cs].
  * [29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao.  ReAct: Synergizing Reasoning and Acting in Language Models.  September 2022.
  * [30] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language Models Can Teach Themselves to Use Tools, February 2023.  arXiv:2302.04761 [cs].
  * [31] Lilian Weng.  LLM Powered Autonomous Agents, June 2023.  Section: posts.