#  How to Leverage High Altitude Platforms in Green Computing?

Wiem Abderrahim *  |, Osama Amin *  | and Basem Shihada *  |
The authors are with the Computer, Electrical and Mathematical Sciences and
Engineering (CEMSE) Divison, King Abdullah University of Science and
Technology (KAUST), Thuwal 23955, Makkah Prov., Saudi Arabia (e-mail:
wiem.abderrahim, osama.amin, basem.shihada @kaust.edu.sa)

###### Abstract

Terrestrial data centers suffer from a growing carbon footprint that could
contribute with $14\%$ to global CO2 emissions by 2040. High Altitude Platform
(HAP) is a promising airborne technology that can unleash the computing
frontier in the stratospheric range by hosting a flying data center. HAP
systems can endorse the sustainable green operation of data centers thanks to
the naturally low atmospheric temperature that saves cooling energy and its
large surface that can host solar panels covering energy requirements.
Throughout this article, we define the operation limitations of this
innovative solution and study the energy-efficiency-related trade-offs. Then,
we shed light on the significance of the scalability of the data center-
enabled HAP architecture by investigating potential bottlenecks and proposing
different deployment scenarios to avoid network congestion. We also highlight
the importance of the management agility of the data center-enabled HAP system
by defining effective management techniques that yield high-performing data
centers. Our results demonstrate that deploying a single data center-enabled
HAP can save $12\%$ of the electricity costs.

##  I Introduction

Climate change is a momentous threat that humankind faces because it severely
affects the health of the planet and jeopardizes all forms of life. Therefore,
prompt and collective actions should be taken to decelerate its effects by
reducing excessive carbon emissions worldwide. One of the major industries
that is raising serious concerns in this regard is the data center industry;
which is anticipated to contribute with $14\%$ to global CO2 emissions by 2040
[1]. Indeed, data centers will consume around $13\%$ of the worldwide
electricity by 2030; which is predominately generated by carbon-intensive
fossil fuels [2]. Moreover, these large-scale computing infrastructures; which
are already behind major energy-efficiency issues, are expected to expand
exponentially to process and store our continuously-growing data generated by
data-intensive applications emerging in different fields such as artificial
intelligence, smart cities and telecommuting [3]. For example, the total
number of installed instances such as virtual machines and containers in
global data centers increased from about 150 million in 2016 to 500 million in
2021 [4]. Moreover, Amazon has deployed large-scale data centers in 25
geographic regions to support 80 availability zones and Google has deployed
more than 18 data centers in Americas, Asia, and Europe recently [1]. Given
this alarming situation, joint efforts from academia and industry are required
to improve the energy-efficiency of data centers beyond recommending the
commonly adopted approaches that are based on renewable energy usage or carbon
offset mechanisms [5].

Pragmatically, non-traditional computation paradigms are needed to provide
innovative and tangible solutions that vigorously tackle the energy-efficiency
issues of current data centers. Different technologies can be incorporated
with the terrestrial data center to contribute to environmental sustainability
by providing carbon-neutral integrated data centers. Underwater data centers
are currently publicized as the future of data storage thanks to their high
reliability. Moreover, satellite data centers have attracted growing interest
recently as the newly defined computing frontier. However, most of these data
center’s architectures present several drawbacks ranging from limited coverage
for underwater data centers to high costs, additional delays, and limited
payload for satellite data centers. Interestingly, high-altitude platforms
(HAP) seem a good trade-off between these incorporated technologies because
they offer larger coverage areas than underwater systems, support more
important payloads, and guarantee easier maintenance and lower delays than
satellites.

Figure 1: Data Center Enabled HAP Use Cases

Therefore, we believe that the data center-enabled High Altitude Platform
(HAP) is a distinguishable green alternative to traditional terrestrial data
centers, given the encouraging merits of the HAPs. Indeed, HAP can be a core
futuristic airborne network component that will unleash the networking
frontier in the stratospheric range at an altitude between 17 km and 20 km [6,
7]. HAP offers several unique advantages, especially from energy and
communication perspectives. Firstly, the HAP location at the stratosphere
saves the cooling energy thanks to the naturally low atmospheric temperature
(between $-50^{\circ}\text{C}$ and $-15^{\circ}\text{C}$). Hence, a data
center-enabled HAP can offload some workload from terrestrial data centers,
saving the associated cooling energy. Moreover, HAP can host large solar
panels that harvest substantial amounts of solar energy thanks to HAP’s large
surface and its location above the clouds. The HAP feeds the servers with the
solar energy harvested during the daytime and stored in the Lithium-Sulphur
batteries during the nighttime. Hence, the harvested solar energy can cover
amply the computational power required by the data center’ servers; while the
necessary energy conversion and management strategies are efficiently applied
[7, 6]. Secondly, the HAP’s location at high altitudes offers LoS
communication links with several receivers thanks to the the availability of
large terrestrial footprint and the absence of obstructions in the horizon.
Hence, HAPs can establish reliable direct links with a large number of
terrestrial base stations [7, 6]. These advantages enable the data center-
enabled HAP to offer a rich panoply of computing services that range from
supporting internet of things applications to intelligent transportation
systems as depicted in Fig .1. Moreover, the data center-enabled HAP improves
the dependability properties of these computing services by boosting the
reliability of the terrestrial data center in urban areas and its
maintainability in disaster areas. Besides, the flying data center hosted in
the HAP guarantees the availability of the computing services in rural and
remote areas in addition to under-connected areas such as maritime area.

In this work, we study the data center-enabled HAP architecture from different
aspects related to energy-awareness, scalability and management agility while
defining the limits and conditions under which this solution can be leveraged
for green computing. First, we investigate the major settings and trade-offs
that impact the performance of the data center-enabled HAP in terms of energy
saving. Then, we analyse the potential bottlenecks of this architecture and we
provide various deployment scenarios to circumvent the anticipated scalability
issues. Finally, we advocate different management techniques to preserve the
desired operation of the data center-enabled HAP over long-duration missions.

##  II Energy-awareness of Data center-enabled HAP

In this section, we study the data center-enabled HAP from an energy-awareness
perspective. Specifically, we analyze the conditions and trade-offs that
impact the energy-saving benefits of the data center-enabled HAP. We conduct
all the simulations based on realistic parameters detailed in Table I.

TABLE I:  Simulation Settings Type | Parameter | Numerical Value
---|---|---
| Supply Temperature |  $299.15$ K
Cooling/ | Server Initial Temperature |  $310$ K
Thermal | CPU Initial Temperature |  $318$ K
Inputs | Thermal Resistance |  $0.34$ K/W
| Server Heat Capacity |  $340$ J/K
| Maximum Payload | $450\>\mathrm{kg}$
| Flying Server Payload | $11\>\mathrm{kg}$
HAP | Area of the Photovoltaic (PV) Surface | $8000\>\mathrm{m}^{2}$
Inputs | Efficiency of the PV | $0.4$
| Propeller efficiency | $0.8$
| Battery capacity | $2\>\mathrm{kWh/kg}$
Trans- | Antennas in TDC | 2
mission | Antennas in HAP | 16
Inputs | Carrier Frequency |  $31$ GHz
| Channel Bandwidth |  $100$ MHz

###  II-A Energy Saving

A breakdown of the energy consumed by a data center shows that the cooling
infrastructure and the computational infrastructure are the main components
that absorb the data center energy with up to $40\%$ and $56\%$ respectively
[8]. Therefore, the data center-enabled HAP can significantly reduce the
energy consumed by a terrestrial data center thanks to two main reasons.
First, the data center-enabled HAP saves the cooling energy thanks to the
naturally low atmospheric temperature of the stratosphere (between
$-50^{\circ}\text{C}$ and $-15^{\circ}\text{C}$); where the HAP is located.
This advantage dismisses the need for the presence of cooling units in the HAP
because the average temperature in the stratosphere is substantially lower
than the recommended temperature for the data center (between $18^{\circ}{C}$
and $26^{\circ}\text{C}$) by the American Society of Heating, Refrigerating
and Air-Conditioning Engineers. Second, the data center-enabled HAP saves the
computational energy thanks to the harvested solar energy. Indeed, the HAP
feeds the servers with the solar energy harvested during the daytime and
stored in the Lithium-Sulphur batteries during the nighttime. However, the
terrestrial data center feeds its servers with the electric energy supplied
through the electrical grid constantly.

We note that the energy saved by the data center-enabled HAP is impacted by
the HAP’s location and the considered period of the year. For instance, the
HAP can harvest more solar energy during June if it is located in the Northern
hemisphere. Indeed, the solar radiation and the daylight duration are more
critical since the Northern hemisphere is closer to the Sun in June.

We also highlight the vital role of distributed learning in improving the
energy efficiency of the data center-enabled HAP. Indeed, the optimization of
such complex non-linear systems geographically distributed between the sky and
the ground by conventional optimization frameworks requires complicated
heuristics and tedious calculations. However, implementing a distributed
learning algorithm that trains the data of the airborne servers and the
terrestrial servers yields a global energy-efficiency model that predicts the
necessary policies with less required complexity. Since learning approaches
are generally energy-hungry, the adopted algorithms should be carefully
selected to conserve the energy-efficiency of the data center-enabled HAP [9].

###  II-B HAP Flying Condition

Energy management of HAPs is crucial because these platforms are typically
designed for long-duration missions. Therefore, we establish the HAP flying
condition to examine when the HAP can stay stable and keep flying. Hence, we
study the energy budget of the HAP such that the daily harvested energy is
equal to the consumed energy. The main energy source for the daytime operation
of HAPs is the solar energy harvested through the photovoltaic system
implemented on the surface of the inflatable part of the HAP. The harvested
energy depends mainly on the latitude of the HAP and the considered day of the
year [7, 6, 10]. HAPs also incorporate energy storage components; which are
typically Lithium-Sulphur batteries or hydrogen fuel cells. These batteries
support the nighttime operation of the HAP and are fed by the solar energy
harvested during the daytime. Under the flying condition, the harvested energy
should cover the consumed energy by the HAP; which captures the energy
consumed by its propulsion subsystem, the energy consumed by its payload
subsystem and wireless communication energy. The propulsion energy includes
the energy required to localize and stabilize the HAP. The payload energy is
the computational energy of the servers hosted in the HAP and it depends on
the servers characteristics (e.g. the desirable utilization ratio and the
service rate in million instructions per second) besides the workload
characteristics (e.g. the arrival rate and the mean task length).

To offer qualitative insight into the HAP flying condition, we study the
maximum utilization ratio per server under this condition based on realistic
parameters of HAP and data centers[11, 12]. We plot the probability
distribution function of the utilization ratio in Fig. 2 over the different
days of the year. Indeed, the HAP flying condition determines the maximum
computational energy consumed by the HAP-hosted servers and depends mainly on
the servers’ utilization ratio. We can assess the utilization ratio of a
server by dividing the workload arrival rate by the server’s service rate. We
consider the server to be under-utilized if its utilization ratio is below
$70\%$, and it is over-utilized if its utilization ratio is close to $100\%$.

The HAP flying condition is mainly impacted by the number of airborne servers
hosted in the HAP. For instance, when 40 servers are deployed in the HAP, the
effective usage of the airborne servers (i.e., utilization ratio between
$70\%$ and $100\%$) is possible during most of the year’s days as depicted in
Fig. 2. In this case, the collected solar energy covers exactly the propulsion
energy and the payload energy and the servers are mostly used efficiently
because the HAP is fully loaded with 40 servers. However, when only 35 servers
are deployed in the HAP, the airborne servers need to be over-utilized (i.e.,
utilization ratio close to $100\%$) to fully use the harvested energy.
Therefore, the harvested energy and the servers’ capacities are effectively
used when the number of airborne servers increases.

Figure 2: HAP Flying Condition over the Year’s Days

Figure 3: Data center-enabled HAP Deployment Scenarios

###  II-C Trade-offs

It is important to reduce the energy consumed by the terrestrial data center
by maximizing the number of airborne servers and the computing workload
offloaded to the HAP. However, the adoption of such techniques leads to a
resource utilization dilemma since energy consumption and resource utilization
are strongly coupled. On the one hand, the over-utilization of the available
resources threatens the physical capabilities of the system and might yield
dysfunctional servers or unbalanced HAP. For instance, high central processing
unit (CPU) utilization and/or memory utilization overload(s) the server and
lead(s) to unresponsive or frozen systems. Also, high resource utilization
requires high harvested energy, which may preclude the flying condition. On
the other hand, the under-utilization of the available resources might yield
aging servers and substantial wasted energy since the idle servers can consume
as high as $60\%$ of the peak power [8]. Therefore, it is valuable to adopt
the adequate resource management (e.g. consolidation, containerization)
techniques in data center-enabled HAP to reduce the consumed energy without
overlooking the physical capacities of the available resources.

Another relevant trade-off to energy-efficiency is related to service
availability. Indeed, data centers have strict service level agreements (SLA)
that include for instance providing high availability (24/7), short delays and
maximum security to the hosted services. To meet these requirements and avoid
the SLA’s violation penalties, resources are usually over-provisioned.
Therefore, it is valuable to find the appropriate resource provisioning and
workload scheduling in the airborne servers present in the HAP that not only
reduces energy consumption but also avoids performance degradation especially
in terms of service continuity, reliability and security.

##  III Scalability of Data Center-Enabled HAP

In this section, we study different deployment scenarios to overcome the
scalability issues of the data center-enabled HAP architecture.

###  III-A Bottlenecks in Data Center-Enabled HAP

According to Cisco, global cloud data center traffic was estimated to reach
20.6 zettabytes per year in 2021 given the data-intensive applications such as
artificial intelligence and internet-of-everything applications [13]. This
skyrocketing data consumes huge computing and communication resources and
might create potential bottlenecks in the data center-enabled HAP. For
instance, some airborne servers may become hotspots if the offloaded traffic
to the HAP is not properly load-balanced. This issue leads to overloaded
servers’ buffers and impacts substantially the experienced queuing delay.
Moreover, the wireless link between the HAP and the terrestrial data center is
a typical bottleneck in the data center-enabled HAP. Indeed, link outage
happens if the offloaded workload from the terrestrial data center cannot be
supported by the established transmission link to the HAP. This issue leads to
a supplementary outage delay besides the imposed transmission delay engendered
by the servers distribution between the HAP and the terrestrial data center.
It also deteriorates the system reliability and requires supplementary re-
transmission to recover the lost workload. Therefore, the servers capabilities
and the link characteristics should be carefully considered when scheduling
the workload to the HAP to achieve higher performance in terms of delay and
reliability. Beyond conventional workload scheduling approaches, we need to
develop ambitious solutions that improve the scalability of this integrated
terrestrial-aerial computing system. In this regard, we propose three
deployment scenarios for data center-enabled HAP in the next section.

Figure 4: Year’s Period Impact

###  III-B Multi-HAPs Constellations

Multi-HAPs constellations can coordinate and cooperate to support larger
payloads by offering reliable services [7]. In our airborne computing
scenario, a constellation of multi-HAPs can circumvent the potential
bottlenecks by supporting larger payloads and offering a more prominent
capacity [7]. However, the server payload should be carefully considered in
this configuration given its impact on the flying condition. Specifically, a
distributed air computing platform supported by different HAPs can overcome
the computational overload of one HAP. Moreover, the multi-links established
between different HAPs and the terrestrial data center increase the available
capacity and mitigate the link outage substantially. However, a proper
constellation design is imperative to meet the growing data traffic
requirements. Firstly, it is essential to determine the number of necessary
airships and to optimize their locations while reducing the overlapped
footprint to maximize the resources’ usage. For instance, we should
investigate when is it more effective from an energy perspective and from a
performance perspective to deploy one HAP loaded with some airborne servers
along with a terrestrial data center and when to deploy all the servers of the
terrestrial data center in different HAPs (Fig. 3). Secondly, it is important
to determine the interconnection type between the different HAPs (e.g. optical
link, radio link) while implementing the necessary spectrum sharing and
interference management techniques not only within HAPs but also with adjacent
networks such as satellites’ constellations and low altitude platforms
constellations. Moreover, the LOS conditions with the terrestrial data centers
and between the HAPs leads to crucial interference issues that must be
mitigated through proper multiplexing techniques that meet the energy-
efficiency objectives of the data center-enabled HAP. It is indispensable to
determine if the control strategy of the multi-HAPs-enabled data center should
be centralized or distributed within the HAPs [14]. Specifically, in a
centralized strategy (Fig. 3), the control is assigned to one of the HAPs that
has the necessary information about the remaining HAPs capabilities and
schedules the offloaded workload from the terrestrial data center accordingly.

In a distributed strategy, all the HAPs coordinate their communication and
cooperate to manage the offloaded workload from the terrestrial data. The
centralized strategy is advantageous because the controller-HAP has a global
view about the constellation and can achieve a near optimal control solution
at the expense of a relatively long response time. The distributed strategy is
advantageous because it circumvents the single point of failure problem at the
expense of higher complexity and supplementary communication overheads. The
centralized-control strategy can be improved through distributed learning
approaches to optimize the control rules regarding response time and energy
efficiency by offloading the appropriate workload amount to the appropriate
HAP.

###  III-C Deployment Scenarios

To offer qualitative insight about the scalability’s gain of data center-
enabled HAP, we compare the system electricity cost; which represents the main
operational expenditure source in data centers; according to the deployment
scenarios shown in Fig 3 [15]. In the first scenario, a terrestrial data
center is considered to process all the incoming workload. In the second
scenario, some servers are placed in the HAP and most of the servers are
hosted in the terrestrial data center. In the third scenario, 4 HAPs
collaborate with the terrestrial data center to process the incoming workload.
Throughout these simulations, the system electricity cost is evaluated for the
maximum workload arrival rate determined by the HAP flying condition according
to the variation of latitudes and days as shown in Fig. 4. It is worthy to
note that the electricity cost of the terrestrial data center includes the
cost of the computational energy and the cooling energy. However, the
electricity cost of the data center-enabled HAP system captures the saved
energy thanks to using the HAP while considering the wireless transmission
energy cost for the communication link from terrestrial base-station to the
HAP in addition to cost of utilized terrestrial servers.

First, by carrying out our own simulations, we notice from Fig. 4 the
deployment of a flying data center in the HAP saves substantially the
electricity cost of a fully-terrestrial data center with $=12\%$ achieved when
one HAP is deployed and with $=36\%$ when 4 HAPs are deployed respectively for
the maximum workload arrival rate. Therefore, the electricity costs of
terrestrial data centers can be further cut down with lower arrival rates
thanks to HAPs deployment, which reduces the tasks arrival rates to
terrestrial data centers. Moreover, we notice from Fig. 4 that the highest
electricity costs are recorded at the beginning and the end of the year
because the lowest solar energy can be harvested during this period in the
Northern hemisphere.

##  IV Management Agility of Data Center-Enabled HAP

In this section, we provide different management techniques that entail
workload management, network management and airship management to enable the
desired performance of the data center-enabled HAP.

###  IV-A Workload Management

Incoming workload to data centers are time varying and heterogeneous because
they are originated from different applications and serve various users.
Therefore, different quality of service levels are required from the data
center enabled-HAP. However, the characteristics of the servers hosted in the
HAP are different from those hosted in the terrestrial data center. For
instance, the servers hosted in the HAP are lighter in terms of weight, have
less advanced computing features and are limited in number to keep the HAP
balanced. Hence, an agile workload management is imperative to improve the
aggregated performance of the data center-enabled HAP in terms of reliability
and delay.

To investigate the data center-enabled HAP delay performance, we consider the
task length characteristic of the workload and study its impact on the delay
performance versus the workload arrival rate; as shown in Fig. 5. For a short
task length workload, the transmission delay needed to send the workload to
the servers hosted in the HAP and symbolized with round trip time (RTT) is
significantly lower compared to the queuing delay experienced in the
terrestrial data center. Therefore, the transmission delay can be tolerated
because the workload spends substantial time in the terrestrial servers’
queues before processing. However, this observation is not valid when the
incoming workload is characterised with a long task length. Specifically, we
notice that the transmission delay exceeds the queuing delay for some arrival
rates as depicted in Fig. 5.

Another type of delays that needs careful consideration in the multi-HAPs
scenario is the relaying delay. We note that the relaying delay represents the
time required to relay the workload inter-HAPs. We notice that the relaying
delay is lower than the queuing delay for a 3-HAPs constellation and long task
length (2 hops) or 8-HAPs constellation (7 hops) for a short task length
workload; as shown in Fig. 5. Thus, we conclude that the workload can be
relayed between 3 HAPs for long task length and 8 HAPs for short task length
without impacting the total delay performance. However, the relaying delay
between 3 HAPs exceeds the queuing delay if the workload is characterised with
a long task length.

Figure 5: Workload Impact on Delays of Data center-Enabled HAP

The agile workload management is also fundamental for delay-tolerant
applications used in far-away areas such as disruption-tolerant networking.
The data center-enabled-HAP can be useful in this case when a HAPs’
constellation is deployed over different regions to widen the coverage.
Accordingly, the delay-tolerant workload is offloaded to the nearest HAP in
the constellation such that the experienced delay is minimized.

Moreover, an agile workload management that tracks the renewable energy
sources is gaining momentum to address the energy concerns of data centers.
However, the trade-off between energy demand and renewable energy supply is
challenging for different reasons. First, the incoming workload to data
centers are time varying and heterogeneous. Second, renewable energy sources
are intermittent and depend tightly on weather conditions and geographical
locations.

###  IV-B Network Management

The distribution of the computing resources between the HAP and the
terrestrial data center engenders supplementary overhead to synchronize and
replicate the offloaded data to the HAP. Therefore, effective network
management strategies based on self-organizing approaches should be adopted in
the data center-enabled HAP. On the one hand, the control strategy of the data
center-enabled HAP’s network is fundamental to pool and orchestrate the
distributed resources with agility and flexibility and provide them as
disaggregated resources in the entire data center-enabled HAP and within a
data center-enabled multi-HAPs. Different technologies can be used to
implement disaggregation such as composable data center infrastructure. This
technology is based on the software-defined paradigm to enable the abstraction
of the disaggregated resources and their composition/decomposition and
management over the HAP or the multi-HAPs and the terrestrial data center.

On the other hand, higher bandwidth requirements are imposed on the
communication link between the HAP and the terrestrial data center and between
the HAPs to embrace the continuously-increasing data traffic and its control
information. In this regard, the terahertz and optical wireless communications
of 6G networks will play a key role to handle incoming workload to terrestrial
data center when transferred to the stratosphere. Besides, different
transmission technologies can be used to enhance the link capacity and
optimize its performance. For instance, massive multiple-input multiple-output
(MIMO) offers a high link robustness based on the aggressive spatial
multiplexing and can be easily implemented on the HAP thanks to its large
surface. Massive MIMO can be combined with various access techniques such as
orthogonal multiple access to achieve a high spectral efficiency between the
HAP and the terrestrial data center by using maximum ratio combing and zero-
forcing precoding methods.

###  IV-C HAP Management

Even though HAPs are designed for long duration missions, regular maintenance
is required to guarantee the reliable performance of the data center enabled-
HAP. First, it is important to supervise if the airship is operating under the
prescribed service conditions such as the supported stratospheric turbulence
and load. It is also crucial to check the energy storage system of the airship
and the electric engines and to monitor the loss percentage of the gas
responsible for lifting the airship.

Moreover, it is important to supervise the operation of the data center hosted
in the HAP by monitoring the security threats and the non-malicious failures
such as memory leakage and CPU overload. Therefore, secure and dependable
mechanisms adapted to the capabilities of the HAP-flying data center should be
implemented and periodically updated to offer secure, recoverable and reliable
computing services. For instance, the blockchain technology can be deployed in
the data center enabled-HAP to secure the data hosted in the HAP.

Besides, advanced redundancy mechanisms can be deployed within different HAPs
in a multi-HAPs constellation setting to fight against the non-malicious
failures. Interestingly, HAPs have the asset to offer an agile maintenance
compared to satellites because the possibility of return to Earth is easier.
However, it is valuable to consider the trade-off between the maintenance
costs; which contribute significantly to the operational expenditures; besides
the takeoff/landing costs on the one hand and the data center enabled-HAP
performance on the other hand.

##  V Conclusion and Future Directions

In this article, we shed light on the potential of the data center-enabled HAP
to unleash the green computing frontier to the stratosphere. The main goal of
this first stage study is to investigate the energy-efficiency and the
performance of data center-enabled-HAP. However, it is important to explore
the potential research challenges towards a comprehensive study. From a
financial perspective, it is crucial to analyze the capital and operational
expenditures of the data center-enabled HAP because they may protract its
adoption given the supplementary generated costs of the HAP, the flying
servers, the required regular maintenance. From a technical perspective, it is
important to consider the impact of the extreme weather in the stratosphere
over the operation of the flying servers and the supported mission duration.
It is also essential to manage the workload in the data center-enabled HAP
based on the application requirements and the originating regions. For
instance, the multi-HAP scenario can be efficient to handle the workload
generated by disruption-tolerant networking in disconnected areas if the
necessary network management strategies are adopted. Eventually, it is
fundamental to implement artificial intelligence in workload prediction and
network management to improve the performance of the data center-enabled HAP.
However, energy-aware AI algorithms should be applied to conserve the energy-
efficiency of the data center-enabled HAP.

## References

  * [1] T. Yang, H. Jiang, Y. Hou, and Y. Geng, “Carbon management of multi-datacenter based on spatio-temporal task migration,” _IEEE Trans. Cloud Comput._ , 2021\.
  * [2] C. Koronen, M. Åhman, and L. J. Nilsson, “Data centres in future european energy systems—energy efficiency, integration and policy,” _Energy Efficiency_ , vol. 13, no. 1, pp. 129–144, 2020.
  * [3] V. Dukic, G. Khanna, C. Gkantsidis, T. Karagiannis, F. Parmigiani, A. Singla, M. Filer, J. L. Cox, A. Ptasznik, N. Harland _et al._ , “Beyond the mega-data center: networking multi-data center regions,” in _Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, SIGCOMM’20_ , 2020, pp. 765–781.
  * [4] R. Lin, Y. Cheng, M. De Andrade, L. Wosinska, and J. Chen, “Disaggregated data centers: Challenges and trade-offs,” _IEEE Commun. Mag._ , vol. 58, no. 2, pp. 20–26, 2020.
  * [5] Z. Cao, X. Zhou, H. Hu, Z. Wang, and Y. Wen, “Towards a systematic survey for carbon neutral data centers,” _IEEE Commun. Surv. Tutor._ , 2022.
  * [6] K. Mershad, H. Dahrouj, H. Sarieddeen, B. Shihada, T. Al-Naffouri, and M.-S. Alouini, “Cloud-enabled high-altitude platform systems: Challenges and opportunities,” _Frontiers Commun. Netw._ , vol. 2, p. 30, 2021.
  * [7] G. K. Kurt, M. G. Khoshkholgh, S. Alfattani, A. Ibrahim, T. S. Darwish, M. S. Alam, H. Yanikomeroglu, and A. Yongacoglu, “A vision and framework for the high altitude platform station (HAPS) networks of the future,” _IEEE Commun. Surv. Tutor._ , vol. 23, no. 2, pp. 729–779, 2021.
  * [8] S. Khalid and I. Ahmad, “Dual optimization of revenue and expense in geo-distributed data centers using smart grid,” _IEEE Trans. Cloud Comp._ , 2022.
  * [9] H. B. Barua, K. C. Mondal, and S. Khatua, “Green computing for big data and machine learning,” in _5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)_, 2022, pp. 348–351.
  * [10] S. C. Arum, D. Grace, P. D. Mitchell, M. D. Zakaria, and N. Morozs, “Energy management of solar-powered aircraft-based high altitude platform for wireless communications,” _Electron._ , vol. 9, no. 1, p. 179, 2020.
  * [11] C. Ding, J.-B. Wang, H. Zhang, M. Lin, and G. Y. Li, “Joint optimization of transmission and computation resources for satellite and high altitude platform assisted edge computing,” _IEEE Trans. Wirel. Commun._ , 2021.
  * [12] K. Sun, J. Li, H. Liang, and M. Zhu, “Simulation of a hybrid energy system for stratospheric airships,” _IEEE Trans. Aerosp. Electron. Syst._ , vol. 56, no. 6, pp. 4426–4436, 2020.
  * [13] K. Kaur, S. Garg, G. Kaddoum, E. Bou-Harb, and K.-K. R. Choo, “A big data-enabled consolidated framework for energy efficient software defined data centers in IoT setups,” _IEEE Trans. Ind. Inform._ , vol. 16, no. 4, pp. 2687–2697, 2020.
  * [14] M. S. Alam, G. K. Kurt, H. Yanikomeroglu, P. Zhu, and N. D. Đào, “High altitude platform station based super macro base station constellations,” _IEEE Commun. Mag._ , vol. 59, no. 1, pp. 103–109, 2021.
  * [15] N. S. Hogade, S. Pasricha, and H. J. Siegel, “Energy and network aware workload management for geographically distributed data centers,” _IEEE Trans. Sustain. Comput._ , 2021.

Wiem Abderrahim (S’14 - M’18) accomplished her undergraduate studies in
electrical engineering at the Higher School of Communications of Tunis,
Carthage University, Tunisia in 2013. She received her Doctoral Degree in
Information and Communication Technologies from the same university in 2017.
She worked as a lecturer and then as an adjunct professor at the Higher School
of Communications of Tunis between 2014 and 2018. Currently, she is a
postdoctoral fellow within King Abdullah University of Science and Technology
(KAUST), Thuwal, Saudi Arabia.
---
Osama Amin (S’07, M’11, SM’15) received his B.Sc. degree in electrical and
electronic engineering from Aswan University, Egypt, in 2000, his M.Sc. degree
in electrical and electronic engineering from Assiut University, Egypt, in
2004, and his Ph.D. degree in electrical and computer engineering, University
of Waterloo, Canada, in 2010. In June 2012, he joined Assiut University as an
assistant professor in the Electrical and Electronics Engineering Department.
Currently, he is a research scientist in the CEMSE Division at KAUST, Saudi
Arabia.
---
Basem Shihada (SM’12) received the Ph.D. degree in computer science from the
University of Waterloo, Waterloo, ON, Canada. He is an Associate and a
Founding Professor with CEMSE Division at KAUST, Saudi Arabia. In 2009, he was
appointed as a Visiting Faculty Member with the Department of Computer
Science, Stanford University, Stanford, CA, USA.
---


# Revisiting subword tokenization: A case study on affixal negation in large
language models

Thinh Hung Truong1 Yulia Otmakhova1 Karin Verspoor2,1
Trevor Cohn1, Timothy Baldwin3,1
1The University of Melbourne 2RMIT University 3MBZUAI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Also at
Google Research.

###### Abstract

In this work, we measure the impact of affixal negation on modern English
large language models (LLMs). In affixal negation, the negated meaning is
expressed through a negative morpheme, which is potentially challenging for
LLMs as their tokenizers are often not morphologically plausible. We conduct
extensive experiments using LLMs with different subword tokenization methods,
which lead to several insights on the interaction between tokenization
performance and negation sensitivity. Despite some interesting mismatches
between tokenization accuracy and negation detection performance, we show that
models can, on the whole, reliably recognize the meaning of affixal negation.

##  1 Introduction

Negation is central to language understanding but is not properly captured by
modern NLP methods (Hossain et al., 2022; Truong et al., 2023, inter alia).
While state-of-the-art large language models (LLMs) have improved negation-
related capabilities, challenges remain, such as the ability to correctly
determine the enclosed scope of negation, or when negation interacts with
other linguistic constructions like quantifiers (She et al., 2023; Truong et
al., 2023). Negations in common English NLP benchmarks are typically marked by
separate negation cues such as not or no. However, in practice, negation can
also be expressed through morphemes (or affixes) of words, i.e. by negative
prefixes or suffixes such as in uninteresting or effortless.

Figure 1: Example of our affixal negation prediction task, with the tokenization output for each model. Model | Type | Variant |  Data source | Vocab. size | Output | NegMorph
---|---|---|---|---|---|---
BERT | BPE | WordPiece |  books, wiki |  030K | {anti, clin, al } | Correct
RoBERTa | BPE | Byte-level BPE |  books, wiki |  050K | {antic, l, inal } | Under-segmented
XLNet | ULM | SentencePiece |  book, wiki, web text |  032K | {anti, clin, al } | Correct
AlBERT | ULM | SentencePiece |  book, wiki |  032K | {anti, clin, al } | Correct
T5 | BPE | SentencePiece |  web text |  032K | {anti, clin, al } | Correct
Llama-2 | BPE | SentencePiece |  web text, code, books, wiki, scientific publications |  032K | {ant, ic, l, inal } | Over-segmented
GPT-2 | BPE | Byte-level BPE |  web text |  050K | {antic, l, inal } | Under-segmented
GPT-4 | BPE | Byte-level BPE |  undisclosed | 100K | {antic, l, inal } | Under-segmented
Table 1: Summary of different tokenizers used in our experiments. Output are
tokenized version of the word anticlinal (model-specific special tokenization
characters are removed for clarity purposes). All models are the base version
unless otherwise specified.

While humans can identify affixal negation by leveraging morphological cues,
NLP systems only rarely consider word-internal structure, beyond normalizing
syntactic variation (Liu et al., 2012). Modern NLP methods such as language
models employ subword tokenization, in which words are broken down into
smaller units. This has an advantage of reducing vocabulary size, as well as
learning shared representation between words with similar subwords. The intent
to improve such representation by making tokenization methods more
linguistically sound has driven the invention of several morphology
segmentation methods, such as Morfessor (Grönroos et al., 2014). However,
these have not been broadly adopted in modern LLMs as they do not scale well.

We hypothesize that current subword tokenization methods could lead to sub-
optimal performance on language understanding tasks involving negation,
because they do not correctly break words down morphologically. For instance,
Table 1 demonstrates how different models employing different subword
tokenization methods tokenize the word anticlinal. Another known challenge
which could affect models is the high false positive rate in detecting affixal
negations (Blanco and Moldovan, 2011), for example misinterpreting de in
deserve as a negative affix, where in practice the de prefix derives from the
Latin root deservire and should not be interpreted as negating serve.

In this work, we analyze the impact of affixal negations on transformer-based
language models, where two main tokenization methods are employed, namely:
byte-pair encoding (Gage, 1994; Sennrich et al., 2016) and unigram language
model (Kudo, 2018). We consider three research questions:

##### RQ1: Are current subword tokenization methods able to preserve negative
affixes?

We analyzed the performance of various subword tokenization methods used in
modern LMs. We find that most do not effectively produce the correct negative
affixes.

##### RQ2: Are modern LMs aware of the presence of negation in affixal
negations?

We design a negation prediction task to probe models’ awareness of affixal
negation. We find that despite not performing well on the tokenization task,
current LLMs can reliably infer the negated meaning of words with negative
affixes. For this task, there is only a weak positive correlation between
tokenizer and classifier performance.

##### RQ3: What are the impacts of affixal negation on downstream tasks?

As negation and sentiment are closely related, we measure the impact on a
downstream sentiment analysis task by looking at samples containing affixal
negations from common datasets. Results show that models perform well on those
samples, implying that the impact of affixal negation is minimal. However,
there exists a bias in predicting negative sentiment for affixal negations.

##  2 Related work

There are two popular ways of constructing the vocabulary for LMs using
subword tokenization methods: byte-pair encoding (“BPE”: Sennrich et al.
(2016)) and unigram language model (“Unigram LM”: Kudo (2018)). BPE starts
from a base character set, then merges those characters based on bigram
frequency to form subword units (bottom-up), whereas unigram language models
start from a large subword vocabulary, which is then reduced based on a
regularization method (top-down). There are multiple variants of BPE,
differing in how the base vocabulary is represented and how the merging is
done. WordPiece (Schuster and Nakajima, 2012) uses characters to represent the
base vocabulary, then selects pairs that maximize the likelihood of training
data, Byte-level BPE uses bytes instead of Unicode to represent the base
vocabulary; the merging is done based on the frequency count of bigrams. In
contrast, the unigram language model starts from a large base vocabulary and
iteratively trims down tokens based on unigram LM perplexity until a target
vocabulary size is reached.

(a) NegMorph by models (ALBERT and XLNet use ULM, while the rest employ BPE)

(b) NegMorph by affixes

Figure 2: Negative affix-preserving segmentation performance on the set of
affixal negations (van Son et al., 2016).

Both methods assume that the input text uses spaces to separate words, which
is not true for languages such as Chinese or Vietnamese. Therefore, a word
segmentation step must be performed in advance. SentencePiece (Kudo and
Richardson, 2018) was introduced to solve this problem by considering
whitespace as part of words, essentially treating the whole input stream as
the smallest unit to perform tokenization on. Then, either BPE or unigram LM
can be applied to construct the vocabulary. Regardless of method, they purely
rely on statistical information and thus are not expected to produce
morphologically-aligned subword tokens.

There have been efforts to build linguistically-sound word tokenization
methods, most notably Morfessor and its variants (Grönroos et al., 2014,
2020). Building morphology-aligned segmentation methods, especially in a
multilingual setting, is an active line of research through recent SIGMORPHON
shared tasks (Batsuren et al., 2022). These methods outperform general
tokenizers in producing morphologically-aligned tokens, but their benefit on
downstream tasks is often negligible (Domingo et al., 2019; Saleva and Lignos,
2021). In this work, we examine if morphologically correct tokenization is
important for LLMs to deal with negation.

BERT and its variants have been shown to be insensitive to negation (Kassner
and Schütze, 2020; Ettinger, 2020), affecting many downstream NLP tasks such
as sentiment analysis, NLI, or QA (Hossain et al., 2020, 2022; Ravichander et
al., 2022; Truong et al., 2022). Compared to previous models, current LLMs
have improved negation handling ability, but still struggle with some
unconventional types of negation and linguistic constructions (Truong et al.,
2023). Here, we investigate the treatment of affixal negation in modern LMs,
with the intuition that subword tokenization methods that don’t appropriately
reflect this morphology will lead to misinterpretation of their semantics.

##  3 Experimental settings

We focus our analysis particularly on how affixal negations are represented in
modern LLMs, designing probing tasks to test their awareness of negation, and
the effect on downstream tasks. All code for the experiments is available at
https://github.com/joey234/affixal-negation.

###  3.1 A lexicon of affixal negation

We use the lexicon created in van Son et al. (2016). The dataset contains a
list of affixal negation and their non-negated counterparts (e.g.
unintended–intended). For each affixal negation, the corresponding negative
affix is also annotated. In total, there are 2089 affixal negations, and 2055
non-negated words which are antonyms of the negations. These numbers are not
equal because one word can have multiple corresponding negated counterparts,
e.g. intrusive–{extrusive, unintrusive}.

###  3.2 Tokenization methods

For each tokenizer type (along with their variants), we consider the most
representative models that use them, based on their popularity. Although some
models use the exact same tokenizer, it is worth investigating them as
differences in training corpora can lead to differences in tokenization
results.

##### BPE

We consider models using different flavors of BPE. For WordPiece, we consider
BERT (Devlin et al., 2019), ELECTRA (Clark et al., 2020); for Byte-level BPE,
we consider RoBERTa (Liu et al., 2019), and GPT-family models including GPT-2
(Radford et al., 2019) and GPT-4 (OpenAI, 2023); and for SentencePiece, we
examine Flan-T5 (Chung et al., 2022) and LLaMA-2 (Touvron et al., 2023).

##### Unigram LM

Models using unigram LM tokenization methods considered in this work are
always used in combination with SentencePiece: XLNet (Yang et al., 2019) and
AlBERT (Lan et al., 2020).

###  3.3 Negative affix-preserving segmentation

We consider a segmentation of an affixal negation to be “affix preserving”
(Correct) only if the negative affix matches with one of the produced tokens
(e.g. anticlimatic $\rightarrow$ anti, clima, tic). Otherwise, it is either
Under-segmented if the negative affix is a substring of one of the produced
tokens (e.g. anticlima, tic), or Over-segmented (e.g. ant, i, clima, tic).
Formally, given an affixal negation word $w$ having the negative affix $a$, if
$w$ is tokenized into $T_{k}=\\{t_{i},t_{i+1},...,t_{n}\\}$ under tokenizer
$k$ then we define $\text{NegMorph}_{k}(w)$ as follows:

$\text{NegMorph}_{k}(w)=\begin{cases}\text{Correct}&\text{if $a\in T_{k}$.}\\\
\parbox[t]{45.52458pt}{Under-segmented}&\parbox[t]{56.9055pt}{if $a$ is a
substring of any $t_{i}\in T_{k}$.}\\\ \parbox{45.52458pt}{Over-
segmented}&\text{otherwise}\end{cases}$

##  4 Findings

###  4.1 Current subword tokenization methods are not negative affix-
preserving

As shown in Figure 2(a), T5 has the best performance in producing negative
affix-preserving tokens, while for the remaining, models employing the unigram
LM method outperform those using BPE. This is in line with previous findings
that the unigram LM produces subword units that align with morphology better
than BPE (Bostrom and Durrett, 2020). Moreover, models that employ
SentencePiece (T5, ALBERT, XLNet, LLaMA-2) outperform those that don’t (BERT,
RoBERTa, GPT-2). However, the best-performing models are only up to 75%
correct relative to NegMorph, with considerable room for improvement. Most
failed cases relate to under-segmentation.

An analysis of what types of negative affix are hard to tokenize is provided
in Figure 2(b), and their most frequent incorrect tokenizations are shown in
Figure 3. Some common affixes that are incorrectly tokenized are il
$\rightarrow$ ill (illicit, illogical), ir $\rightarrow$ irre (irresolute,
irreponsibly, irregular), a $\rightarrow$ as (asymmetric), and a $\rightarrow$
at (atypically). Overall, we see that some affixes can be wrongly tokenized in
a wide range of ways (represented by the large number of substacks), showing
that current tokenization methods are inefficient. Overcoming this problem
could result in embeddings that better encapsulate word morphology.

Figure 3: Top 10 most frequent affixes in the dataset and the distribution of
tokens that they are wrongly tokenized into. Each substack denotes the
percentage that the corresponding token is accounted for.  Yellow bar denotes
Under-segmented, while  Red bar denotes Over-segmented. Figure 4: Ratio of
correct/incorrect prediction on the Affix (fewshot) task, breakdown by
affixes. The left greyed-out side of each subplot corresponds to wrong
predictions. Affix (zero-shot)

    The word {word} contains negation. True or
    False?
    Answer:

Figure 5: Zero-shot prompt Affix (few-shot)

    A word contains negation if it has a negated
    meaning, usually expressed through a negative
    prefix (such as un, in) or suffix (such as
    less).

    The word decentralize contains negation. True
    or False?
    Answer: True
    Explanation: decentralize is created by
    prepending the root word centralize with the
    negative prefix de.

    The word deserve contains negation. True or
    False?
    Answer: False
    Explanation: deserve just coincidentally starts
    with de.

    The word {word} contains negation. True or
    False?
    Answer:

Figure 6: Few-shot prompt

###  4.2 Negative affixes signify negation, but word knowledge is essential

We design a binary classification task on the lexicon described in Section 3.1
to probe the ability of models to understand affixal negation, denoted Affix.
First, for smaller models ($<$1B parameters), we conducted a fully fine-tuned
setting with a 80/20 split and see that they achieve good results on the test
set ($>$93% accuracy), showing that models can learn the patterns of negative
prefixes and suffixes with enough supervision.

For larger models, we evaluate three state-of-the-art LLMs in a zero- and few-
shot manner. The prompts are presented in Figures 5 and 6, respectively.

Figure 7: Zero- and Few-shot results on the affixal negation prediction task.

For the few-shot prompt, we provide explicit instructions to explain what
negation means in this context, as well as two demonstrating samples, to avoid
ambiguity (such as confusion with negative sentiment).

Results are summarized in Figure 7 (full numerical results, including in the
fine-tuned setting, are in Table 4). Overall, we find that the performance on
Neg (the subset containing only affixal negation) is much lower compared to
its Non-neg counterpart, where the best models achieve near-perfect
performance.

For the zero-shot setting, surprisingly, Flan-T5 outperforms both LLaMA-2 and
GPT-4, despite being the smallest in size. After adding more explicit
instructions and examples (Affix (few-shot)), we observe large increases in
performance for GPT-4 and LLaMA-2, and little to no difference for Flan-T5.
For the non-negated subset, on the other hand, all models have near-perfect
performance, with GPT-4 slightly outperforming Flan-T5. LLaMA-2 performance
for this task is much lower than the other two.

We further break down the results based on affixes. Figure 4 illustrates the
percentage of correct/incorrect prediction for each affix, divided by NegMorph
categories. Compared to the relatively high results for Neg in Table 4, we
have a clearer view on the actual performance of models. On average, we see
that models made errors equally as likely for all affixes (as shown by the
last Overall bar, where the percentages of incorrect and correct predictions
are roughly 50%). From the figure, we can also observe that the
correct/incorrect prediction distribution is similar across models (especially
between GPT-4 and Flan-T5), showing that they tend to make the same errors. In
general, we see a larger portion of correct segmentation when models predict
negation correctly, and more under-segmentation when models predict non-
negation, while over-segmentation appears equally likely regardless of
prediction. We also attempted to calculate the Pearson’s coefficient between
NegMorph and Accuracy on the Neg set but did not yield any statistically
significant correlation.

##### Error analysis

We inspect the errors made by GPT-4 in the Affix (few-shot) setting to perform
a qualitative analysis. We adopt the classification introduced in Joshi (2012)
and summarize in Table 2.

Error type |  Example |  Ratio
---|---|---
Reversal of action |  divest, diverge, detach |  0.213
Reversal of direction |  outdoors, descending, downstairs |  0.204
Insufficiency |  hypoglycemia, inferior, underpay |  0.132
Positive sentiment |  fearless, indispensable, unselfishly |  0.081
Wrongness |  infamy, malignant, misconstruction |  0.047
Rare words |  asyndetic, abactinal, syncategorem |  0.047
Noise (annotation errors) |  uncle, intense, increment |  0.106
Other (normal affixal negation) |  illicit, immortal, informality |  0.17
Table 2: Error analysis of the 235 errors made by GPT-4 on Affix (few-shot).

In total the model made 235 errors. Aside from the errors on normal affixal
negation where the negative suffix can be replaced by not without changing the
word’s meaning (uncritically, infinitely), the bulk of the errors were caused
by cases where the affix has more complex semantics. The most common source of
errors is from affixes which show the reversal of action (e.g. diverge,
detach), or the reversal of direction (e.g. descending, downstairs). In
addition, a large proportion of errors come from affixes that negate reaching
some normal or default state (e.g. hypoglycemia, inferior, underpay). Another
interesting pattern is caused by words with positive sentiment (e.g. fearless,
incredibly, infallibility) showing that models are confused between negation
and negative sentiment, likely due to over-exposure to sentiment analysis
data. In addition, some errors are attributed to words where the negative
affix has an additional sentiment of “wrongness” (e.g. malignant,
misconstruction). The remaining errors are attributed to rare words and noise
in annotation.

##### Hyphenated words

To make sure that the negative affix is not further broken down by tokenizers,
we convert words into their “hyphenated” form (e.g. unintended $\rightarrow$
un-intended). From Figure 8, we see that this greatly increases the
performance of different tokenizers on the NegMorph metric (by as much as
32%). Compared to the normal setting, the accuracy of all models also
increases on the Affix task, suggesting a positive correlation between
NegMorph and Accuracy. LLaMA-2 benefited the most from this setting, having
the largest increases in both Accuracy and NegMorph.

Figure 8: Results of Few-shot and Few-shot Hyphen on the affixal negation
prediction task. Bars denote the accuracy on the prediction task , while Dots
denote the Correct NegMorph scores for the segmentation task.

##### Nonce words

Nonce words are words that look and sound like real words, but are created for
a single-purpose use and not recognized as words within a language (e.g.
roagly). To measure the effect of negative affix on word semantics, we
construct a list of “affixal nonce words” by prepending or appending negative
affixes to a list of nonce words. We collect a list of adjective nonce words
from Cremers (2022) . For affixes, we used the list of 40 negative affixes
provided in van Son et al. (2016) and collected 40 non-negative affixes (e.g.
auto-, bi-, -ism, -ful).111We collected the affixes from
https://litinfocus.com/120-root-words-prefixes-and-suffixes-pdf-list/ For each
nonce word, we prepend (or append) the affixes to form an “affixal nonce
word”. In total, the set consists of 11 nonce words $\times$ 80 affixes = 880
samples, evenly distributed between negated (e.g. dis-roagly) and non-negated
(e.g. auto-roagly). We adopt the Affix (few-shot) prompt and add an
instruction to prevent models from refusing to answer the questions because of
invalid words (full prompt in Appendix B). Similarly, we also report the
results of two subsets of negative affixes (Neg) and non-negative affixes
(Non-neg) in Figure 9. For the Neg set, we find that the performance of all
models is relatively low, despite them being able to correctly tokenize the
negative affixes. Whereas for the Non-neg set, performance is near-perfect for
all models, similar to the previous Affix-Hyphen task. Looking at the results,
however, we found that most errors made by the models are when the negative
affixes are ambiguous, i.e. their meaning depends on which words they are
attached to (e.g. a-, di-, ef-, para-, re-). This reveals an important insight
that whether something is considered to be a negation should be judged with
context (which is parametric knowledge about words in this case).

Figure 9: Accuracy on the affixal nonce words prediction task. Figure 10:
Performance on word-level sentiment task

##### Non-negated words with tokens homonymous with negative affixes

To explore the false positive problem i.e., words coincidentally contain
negative affixes, we collect words from the Non-neg subset and WordNet
(Miller, 1995) which do not have negated meaning, but have a negative
prefix/suffix as the first/last subword token. We tokenize WordNet using the
T5 tokenizer and select all words that start/end with a negative
prefix/suffix, then subtract all words in the list of affixal negations. We
manually go through the extracted list again to remove errors, resulting in a
set of 330 words.222We didn’t consider other models as this list of words
would be different between models. Following the same affixal negation
prediction task, we find that Flan-T5 has very good performance (0.958
accuracy), showing that it can synthesize information from all subword tokens
instead of only relying on the negative affixes. Most errors come from the
uni- cases, where the model tokenizes them as un- (e.g. unidirectional,
univalent).

###  4.3 Impact on downstream tasks

One main drawback of our probing task is that the words lack context. Negation
is a context-dependant concept, that is, what is considered negation can
differ depending on the context of use. Investigating the impact of affixal
negation in the context of downstream tasks is thus an essential component of
this work.

####  4.3.1 Sentiment analysis

Previous work has shown that negation is a strong indicator of negative
sentiment (Wiegand et al., 2010). Furthermore, the fact that sentiment
analysis is part of many NLP benchmarks could create a bias in models, leading
to negation being conflated with negative sentiment, which is not always the
case. For instance, the word incredible is constructed by prepending the root
word credible with the negative affix in-, meaning “not credible” but used to
express a positive meaning. This inspired us to extend our analysis to a
downstream sentiment analysis task. We evaluate the few-shot performance of
LLMs in two settings of word- and sentence-level sentiment analysis (full
prompts in Appendix C).

##### Word-level sentiment

We first use SentiWordNet 3.0 (Baccianella et al., 2010) to automatically
assign a sentiment label for the lexicon of affixal negation described in
Section 3.1. After that, two graduate researchers went over the list to
determine the final labels (positive, negative, or neutral). In general, we
find that GPT-4 outperforms Flan-T5 and LLaMA-2 on this word-level task. As
seen in Figure 10, all models have almost perfect performance at predicting
negative words, but struggle with the other two classes. In particular, we
find Flan-T5 and LLaMA-2 overpredict Negative for Neutral words, while GPT-4
often mistakes Positive for Neutral.

##### Sentence-level sentiment

Figure 11: Accuracy on sentence-level sentiment analysis task. Results are
averaged across 3 models.

For this task, we look at common sentence-level sentiment analysis datasets
including SST-2 (Socher et al., 2013), and Rotten Tomatoes (RT) (Pang and Lee,
2005). One drawback of this evaluation is that samples tend to contain many
sentiment signals, making it hard to gauge the effect of affixal negations.

We consider 3 settings: (1) Affix = only samples containing affixal negation;
(2) Non affix = only samples without affixal negation; and (3) Replace affix =
similar to Affix, but we replace all instances of affixal negations with
equivalent syntactic negations, i.e. not \+ word (uninteresting $\rightarrow$
not interesting). We present the results in Figure 11. While it is true that
replacing negative affixes with not does not always result in a direct
paraphrase, we argue that the change in meaning is minimal, and that samples
will likely preserve the sentiment. Furthermore, we mostly find adjectives in
the datasets rather than nouns, which ensures that the sentences are
grammatically correct after replacement. Overall, we can conclude that affixal
negation is a strong signal to guide model prediction. We observe good
performance for Affix in both datasets, where the accuracy are comparable to
Non Affix in SST-2 and higher in RT. Attempting to replace affixal negations
slightly decreases the performance of models in both datasets. This suggests
that affixal negation is actually a stronger sentiment cue compared to
syntactic negation. We further report class-wise performance of the Affix set
in Figure 12. Accuracy on samples with Negative sentiment is higher than
Positive, once again showing that affixal negation is a strong cue for
predicting negative sentiment.

Figure 12: Accuracy of Neg/Pos class of the Affix set. Results are averaged
across 3 models.

##  5 A look into token attribution

We perform an interpretation analysis to attain insights into what drives
model predictions. For this analysis, we use the Flan-T5-xxl model, as we
could not obtain probabilities (logprobs) from GPT-4. We calculate the
attribution for each token corresponding to the predictions using the
Integrated Gradient method (Sundararajan et al., 2017), with probability as
the scoring function, implemented in Inseq (Sarti et al., 2023). Overall, we
observe high attribution scores from relevant tokens, such as the subword
tokens of the target words, showing that models know where to pay attention to
when performing inference.

Figure 13: Token attribution of selected samples on the word-level sentiment
prediction task. Only parts of the prompts are shown for clarity purposes.

##### Negative affixes have flipping sentiment effect

In Section 4.3.1, we see that models tend to overpredict negative sentiment on
the list of affixal negations. Through the saliency heatmap in Figure 13, we
can see high token attributions for the negative affixes that change the
sentiment of the root words (either positive or neutral) into negative. This
is in line with previous findings that negation flips the polarity of
sentiment (Tigges et al., 2023). This effect could be the main cause for the
low performance on the Neutral class observed in our word-level sentiment
analysis task (Section 4.3.1). When applied to the negation prediction task,
however, we did not observe a similar effect and did not see any clear pattern
for token attribution.

##### Correct tokenization is not essential for negation awareness

Through many experiments, we have shown that overall, correct tokenization
leads to better awareness of models to the presence of negation. This effect,
however is not significant. By comparing token attributions between 3 cases of
NegMorph (Figure 14), we saw that models are able to combine information from
relevant subword tokens corresponding to a word to make the correct inference.

Figure 14: Token attribution of selected sample samples on the negation
prediction task. The three subplots correspond to Correct, Under-segmented,
and Over-segmented case respectively. Only parts of the prompts are shown for
clarity purposes.

##  6 Conclusion

In this work, we conducted an in-depth analysis into how well modern LLMs
handle affixal negation, a type of negation where morphology is essential to
understanding word semantics. We have shown that there is significant room for
improvement in current tokenization methods in terms of producing negative
affix-preserving tokens. Despite that, the effect of morphologically incorrect
tokenization on the ability of models to understand word meaning in downstream
tasks, including sentiment analysis, is minimal. Regardless, designing better
subword tokenization methods may have many immediate benefits such as reducing
vocabulary size, learning better word representations, and improving model
interpretability.

##  7 Limitations

##### Prompting

As this work involves experiments with LLMs, there is always a possibility
that the prompts we used are not optimal (and also, the problem of
reproducibility). We attempted to reuse prompt templates from existing work
where possible, and strove to design prompts that are intuitive and specific
otherwise.

##### Multilinguality

Morphology is a language-dependent problem. We recognize that the lack of
investigation in other languages other than English is a drawback of this
work.

##### Broader impact

Given that our focus is on presenting and analysing the problem of poor
treatment of affixal negation in LLMs, we did not propose any immediate
solutions to improve the status quo. The finding on the impact on downstream
tasks could be limited by the lack of samples (both in size and meaningful
patterns) in the test data.

## Acknowledgements

We would like to thank Khuyagbaatar Batsuren for the talk on morphological
segmentation at The University of Melbourne that inspired this work, and
Daniel Beck and Simon Šuster for discussions that helped refine our
experiments. We also thank the anonymous reviewers and area chairs of ARR
December cycle and NAACL for an engaging and constructive review period. This
research was conducted by the Australian Research Council Training Centre in
Cognitive Computing for Medical Technologies (project number ICI70200030) and
funded by the Australian Government. This research was undertaken using the
LIEF HPC-GPGPU Facility hosted at The University of Melbourne, established
with the assistance of LIEF Grant LE170100200.

## References

  * Baccianella et al. (2010) Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010.  SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Batsuren et al. (2022) Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdeněk Žabokrtský, Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševčíková, Kateřina Pelegrinová, Fausto Giunchiglia, Ryan Cotterell, and Ekaterina Vylomova. 2022.  The SIGMORPHON 2022 shared task on morpheme segmentation.  In _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 103–116, Seattle, Washington. Association for Computational Linguistics.
  * Blanco and Moldovan (2011) Eduardo Blanco and Dan Moldovan. 2011.  Some issues on detecting negation from text.  In _Twenty-Fourth International FLAIRS Conference_.
  * Bostrom and Durrett (2020) Kaj Bostrom and Greg Durrett. 2020.  Byte pair encoding is suboptimal for language model pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4617–4624, Online. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _ArXiv preprint_ , abs/2210.11416.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations (ICLR 2020)_.
  * Cremers (2022) Alexandre Cremers. 2022.  Interpreting gradable adjectives: rational reasoning or simple heuristics?  _Empirical Issues in Syntax and Semantics_ , 14:31–61.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Domingo et al. (2019) Miguel Domingo, Mercedes García-Martínez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2019.  How much does tokenization affect neural machine translation?  In _International Conference on Computational Linguistics and Intelligent Text Processing_ , pages 545–554. Springer.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Gage (1994) Philip Gage. 1994.  A new algorithm for data compression.  _C Users Journal_ , 12(2):23–38.
  * Grönroos et al. (2020) Stig-Arne Grönroos, Sami Virpioja, and Mikko Kurimo. 2020.  Morfessor EM+Prune: Improved subword segmentation with expectation maximization and pruning.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 3944–3953, Marseille, France. European Language Resources Association.
  * Grönroos et al. (2014) Stig-Arne Grönroos, Sami Virpioja, Peter Smit, and Mikko Kurimo. 2014.  Morfessor FlatCat: An HMM-based method for unsupervised and semi-supervised learning of morphology.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1177–1185, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Hossain et al. (2022) Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022.  An analysis of negation in natural language understanding corpora.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Joshi (2012) Shrikant Joshi. 2012.  Affixal negation–direct, indirect and their subtypes 1.  _Syntaxe & sémantique_, (1):49–63.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations (ICLR 2020)_.
  * Liu et al. (2012) Haibin Liu, Tom Christiansen, William A Baumgartner, and Karin Verspoor. 2012.  Biolemmatizer: a lemmatization tool for morphological processing of biomedical text. _Journal of Biomedical Semantics_ , 3:3.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Miller (1995) George A Miller. 1995.  WordNet: a lexical database for English.  _Communications of the ACM_ , 38(11):39–41.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the ACL_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  OpenAI blog.
  * Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marasovic. 2022.  CONDAQA: A contrastive reading comprehension dataset for reasoning about negation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8729–8755, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Saleva and Lignos (2021) Jonne Saleva and Constantine Lignos. 2021.  The effectiveness of morphology-aware segmentation in low-resource neural machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 164–174, Online. Association for Computational Linguistics.
  * Sarti et al. (2023) Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. 2023.  Inseq: An interpretability toolkit for sequence generation models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 421–435, Toronto, Canada. Association for Computational Linguistics.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _International Conference on Acoustics, Speech and Signal Processing_ , pages 5149–5152.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * She et al. (2023) Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023.  ScoNe: Benchmarking negation reasoning in language models with fine-tuning and in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1803–1821, Toronto, Canada. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR.
  * Tigges et al. (2023) Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023.  Linear representations of sentiment in large language models.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  LLaMA 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Truong et al. (2023) Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023.  Language models are not naysayers: an analysis of language models on negation benchmarks.  In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_ , pages 101–114, Toronto, Canada. Association for Computational Linguistics.
  * Truong et al. (2022) Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022.  Not another negation benchmark: The NaN-NLI test suite for sub-clausal negation.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 883–894, Online only. Association for Computational Linguistics.
  * van Son et al. (2016) Chantal van Son, Emiel van Miltenburg, and Roser Morante. 2016.  Building a dictionary of affixal negations.  In _Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics (ExProM)_ , pages 49–56, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Wiegand et al. (2010) Michael Wiegand, Alexandra Balahur, Benjamin Roth, Dietrich Klakow, and Andrés Montoyo. 2010.  A survey on the role of negation in sentiment analysis.  In _Proceedings of the Workshop on Negation and Speculation in Natural Language Processing_ , pages 60–68, Uppsala, Sweden. University of Antwerp.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  XLNet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.

##  Appendix A Model endpoints

  * •

GPT-4: We accessed GPT-4 through the offical API with the name gpt-4. Note
that this is different from the GPT-4 turbo model with the name
gpt-4-1106-preview.

  * •

LLaMA-2-13B: We used the official instruction fine-tuned LLaMA-2-13B available
on the HuggingFace hub with the name: meta-llama/LLaMA-2-13b-chat-hf.

  * •

Flan-T5-xxl: We used the official xxl version (11.3B) of the Flan-T5 model
available on the HuggingFace hub with the name: google/flan-t5-xxl.

##  Appendix B Details of Affixal Nonce word prediction task

##### List of nonce words

roagly, vibble, drok, scrop, plard, hif, tepable, plawic, bluth, sprat, flurf

##### List of non-negative affixes

Prefix: ambi-, aqu-, ast-, aud-, auto-, bi-, bio-, cent-, circum-, co-, cred-,
cycl-, dec-, dia-, equ-, geo-, grad-, hydro-, inter-, medi-, mega-, min-,
micro-, pan-, semi-, tele-, uni-, tri-. Suffix: -able, -al, -ance, -ful, -ian,
-ic, -tic, -ile, -ism, -ist, -junct, -ly

Nonce

    A nonce word is a word occurring, invented,
    or used just for a particular occasion, or a
    word with a special meaning used for a special
    occasion. Infer whether the given nonce word
    contains negation or not.

    A word contains negation if it has a negated
    meaning, usually expressed through a negative
    prefix (such as un, in) or suffix (such as
    less).

    The word decentralize contains negation. True
    or False?
    Answer: True
    Explanation: decentralize is created by
    prepending the root word centralize with the
    negative prefix de.

    The word deserve contains negation. True or
    False?
    Answer: False
    Explanation: deserve just coincidentally starts
    with de.

    The word {word} contains negation. True or
    False?
    Answer:

##  Appendix C Prompts for sentiment analysis

Word-level sentiment

    {Few-shot samples}

    The sentiment of the word {word} is positive,
    negative, or neutral.

    Answer:

Sentence-level sentiment

    {Few-shot samples}

    {sentence}
    Question: Is this sentence positive or
    negative?

    Answer:

##  Appendix D Full results

Model |  Neg. Nonce |  Non-neg. Nonce |  All
---|---|---|---
GPT-4 |  0.434 |  1 |  0.717
LLaMA-2-13B |  0.575 |  0.991 |  0.783
Flan-T5-xxl |  0.627 |  0.964 |  0.795
Table 3: Affixal nonce word prediction task Model | Accuracy | NegMorph
---|---|---
| Neg | Non-neg | All | Correct
Affix (fine-tuned) |  |  |  |
BERT | 0.940 | 0.959 | 0.949 | 0.516
RoBERTa | 0.931 | 0.964 | 0.947 | 0.614
AlBERT | 0.933 | 0.956 | 0.945 | 0.737
XLNet | 0.950 | 0.915 | 0.932 | 0.707
GPT-2 | 0.928 | 0.949 | 0.938 | 0.614
Affix (zero-shot) |  |  |  |
GPT-4 | 0.783 | 0.994 | 0.888 | 0.671
LLaMA-2-13B | 0.707 | 0.770 | 0.738 | 0.658
Flan-T5-xxl | 0.867 | 0.976 | 0.921 | 0.751
Affix (fewshot) |  |  |  |
GPT-4 | 0.890 ( +0.107) | 0.997 ( +0.003) | 0.943 ( +0.055) | 0.670
LLaMA-2-13B | 0.767 ( +0.060) | 0.938 ( +0.168) | 0.852 ( +0.114) | 0.658
Flan-T5-xxl | 0.855 ( -0.012) | 0.993 ( +0.017) | 0.924 ( +0.003) | 0.750
Affix (fewshot)-Hyphen |  |  |  |
GPT-4 | 0.916 ( +0.133) | - | - | 0.929 ( +0.258)
LLaMA-2-13B | 0.956 ( +0.249) | - | - | 0.984 ( +0.326)
Flan-T5-xxl | 0.948 ( +0.081) | - | - | 0.968 ( +0.217)
Table 4: Results of our affixal negation prediction task. (+/- denote the
change compared to the Affix (zero-shot) setting