# The Simplex Projection: Lossless Visualization of
4D Compositional Data on a 2D Canvas

Marvin Schmitt
University of Stuttgart &Yuga Hikida
TU Dortmund University Stefan T. Radev
Rensselaer Polytechnic Institute &Filip Sadlo
Heidelberg University &Paul-Christian Bürkner
TU Dortmund University Corresponding author: Marvin Schmitt,
<EMAIL_ADDRESS>
###### Abstract

The simplex projection expands the capabilities of simplex plots (also known
as ternary plots) to achieve a lossless visualization of 4D compositional data
on a 2D canvas. Previously, this was only possible for 3D compositional data.
We demonstrate how our approach can be applied to individual data points,
point clouds, and continuous probability density functions on simplices. While
we showcase our visualization technique specifically for 4D compositional
data, we offer rigorous proofs that support its extension to compositional
data of any (finite) dimensionality.

##  1 Introduction

The visualization of high-dimensional data is a key task in countless domains
of scientific research. Yet, the representation of multi-dimensional data in a
two-dimensional canvas (e.g., static screens or paper) can pose a significant
challenge, leading to substantial information loss or distortions, which, in
turn, can skew the interpretation and analysis of the data.

In this paper, we address this challenge by developing a novel approach for
visualizing 4D compositional data on a 2D canvas. Compositional data consists
of vectors with strictly positive entries that sum to one [7]. This data type
naturally arises for proportions, normalized data, or discrete probabilities.
Examples for compositional data include (i) the relative composition of the
gut microbiome [6]; (ii) proportion of peoples’ activities throughout the day
[5, e.g., activity, rest, and sleep;]; or (iii) discrete probability vectors,
such as posterior model probabilities arising in Bayesian model comparison [4,
17].

Our technique, which we call _simplex projection_ , is a lossless
visualization method that accurately represents the compositional data while
preserving its geometrical and topological properties. We prove mathematically
that our mapping from 4D compositional data to its 2D representation is a
bijection (invertible one-to-one correspondence) that incurs no loss of
information. We demonstrate the effectiveness of our approach, highlighting
the simplex projection as a potent tool for exploring and analyzing 4D
compositional data. While the underlying mathematical treatment holds for
arbitrary finite dimensions, throughout the paper, we will focus chiefly on
illustrations and intuitions for the 4D case.

##  2 Preliminaries

Throughout this manuscript, let $J\in\mathbb{N}$, and the points
$v_{1},\ldots,v_{J}\in\mathbb{R}^{J}$ be affinely independent, that is,
$(v_{2}-v_{1}),\ldots,(v_{j}-v_{1})$ are linearly independent. Further, the
points $v_{1},\ldots,v_{J}$ are the vertices of the $(J-1)$-dimensional
simplex $\Delta^{J-1}$ defined by the set

$\Delta^{J-1}=\Big{\\{}x\in\mathbb{R}^{J}:x=\sum_{j=1}^{J}\pi_{j}v_{j}\Big{\\}},$
(1)

with weights $\pi_{j}\in(0,1)$ such that $\sum_{j=1}^{J}\pi_{j}=1$. When the
dimension of the simplex $\Delta^{J-1}$ is sufficiently clear from the
context, we drop the superscript and simply write $\Delta$.

The representation of a point $x$ through weighted vertices, that is, via
coefficients $(\pi_{1},\ldots,\pi_{J})$ in Equation 1, is commonly referred to
as _barycentric coordinates_ with respect to the vertices
$v_{1},\ldots,v_{J}$. For brevity, we will slightly overload the notation and
use the vector of barycentric coordinates $(\pi_{1},\ldots,\pi_{J})$ to refer
to a point $x=\sum_{j=1}^{J}\pi_{j}\,v_{j}$ in the simplex with vertices
$v_{1},\ldots,v_{J}$ as defined in Equation 1. Since any two simplices of
equal dimension $J$ are homeomorphic by a simplicial homeomorphism [15], the
exact location of the vertices $v_{1},\ldots,v_{J}$ is irrelevant and we will
use regular (aka. equilateral) simplices for all illustrations.

The convex hull of each non-empty subset of size $N$ from the $J$ vertices
$v_{1},\ldots,v_{J}$ of a simplex $\Delta^{J-1}$ is called a $(N-1)$-face. In
particular, the $0$-faces are the vertices, the $1$-faces are the edges, the
$(J-2)$-faces are the facets which we denote as $\sigma$, and the only
$(J-1)$-face is the simplex $\Delta^{J-1}$ itself. We denote the facet
opposing a given vertex $v_{j}$ as $\sigma_{-j}$.

Figure 1: Projection of $x=(\pi_{1},\pi_{2},\pi_{3})$ onto the facet (edge)
$\sigma_{-1}=\overline{v_{2}v_{3}}$ in a triangle (2-simplex). It can be seen
clearly that the perspective projection $\psi_{1}$ does not change the points’
coordinate ratios, $\pi_{2}/\pi_{3}=\tilde{\pi}_{2}/\tilde{\pi}_{3}$.

(a) Simplicial complex.

(b) No simplicial complex.

Figure 2: Not all sets of simplices are simplicial complexes. In (a), three
$2$-simplices form a (pure) simplicial complex. In (b), the dashed
intersection of the 2-simplices is neither empty nor a face of the two
simplices.

#### Renormalized Barycentric Coordinates

Let $(\pi_{1},\ldots,\pi_{J})$ be barycentric coordinates with $J$ components,
as defined above. For an index subset $K\subseteq\\{1,\ldots,J\\}$, we define

$\Big{(}\tilde{\pi}_{k}\Big{)}_{k\in
K}=\left(\dfrac{\pi_{k}}{\sum\limits_{k{{}^{\prime}}\in
K}\pi_{k{{}^{\prime}}}}\right)_{k\in K}$ (2)

as _renormalized barycentric coordinates_ (mind the tilde to differentiate
between vanilla and renormalized coordinates). The term “renormalized” is
motivated by the normalizing effect of the denominator in Equation 2. While a
simple subset of compositional data does not generally sum to one, $\sum_{k\in
K}\pi_{k}\leq 1$, it is easy to show that the renormalized barycentric subset
sums to one, $\sum_{k\in K}\tilde{\pi}_{k}=1$. Moreover, the ratio of every
two renormalized coordinates $\tilde{\pi}_{n},\tilde{\pi}_{m}$ equals the
ratio of the original coordinates $\pi_{n},\pi_{m}$ since these ratios are
clearly invariant to the division by the same normalizing constant,

$\frac{\pi_{n}}{\pi_{m}}=\frac{\pi_{n}/\sum_{k\in
K}\pi_{k}}{\pi_{m}/\sum_{k\in
K}\pi_{k}}\overset{\text{def}}{=}\frac{\tilde{\pi}_{n}}{\tilde{\pi}_{m}},$ (3)

as depicted in Figure 1.

#### Simplicial complex

Following the definition of [15], a simplicial complex $\mathcal{K}$ is a set
of simplices in an Euclidean space that satisfies (see Figure 2 for an
illustration):

  1. 1.

For every simplex $\sigma\in\mathcal{K}$, every face of $\sigma$ is in
$\mathcal{K}$;

  2. 2.

the intersection of any two simplices $\sigma_{1},\sigma_{2}\in\mathcal{K}$ is
either empty or a face of both $\sigma_{1}$ and $\sigma_{2}$.

A simplicial complex $\mathcal{K}$ is a _pure simplicial complex_ if all
simplices $\sigma\in\mathcal{K}$ have equal dimension. In what follows, we
will project a point $x$ from the $(J-1)$-dimensional simplex $\Delta$ onto
$J$ points $x_{1},\ldots,x_{J}$, where each projected point $x_{j}$
($j=1,\ldots,J$) lies in a $(J-2)$-dimensional simplex $\sigma_{-j}$, and all
simplices $\sigma_{-1},\ldots,\sigma_{-J}$ form a pure simplicial complex.
Since the number of lower-dimensional simplices $\sigma$ and the
dimensionality of the higher-dimensional point $x$ are equal in our
application, we refer to both as $J$. Further, we will need to refer to the
set $\\{x_{1},\ldots,x_{J}\\}$, where each point lies in exactly one simplex
$\sigma_{-j}$ of the simplicial complex
$\mathcal{K}=\\{\sigma_{-1},\ldots,\sigma_{-J}\\}$, and the indices $j$
coincide (see previous section for details on the $-j$ index notation). We
capture this with the following product notation:

$\prod\limits_{j=1}^{J}\sigma_{-j}=\sigma_{-1}\times\ldots\times\sigma_{-J}=\Big{\\{}x_{j}\,\Big{|}\,x_{j}\in\sigma_{-j}\Big{\\}}_{j=1}^{J}$
(4)

#### Perspective Projection

Let $\Delta$ be a ($J-1$)-simplex and $(\pi_{1},\ldots,\pi_{J})$ be
barycentric coordinates of a point $x\in\Delta$. Then, we define

$\displaystyle\psi_{j}:\Delta$ $\displaystyle\rightarrow\sigma_{-j}$ (5)
$\displaystyle x$ $\displaystyle\mapsto\psi_{j}(x)$
$\displaystyle(\pi_{1},\ldots,\pi_{J})$
$\displaystyle\mapsto\left(\tilde{\pi}_{1},\ldots,\tilde{\pi}_{j-1},\tilde{\pi}_{j+1},\ldots,\tilde{\pi}_{J}\right)$

as the _perspective projection_ of $x$ about the vertex $v_{j}$ onto the
opposing facet $\sigma_{-j}$. This corresponds to shooting a ray from the
vertex $v_{j}$ through the point $x$, and the intersection of that ray with
the opposing edge is the image $\psi_{j}(x)$. Figure 1 provides an
illustration for a triangle (2-simplex) $\Delta$ with edges
$v_{1},v_{2},v_{3}$, where the perspective projection $\psi_{1}$ projects the
point $x$ about the vertex $v_{1}$ onto the opposing edge
$\sigma_{-1}=\overline{v_{2}v_{3}}$ ($\overline{AB}$ denotes a line segment
from $A$ to $B$). It is evident that perspective projection about a vertex
$v_{j}$ is the geometrical equivalent to renormalization (Equation 2) after
removing the $j^{\text{th}}$ component. For the theorems below, it is crucial
that perspective projection does not affect the ratios of the remaining
components’ barycentric coordinates.

##  3 Related Work

(a) Parallel coordinates.

(b) Stacked plot.

(c) Simplex plot.

Figure 3: Existing visualization techniques for compositional data. The data
are equal for all three visualization methods and they stem from a Bayesian
model comparison experiment: For $i=1,\ldots,100$, we have posterior model
probabilities $p(M_{1}\,|\,y^{(i)}),p(M_{2}\,|\,y^{(i)}),p(M_{3}\,|\,y^{(i)})$
which are, per construction, non-negative and sum to one for each $i$. (a)
Parallel coordinates convey the dispersion for each compartment $M_{j}$. Here,
$M_{3}$ has less variance across simulation indices $i$. (b) Stacked plots
illustrate the dependency on a third variable. Here, there is no systematic
dependency on the simulation index $i$. (c) Simplex plots visualize the
relation between compartments. Here, we observe a banana-shaped relation where
some combinations of compartments are more likely than others.

In the following, we will briefly compare three fundamental visualization
techniques to plot compositional data: parallel coordinate plots, stacked
plots, and simplex (aka ternary) plots. An illustration of each method for a
fixed data set is displayed in Figure 3.

Parallel coordinate plots [13, 12] depict the dimension as a variable on the
$x$-axis (see 3(a)). This means that, in principle, there is no upper bound on
the number of data dimensions. However, correlations and clusters are not
immediately visible, which is a major conceptual drawback of parallel plots.

Stacked plots [3] are vertically stacked area plots with a continuous $x$-axis
(see 3(b)). The dimension is encoded in the fill color or pattern of the
respective area. Stacked plots can communicate a relatively large number of
dimensions, with an upper bound given by the number of discrete levels for the
area fill (e.g., color or fill pattern). Global trends, such as high values on
one dimension throughout many data sets, are easily visible. Furthermore, an
additional (continuous) variable of interest can be plotted on the $x$-axis.
As a drawback, correlations between data instances as well as clusters are not
immediately visible. Moreover, the usability of stacked plots is influenced by
crucial design choices, such as ordering [18], layout [9], or type [11,
classical, inverting, and diverging;].

Simplex plots [10, aka. ternary plots;] leverage the fact that compositional
data $(\pi_{1},\ldots,\pi_{J})$ fulfil the properties of barycentric
coordinates (they sum to 1 and are non-negative) according to Equation 1.
Consequently, the simplex plot visualizes the data as a point in an
equilateral triangle ($2$-simplex) by interpreting the data as barycentric
coordinates (see 3(c)). The main advantage of simplex plots is that
correlations are immediately visible. The crucial drawback of a simplex plot
is the data dimension in the visualization: In an $n$-dimensional
visualization, the conventional simplex plot is limited to $n+1$ components.
In a printed $2$D medium, this implies an upper bound of $3$ dimensions for
compositional data.

Within the scope of this paper, we will extend simplex plots in order to
preserve their advantages (i.e., conveying correlations between components)
while pushing the envelope on their main drawback, namely the limited number
of dimensions.

(a) Projection of a single point $x$ onto each facet.

(b) Unfolded projections.

Figure 4: Illustration of the simplex projection
$\phi=(\psi_{1},\psi_{2},\psi_{3},\psi_{4})$ for a single point $x$ in the
tetrahedron $\Delta^{3}$.

##  4 Simplex Projection

We leverage the structure of compositional data to propose a visualization
method with less image dimensions than data dimensions. Precisely, we show
that all the points in a higher dimensional simplex can be projected onto its
facet without loss of information by proving that our simplex projection is
bijective. Consequently, compositional data with any dimension can be
projected onto a 2D canvas. However, in this paper, we specifically highlight
the case of $J=4$ components, that is, data that could be naïvely visualized
via a tetrahedron. After proving that the method acts as a bijective mapping
between the full-order simplex and lower-order multivariate marginals for
single points, we show how the method generalizes to entire sets of points and
even to continuous probability density functions.

###  4.1 Single Point

Consider a point $x\in\Delta$ in the $(J-1)$ simplex. In the following, we
will prove that a _specific_ set of perspective projections
$\big{(}\psi_{1},\ldots,\psi_{J}\big{)}$ constitutes an invertible mapping
function $\phi$. That is, we can reconstruct the original point $x$ from its
image $\phi(x)$ under the mapping function $\phi$. The exact form of $\phi$ is
illustrated in Figure 4 and formalized below.

######  Theorem 1 (Bijective simplex projection for labeled points).

Let $\Delta$ be a $(J-1)$-simplex,
$\mathcal{K}=\left\\{\sigma_{-j}\right\\}_{j=1}^{J}$ be a pure simplicial
complex of the facets of $\Delta$, and $\psi_{j}(x)$ the perspective
projection of $x$ onto $\sigma_{-j}$ about the vertex $v_{j}$. Further, let
$\mathrm{Img}_{\phi}$ be the image of $\phi$, as detailed in Appendix A.1.
Then,

$\displaystyle\phi:\Delta$
$\displaystyle\rightarrow\mathrm{Img}_{\phi}(\Delta)$ (6) $\displaystyle x$
$\displaystyle\mapsto\left(\psi_{j}\left(x\right)\in\sigma_{-j}\right)_{j=1}^{J}$

is a bijective mapping from the $(J-1)$-simplex $\Delta$ to the set of
compatible projections in the product set of the $(J-2)$-facets of $\Delta$.
What is more, only two matching projections $T\in\sigma_{-1},R\in\sigma_{-2}$
onto different simplices $\sigma_{-1},\sigma_{-2}\in\mathcal{K}$ suffice to
uniquely define the original point $x\in\Delta$.

The proof of Theorem 1 can be found in Appendix A.2.

###  4.2 Set of Points

We previously treated the simplex projection $\phi$ of a single point
$x\in\Delta$ and proved that $\phi$ is a bijection. The next step generalizes
the simplex projection to a _set_ of $L$ points
$\\{x^{(1)},\ldots,x^{(L)}\\}\equiv\\{x^{(l)}\\}_{l=1}^{L}$ with shorthand
notation $\\{x^{(l)}\\}$ when the context is clear. One might be tempted to
assume that we can simply apply $\phi$ to each point in the set individually
and preserve the bijection, and this is technically true—under one critical
assumption, which does not hold in practical applications: We do not know
which projections match to the same original point $x$ in the preimage, as
explained more technically in the following.

For each single point $x^{(l)}$, we can calculate $\phi(x^{(l)})$ according to
Equation 6, and we can readily apply Theorem 1 to recover
$x^{(l)}=\phi^{-1}\big{(}\phi(x^{(l)})\big{)}$. Figure 4 illustrates this for
the tetrahedron ($J=4$). Now consider two points $x^{(n)},x^{(m)}$ from the
preimage, and their projections $\phi(x^{(n)}),\phi(x^{(m)})$. In order to
recover the original points with Theorem 1, we need the barycentric coordinate
ratios of each image, which is where the critical assumption becomes salient.
On each facet $\sigma_{-j}$, we now have _two_ projections, namely
$\psi_{j}(x^{(n)})$ and $\psi_{j}(x^{(m)})$. However, the two projections are
indistinguishable111Theoretically, it is possible to color all projections
arising from $x^{(n)}$ in one color, all projections arising from $x^{(m)}$ in
another color, and so forth, but this quickly becomes infeasible for a larger
set of projected points., and it is not clear which one arose from $x^{(n)}$
and which one arose from $x^{(m)}$. Yet, the argumentation in Theorem 1 builds
on using coordinate ratios of an original point from its different
projections. Therefore, the previous proof cannot be readily applied to a set
of images when the _labels_ (i.e., which projections arose from which original
point) are unknown.

(a) Projection of a set of points $\\{x^{(l)}\\}$ onto each facet.

(b) Unfolded projections.

Figure 5: Illustration of the simplex projection $\Phi$ for a set of points
$\\{x^{(l)}\\}$. The labels of the projections onto the facets in (c) are
lost: We do not know which points across facets originate from the same
original point in the preimage. Yet, Theorem 2 shows that the original points
in (a) can still be recovered from the unlabeled projections in (c)

.

As an extension to Theorem 1, we define a function $\Phi$ which applies the
simplex projection $\phi$ to each element of a _set_ of $L$ points
$\\{x^{(l)}\\}$,

$\Phi\left(\left\\{x^{(l)}\right\\}\right)=\left\\{\phi(x^{(l)})\right\\},$
(7)

where we use the shorthand set notation $\\{\cdot\\}$ for brevity. If the
point labels $l=1,\ldots,L$ are known for all images (up to permutation), the
function $\Phi$ will be invertible as well because $\phi^{-1}$ is invertible:

$\displaystyle\Phi^{-1}\left(\left\\{\phi(x^{(l)})\right\\}\right)$
$\displaystyle=\left\\{\phi^{-1}\left(\phi(x^{(l)})\right)\right\\}=\left\\{x^{(l)}\right\\}.$
(8)

Consequently, it remains to be proven that the labels across projections onto
different facets are unique (up to permutations of labels) such that the
problem can be reduced to Theorem 1. Since performing a perspective projection
on a point $\Delta^{0}(J=1)$ is not sensible and a perspective projection of
points on a line $\Delta^{1}(J=2)$ would project all points to a single
renormalized coordinate $\tilde{\pi}_{j}=1$, we formulate the following
theorem for $J\geq 3$.

######  Theorem 2 (Bijective simplex projection for sets of points).

Let $\Delta$ be a $(J-1)$-simplex ($J\geq 3$),
$\mathcal{K}=\left\\{\sigma_{-j}\right\\}_{j=1}^{J}$ be a pure simplicial
complex of the facets of $\Delta$, $\phi$ a bijective mapping (Equation 6)
with inverse function $\phi^{-1}$ and $\psi_{j}(x)$ the perspective projection
of $x$ onto $\sigma_{-j}$ about the vertex $v_{j}$. Further, let
$\mathrm{Img}_{\phi}$ denote the image of $\phi$, as described in Appendix
A.1. Then,

$\displaystyle\Phi:\left\\{\Delta^{(l)}\right\\}_{l=1}^{L}$
$\displaystyle\rightarrow\left\\{\mathrm{Img}_{\phi}(\Delta^{(l)})\right\\}_{l=1}^{L}$
(9) $\displaystyle\left\\{x^{(l)}\right\\}_{l=1}^{L}$
$\displaystyle\mapsto\left\\{\phi\left(x^{(l)}\right)\right\\}_{l=1}^{L}$

is a bijective mapping.

The proof of Theorem 2 is presented in Appendix A.3.

Algorithm 1 Marginal density approximation

0: Simplex $\Delta$, probability density function $p(x)$ over $\Delta$,
integral accuracy $M$

0: Approximate marginal density $\hat{p}(z)$

1: for $\sigma_{-j}\in\mathcal{K}$ do

2: for $z\in\mathcal{S}(\sigma_{-j})$ do

3: $s=\overline{zv_{j}}$

4: $\hat{p}(z)=0$

5: for $m=1,\ldots,M$ do

6: $x_{m}\leftarrow z+\frac{s}{M}${equidistant points on $s$}

7: $\hat{p}(z)\mathrel{+}=p(x_{m})\,\frac{|s|}{M}$

8: end for

9: end for

10: end for

11: return $\hat{p}(z)$

Figure 6: Validation of the marginal density approximation: For the joint
Dirichlet density over $\Delta^{2}$ (triangle interior), the $2$-variate
marginal distributions are approximated (dashed green) and plotted against the
analytic ground-truth density (solid purple) orthogonal to the respective
edge. Our approximation is indistinguishable from the analytic ground-truth
density.

###  4.3 Continuous Probability Densities

In what follows, we will extend our method from discrete sets of points to
continuous densities, such as probability density functions over the simplex
$\Delta$. Given a density $p(x)$ defined over the simplex $\Delta$, we can
approximate the (multivariate) marginal densities of the projections in the
simplices of the simplicial complex $\mathcal{K}$ numerically (see Algorithm
1). This approach is conceptually similar to pre-integration [8] for each
component, but offers a strong statistical foundation through the principled
statistically equivalent operation of _marginalization_. Furthermore, the
simplex projection method is information-preserving in finite dimensions (as
opposed to pre-integration in general), as shown in Theorem 1 and Theorem 2.

Let $\Delta$ be a $(J-1)$-simplex, $p(x)$ be a probability density function
over $\Delta$, and $\mathcal{K}=\left\\{\sigma_{-j}\right\\}_{j=1}^{J}$ be a
pure simplicial complex of the facets of $\Delta$. For each simplex
$\sigma_{-j}\in\mathcal{K}$, we define a node-based subdivision
$\mathcal{S}(\sigma_{-j})$ of depth $D$.

The depth $D$ controls the number of nodes and thus the resolution of the
approximated marginal density. For each node $z$ of the subdivision
$\mathcal{S}(\sigma_{-j})$, we construct a line segment to the opposing vertex
$s=\overline{zv_{j}}$ in the space of the original higher-order simplex
$\Delta$. Then, we choose $M$ equidistant points
$(x_{m})_{m=1}^{M},x_{m}\in\mathbb{R}^{J-1}$ on the line segment $s$ and
evaluate the density $p(x_{m})$ for each point $x_{m}$. The line integral of
$p(x)$ along $s$ is then approximated by

$\int_{s}p(x)\mathrm{d}x=\lim_{M\rightarrow\infty}\sum\limits_{m=1}^{M}p(x_{m})\,\frac{|s|}{M}$
(10)

with integration step size $\delta=|s|/M$, where $|s|$ is the length of the
line segment $s$. In the implementation, $M$ is finite and acts as a
hyperparameter which controls the accuracy of the approximation in Equation 10
through the number of equidistant points $x_{m}$.

In the software implementation, the density at the bounds $m=1$ and $m=M$ in
Equation 10 might be undefined. We tackle this by setting
$p(x_{1})=p(x_{M})=0$, and the de-facto effect of this practical adjustment
vanishes for increasing accuracy $M$.

#### Proof-of-Concept with an Analytic Density

We validate our approximation for a Dirichlet distribution which has known
analytic marginal distributions as a ground-truth. For
$(\pi_{1},\ldots,\pi_{J})\in\Delta^{J-1},J\geq 2$, the Dirichlet distribution
$\mathrm{Dir}(\alpha_{1},\ldots,\alpha_{J})$ has the probability density
function

$\displaystyle p(\pi_{1},\ldots,\pi_{J}\,|\,\alpha_{1},\ldots,\alpha_{J})$
$\displaystyle=\mathbf{B}(\alpha_{1},\ldots,\alpha_{J})^{-1}\,\prod\limits_{j=1}^{J}\pi_{j}^{\alpha_{j}-1},\alpha_{j}>0\,\forall\,j=1,\ldots,J\;\;\text{with}$
(11) $\displaystyle\mathbf{B}(\alpha_{1},\ldots,\alpha_{J})$
$\displaystyle=\left(\prod\limits_{j=1}^{J}\Gamma(\alpha_{j})\right)\,\Gamma\left(\prod\limits_{j=1}^{J}\alpha_{j}\right)^{-1},$

where $\Gamma$ denotes the Gamma function. The Dirichlet distribution is a
multivariate generalization of the Beta distribution [14]. To ease the
illustration, we study a Dirichlet distribution
$\mathrm{Dir}(\alpha_{1},\alpha_{2},\alpha_{3})$ over the 2-simplex
$\Delta^{2}$ with parameter vector
$(\alpha_{1},\alpha_{2},\alpha_{3})=(2,5,3)$. This implies the following
analytic multivariate marginal distributions [1], which we will use as a
ground-truth to benchmark our approximation against:

$\displaystyle(\pi_{1},\pi_{2})$
$\displaystyle\sim\operatorname{Dir}(\alpha_{1},\alpha_{2})=\operatorname{Dir}(2,5),$
(12) $\displaystyle(\pi_{1},\pi_{3})$
$\displaystyle\sim\operatorname{Dir}(\alpha_{1},\alpha_{3})=\operatorname{Dir}(2,3),$
$\displaystyle(\pi_{2},\pi_{3})$
$\displaystyle\sim\operatorname{Dir}(\alpha_{2},\alpha_{3})=\operatorname{Dir}(5,3).$

Figure 6 illustrates that the proposed approximation technique yields
essentially equal results to the analytic marginal distributions for a subgrid
depth of $D=10$ ($2^{10}=1024$ evaluation nodes on each edge) and an
integration accuracy of $M=1000$. For the practical purpose of visualizing
(probability) densities over higher-order simplices in a lower-order canvas,
this shows that (i) the density information can be preserved through the
simplex projection; and (ii) the necessary numerical approximation does not
introduce substantial inaccuracies.

(a) $x_{m}$ is between nodes

(b) Barycentric approximation

(c) Recursive density approximation

Figure 7: Illustration of the recursive method to interpolate lower-order
marginal densities from a joint density over a $3$-simplex (tetrahedron). (a)
Since $p(x_{m})$ cannot be evaluated directly, the density will be
approximated through the $J-1=3$ nearest neighbors $z_{1},z_{2},z_{3}$. (b)
The density of $x_{m}$ is approximated through barycentric interpolation of
the densities of the nearest neighbor vertices $z_{1},z_{2},z_{3}$. (c)
Applying Algorithm 1 recursively yields approximations for lower-order
marginal distributions as recursive interpolations. The analytic density is
defined over the tetrahedron $\Delta^{3}$. The $3$-variate marginal density
approximations $\hat{p}(\pi_{1},\pi_{2},\pi_{3})\approx\int
p(\pi_{1},\pi_{2},\pi_{3},\pi_{4})\mathrm{d}\pi_{4}$ are depicted on the area
of the triangular facets using the viridis color palette. The subsequent
$2-$variate marginal density approximations are defined over the edges. These
densities are obtained via recursive approximation (see (a) and (b)) and
plotted on the edges with the viridis-plasma palette. For instance, the edge
$\overline{v_{1}v_{2}}$ shows the recursive approximation of the marginal
density
$\hat{p}(\pi_{1},\pi_{2})\approx\int\hat{p}(\pi_{1},\pi_{2},\pi_{3})\mathrm{d}\pi_{3}$.

#### Recursive Marginal Approximations are Possible through Interpolation

For $J=4$ dimensions, the full-order simplex $\Delta$ is a tetrahedron.
Accordingly, the $3$-variate marginal density distributions are defined over
the $2$-simplices (triangles) at the facets of the tetrahedron, and
approximated as described above. In theory, this algorithm could be carried
out again in order to obtain $2$-variate marginal distributions along the
edges of each $2$-simplex. However, Algorithm 1 assumes that we can readily
evaluate the density $p(x)$ at any point $x$ in the $2$-simplex (triangle).
This is not generally the case because we only have access to a numeric
density for the subgrid $Z\subseteq\Delta$ which we approximated via Algorithm
1.

We address the problem of accessing a density $p(x)$ for arbitrary $x$ through
a straightforward barycentric interpolation, as explained in the following.
First, we apply Algorithm 1 with the recursion step
$\Delta\leftarrow\sigma_{-j}$ and proceed as usual until we need to evaluate
the density $p(x_{m})$ at the node $x_{m}$. Because we do not generally have
access to the density $p(x_{m})$, we approximate $p(x_{m})$ with an
interpolation from its $J-1$ nearest neighbors $z_{1},\ldots,z_{J-1}$ (i.e.,
the vertices of the enclosing $J-2$ simplex of the subdivision, see 7(c)). The
notation $z_{j}$ emphasizes that the density $\hat{p}(z_{j})$ of these
vertices has already been approximated through the previous iteration of
Algorithm 1.

With $\lambda_{1},\ldots,\lambda_{J-1}$ as the barycentric coordinates of
$x_{m}$ with reference vertices $z_{1},\ldots,z_{J-1}$, we obtain the linear
barycentric interpolation
$\hat{p}(x_{m})~{}=~{}\sum_{j=1}^{J-1}\lambda_{j}\hat{p}(z_{j})$ for the
probability density of $x_{m}$. More sophisticated interpolation schemes are
possible as well but do not fall within the scope of this paper. This process
can be repeated recursively (with recursion $\Delta\leftarrow\sigma_{-j}$)
until $\Delta=\Delta^{1}$ (line segment) in Algorithm 1. The result of
applying the recursive approximation technique to an arbitrary density over
the $3$-simplex is illustrated in 7(c).

##  5 Conclusion

Current visualization techniques for compositional data on a 2D canvas are
either (i) limited to 3D data (simplex plots); or (ii) they fail to express
structures across dimensions, such as correlations (parallel coordinates,
stacked plots). In this work, we overcome the limited expressiveness (i) of
simplex plots while preserving the structure capturing property (ii).
Exploiting the inherent structure of compositional data, we proposed a
mathematically sound perspective projection approach which corresponds to
visualizing marginal densities in statistics. The resulting visualization
enjoys the remarkable property that the original joint data distribution can
be reconstructed without loss of information, which we proved mathematically.
While our mathematical proof holds for an arbitrary number of dimensions, we
focused our novel visualization method to representing 4D compositional data
on a 2D canvas. Future research can aim to push the envelope and propose
structure- and information-preserving visualizations for higher-dimensional
compositional data on a low-dimensional canvas.

## Acknowledgments

We thank Maximilian Scholz and Javier Enrique Aguilar for stimulating
discussions that played a pivotal role in transforming this method from
Cthulhu triangles into a mathematically sound visualization technique. We
thank Egzon Miftari for helpful advice and feedback on mathematical
intricacies of the early conceptualization. MS thanks the Cyber Valley
Research Fund (grant number: CyVy-RF-2021-16) and the ELLIS PhD program for
support. MS and PCB were supported by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) under Germany’s Excellence Strategy –
EXC-2075 - 390740016 (the Stuttgart Cluster of Excellence SimTech).

## References

  * [1] J. Aitchison  “The Statistical Analysis of Compositional Data”  In _Journal of the Royal Statistical Society_ 44.2, 1982, pp. 139–177
  * [2] Malgorzata Buba-Brzozowa  “Ceva’s and Menelaus’ Theorems for the n-Dimensional Space”  In _Journal for Geometry and Graphics_ 4.2, 2000, pp. 115–118
  * [3] L. Byron and M. Wattenberg  “Stacked Graphs – Geometry & Aesthetics”  In _IEEE Transactions on Visualization and Computer Graphics_ 14.6, 2008, pp. 1245–1252  DOI: 10.1109/TVCG.2008.166
  * [4] Alexander Philip Dawid  “Posterior Model Probabilities”  In _Philosophy of Statistics_ Elsevier, 2011, pp. 607–630  DOI: 10.1016/b978-0-444-51862-0.50019-8
  * [5] Dorothea Dumuid, Željko Pedišić, Javier Palarea-Albaladejo, Josep Antoni Martín-Fernández, Karel Hron and Timothy Olds  “Compositional Data Analysis in Time-Use Epidemiology: What, Why, How”  In _International Journal of Environmental Research and Public Health_ 17.7 MDPI AG, 2020, pp. 2220  DOI: 10.3390/ijerph17072220
  * [6] Gregory B. Gloor, Jean M. Macklaim, Vera Pawlowsky-Glahn and Juan J. Egozcue  “Microbiome Datasets Are Compositional: And This Is Not Optional”  In _Frontiers in Microbiology_ 8 Frontiers Media SA, 2017  DOI: 10.3389/fmicb.2017.02224
  * [7] Michael Greenacre  “Compositional Data Analysis”  In _Annual Review of Statistics and Its Application_ 8.1 Annual Reviews, 2021, pp. 271–299  DOI: 10.1146/annurev-statistics-042720-124436
  * [8] Andreas Griewank, Frances Y. Kuo, Hernan Leövey and Ian H. Sloan  “High dimensional integration of kinks and jumps—Smoothing by preintegration”  In _Journal of Computational and Applied Mathematics_ 344, 2018, pp. 259–274  DOI: 10.1016/j.cam.2018.04.009
  * [9] Yutian He and Hongjun Li  “Optimal layout of stacked graph for visualizing multidimensional financial time series data”  In _Information Visualization_ 21.1, 2022, pp. 63–73  DOI: 10.1177/14738716211045005
  * [10] Richard J. Howarth  “Sources for a history of the ternary diagram”  In _The British Journal for the History of Science_ 29.3, 1996, pp. 337–356  DOI: 10.1017/S000708740003449X
  * [11] Indratmo, Lee Howorko, Joyce Maria Boedianto and Ben Daniel  “The efficacy of stacked bar charts in supporting single-attribute and overall-attribute comparisons”  In _Visual Informatics_ 2.3, 2018, pp. 155–165  DOI: 10.1016/j.visinf.2018.09.002
  * [12] Alfred Inselberg  “Parallel Coordinates”  New York, NY: Springer New York, 2009  DOI: 10.1007/978-0-387-68628-8
  * [13] Alfred Inselberg and Bernard Dimsdale  “Parallel Coordinates”  In _Human-Machine Interactive Systems_ Boston, MA: Springer US, 1991, pp. 199–233  DOI: 10.1007/978-1-4684-5883-1˙9
  * [14] Samuel Kotz, Norman Lloyd Johnson, N. Balakrishnan and Norman Lloyd Johnson  “Continuous multivariate distributions”, Wiley series in probability and statistics  New York: Wiley, 2000
  * [15] John M Lee  “Introduction to topological manifolds” OCLC: 1295480606  New York: Springer, 2000
  * [16] Igorʹ Rostislavovič Šafarevič and Alexey O. Remizov  “Linear algebra and geometry”  Berlin New York: Springer, 2013
  * [17] Marvin Schmitt, Stefan T. Radev and Paul-Christian Bürkner  “Meta-Uncertainty in Bayesian Model Comparison”  In _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (AISTATS)_ 206, PMLR, 2023, pp. 11–29
  * [18] Steffen Strunge Mathiesen and Hans-Jörg Schulz  “Aesthetics and Ordering in Stacked Area Charts” Series Title: Lecture Notes in Computer Science  In _Diagrammatic Representation and Inference_ 12909 Cham: Springer International Publishing, 2021, pp. 3–19  DOI: 10.1007/978-3-030-86062-2˙1

Appendix

##  Appendix A Further Details and Proofs

###  A.1 Image of the Perspective Projection $\phi$

#### The “is compatible with” Relation $\sim$

Let $\mathcal{K}=\left\\{\sigma_{1},\ldots,\sigma_{J}\right\\}$ be a pure
simplicial complex, and $I(\sigma_{j})$ be the set of vertex indices of the
simplex $\sigma_{j}$. The _“is compatible with” relation_

$\displaystyle\sim\;=\Big{\\{}(A,B)\;\Big{|}$ $\displaystyle
A=(\alpha_{1},\ldots,\alpha_{n})\in\sigma_{1},B=(\beta_{1},\ldots,\beta_{m})\in\sigma_{2},$
(13) $\displaystyle\text{for}\;U=I(\sigma_{1})\cap
I(\sigma_{2}):(\tilde{\alpha}_{k})_{k\in U}=(\tilde{\beta}_{k})_{k\in
U}\Big{\\}}\subseteq\sigma_{1}\times\sigma_{2}$

describes equivalence of points across simplices
$\sigma_{1},\sigma_{2}\in\mathcal{K}$ with respect to the ratio of their
shared components, which we call “compatibility”. According to conventions, we
define the infix notation $A\sim B\Longleftrightarrow(A,B)\in\;\sim$.

$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{4}$$v_{4}$$T$$R^{\prime}$$t$r

Figure 8: If two points are compatible according to the relation $\sim$, they
have identical projections onto shared faces. $\forall R\in r:T\sim R$ because
the projections on the shared face $\overline{v_{1}v_{3}}$ are equal,
$\psi_{2}(T)=\psi_{4}(R)=R^{\prime}$. What is more, we can extend this to all
points in the simplex $\sigma_{-4}$ which project to $R^{\prime}$ as well,
$\forall\,R\in r\;\forall\,\tilde{T}\in t:R\sim\tilde{T}$.

While the preimage of the map $\phi$ is clearly the $(J-1)$-simplex $\Delta$,
determining the exact image is not as straightforward. First of all, the
function $\phi$ takes a $J$-dimensional point $x$ and performs $J$ perspective
projections, yielding a vector

$\phi(x)=\big{(}\psi_{1}(x),\ldots,\psi_{J}(x)\big{)}\overset{\text{def}}{=}\big{(}\phi(x)_{1},\ldots,\phi(x)_{J}\big{)}.$

Each element of this vector is an element of the corresponding facet,
$\psi_{j}(x)\overset{\text{def}}{=}\phi(x)_{j}\in\sigma_{-j}$, making the
vector of projections an element of the product space of the facets
$\prod\limits_{j=1}^{J}\sigma_{-j}$. However, due to the invariance of the
barycentric coordinate ratios to perspective projection, not all combinations
in the product space are possible images (cf. Theorem 2). Instead, we need to
restrict the image to objects where all combinations of projections
$\phi(x)_{n},\phi(x)_{m},(n,m)\in\\{1,\ldots,J\\}^{2}$ are compatible to each
other according to the $\sim$ relation (see below). The image of $\Delta$
under the function $\phi$ follows as

$\mathrm{Img}_{\phi}(\Delta)=\left\\{\big{(}\phi(x)_{1},\ldots,\phi(x)_{J})\in\prod\limits_{j=1}^{J}\sigma_{-j}\,\middle|\,\phi(x)_{n}\sim\phi(x)_{m}\
\forall(n,m)\in\\{1,\ldots,J\\}^{2}\right\\}$ (14)

###  A.2 Proof of Theorem 1

###### Proof.

The proof will show that $\phi$ is a bijective map by proving that $\phi$ is
invertible.

Let $x=(\pi_{1},\ldots,\pi_{J})\in\Delta^{J-1}$ and $\phi(x)$ be the
projections of $x$ onto the respective facets as defined above. The projection
on each facet $\sigma_{j}$ is described by the renormalized barycentric
coordinates after removing the $j^{\text{th}}$ component

$\psi_{j}(x)=(\tilde{\pi}_{1},\ldots,\tilde{\pi}_{j-1},\tilde{\pi}_{j+1},\ldots,\tilde{\pi}_{J})$
(15)

by definition in Equation 6 and Equation 5. By repeatedly removing another
component and re-normalizing the coordinates, we can extract numerical values

$\nicefrac{{\pi_{n}}}{{\pi_{m}}}=r_{n,m}$

for the ratios of all pairs of barycentric coordinates of the original point
$x$ because the barycentric coordinate _ratios_ are invariant to projection.
In the following, we will only consider the ratios of subsequent components,
i.e.,
$\\{\nicefrac{{\pi_{1}}}{{\pi_{2}}},\nicefrac{{\pi_{2}}}{{\pi_{3}}},\ldots,\nicefrac{{\pi_{J-1}}}{{\pi_{J}}}\\}$.
The other ratios are not required to solve the problem at hand. Recall that
the constraint $\sum_{j=1}^{J}\pi_{j}=1$ still holds for the barycentric
coordinates of $x$. This yields a system of $J$ equations

$\left\\{\begin{array}[]{@{}l@{}}\pi_{1}/\pi_{2}=r_{1,2}\\\
\pi_{2}/\pi_{3}=r_{2,3}\\\ \hskip 28.45274pt\vdots\\\
\pi_{J-1}/\pi_{J}=r_{J-1,J}\\\
\pi_{1}+\ldots+\pi_{J}=1\end{array}\right.\Rightarrow\left\\{\begin{array}[]{@{}l@{}}\pi_{1}=r_{1,2}\pi_{2}\\\
\pi_{2}=r_{2,3}\pi_{3}\\\ \hskip 28.45274pt\vdots\\\
\pi_{J-1}=r_{J-1,J}\pi_{J}\\\
\pi_{1}+\ldots+\pi_{J}=1\end{array}\right.\Rightarrow\left\\{\begin{array}[]{@{}l@{}}\pi_{1}-r_{1,2}\pi_{2}=0\\\
\pi_{2}-r_{2,3}\pi_{3}=0\\\ \hskip 28.45274pt\vdots\\\
\pi_{J-1}-r_{J-1,J}\pi_{J}=0\\\ \pi_{1}+\ldots+\pi_{J}=1\end{array}\right.$
(16)

with $J$ unknowns and the matrix representation

$\displaystyle\underbrace{\begin{pmatrix}1&-r_{1,2}&0&\cdots&0&0\\\
0&1&-r_{2,3}&\cdots&0&0\\\ \vdots&\vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&0&\cdots&1&-r_{J-1,J}\\\
1&1&1&\cdots&1&1\end{pmatrix}}_{=\mathbf{A}\in\mathbb{R}^{J\times
J}}\begin{pmatrix}\pi_{1}\\\ \pi_{2}\\\ \vdots\\\ \pi_{J-1}\\\
\pi_{J}\end{pmatrix}=\begin{pmatrix}0\\\ 0\\\ \vdots\\\ 0\\\ 1\end{pmatrix}$
(17)

which can be solved since $\mathbf{A}\in\mathbb{R}^{J\times J}$ clearly has
full rank $J$. The solution
$(\pi_{1},\ldots,\pi_{J})^{\top}=\mathbf{A}^{-1}(0,\ldots,0,1)^{\top}$, in
turn uniquely defines $x$ through its barycentric coordinate representation
$x=(\pi_{1},\ldots,\pi_{J})=\sum_{j=1}^{J}\pi_{j}v_{j}\in\Delta^{J-1}$.

We conclude the general case for arbitrary $J$ with two remarks. First, the
barycentric coordinate ratios do not need to be extracted for subsequent
components. It is sufficient if $J-1$ independent ratios are calculated to
determine $J-1$ unknowns, while the last unknown is solved through the
constraint $\sum\pi_{j}=1$. What is more, this implies that any unknown can be
solved through the sum-to-one constraint, and it does not need to be
$\pi_{J}$. After all, the ordering of the components $1,\ldots,J$ is arbitrary
and we can always re-arrange indices to match the notation in the proof above.

Second, it is not necessary to use the projections onto all facets
$\sigma_{-j}$. In the argumentation above, only $J-1$ independent ratios (and
the sum-to-one constraint) are required to recover the original point $x$.
$J-1$ independent ratios can, in turn, be extracted from the projections on
exactly two different facets: From the first projection
$\psi_{n}(x)=(\tilde{\pi}_{1},\ldots,\tilde{\pi}_{n-1},\tilde{\pi}_{n+1},\ldots,\tilde{\pi}_{J})$
onto the facet $\sigma_{-n}$, all necessary ratios except a ratio
$\nicefrac{{\pi_{n}}}{{\pi_{m}}},m\neq n$ involving $\pi_{n}$ can be
extracted. However, the “final” independent ratio including $\pi_{n}$ to solve
for $\pi_{n}$ can be extracted from the projection $\psi_{m}(x)$ onto the
other facet $\sigma_{-m}$ if $n\neq m$. This means that the projections onto
only two facets must always suffice to recover the original point $x$
regardless of the dimensionality $J$.

∎

###  A.3 Proof of Theorem 2

#### Rouché-Capelli Theorem

One critical argument in the proof builds on a corollary of the Rouché-Capelli
theorem [16], which we will denote in the following: “In an euclidean space
$\mathbb{R}^{J-1}$, $J$ hyperplanes $Z_{1},\ldots,Z_{J}$ can have zero, one,
or infinitely many concurrencies”. The $J$ hyperplanes are defined by
$Z_{j}:a_{j,1}x_{1}+\ldots+a_{j,J-1}x_{J-1}=c_{j}$, and the concurrency
(intersection of _all_ hyperplanes simultaneously) is the solution to the
following system of equations:

$\underbrace{\begin{pmatrix}a_{1,1}&\cdots&a_{1,J-1}\\\
\vdots&\ddots&\vdots\\\ a_{J,1}&\cdots&a_{J,J-1}\\\
\end{pmatrix}}_{\mathbf{A}\in\mathbb{R}^{J\times(J-1)}}\begin{pmatrix}x_{1}\\\
\vdots\\\ x_{J-1}\end{pmatrix}=\underbrace{\begin{pmatrix}b_{1}\\\ \vdots\\\
b_{J}\end{pmatrix}}_{b\in\mathbb{R}^{J}}$ (18)

The Rouché-Capelli theorem [16] states that this system has

  1. (i)

no solution, if and only if the rank of its coefficient matrix $\mathbf{A}$
equals the rank of the augmented matrix $[\mathbf{A}|b]$;

  2. (ii)

exactly one solution if and only if $\mathrm{Rank}(\mathbf{A})=J-1$; and

  3. (iii)

infinitely many solutions otherwise.

#### Proof of the Theorem

###### Proof.

The proof will show that $\Phi$ is a bijective mapping by showing that there
is exactly one solution to label the projections onto the facets (up to
permutations), and reduce the problem to Theorem 1.

(a) In $\Delta^{2}$, the hyperplanes which are implied by the ratios on the
edges are lines.

(b) Finding concurrencies of hyperplanes (lines in this case) is equivalent to
identifying consistent points in the preimage. Note that hyperplanes
originating from the same edge may be equal (bottom-most purple line), but not
parallel and unequal.

Figure 9: Illustrations for Theorem 2.

As in Theorem 1, we can repeatedly remove components from all projections and
re-normalize the coordinates to extract numerical values
$\nicefrac{{\pi_{n}}}{{\pi_{m}}}=r_{n,m}$ for the ratios of all pairs of
barycentric coordinates of all projections. Each ratio
$r_{n,m}^{(l)}=\nicefrac{{\pi_{n}^{(l)}}}{{\pi_{m}^{(l)}}}$ implies a
hyperplane $H\in\mathbb{R}^{J-2}$ through the points
$(\tilde{\pi}_{n}^{(l)},\tilde{\pi}_{m}^{(l)})$ and the set of points
$\\{v_{a}\\}_{a\notin\\{n,m\\}}$ of all vertices of the $(J-1)$-simplex which
have been marginalized out to obtain the ratio $r_{n,m}^{(l)}$ (see 9(a)). All
points $h\in H\cap\Delta$ in the union of $H$ and the simplex are compatible
with the point $(\tilde{\pi}_{n}^{(l)},\tilde{\pi}_{m}^{(l)})$ on the edge
$\overline{v_{n}v_{m}}$:

$\forall h\in H\cap\Delta:h\sim(\tilde{\pi}_{n}^{(l)},\tilde{\pi}_{m}^{(l)}).$

Finding intersections of hyperplanes in the simplex $\Delta$ is equivalent to
finding points which are compatible with the ratios inducing these
hyperplanes. Consequently, finding a point where hyperplanes induced by $J$
consecutive ratios forming a cycle (i.e., $r_{1,2},\ldots,r_{J-1,J},r_{J,1}$)
intersect is equivalent to identifying an original point in the preimage which
is compatible with all points on the edges with barycentric coordinates of the
ratios (see 9(b) for an illustration with $J=3$). On each linearly independent
edge $\overline{v_{1}v_{2}},\ldots,\overline{v_{J}v_{1}}$, there is one
extracted ratio (and consequently one implied hyperplane) per original point
$x^{(l)}$. Thus, there is a total of $L$ hyperplanes on each linearly
independent edge, implied by the set of original points. The hyperplane
implied by the ratio $r_{n,m}^{(l)}$ will be referred to as $H_{n,m}^{(l)}$.
The resulting structure of hyperplanes follows as

$\underbrace{\\{H_{1,2}^{(1)},\ldots,H_{1,2}^{(L)}\\}}_{\text{edge}\;v_{1}v_{2}},\ldots,\underbrace{\\{H_{J-1,J}^{(1)},\ldots,H_{J-1,J}^{(L)}\\}}_{\text{edge}\;v_{J-1}v_{J}},\underbrace{\\{H_{J,1}^{(1)},\ldots,H_{J,1}^{(L)}\\}}_{\text{edge}\;v_{J}v_{1}}$
(19)

when we group the $L$ hyperplanes implied by a ratio $r_{n,m}$ in a set, for a
total of $J$ sets of $L$ hyperplanes each. Finding consistent original points
is now equivalent to finding concurrencies of $J$ hyperplanes, each from one
of the $J$ sets above, see 9(b). What is more, no two $J$ hyperplanes in such
an arrangement can be parallel or equal because of the geometry of the
simplex. However, pairs of hyperplanes within one of the $J$ sets may be
equal, yet they cannot be parallel and unequal (see 9(b)). It remains to be
proven that there are exactly $L$ concurrencies where $J$ hyperplanes—one of
each of the sets in Equation 19—intersect.

Statement: “There are exactly $L$ concurrencies between the $J$ sets of $L$
hyperplanes each, implied by the ratios of $L$ images according to Equation
19”. We prove this statement by first arguing that there are at least $L$
concurrencies, and then show that there cannot be more than $L$ concurrencies.
9(b) shows an example where the statement is clearly true. Below, we prove
that it holds for all possible cases with arbitrary $L$ and $J$.

  1. (I)

There are at least $L$ concurrencies. Every preimage point $x^{(l)}$ has a
consistent ratio representation, and thus there must clearly be a concurrency
at each point $x^{(l)}$, resulting in at least $L$ concurrencies.

  2. (II)

There are not more than $L$ concurrencies. We prove this by contradiction.
Assume there was an additional $L+1^{\text{st}}$ concurrency
$z\notin\\{x^{(l)}\\}$. This means that $J$ hyperplanes—one originating from
each edge—intersect at the point $z\in\Delta$. Call these hyperplanes
$Z_{1},\ldots,Z_{J}$ with $Z_{j}\in\\{H_{j,j+1}^{(l)}\\}$ for $j\neq J$ and
$Z_{J}\in\\{H_{J,1}^{(l)}\\}$ for $j=J$ as in Equation 19. Each of the $L$
concurrencies $\\{x^{(l)}\\}$, which must exist as argued in (I), has a set of
$J$ corresponding hyperplanes which intersect at $x^{(l)}$ by definition.
Because one hyperplane of each set is induced by each original point, the
mapping from an original point (one of the $L$ concurrencies) to $J$
hyperplanes (one of each set of $L$ concurrencies $\\{x^{(l)}\\}$ as above) is
surjective (exhaustive) on the set of all $J\cdot L$ hyperplanes.

Figure 10: Illustration of (II) in the proof. Assuming that an additional
concurrency $z$ exists, there are corresponding established concurrencies
$\hat{x}^{(j)}$, one for each hyperplane (line) $Z_{j}$, $j=1,2,3$. For each
hyperplane (line) $Z_{j}$, there must be two other hyperplanes (lines) that
share established concurrencies $\hat{x}^{(j)}$ with $Z_{j}$ by construction.

Consequently, each of the hyperplanes $Z_{1},\ldots,Z_{J}$, which intersect at
the “additional” concurrency $z$, has a corresponding “established”
concurrency with $J-1$ other hyperplanes from the set of $L$ concurrencies
from above. For example, if $Z_{1}=H_{1,2}^{(l^{\prime})}$, then
$x^{(l^{\prime})}\in\\{x^{(l)}\\}$ is an established concurrency corresponding
to $Z_{1}$. The phrase ”established“ emphasizes that we already know about
these $L$ concurrencies through (1). Now consider these $J$ established
concurrencies which correspond to $Z_{1},\ldots,Z_{J}$, which we call
$\hat{x}^{(1)},\ldots,\hat{x}^{(J)}$, all in $\\{x^{(l)}\\}$. One of the
following cases must occur:

    1. (a)

At least one pair of established concurrencies is equal, $\exists
n,m\in\\{1,\ldots,J\\}^{2},n\neq m:\hat{x}^{(n)}=\hat{x}^{(m)}$. This means
that the hyperplanes $Z_{n}$ and $Z_{m})$ intersect at
$\hat{x}^{(n)}=\hat{x}^{(m)}$, and by definition of $z$ they also intersect at
$z$. The corollary on the Rouché-Capelli theorem implies that either
$z=\hat{x}^{(n)}=\hat{x}^{(m)}\in\\{x^{(l)}\\}$ (contradiction; $z$ is assumed
to be a concurrency beyond $\\{x^{(l)}\\}$) or the two hyperplanes $Z_{n}$ and
$Z_{m}$ are equal (contradiction).

    2. (b)

All established concurrencies are different from each other,
$\hat{x}^{(n)}\neq\hat{x}^{(m)}\;\forall n,m\in\\{1,\ldots,J\\}^{2},n\neq m$.
The fact that $z=(\zeta_{1},\ldots,\zeta_{J})$ shares a hyperplane with each
of the $J$ established concurrencies $\hat{x}^{(1)},\ldots,\hat{x}^{(J)}$—each
with barycentric coordinate representation
$(\hat{\pi}_{1}^{(j)},\ldots,\hat{\pi}_{J}^{(j)})$—translates to the ratio
equalities in barycentric coordinates

$\frac{\zeta_{1}}{\zeta_{2}}=\frac{\hat{\pi}^{(1)}_{1}}{\hat{\pi}^{(1)}_{2}},\quad\frac{\zeta_{2}}{\zeta_{3}}=\frac{\hat{\pi}^{(2)}_{2}}{\hat{\pi}^{(2)}_{3}},\quad\ldots,\quad\frac{\zeta_{J}}{\zeta_{1}}=\frac{\hat{\pi}^{(J)}_{J}}{\hat{\pi}^{(J)}_{1}}.$
(20)

Generalized Ceva’s theorem [2] guarantees that

$\frac{\zeta_{1}}{\zeta_{2}}\,\frac{\zeta_{2}}{\zeta_{3}}\cdots\frac{\zeta_{J}}{\zeta_{1}}=1$
(21)

because $z$ is a concurrency by definition. The inverse direction of
generalized Ceva’s theorem states that the hyperplanes $Z_{1},\ldots,Z_{J}$
have a concurrency $\hat{x}$ since the following equality holds:

$\frac{\hat{\pi}^{(1)}_{1}}{\hat{\pi}^{(1)}_{2}}\,\frac{\hat{\pi}^{(2)}_{2}}{\hat{\pi}^{(2)}_{3}}\cdots\frac{\hat{\pi}^{(J)}_{J}}{\hat{\pi}^{(J)}_{1}}=1.$
(22)

It follows from the corollary of the Rouché-Capelli theorem that either
$\hat{x}=z$ (contradiction) or that at least two of the hyperplanes are equal
(contradiction).

∎


# Comparison Study Between Token Classification and Sequence Classification In
Text Classification

Amir Jafari

## I. Introduction

Text classification engines has been used as an alternative and complement to
human annotation for many years. Essay tests are an example of a text
responses where students are given a particular prompt topic to write about
and elaborate on. These texts are evaluated for their writing quality in areas
such as grammar, fluency of elaboration and how well they are organized.
However, evaluating texts in large amount is impractical and classifying it is
mostly very complicated and difficult [1].

Text classifications engines can’t genuinely read and comprehend texts the way
humans or raters can. Text classification engines do not use direct qualities
of a text such as how fluent it is or how grammatically correct. But human
raters may explicitly evaluate numerous of those intrinsic variables to
produce a prediction. This gives a clear distinction between machine scoring
and human labeling. In [2] they studied whether human scoring or machine
scoring classification models are better at classifying scientific research
abstracts according to a fixed set of discipline groups. Text engines are
mostly trained by collecting generally a few couple of thousand sample text
which were previously assessed by human raters. There has always been
skepticism over machine scoring over time mostly in regard to the fact that
computer models does not understand the written text as they human does [3].
However, by looking at labeling between human raters and machine models and
create a an statical metric like Cohen’s Quadratic Weighted Kappa score we
could measure the quality of scoring models (human vs machine) as long as we
prove the sample data is statistically rules significant.

There are two major categories of text classification engines available in the
field natural language processing: traditional and neural network-based
models. Traditional text classification engines examine a text using carefully
crafted features. The effectiveness of such models is closely correlated with
the caliber of the underlying components. The most important element of such
engines is manually designing the features and rules, which is time-consuming.
A feature free method for learning the relationship between a text assigned
label in [4] investigates the use of recurrent neural network model as a
modern methods. For the purpose of text classification engine, various neural
network models are used in the field [5, 6, 7, 8].

In comparison to statistical models with handmade characteristics, neural
network models have recently been utilized for text classifiers [9, 10, 4]
have produced better results. Word representations in particular are utilized
as the input to a neural network model which produces a single dense vector
for each word representation in the whole text. Based on a non-linear
transformation in each layer on the representation, a label is given. It has
been demonstrated that across several domains, neural network models are more
reliable than statistical models without the use of handmade features.

Text classifiers engines have been using both convolutional neural networks
[11] and recurrent neural networks [12] modles. A single-layer LSTM also has
been used over the word sequence to model the text, and they separately model
sentences and documents using a two-level hierarchical CNN structure. It is
well known that LSTMs excel at modeling temporal and ordered data while CNNs
excel at capturing local information. The effectiveness of LSTMs and CNNs
using the identical text classification settings has been discussed [10].

Pre-trained language models have recently been used to significantly increase
the performance of phrase prediction tasks by combining representations from
many layers, creating an auxiliary sentence, using multitask learning, etc.

Large pre-trained language models have recently demonstrated amazing abilities
in representation and generalization, like GPT [13], BERT [14], XLNet [15],
etc. These models now perform a variety of downstream tasks including text
summarization, generation, classification, and regression with enhanced
accuracy. There are numerous innovative methods for improving already-trained
language models. Auxiliary sentence construction was suggested by [16] to
improve sentiment classification problems. To address sequential sentence
classification tasks, Cohan [17] introduced additional distinct tokens to
obtain representations of each sentence. A work in [16] on several fine-tuning
techniques, such as combining text representations from various layers and
leveraging multi-task learning has been done.

It makes sense to use BERT type models or generally transformer-based models
to learn text representations given the enormous success of pre-trained
language models like BERT in learning text representations with deep
semantics. The BERT approach, which relies heavily on self-attention, can
capture interactions between any two words throughout the entire body of
articles (long texts). Previous work on [16] demonstrates that merging text
representations from various layers does not significantly increase
performance. The length of texts for the text classification job is close to
the maximum allowed by the BERT model, making it challenging to create an
auxiliary phrase.

In this article, we suggest using two type classification heads for AES engine
to our base model to learn text representations that fully reflect deep
semantics. These two models are Token Classifier and Sequence Classier models.
A token classifier uses all the states of model (features that are transferred
over many layers of pre-trained models) and maps all the numerical
representation into a label, however the Sequence Classifier utilize a slice
of information from all the states and maps the representations into a label.

In summary, our contributions are:

  * •

Investigate the use two different classifier from different in nature in text
classification engines on a standard dataset which allows us to compare
results.

  * •

Explore the advantages and disadvantages of each method.

  * •

To label long texts which are longer than 256 token using token classification
method.

## II. Data Description

Modern natural language processing systems learn effective models with the
help of annotated data as supervision. These models can only be utilized
within that language because they are typically trained on data that is only
available in that language (typically English). Since it is impractical to
collect data in every language, cross lingual language understanding (XLU) and
low-resource cross-language transfer have gained popularity. The Cross-lingual
Natural Language Inference (XNLI) corpus is the extension of the Multi-Genre
NLI (MultiNLI). There are 122k train, 2490 validation, and 5010 test samples
in the dataset [18] in all 15 languages. The model is trained on the training
set and the test set is used to find the best hyper-parameter and model. The
validation set is used is an unseen set which is used later for testing the
performance of the classifiers.

Table 1: XNLI Data Stats Set | Samples | Tokens | Token > 256 | Average Label | % Label 0 | % Label 1 | % Label 2
---|---|---|---|---|---|---|---
Train | 392702 | 392702 | 66 | 1 | 33.33 | 33.33 | 33.33
Test | 5010 | 5010 | 0 | 1 | 33.33 | 33.33 | 33.33
Validation | 2490 | 2490 | 0 | 1 | 33.33 | 33.33 | 33.33
Table 2: Sample data Premise |  Hypothesis |  Label
---|---|---
Conceptually cream skimming has two basic dimensions - product and geography . |  Product and geography are what make cream skimming work . |  1
’you know during the season and i guess at at your level uh you lose them to the next level if if they decide to recall the the parent team the Braves decide to call to recall a guy from triple A then a double A guy goes up to replace him and a single A guy goes up to replace him’ |  You lose the things to the following level if the people recall . |  0
Gays and lesbians . |  Heterosexuals . |  2

Table 1 shows basic statistics about XNLI dataset on just English data such as
the number of samples in each set, number overall tokens in all samples,
number of texts whose number of tokens is greater than 256 and finally the
distribution of all labels. XNLI dataset is used as sample set of data for
testing the hypothesis of scoring regime on two different methods. The dataset
is chosen since the label distribution and mean of labels are perfectly
balanced.

Table 2 shows a sample set of text data which we used for classification
purpose. Premise and Hypothesis text are combined and construct our sample
text. The labels are integer encoded numbers which zero represents entailment,
one represents neutral and 2 represents contradiction. This sample would be
suitable for comparison study about the different nature of transformer
classifiers.

## III. Background for Text Classification Engines

The development of neural network-based natural language processing has had a
significant impact on the current possibilities for text classification
engines. A few preliminary corresponding comments on the meaning of some of
the terms used as well as some additional methodological background will be
provided beforehand because the terminology used in a machine learning context
is frequently different from that used in psychometrics and is also
occasionally ambiguous. Following that, more background information is
provided along with two significant NLP techniques based on neural networks
and their accompanying AES successes.

### i. Terms

  * •

Model: A particular architecture for a machine learning which has a lot
parameters needs to be tuned to find a mapping between a text and label. This
can be pre-trained with a specific training dataset

  * •

Token/Tokenization: A text sequence is an input to the model in NLP task.
Making this text sequence into a chunk of words or sub words and map them to a
series of numbers called tokenization.

  * •

Padding: The only input sequences that neural networks accept are fixed size.
The nature of text are variable in length to overcome the issue we add pad
word in to a sequence which is less than maximum length size.

  * •

Embeddings: The embeddings give the tokens a meaning by converting a single
token into a high-dimensional vector.

  * •

Feature : In text classification regime, the features are referred to
embedding of each token of a text sequence.

  * •

Language Model: A language model is a model that predicts the next word,
subword, or character in a text.

### ii. Text Classification Modeling Approaches

  * •

Traditional Approaches : Bag of words approach (BOW) was commonly used in text
classification for a number of years and the aim in these models are to
extract features from the text and label the text based on those features [19,
20, 21].

  * •

Recurrent Neural Nets (RNN): The gated recurrent units (GRU) [22]and the long
short-term memory (LSTM) [23] are the two most common types of RNNs models.
They used very often in the field of AES.

  * •

Transformer Models: Transformer models are neural networks based on an
attention mechanism [24]that are pre-trained on the large text data for the
purpose of language modeling.

A transformer-based language models from the various architecture types (BERT,
ELECTRA, ROBERTA, etc.) first undergo self-supervised training on sizable text
corpora. They are then fine-tuned to a particular task (like classification,
question/answering, token classification, generation, summarization) in a
subsequent stage utilizing supervised learning and hand-coded labels. In the
next sections we explain our methodology to use different transformer
classifier on long or short texts.

## IV. Long Text Tokenization

Powerful neural network architectures known as Transformers [24, 25, 26] have
achieved state of art status in a number of machine learning domains,
including Natural Language Processing (NLP), Neural Machine Translation (NMT)
[27] and text generation summarization. Unfortunately, a conventional
Transformer scales quadratically with the number of tokens, making it
unaffordable for a sequence larger than 512 tokens. For this problem, several
solutions have been proposed in [28, 29, 30, 31]. Most methods limit the
attention mechanism to focus on nearby neighborhoods or include structural
priors on attention such as sparsity, pooling-based compression, clustering,
or convolution techniques.

A dataset must first be preprocessed into an input format before it can be
used to train a model. Your data must be transformed and put together into
batches of texts. This preprocessing method is called tokenization, we use a
tokenizer to break text into a series of tokens, then turn those tokens into
numbers and put them together into certain ID numbers. Transformer
architectures are trained on the large set of data with a restriction on the
size of tokenized input. Most of transformers have a limit of 512 input
tokens.

In this work we are not changing the model architecture to address the issues
of long texts by increasing the limit size, however we are proposing to use
the 256 limit size architecture using a method using overlapping segments. In
the Token Classier model, we are introducing an stride hyper parameter that is
the amount the text which we are overlapping. A single long text will be
chunked and as $N$ number of texts of 256 token size, in such way that we
track the overflow tokens.

Figure 1: Long Text Chuncking

In Figure 1, the text containing 532 tokens, was passed through a tokenizer
that returned tokenized inputs of length 512. Since the number of tokens in
the text was more than the max length, the text was truncated to length 512,
and the remaining tokens were put in the next entry. The last 20 token ids of
the first entry were also present at the beginning of the second entry due to
the use of the stride parameter. Since the second entry falls short of the max
length of 512, padding was added to the second entry.

## V. Methodology

### i. Token Classifier For Text Labeling

The token classification model architecture is unquestionably the most popular
one among all the Machine Learning techniques used in NLP. There has been a
lot of work been done in the field of token classification like the the
following research works; dependency parsing [32], detecting label errors in
Token Classification data [33] and token sequence labeling vs. clause
classification for English emotion stimulus detection [34]. Tokenization, or
simply identifying the beginning and end of the words and punctuation in the
text, is the initial stage in token categorization. The next step is to decide
which class (tag or token) belongs to. The choice of classes for issues like
part-of-speech tagging is simple: they are the part-of-speech classes that
each word may have, such as noun, verb, adjective, etc. Since one or more
nearby tokens may appear to be a member of the same sort of phrasal chunk, but
be a part of a different chunk. Other tasks, like chunking, necessitate subtle
alterations when defining their classes. This would be useful for a name
entity recognition system,

However, we propose to use the Token Classifier as a text classier that could
label texts. The XNLI dataset is used in this case as a sample set that
represents texts. Every single text has a label, that label is going to be
assigned for every single token presented in the text shown in Figure 2. The
model predicts label for every single token and finally the average of all the
token labels will be assigned a final label.

Figure 2: Token Classifier Model

For texts that the length is more than 256 tokens, the overflow option of the
tokenizer is used and mapped all the overflow token to a unique id. Now we can
train an text with more than 256 token by introducing the stride hyper
parameter. Therefore, we could scan a long texts by starting the first 256
token and then use stride of $n$ to overlap the texts again and get another
256 chunks till the and long texts will be a chunk of 256 texts.

Finally, we can train the Token Classifier concept in token classification
theme with the chunking mechanism which addresses the issue of using the first
256 tokens.

### ii. Sequence Classifier For Text Labeling

Text classification is a well-known and extensively studied job in Natural
Language Processing (NLP) and text mining. It refers to the act of assigning
one or more labels from a set of classes to a document [35]. Simple binary
classification, multi-class classification, and multi-label classification are
examples of text classification problem types. Simple binary classification,
for instance, asks if a document has positive sentiment or not. The latter
allows for the assignment of many labels to a single document. Text
Classification engines are the subcategory of text classification.

Tokenizing the text input is the initial stage in the classification process
using a transformer model, just like in the other traditional methods. The
chosen pretrained transformer has a strict binding effect on the
tokenization’s format. The next step is to use the Sequence Classifier model
that will be fine-tuned on the set of labels.

Figure 3: Sequence Classifier Model

As you can see in Figure 3, the first predicted token is used to map the
features to the target labels however in the token classier all the prediction
token are used in prediction the the final label for the text.

## VI. Results

The results of scoring using Sequence Classification and Token Classification
are compared. First the pre-trained Electra model [36] is fine-tuned on
training set. A grid search technique is used to find a best model on our
metric. QWK metric is our main metric to evaluate the performance of the
model. The hyper parameter that we are optimizing our models are:

  * •

Learning Rate

  * •

Batch Size

  * •

Stride

### i. Grid Search

Table 3: Grid Search - Sequence Classification experiment_id | epoch | num label | test acc | test QWK | batch size | lr
---|---|---|---|---|---|---
3 | 1 | 3 | 0.805 | 0.766 | 60 | 0.000035
2 | 1 | 3 | 0.807 | 0.766 | 50 | 0.000025
1 | 1 | 3 | 0.805 | 0.766 | 50 | 0.000035
4 | 2 | 3 | 0.799 | 0.760 | 60 | 0.000025
Table 4: Grid Search - Token Classification experiment | epoch | num label | test acc | test QWK | stride | max token | batch size | lr
---|---|---|---|---|---|---|---|---
8 | 4 | 3 | 0.825 | 0.800 | 250 | 256 | 12 | 0.000035
4 | 5 | 3 | 0.810 | 0.792 | 200 | 256 | 10 | 0.000025
5 | 3 | 3 | 0.821 | 0.790 | 250 | 256 | 10 | 0.000025
1 | 4 | 3 | 0.819 | 0.789 | 200 | 256 | 10 | 0.000035
3 | 5 | 3 | 0.815 | 0.789 | 300 | 256 | 10 | 0.000035
2 | 7 | 3 | 0.817 | 0.788 | 250 | 256 | 10 | 0.000035
6 | 10 | 3 | 0.817 | 0.787 | 300 | 256 | 10 | 0.000025
7 | 4 | 3 | 0.814 | 0.780 | 200 | 256 | 12 | 0.000035

Table 3 shows the result of training a Sequence Classier Electra model on the
training set and save the best model on the test set. The highest test QWK
metric on the Sequence Classification method 0.76.

Table 4 shows the result of training a Token Classier Electra model on the
train set and save the best model on the test set. As you can in the token
classification, stride is another hyper parameter that is introduced to deal
with a long text. The highest test QWK metric on the the token classification
method 0.80.

Now it is the time to evaluate the best models on the validation set which the
network has not seen before to draw a final decision on the performance of
each model.

### ii. Validation Results

Final models are tested on the validation set and Table 5 shows each model
performance on the validation set. As you can see the Token Classifier has
higher accuracy (test acc and val acc) and perform well on scoring texts.
Beside QWK, standardized mean difference (SMD) is used to measure efficacy in
terms of a continuous measurement (lower better).

Table 5: Validation Results Model | val acc | val qwk | val smd
---|---|---|---
Token Classifier | 0.827 | 0.803 | 0.043
Sequence Classifier | 0.814 | 0.781 | 0.052

## VII. Conclusion and Future Work

In this work, the performance between the Sequence Classifiers and Token
Classifiers are studied. The token classification model has better performance
on text classification and labeling texts. The use of all of the prediction
sequences that models’ outputs help to extract more information for predicting
labels. The sequence classifier is the most common used model for scoring but
its performance is not as well as the token classification method, because
sequence classifier just uses a first sequence prediction token which ignores
all the other sequences information and then maps the texts into its label.
Certainly, the use all the sequences that are extracted from the features
provided better performance because the base model are exactly the same for
the this study . For future work, different type of aggregation in token
classification can be explored instead of using a mean of label for every
single token. Also, we could verify the performance on the Kaggle essay data
set.

## References

  * [1] Y. Attali and J. Burstein, “Automated essay scoring with e-rater® v. 2,” _The Journal of Technology, Learning and Assessment_ , vol. 4, no. 3, 2006.
  * [2] Y. C. Goh, X. Q. Cai, W. Theseira, G. Ko, and K. A. Khor, “Evaluating human versus machine learning performance in classifying research abstracts,” _Scientometrics_ , vol. 125, no. 2, pp. 1197–1212, 2020.
  * [3] E. B. Page and N. S. Petersen, “The computer moves into essay grading: Updating the ancient test,” _Phi delta kappan_ , vol. 76, no. 7, p. 561, 1995\.
  * [4] K. Taghipour and H. T. Ng, “A neural approach to automated essay scoring,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 1882–1891.
  * [5] G. Liang, B.-W. On, D. Jeong, H.-C. Kim, and G. S. Choi, “Automated essay scoring: A siamese bidirectional lstm neural network architecture,” _Symmetry_ , vol. 10, no. 12, p. 682, 2018.
  * [6] P. U. Rodriguez, A. Jafari, and C. M. Ormerod, “Language models and automated essay scoring,” _arXiv preprint arXiv:1909.09482_ , 2019.
  * [7] M. Uto and M. Okano, “Robust neural automated essay scoring using item response theory,” in _International Conference on Artificial Intelligence in Education_. Springer, 2020, pp. 549–561.
  * [8] Y. Farag, H. Yannakoudakis, and T. Briscoe, “Neural automated essay scoring and coherence modeling for adversarially crafted input,” _arXiv preprint arXiv:1804.06898_ , 2018.
  * [9] D. Alikaniotis, H. Yannakoudakis, and M. Rei, “Automatic text scoring using neural networks,” _arXiv preprint arXiv:1606.04289_ , 2016.
  * [10] F. Dong and Y. Zhang, “Automatic features for essay scoring–an empirical study,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 1072–1077.
  * [11] Y. Chen, “Convolutional neural network for sentence classification,” Master’s thesis, University of Waterloo, 2015.
  * [12] T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur, “Recurrent neural network based language model.” in _Interspeech_ , vol. 2, no. 3. Makuhari, 2010, pp. 1045–1048.
  * [13] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [15] F. Dong, Y. Zhang, and J. Yang, “Attention-based recurrent convolutional neural network for automatic essay scoring,” in _Proceedings of the 21st conference on computational natural language learning (CoNLL 2017)_ , 2017, pp. 153–162.
  * [16] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” in _China national conference on Chinese computational linguistics_. Springer, 2019, pp. 194–206.
  * [17] A. Cohan, I. Beltagy, D. King, B. Dalvi, and D. S. Weld, “Pretrained language models for sequential sentence classification,” _arXiv preprint arXiv:1909.04054_ , 2019.
  * [18] A. Conneau, G. Lample, R. Rinott, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “Xnli: Evaluating cross-lingual sentence representations,” _arXiv preprint arXiv:1809.05053_ , 2018.
  * [19] Y.-Y. Chen, C.-L. Liu, C.-H. Lee, T.-H. Chang _et al._ , “An unsupervised automated essay-scoring system,” _IEEE Intelligent systems_ , vol. 25, no. 5, pp. 61–67, 2010.
  * [20] C. Leacock and M. Chodorow, “C-rater: Automated scoring of short-answer questions,” _Computers and the Humanities_ , vol. 37, no. 4, pp. 389–405, 2003.
  * [21] V. Ramalingam, A. Pandian, P. Chetry, and H. Nigam, “Automated essay grading using machine learning algorithm,” in _Journal of Physics: Conference Series_ , vol. 1000, no. 1. IOP Publishing, 2018, p. 012030.
  * [22] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [25] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” _arXiv preprint arXiv:1710.10903_ , 2017.
  * [26] J. Vig, “A multiscale visualization of attention in the transformer model,” _arXiv preprint arXiv:1906.05714_ , 2019.
  * [27] M. X. Chen, O. Firat, A. Bapna, M. Johnson, W. Macherey, G. Foster, L. Jones, N. Parmar, M. Schuster, Z. Chen _et al._ , “The best of both worlds: Combining recent advances in neural machine translation,” _arXiv preprint arXiv:1804.09849_ , 2018.
  * [28] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” _arXiv preprint arXiv:2004.05150_ , 2020.
  * [29] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” _arXiv preprint arXiv:1904.10509_ , 2019.
  * [30] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3286–3295.
  * [31] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [32] R. L. Milidiu, C. E. M. Crestana, and C. N. dos Santos, “A token classification approach to dependency parsing,” in _2009 Seventh Brazilian Symposium in Information and Human Language Technology_. IEEE, 2009, pp. 80–88.
  * [33] W.-C. Wang and J. Mueller, “Detecting label errors in token classification data,” _arXiv preprint arXiv:2210.03920_ , 2022.
  * [34] L. A. M. Oberländer and R. Klinger, “Token sequence labeling vs. clause classification for english emotion stimulus detection,” in _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics_ , vol. 1, 2020, pp. 58–70.
  * [35] F. Sebastiani, “Machine learning in automated text categorization,” _ACM computing surveys (CSUR)_ , vol. 34, no. 1, pp. 1–47, 2002.
  * [36] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” _arXiv preprint arXiv:2003.10555_ , 2020.