# Competition of alliances in a cyclically dominant eight-species population

Junpyo Park  Xiaojie Chen  and Attila Szolnoki  Department of Applied
Mathematics, Kyung Hee University, Yongin 17104, Republic of Korea  School of
Mathematical Sciences, University of Electronic Science and Technology of
China, Chengdu 611731, China  Institute of Technical Physics and Materials
Science, Centre for Energy Research, P.O. Box 49, H-1525 Budapest, Hungary

###### Abstract

In a diverse population, where many species are present, competitors can fight
for surviving at individual and collective levels. In particular, species,
which would beat each other individually, may form a specific alliance that
ensures them stable coexistence against the invasion of an external species.
Our principal goal is to identify those general features of a formation which
determine its vitality. Therefore, we here study a traditional Lotka-Volterra
model of eight-species where two four-species cycles can fight for space.
Beside these formations, there are other solutions which may emerge when
invasion rates are varied. The complete range of parameters is explored and we
find that in most of the cases those alliances prevail which are formed by
equally strong members. Interestingly, there are regions where the symmetry is
broken and the system is dominated by a solution formed by seven species. Our
work also highlights that serious finite-size effects may emerge which prevent
observing the valid solution in a small system.

###### keywords:

cyclic dominance , alliances , competition

††journal: Chaos Solitons and Fractals

##  1 Introduction

If a species is stronger and beats the other one then how can the “weaker”
species survive? This is a fundamental problem of ecology to explain the
amazing biodiversity of life [1, 2]. A possible explanation could be the so-
called intransitive or cyclically dominated relation among competing species
where a third (or more) species beat the predator species, hence establishing
a delicate balance among all competitors. In the simplest case, these
relations remind the well-known rock-scissors-paper game and this type of
interaction can really be observed among animals, plants, or even among
bacterias [3, 4, 5, 6, 7, 8, 9]. Evidently, a three-member loop can be easily
enlarged for more participants, as it was made in the extended Lotka-Volterra
model [10, 11, 12, 13]. Motivated by the complexity of such interactions,
researchers have studied related models actively over the last decade and
several interesting observations have been made [14, 15, 16, 17, 18].

It would be almost impossible to sum all important observations, such as
reported in Refs. [19, 20, 21, 22, 23, 24, 25]. Instead, the reader is
referred to specific review papers about the milestones along this research
path [26, 27, 28, 29]. Nevertheless, it is also worth noting here that the
mentioned cyclic or intransitive relation among participants should not
necessarily be defined by a Lotka-Volterra-type microscopic rule, but could be
the result of collective behavior among competing strategies in evolutionary
game models [30, 31, 32, 33, 34, 35, 36, 37, 38]

The most exciting question is whether we can predict the direction of
evolution based on the food-web that defines the microscopic dynamics? Can we
identify general principles which may inform us about the vitality of an
alliance? An early observation was, for instance, that when two three-member
cycles fight then the one, where the inner invasion is faster, is more viable
[39]. The picture, however, is more subtle, because a faster general rotation
does not necessarily provide an advantage if the members of the trio are
unequal. In the latter case, the triplet becomes vulnerable no matter the
average of inner invasion rates is higher than in the rival triplet which is
formed by less aggressive, but equally strong partners [40]. Notably,
heterogeneous invasion rates may emerge temporarily or locally [41, 42]. Or,
the extension of the original food-web by adding a reverse link can also
change a stable solution [43]. Furthermore, the number of members could also
be a decisive factor because a smaller loop is generally stronger than a large
one [44].

In this work we introduce a minimal eight-species model where two four-member
loops fight for space. The interaction between these quartets practically
establishes a traditional Lotka-Volterra circle of eight species, which can be
described by an invasion rate. Not to break the balance between the quartets,
we assume that the inner invasion is uniform and equally strong in both
formations. The only difference between them is we introduce two short-cuts in
one of the loops which generates extra interactions within the related
alliance. Our major question is how the stability of alliances change due to
this extension and can we identify new solutions which would be hidden
otherwise? It could also be interesting whether the previously established
principles, observed for simpler systems formed by trials and duets, remain
valid when we apply them to alliances formed by larger groups.

##  2 Alliances formed at different levels

To make our observations comparable with previous works [45, 46, 47, 48, 49,
50], we assume that species are distributed on an $L\times L$ square lattice
where periodic boundary conditions are applied. Each node is occupied by one
of the species which are denoted by $X_{i}$ where indexes are from $i=0$ to 7.
During an elementary step, we select a neighboring pair nodes. If they are
occupied by the same species or represent species which are neutral then
nothing happens. Otherwise, an invasion happens with a specific probability
and predator species invades prey species. The microscopic rules are defined
in the following way:

$\displaystyle X_{i}X_{i+1}\xrightarrow{\gamma}X_{i}X_{i}$ (1) $\displaystyle
X_{i}X_{i+2}\xrightarrow{\alpha}X_{i}X_{i}$ (2) $\displaystyle
X_{2}X_{6}\xrightarrow{\beta}X_{2}X_{2},X_{4}X_{0}\xrightarrow{\beta}X_{4}X_{4},$
(3)

where $i+1$ and $i+2$ are calculated in cyclic manner. The parameters
$\alpha,\beta$ and $\gamma$ define the probabilities of successful invasions
between the involved predator-prey neighbors.

Figure 1: Food-web of competing species. In the basic model eight species are
invade cyclically each other with probability $\gamma$, similar to the
extended Lotka-Volterra model. Additionally, we introduce a cyclic inner
invasion among odd and separately among even species with probability
$\alpha$. To break the symmetry among the quartets, we also introduce an inner
invasion between species “2” and “6” and between species “4” and “0” with
probability $\beta$. Importantly, we use the color-code of species in later
figures where spatial distributions are presented.

For a deeper insight about the model, in Fig. 1 we show the food-web of
competing species. Our first note is about the biggest loop where all species
are involved, according to the extended Lotka-Volterra type system. Here every
species is simultaneously a predator and a prey of another one, hence
establishing a possible solution. The invasion rate is $\gamma$ for all
interactions here. Another solution is formed by “1”, “3”, “5”, and “7”
species who invade each other cyclically with probability $\alpha$. For later
reference, we denote this alliance as $A^{4}_{1,3,5,7}$. A similar
$A^{4}_{0,2,4,6}$ quartet is formed among species “0”, “2”, “4” and “6”.
Additionally, we introduce an extra chance of invasions among group members
here. In particular, species “2” invades species “6” and species “4” invades
species “0” with probability $\beta$. In this way, there are no neutral pairs
in the mentioned quartet anymore. As a consequence, the average inner invasion
is augmented among the four species, which could be a support to their
viability. Interestingly, however, there are many other possible solutions in
this system. For example, the new invasions among even-numbered species offer
the chance for trials to emerge: $A^{3}_{0,2,4}$ or $A^{3}_{0,2,6}$ can work
as a rock-scissors-paper-type solution. Beside the mentioned formations, there
are $A^{5}$ (five-), $A^{6}$ (six-), or $A^{7}$ seven-member solutions, as
well. Perhaps, it is not necessary to specify them because the reader can
easily construct examples, based on the food-web, shown in Fig. 1.

As we noted, there are three parameters in our model and in the following we
systematically scan the whole parameter field to identify the dominant
solution for each combination of parameters. For this reason, we executed
Monte Carlo simulations where a Monte Carlo step ($MCS$) means that on average
every node has a chance to update its state. The linear system size varied
between $L=100$ to $L=3200$ and the necessary relaxation steps were between
$10^{3}$ to $3\times 10^{5}$ $MCS$s. According to the standard protocol,
simulations were launched from an initial state where species are distributed
randomly on the lattice and monitor the $\rho_{i}$ concentration of $X_{i}$
for all species. We, however, would like to stress that this initial state
does not always could be a good choice to find the solution which is valid in
the large size limit. It is because a small system size does not necessarily
“offer” equal chance for all possible solutions to emerge. Some solution,
which involves large typical lengths, would emerge just later, but to that
stage of the evolutionary process other solutions may invade the whole
available space. Therefore, a more complex solution practically has no chance
to emerge at a small system size. As a consequence, other solutions may win
and an independent run can easily terminates onto a different solution, no
matter we use the same set of model parameters. To overcome this uncertainty,
we not simply enlarged the system size, but also used alternative initial
states where larger and homogeneous domains of competing species were
distributed randomly on the lattice. Similar approach was used previously in
Refs.[51, 52], for instance. In this way, we can create all types of
interfaces which could be the foundation to build more sophisticated
solutions. To make our analysis more complete, at the critical values of
control parameters we also generated sub-solutions independently in restricted
areas of space and after we let them fight directly along the separating
interface [53, 54, 55]. This technique can identify the dominant solution
unambiguously. The details of the above mentioned protocols will be specified
in the next section.

##  3 Results

###  3.1 Phase diagrams

We first summarize our main findings and later we give further details about
the microscopic mechanisms which are responsible how dominant solutions
emerge. According to our simulations, we can distinguish three main cases
which determine the system behavior. The decisive condition is the intensity
of interaction between the quartets mentioned in Sec. 2. If this interaction
is weak, means the value of $\gamma$ is small, then the
$A^{4}_{\\{1,3,5,7\\}}$ solution dominates the whole $\beta-\alpha$ parameter
plane. In other words, only the quartet of “1+3+5+7” survive independently of
the $\alpha$ and $\beta$ values.

For intermediate values of $\gamma$, when the invasion flow in the large cycle
becomes substantial, a conceptually new behavior emerges. We represent this
phenomenon by showing the phase diagram obtained at $\gamma=0.5$. Figure 2
shows that beside the mentioned $A^{4}_{\\{1,3,5,7\\}}$ quartet, other
solutions become dominant at certain values of $\beta-\alpha$ pairs. When both
$\alpha$ and $\beta$ are small then the “large circle” is the winner, hence
all eight species coexist. For intermediate $\beta$ values this solution is
replaced by $A^{7}_{\\{0,1,2,3,4,6,7\\}}$ where only species “5” is missing.

Figure 2: Phase diagram on $\beta-\alpha$ parameter plane obtained at
$\gamma=0.5$. If $\alpha$ is large enough, the quartet of “1+3+5+7” species
always win. If both $\beta$ and $\alpha$ are small, all species survive.
Interestingly, for low $\alpha$ and intermediate $\beta$ a seven-member
solution dominates where species “5” is missing. Dashed blue line denotes
discontinuous, while solid red line marks the positions of continuous phase
transition points.

Qualitatively similar system behavior can be observed when $\gamma$ is large,
hence the flow in the external loop is intensive. In other words, the
interaction between the quartets becomes large. This is illustrated in Fig. 3
where we present the complete phase diagram on the $\beta-\alpha$ plane. The
only difference between the diagrams is the area, where complete eight-species
solution dominates, is larger and the seven-member solution is shifted toward
larger $\beta$ values.

Figure 3: Phase diagram on $\beta-\alpha$ parameter plane obtained at
$\gamma=1$. The diagram is similar to the one shown in Fig. 2. The only
difference is the eight-member full solution occupies larger area in the
parameter field, while seven-member solution is shifted toward higher $\beta$
values.

We must stress that the presented phase diagrams are valid in the large system
size limit. If the system size is small then we may observe that the system
can easily terminate onto different solutions. To illustrate it, in Fig. 4 we
present the survival probabilities at each $(\alpha,\beta)$ parameter pairs
for $\gamma=1$, when the linear system size was $L=100$. More precisely, we
launched the evolution from a random initial state and monitored the fractions
of species until $N=L\times L$ $MCS$s. After, we recorded the number of
surviving species and repeated the whole process 100 times. In this way we can
estimate the probability that $S$ species survive for long time. The panels of
Fig. 4 show these surviving probabilities for all possible $S$ values at $x$
different $(\alpha,\beta)$ pairs on the parameter plane. These panels indicate
that different solutions may emerge no matter we use the same values of
$(\alpha,\beta)$ parameters. Therefore, the destination is rarely unambiguous,
hence the surviving probability equals to 1 only for $S=4$ when $\alpha$ is
high and $\beta$ is low. Nevertheless, the contour of areas around “maximum”
values are roughly agree with the diagram shown in Fig. 3. But, as Fig. 4
illustrated, this system size is insufficient to make reliable conclusions
about the proper system behavior.

Figure 4: Heat maps of the survival probability on $\beta-\alpha$ parameter
plane obtained for $\gamma=1$ by using $100\times 100$ system size. Each panel
shows the probability to reach a state which contains $S$ species after
$N=L^{2}$ $MCS$s. The value of $S$ is indicated on each panel. The results are
averaged over 100 independent runs.

###  3.2 Detecting phase transitions

Therefore, to detect the phase transition points more precisely, we need to
apply systematic finite-size analysis. An example is given in Fig. 5 where we
present the “lack” of species “5” in dependence of $\alpha$ at fixed
$\beta=0.75$ and $\gamma=1$ values. When the value of $1-\rho_{5}$ reaches 1
then the system enters onto the mentioned $A^{7}_{\\{0,1,2,3,4,6,7\\}}$
solution. Evidently, we also checked the portions of other species, because
$\rho_{5}=0$ is fulfilled in other solutions, too. Here the symbols are the
average of many independent runs. As an example, for $L=100$ we executed 2000
times. Importantly, the average hides the proper system behavior for small
system size, because it mixes different destinations. For example, at $L=100$,
$\alpha=0.15$ we never measured $\rho_{5}=0.063$. Instead, the majority of
independent runs terminated onto a state where $\rho_{5}=0$ or
$\rho_{5}=0.25$. This ambiguity disappears for large system sizes. The plot
also highlights that the transition from the eight-member to seven-member
solution is continuous because species “5” vanishes gradually as we increase
$\alpha$. The transition, however, between $A^{7}_{\\{0,1,2,3,4,6,7\\}}$ and
$A^{4}_{\\{1,3,5,7\\}}$ phases is discontinuous.

Figure 5: The absence of species “5”, as an order parameter, in dependence of
$\alpha$ at $\gamma=1$, $\beta=0.75$ for different system sizes. The linear
sizes are indicated in the legend. The dominant solutions are marked on the
top. The plots are the average of 100-2000 runs depending on the system size.
Lines are just to guide the eye.

To illustrate another type of phase transition, in Fig. 6 we show an
alternative order parameter in dependence of $\beta$ at $\gamma=0.5$ and
$\alpha=0.01$. Here, we calculate the difference between the portions of
quartets formed by odd- and even-indexed species, respectively. When $\beta$
is small, all available species coexist, the system is in the “all” phase,
hence the mentioned difference is small. When this parameter becomes 1, then
species with even indexes vanish and the quartet of “1+3+5+7” species becomes
dominant. Interestingly, this formation is viable even if the inner invasion,
due to the tiny $\alpha=0.01$, is extremely slow. The explanation of this
surprising behavior is given in the next subsection.

Similarly to the previously discussed Fig. 5 case, the average of the order
parameter could be misleading when the system size is small. This can be
clearly seen for $L=100$ and $L=200$, where the average is larger than the
value for larger system sizes. It is because the system can easily be trapped
in the $A^{4}_{\\{1,3,5,7\\}}$ state already at small $\beta$ values. The
possible destinations, however, are less ambiguous for larger sizes, but the
jump in the order parameter is valid signaling a discontinuous phase
transition at $\beta=0.66$.

Figure 6: The difference between symmetric quartets in dependence of $\beta$
at $\gamma=0.5$, $\alpha=0.1$. When this order parameter becomes 1, the system
enters to a four-species state. The dominant solutions are marked on the top.
The plots are the average over 20-2000 runs, depending on the system size. The
linear size of squares are indicated.

###  3.3 Battles of solutions

In the following, we analyze the possible mechanisms which explain the system
behavior summarized in Fig. 2 and in Fig. 3. To get a deeper insight about the
dominant processes, which drive the patter formation, it is instructive to use
a specific initial state where we divide the available space into two halves
and both regions are occupied by one of the main quartets formed by odd- or
even-indexed species. In this way, we can follow how these solutions compete,
or how new possibilities may emerge due to their interaction.

The $\beta-\alpha$ parameter plane can be divided into four main sectors which
fundamentally determine the relation of different solutions. We first consider
the low $\alpha$ – low $\beta$ region. It is important to note that the
$A^{4}_{\\{0,2,4,6\\}}$ quartet formed by even-indexed species is not a proper
solution here. Instead, we can see the battle of $A^{3}_{\\{0,2,4\\}}$ and
$A^{3}_{\\{0,2,6\\}}$ triplets when even-indexed species are present. Since
species “4” beats species “6”, this fight ends up with the victory of the
former solution. This domain is marked by “$A$” on the left panel of Fig. 7.
When $\gamma$ is small then there is just a weak interaction between odd- and
even-indexed species. Therefore, the mentioned domains remain compact and they
fight along the separating interface. Finally, $A^{4}_{\\{1,3,5,7\\}}$ quartet
win this battle and the whole space will be occupied by the domain, marked by
“$B$” on the left panel.

Figure 7: Pattern formation in the low $\alpha$ – low $\beta$ region when we
launch the evolution from a prepared state where left (right) side of the
space is occupied by even-indexed (odd-indexed) species. In the former case
only the $A^{3}_{\\{0,2,4\\}}$triplet survive, shown by “$A$” on the left
panel. For small $\gamma$, when the interaction is weak around the main loop,
the $A^{4}_{\\{1,3,5,7\\}}$ quartet, marked by “$B$”, can beat the other
solution and gradually invades the whole space. If $\gamma$ is high enough,
species “7” can easily invade the triplet leaving a neutral
$A^{2}_{\\{2,7\\}}$ pair behind. But this solution, marked by “$C$”, cannot
survive because the interface between the original quartets is a birthplace of
the complete eight-species solution. This is marked by “$D$”. Since $\gamma$
is high enough, this solution is viable and prevails. Color codes agree with
those we introduced in Fig. 1. Parameters are: $\alpha=0.1$, $\beta=0.1$,
$\gamma=0.05$ (left panel), and $\gamma=1$ (right panel).

If $\gamma$ is high enough then the evolution changes drastically. This is
illustrated on the right panel of Fig. 7. Because of high $\gamma$, species
“7”, who has no predator in the “0+2+4” triplet, can easily enter into the
$A^{3}_{\\{0,2,4\\}}$ domain. The raid of species “7” results in a neutral
duo. This $A^{2}{\\{2,7\\}}$ solution is marked by “$C$” on this panel.
Interestingly, however, this solution has limited life, because the intensive
interactions of the original quartets at the border offer a chance for the
complete eight-member solution to emerge. This solution is marked by “$D$” on
this panel. Because of the high $\gamma$ value, the general invasion flow is
intensive in the largest loop, which makes it strong against
$A^{4}_{\\{1,3,5,7\\}}$. Notably, the latter quartet is weak due to small
$\alpha$. The above described mechanism explains the left-down corners of
phase diagrams shown in Fig. 2 and in Fig. 3.

We can face a conceptually different situation in the large $\alpha$ – small
$\beta$ corner of the parameter plane, because it provides supporting
conditions both for $A^{4}_{\\{0,2,4,6\\}}$ and $A^{4}_{\\{1,3,5,7\\}}$
quartets. Therefore, both solutions would be viable in the absence of the
other. Importantly, the high $\alpha$ value makes the difference from the
previously discussed cases, which generates a fast rotation withing both
quartets. There is, however, a crucial difference between these solutions:
while $A^{4}_{\\{1,3,5,7\\}}$ is formed by equal partners, the extra inner
invasions described by $\beta$ probability break this delicate balance for the
benefit of species “0” at the expense of species “6”. In this way the pattern
formed by even-indexed species becomes heterogeneous, including larger domains
of “0” species. This makes the whole alliance vulnerable against the attack of
the rival well-balanced loop.

This process is illustrated in Fig. 8, where we monitor the battle of these
quartets. Initially, we allowed both solutions to emerge peacefully in
restricted areas and the stationary portions of species “0” and “6” change, as
we described previously. Let us stress that in the initial state all species
represented equally, which is practically invisible because the unequal
stationary distributions of “0+2+4+6” species evolve very fast. After 20000
$MCS$s, when the separating borders opened, the symmetric solution gradually
invades the whole space. Notably, the presented simulation was recorded at
$\gamma=0.1$, where the interaction between the quartets is moderate. Still,
this light communication is capable to reveal the advantage of
$A^{4}_{\\{1,3,5,7\\}}$ solution. If we apply larger $\gamma$ values then
nothing changes conceptually, but the victory of $A^{4}_{\\{1,3,5,7\\}}$
becomes faster. Hence, in agreement with the phase diagrams, the symmetric
four-member solutions is always the winner in the mentioned corner of the
$\beta-\alpha$ parameter plane.

The lastly discussed observation also answers one of our original questions.
Namely, one may argue that the introduction of an additional inner invasion
within $A^{4}_{\\{0,2,4,6\\}}$ solution results in a faster rotation among
these species, which could be useful them [39]. But the weakening consequence
of symmetry breaking is stronger, as it was also the case for three-member
loops [40].

Figure 8: The time evolution of species when quartets are fighting at high
$\alpha$, low $\beta$ values. Initially both $A^{4}_{\\{0,2,4,6\\}}$ and
$A^{4}_{\\{1,3,5,7\\}}$ solutions evolve independently in separated areas of
available space. When separating borders opened, marked by an arrow, symmetric
quartet gradually crowd out the rival gang. Color codes are the usual, except
white line is replaced by grey one. Parameters are: $\alpha=0.5$, $\beta=0.1$,
$\gamma=0.1$, $L=1600$.

Next, we discuss the small $\alpha$ – large $\beta$ corner of the parameter
plane. Here, the situation is partly similar to the first discussed case. More
precisely, $A^{4}_{\\{0,2,4,6\\}}$ solution is unstable and replaced very soon
by $A^{3}_{\\{0,2,4\\}}$. But this triplet is vulnerable, because the large
heterogeneity of inner flow, ($\alpha\ll\beta$), results in huge homogeneous
domains, as it was already reported in [56] for traditional rock-scissors-
paper game. Such a large homogeneous domain is also an easy prey of the rival
$A^{4}_{\\{1,3,5,7\\}}$ quartet. This picture is valid for small and medium
$\gamma$ values, where the interaction between the even- and odd-indexed
species is not too strong. For large $\gamma$ values, however, the evolution
could be more complex, which can be observed more easily from a “random-patch”
initial state. This starting pattern is illustrated in Fig. 9. As we already
mentioned in Sec. 2, this prepared initial state can offer all possible
interfaces to be present at the very beginning, which is extremely useful when
there are large difference among the invasion rates. In this way, we can
observe the valid solution already at smaller system size. We stress, however,
that the mentioned state should be reached from all kind of initial states if
all available species are present and the system size is large enough.

Figure 9: Alternative initial states from where evolution is launched when
$L=200$. Panel (a) shows the traditional starting state where every node is
uploaded by a randomly chosen species. Panel (b) shows a state where two
competing quartets are generated first. Panel (c) shows a state where larger
patches of species are distributed randomly. The solution, which is valid in
the large size limit, can be reached from all starting points, but the
necessary system size could be largely different.
Figure 10: Stationary fractions of species in dependence of $\beta$ at
$\gamma=1$, $\alpha=0.2$. The results are obtained at $L=3200$ system size.
Species are denoted to every lines. For clarity we only show the lines without
symbols here. The stable phases are shown on the top.

Turning back to the large $\gamma$, large $\beta$, small $\alpha$ region, the
above specified prepared initial state can help us to identify the valid
$A^{7}_{\\{0,1,2,3,4,6,7\\}}$ solution already at relatively small system
sizes. At first sight, this solution may seem weird or exotic, but its
emergence can be understood if we follow how the portion of species change by
increasing $\beta$. This phenomenon is illustrated in Fig. 10 where we present
the stationary fractions of all species when we increase the intensity of
additional invasions at fixed $\gamma=1$ and $\alpha=0.2$. At $\beta=0$ we
have a completely symmetric food-web where there are equally strong quartets.
Consequently, all species are present at the same level here. When we
introduce a non-zero $\beta$ then not only the fractions of species “0” and
“6” start decaying, but simultaneously, the portions of their principal preys,
species “1” and “7” start growing. This increment involves the decay of their
preys, which are species “2” and “0”. This double stress explains why species
“0” suffers the most at small $\beta$ values. Naturally, the decay of species
“6” affects the population of its principal predator, hence the portion of
species “5” also a decreasing function. One may argue that the decay of
species “0” should affect its main predator species “7”. But the predator of
the latter, which is species “6”, cannot grow here, hence in sum species “7”
should not necessarily decrease. This difference between the status of species
“5” and “7” explains why there are diverse consequences when both of their
principal preys, species “6” and “0”, are attacked directly via an inner
invasion described by parameter $\beta$. Importantly, the direct support of
species “4” via the intensive “4”$\to$“2” helps the spreading of species “4”.
This process is also dangerous for species “5”. On the other hand, the
intensive “2”$\to$“6” process will not simply strengthen species “2”, but also
weaken its prey species “3”, which consequence is also enjoyed by species “4”.
When the latter species die out, the remaining seven species form a
heterogeneous seven-member Lotka-Volterra loop where the “weakest” species
(species “4”, who beats species “6” with probability $\alpha$) occupies the
largest fraction of space. This effect is an example for the phenomenon
observed first by Tainaka in the simplest three-member loop [57].

Last, we discuss the remaining large $\alpha$ – large $\beta$ corner of the
parameter plane. Here the $A^{4}_{\\{0,2,4,6\\}}$ solution is not stable
again, hence the remaining $A^{3}_{\\{0,2,4\\}}$ solution fights against
$A^{4}_{\\{1,3,5,7\\}}$ quartet. When $\gamma$ is low and the interaction is
weak between these groups then the latter formation wins. This evolution is
similar to the case we reported in the left panel of Fig. 7. Practically, the
same happens when $\gamma$ is high, but in this case a large set of solutions
may emerge temporarily. Despite of this diversity, however, the winning
alliance remains the symmetric quartet. To give an impression about the
variety of different candidates, we present an intermediate snapshot about the
“battlefield” taken at $\alpha=0.9$, $\beta=0.9$, and $\gamma=1$. Figure 11
shows that in the intermediate stage of the evolution at least nine(!)
solutions emerge locally and fight for space. But eventually the symmetric
$A^{4}_{\\{1,3,5,7\\}}$ solution crowds out all other competitors.

Figure 11: Intermediate stage of the evolution taken from the battlefield when
different solutions emerge and fight for space. The domains mark the following
solutions: $A^{4}_{\\{0,1,3,4\\}}$ ($A$), $A^{5}_{\\{1,3,4,5,7\\}}$ ($B$),
$A^{4}_{\\{1,3,5,7\\}}$ ($C$), $A^{5}_{\\{0,2,3,5,7\\}}$ ($D$),
$A^{5}_{\\{0,1,3,4,6\\}}$ ($E$), $A^{3}_{\\{0,2,4\\}}$ ($F$),
$A^{6}_{\\{1,2,3,4,5,7\\}}$ ($G$), $A^{5}_{\\{1,2,4,5,7\\}}$ ($H$), and
$A^{5}_{\\{1,2,3,5,7\\}}$ ($I$). Finally “$C$” domain wins. Parameters are
$\alpha=0.9$, $\beta=0.9$, $\gamma=1$, and $L=400$.

##  4 Conclusions

Which are the most important features of a cyclically dominant alliance that
determine its viability against alternative formations? Motivated by this
question, we introduced an eight-species model where there are two four-member
quartets who interact each other, hence forming a complete eight-member loop,
too. The original model is completely symmetric where the inner invasion
within the quartets are equally strong. Additionally, we break the symmetry
and introduced an extra inner invasion within one of the loops. Based on
previous observations about three-member loops, one may argue that this faster
rotation of alliance members could be positive to make them stronger. From the
other viewpoint, the broken symmetry is always detrimental, hence the new
opportunity would just weaken the involved quartet. It is also worth noting
that the slight alteration of the original “symmetric” food-web offers the
chance several potential solutions to emerge. Indeed, an armada of candidates
can be observed at intermediate stage of the evolution, but it is believed
that the final pattern is determined by some basic concepts.

According to our findings, keeping the symmetry is vital and the solutions,
which maintain a balance among their members, are fitter. In the complete
parameter space we studied, it can happen that there are more than one
solution which possess this attractive character. In this case the general
speed of inner invasion could be a decisive factor. If the rotations are
equally strong then we could give examples when a short or a larger loop is
the winner. For example, a quartet can beat a trio, but a quartet can also
beat an octet. Therefore, it cannot be made a simple conclusion that a smaller
or larger alliance is better. Probably, there are two competing effects whose
relation determines the actual fitness of a loop. On one hand, a short loop
may involve relatively large homogeneous domains, which could always be
dangerous. On the other hand, too large loop also means that the reaction of
the alliance could be delayed, because several inner invasions should happen
to produce the predator of the external intruder. Therefore, the sum of these
adverse impacts could be case-sensitive.

Naturally, our present study is a sterile theoretical model, but we strongly
believe that the observations we made could be useful when real systems are
studied. Our another important message is the importance of appropriately
chosen system size, which is always a central issue when cyclic dominance is
present. Otherwise, the observations could be diverse without deeper
understanding. This is why we should always consider the actual size when we
want to give predictions about the pattern formation in a finite system driven
by intransitive interactions.

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (No. NRF-2021R1A4A1032924). J.P.
was also supported by a grant from Kyung Hee University in 2022
(KHU-20220901). X.C. was supported by the National Natural Science Foundation
of China (Grants Nos. 61976048 and 62036002) and the Fundamental Research
Funds of the Central Universities of China. A.S. was supported by the National
Research, Development and Innovation Office (NKFIH) under Grant No. K142948.

## References

  * Sigmund [1993] K. Sigmund, Games of Life: Exploration in Ecology, Evolution and Behavior, Oxford University Press, Oxford, UK, 1993.
  * Bascompte and Solé [1998] J. Bascompte, G. Solé (Eds.), Modeling Spatiotemporal Dynamics in Ecology, Springer, New York, 1998\.
  * Sinervo and Lively [1996] B. Sinervo, C. M. Lively,  The rock-paper-scissors game and the evolution of alternative male strategies,  Nature 380 (1996) 240–243.
  * Burrows and Hawkins [1998] M. T. Burrows, S. J. Hawkins,  Modelling patch dynamics on rocky shores using deterministic cellular automata,  Mar. Ecol. Prog. Ser. 167 (1998) 1–13.
  * Kerr et al. [2002] B. Kerr, M. A. Riley, M. W. Feldman, B. J. M. Bohannan,  Local dispersal promotes biodiversity in a real-life game of rock-paper-scissors,  Nature 418 (2002) 171–174.
  * Cameron et al. [2009] D. D. Cameron, A. White, J. Antonovics,  Parasite-grass-forb interactions and rock-paper-scissor dynamics: predicting the effects of the parasitic plant Rhinanthus minor on host plant communities,  J. Ecol. 97 (2009) 1311–1319.
  * Ruifrok et al. [2015] J. L. Ruifrok, T. Janzen, D. P. Kuijper, M. Rietkerk, H. Olff, C. Smit,  Cyclical succession in grazed ecosystems: The importance of interactions between different-sized herbivores and different-sized predators,  Theor. Popul. Biol. 101 (2015) 31–39.
  * Garde et al. [2020] R. Garde, J. Ewald, Á. T. Kovács, S. Schuster,  Modelling population dynamics in a unicellular social organism community using a minimal model and evolutionary game theory,  Open Biol. 10 (2020) 200206.
  * Liao et al. [2020] M. J. Liao, A. Miano, C. B. Nguyen, L. Chao, J. Hasty,  Survival of the weakest in non-transitive asymmetric interactions among strains of E. coli,  Nat. Com. 11 (2020) 6055.
  * Frachebourg et al. [1996] L. Frachebourg, P. L. Krapivsky, E. Ben-Naim,  Spatial organization in cyclic Lotka-Volterra systems,  Phys. Rev. E 54 (1996) 6186–6200.
  * Esmaeili et al. [2018] S. Esmaeili, B. L. Brown, M. Pleimling,  Perturbing cyclic predator-prey systems: How a six-species coarsening system with nontrivial in-domain dynamics responds to sudden changes,  Phys. Rev. E 98 (2018) 062105.
  * Baker and Pleimling [2020] R. Baker, M. Pleimling,  The effect of habitats and fitness on species coexistence in systems with cyclic dominance,  J. Theor. Biol. 486 (2020) 110084.
  * Bayliss et al. [2020] A. Bayliss, A. Nepomnyashchy, V. Volpert,  Beyond rock-paper-scissors systems - deterministic models of cyclic ecological systems with more than three species,  Physica D 411 (2020) 132585.
  * Durney et al. [2011] C. H. Durney, S. O. Case, M. Pleimling, R. Zia,  Saddles, arrows, and spirals: Deterministic trajectories in cyclic competition of four species,  Phys. Rev. E 83 (2011) 051108.
  * Cazaubiel et al. [2017] A. Cazaubiel, A. F. Lütz, J. J. Arenzon,  Collective strategies and cyclic dominance in asymmetric predator-prey spatial games,  J. Theor. Biol. 430 (2017) 45–52.
  * Choi et al. [2017] C. W. Choi, C. Xu, P. M. Hui,  Adaptive cyclically dominating game on co-evolving networks: numerical and analytic results,  Eur. Phys. J. B 90 (2017) 190.
  * Roman et al. [2016] A. Roman, D. Dasgupta, M. Pleimling,  A theoretical approach to understand spatial organization in complex ecologies,  J. Theor. Biol. 403 (2016) 10–16.
  * Szabó et al. [2007] G. Szabó, A. Szolnoki, G. A. Sznaider,  Segregation process and phase transition in cyclic predator-prey models with even number of species,  Phys. Rev. E 76 (2007) 051921.
  * Park [2022] J. Park,  Correlation between the formation of new competing group and spatial scale for biodiversity in the evolutionary dynamics of cyclic competition,  Chaos 32 (2022) 081101\.
  * Yoshida et al. [2022] T. Yoshida, T. Mizoguchi, Y. Hatsugai,  Non‐Hermitian topology in rock-paper-scissors games,  Sci. Rep. 12 (2022) 560.
  * Tainaka et al. [2021] K. Tainaka, N. Nakagiri, H. Yokoi, K. Sato,  Multi-layered model for rock-paper-scissors game: A swarm intelligence sustains biodiversity,  Ecological Informatics 66 (2021) 101477.
  * Serrao and Täuber [2021] S. R. Serrao, U. C. Täuber,  Stabilizing spiral structures and population diversity in the asymmetric may–leonard model through immigration,  Eur. Phys. J. B 94 (2021) 175.
  * Park et al. [2018] J. Park, Y. Do, B. Jang,  Multistability in the cyclic competition system,  Chaos 28 (2018) 113110\.
  * Nagatani et al. [2018] T. Nagatani, G. Ichinose, K. Tainaka,  Heterogeneous network promotes species coexistence: metapopulation model for rock-paper-scissors game,  Sci. Rep. 8 (2018) 7094.
  * Mobilia et al. [2016] M. Mobilia, A. M. Rucklidge, B. Szczesny,  The influence of mobility rate on spiral waves in spatial rock-paper-scissors games,  Games 7 (2016) 24\.
  * Szolnoki et al. [2014] A. Szolnoki, M. Mobilia, L.-L. Jiang, B. Szczesny, A. M. Rucklidge, M. Perc,  Cyclic dominance in evolutionary games: a review,  J. R. Soc. Interface 11 (2014) 20140735.
  * Szabó and Fáth [2007] G. Szabó, G. Fáth,  Evolutionary games on graphs,  Phys. Rep. 446 (2007) 97–216.
  * Dobramysl et al. [2018] U. Dobramysl, M. Mobilia, M. Pleimling, U. C. Täuber,  Stochastic population dynamics in spatially extended predator-prey systems,  J. Phys. A: Math. Theor. 51 (2018) 063001.
  * Szolnoki et al. [2020] A. Szolnoki, B. F. de Oliveira, D. Bazeia,  Pattern formations driven by cyclic interactions: A brief review of recent developments,  EPL 131 (2020) 68001\.
  * Hauert et al. [2002] C. Hauert, S. De Monte, J. Hofbauer, K. Sigmund,  Volunteering as Red Queen mechanism for cooperation in public goods game,  Science 296 (2002) 1129–1132.
  * Szolnoki and Chen [2020] A. Szolnoki, X. Chen,  Strategy dependent learning activity in cyclic dominant systems,  Chaos Soliton. Fract. 138 (2020) 109935.
  * Palombi et al. [2020] F. Palombi, S. Ferriani, S. Toti,  Coevolutionary dynamics of a variant of the cyclic Lotka-Volterra model with three-agent interactions,  Eur. Phys. J. B 93 (2020) 194.
  * Szolnoki and Chen [2017] A. Szolnoki, X. Chen,  Alliance formation with exclusion in the spatial public goods game,  Phys. Rev. E 95 (2017) 052316.
  * Canova and Arenzon [2018] G. A. Canova, J. J. Arenzon,  Risk and interaction aversion: Screening mechanisms in the prisoner’s dilemma game,  J. Stat. Phys. 172 (2018) 279–292.
  * Szolnoki and Perc [2017] A. Szolnoki, M. Perc,  Second-order free-riding on antisocial punishment restores the effectiveness of prosocial punishment,  Phys. Rev. X 7 (2017) 041027.
  * Tao et al. [2021] Y. Tao, K. Hu, L. Shi,  Risk-preference-driven participate willingness provides alternative routes to solve social dilemma,  EPL 135 (2021) 28001\.
  * Szolnoki et al. [2010] A. Szolnoki, Z. Wang, J. Wang, X. Zhu,  Dynamically generated cyclic dominance in spatial prisoner’s dilemma games,  Phys. Rev. E 82 (2010) 036110.
  * Liu et al. [2022] L. Liu, Z. Xiao, X. Chen, A. Szolnoki,  Early exclusion leads to cyclical cooperation in repeated group interactions,  J. R. Soc. Interface 19 (2022) 20210755.
  * Perc et al. [2007] M. Perc, A. Szolnoki, G. Szabó,  Cyclical interactions with alliance specific heterogeneous invasion rates,  Phys. Rev. E 75 (2007) 052102.
  * Blahota et al. [2020] M. Blahota, I. Blahota, A. Szolnoki,  Equal partners do better in defensive alliances,  EPL 131 (2020) 58002\.
  * Szolnoki and Perc [2016a] A. Szolnoki, M. Perc,  Biodiversity in models of cyclic dominance is preserved by heterogeneity in site-specific invasion rates,  Sci. Rep. 6 (2016a) 38608.
  * Szolnoki and Perc [2016b] A. Szolnoki, M. Perc,  Zealots tame oscillations in the spatial rock-paper-scissors game,  Phys. Rev. E 93 (2016b) 062307.
  * Bazeia et al. [2020] D. Bazeia, B. F. de Oliveira, J. V. O. Silva, A. Szolnoki,  Breaking unidirectional invasions jeopardizes biodiversity in spatial May-Leonard systems,  Chaos, Solitons and Fractals 141 (2020) 110356.
  * de Oliveira and Szolnoki [2022] B. F. de Oliveira, A. Szolnoki,  Competition among alliances of different sizes,  Chaos, Solitons and Fractals 157 (2022) 111940.
  * Intoy and Pleimling [2015] B. Intoy, M. Pleimling,  Synchronization and extinction in cyclic games with mixed strategies,  Phys. Rev. E 91 (2015) 052135.
  * Lütz et al. [2013] A. Lütz, S. Risau-Gusman, J. Arenzon,  Intransitivity and coexistence in four species cyclic games,  J. Theor. Biol. 317 (2013) 286–292.
  * Mir et al. [2022] H. Mir, J. Stidham, M. Pleimling,  Emerging spatiotemporal patterns in cyclic predator-prey systems with habitats,  Phys. Rev. E 105 (2022) 054401.
  * Szabó and Szolnoki [2008] G. Szabó, A. Szolnoki,  Phase transitions induced by variation of invasion rates in spatial cyclic predator-prey models with four or six species,  Phys. Rev. E 77 (2008) 011906.
  * Brown et al. [2019] B. L. Brown, H. Meyer-Ortmanns, M. Pleimling,  Dynamically generated hierarchies in games of competition,  Phys. Rev. E 99 (2019) 062116.
  * Park and Jang [2019] J. Park, B. Jang,  Robust coexistence with alternative competition strategy in the spatial cyclic game of five species,  Chaos 29 (2019) 051105\.
  * Szolnoki et al. [2011] A. Szolnoki, G. Szabó, M. Perc,  Phase diagrams for the spatial public goods game with pool punishment,  Phys. Rev. E 83 (2011) 036101.
  * Szolnoki and Chen [2018] A. Szolnoki, X. Chen,  Competition and partnership between conformity and payoff-based imitations in social dilemmas,  New J. Phys. 20 (2018) 093008.
  * Szolnoki and Perc [2015] A. Szolnoki, M. Perc,  Reentrant phase transitions and defensive alliances in social dilemmas with informed strategies,  EPL 110 (2015) 38003\.
  * Szolnoki and Perc [2016] A. Szolnoki, M. Perc,  Competition of tolerant strategies in the spatial public goods game,  New J. Phys. 18 (2016) 083021.
  * Szolnoki and Danku [2018] A. Szolnoki, Z. Danku,  Dynamic-sensitive cooperation in the presence of multiple strategy updating rules,  Physica A 511 (2018) 371–377.
  * Juul et al. [2012] J. Juul, K. Sneppen, J. Mathiesen,  Clonal selection prevents tragedy of the commons when neighbors compete in a rock-paper-scissors game,  Phys. Rev. E 85 (2012) 061924.
  * Tainaka [1993] K. Tainaka,  Paradoxial effect in a three-candidate voter model,  Phys. Lett. A 176 (1993) 303–306.


# Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and
Language Models

Muhammad Maaz*, Hanoona Rasheed*, Salman Khan, Fahad Shahbaz Khan
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Mohamed bin Zayed University of AI

###### Abstract

Conversation agents fueled by Large Language Models (LLMs) are providing a new
way to interact with visual data. While there have been initial attempts for
image-based conversation models, this work addresses the underexplored field
of _video-based conversation_ by introducing Video-ChatGPT. It is a multimodal
model that merges a video-adapted visual encoder with a LLM. The resulting
model is capable of understanding and generating detailed conversations about
videos. We introduce a new dataset of 100,000 video-instruction pairs used to
train Video-ChatGPT acquired via manual and semi-automated pipeline that is
easily scalable and robust to label noise. We also develop a quantitative
evaluation framework for video-based dialogue models to objectively analyse
the strengths and weaknesses of video-based dialogue models. Our code, models,
instruction set and demo are released at https://github.com/mbzuai-oryx/Video-
ChatGPT. 00footnotetext: *Equally contributing first authors

##  1 Introduction

The surge of deep learning applications for video understanding has lead to
major advancements in video-related tasks. However, the current video
understanding models are still unable to hold an open-ended conversation about
the video content in a coherent manner. A video-based dialogue model can
revolutionize video search, surveillance operations and help summarize key
events and abnormal event detection. Above all, it can provide a unified
human-understandable interface to video-related tasks such as action
recognition, localization, detection, segmentation, retrieval, and tracking.
Further, such a capability is of great interest as it will demonstrate the
model’s ability to encode temporal and spatial cues, contextual relationships
and long-term dependencies.

Recent advancements in multimodal understanding are largely based on the
combination of pretrained _image_ models with Large Language Models (LLMs) but
generally do not consider video inputs [1, 2, 3, 4, 5]. It is therefore
interesting to leverage the vast capabilities of LLMs for video understanding
tasks in a way that would not only maintain the temporal and spatial
characteristics but also be adept at generating human-like conversations about
videos. In this paper, we introduce Video-ChatGPT, a novel multimodal model
that merges the representational abilities of a pretrained visual encoder and
the generative powers of an LLM, capable of understanding and conversing about
videos.

Video-ChatGPT leverages an adapted LLM [1] that integrates the visual encoder
of CLIP [6] with Vicuna [7] as a language decoder, fine-tuned on generated
instructional image-text pairs. Our approach further adapts the desgin for
spatiotemporal video modeling and fine-tunes the model on video-instruction
data to capture temporal dynamics and frame-to-frame consistency relationships
available in video data. In contrast to other concurrent works for video-based
conversation [8, 9], Video-ChatGPT excels at temporal understanding, spatial
consistency and contextual comprehension as demonstrated by our extensive
evaluations.

A fundamental contribution of this work is the creation of a dataset of
100,000 video-instruction pairs using a combination of human-assisted and
semi-automatic annotation methods. Each pair consists of a video and its
associated instruction in the form of a question-answer. This provides Video-
ChatGPT with a large and diverse dataset to learn from, increasing its video-
specific understanding, attention to temporal relationships and conversation
capabilities.

Moreover, we introduce the first quantitative video conversation evaluation
framework for benchmarking, allowing for a more accurate evaluation of the
performance of video conversation models. This framework evaluates models on a
variety of capabilities, such as correctness of information, detail
orientation, contextual understanding, temporal understanding, and
consistency.

The contributions of this work are as follows,

  * •

We propose Video-ChatGPT, a video conversation model capable of generating
meaningful conversations about videos. It combines the capabilities of LLMs
with a pretrained visual encoder adapted for spatiotemporal video
representations.

  * •

We introduce 100,000 high-quality video instruction pairs together with a
novel annotation framework that is scalable and generates a diverse range of
video-specific instruction sets.

  * •

We develop the first quantitative video conversation evaluation framework for
benchmarking video conversation models. We demonstrate Video-ChatGPT to
perform well compared to concurrent conversational engines for videos such as
Video Chat [8].

##  2 Related Work

Vision Language Models: Significant advancements in the field of computer
vision have recently been observed due to the development of many foundational
vision-language models. These models represent a significant leap towards
creating general-purpose vision models capable of tackling various tasks
simultaneously [6, 10, 11, 12]. A prime example is CLIP [6], which is trained
on 400M image-text pairs and has demonstrated impressive zero-shot performance
on numerous benchmarks. It has been employed in various downstream
applications, from image-based object detection and segmentation [13, 14] to
3D applications [15, 16]. Numerous attempts have also been made to adapt CLIP
for video applications [17, 16]. Similar to our design, ViFi-CLIP [18]
suggests employing temporal pooling across video frames to adapt the image-
based CLIP model for video-based tasks.

Large Language Models: The field of natural language processing has witnessed
a paradigm shift with the advent of pretrained Large Language Models (LLMs)
such as GPT [19], LLaMA [20], OPT [21], and MOSS [22]. These models exhibit
extraordinary abilities like language generation and in-context learning, and
their knack for understanding intricate tasks given user prompts in a zero-
shot manner reflects their impressive adaptability and generalization. The
proven capabilities of LLMs have encouraged researchers to fine-tune them to
maximize their proficiency.

A key strategy in this pursuit is instruction tuning. This approach focuses on
improving the model’s alignment with user intentions and optimizing their
output quality. For instance, InstructGPT [23] and ChatGPT [24] significantly
benefit from this technique, showcasing improvements in diverse conversational
interaction capabilities and their aptitude to answer a broad range of complex
questions. This effective approach has recently been employed in open-source
models like Alpaca [25] and Vicuna [7], both developed using the LLaMA [20]
framework, resulting in performance improvements.

Pre-trained LLMs in Vision-Language Tasks: The recent strides in multimodal
understanding have primarily been driven by the integration of image-based
vision models with LLMs. Seminal contributions such as Flamingo [10] and
BLIP-2 [4] have demonstrated the power of utilizing web-scale image-text data,
as well as pioneering techniques in cross-modal alignment, to exhibit dynamic
abilities in conversational and few-shot learning contexts. Building on this
foundation, MiniGPT-4 [2] allows image-based conversations by integrating
BLIP-2 and Vicuna for zero-shot image comprehension.

Equally significant is the emergence of LLaVA [1], a model derived from the
LLaMa architecture, leveraging GPT-4’s language proficiency to generate
multimodal instruction-following data. With instruction tuning applied on the
derived data, LLaVA has displayed interesting multimodal chat capability,
hinting at the scalability potential of such a methodology. In addition,
InstructBLIP [5] model has demonstrated strong image-based dialogue
capabilities via vision-language instruction tuning by innovating with
instruction-aware visual feature extraction.

More closely related to our work, VideoChat [8] employs selective components
of video foundational models [26] and image foundation models [4], and
integrates them with LLMs [7] in conjunction with few learnable layers, tuned
using a two-stage lightweight training. Additionally, they construct a video-
specific dataset using off-the-shelf vision-language models [27, 4, 28, 26]
for generating noisy detailed textual descriptions to enhance the training of
video-centric conversational models.

Different from VideoChat, we propose a novel human assisted and semi-automatic
annotation framework for generation high quality instruction data for videos
(see Sec. 4). Our simple and scalable architecture design utilizes pretrained
CLIP [6] to generate spatiotemporal features which help Video-ChatGPT in
generating meaningful video conversation. Further, we are the first to propose
quantitative framework for evaluating video conversation tasks (see Sec. 4).

Figure 1: Architecture of Video-ChatGPT. Video-ChatGPT leverages the CLIP-L/14
visual encoder to extract both spatial and temporal video features. This is
accomplished by averaging frame-level features across temporal and spatial
dimensions respectively. The computed spatiotemporal features are then fed
into a learnable linear layer, which projects them into the LLMs input space.
In our approach, we utilize the Vicuna-v1.1 model, comprised of 7B parameters,
and initialize it with weights from LLaVA [1].

##  3 Video-ChatGPT

Video-ChatGPT is a large vision-language model that aligns video
representations with a Large Language Model (LLM), thus enhancing its ability
to generate meaningful conversation about videos. Our approach draws from the
approach employed in designing vision-language (VL) models for the video
domain. Given the limited availability of video-caption pairs and the
substantial resources required for training on such data from scratch, these
models commonly adapt pretrained image-based VL models for video tasks [16,
17, 18]. We adopt a similar approach, starting with the Language-aligned Large
Vision Assistant (LLaVA)[1] as our foundation.

LLaVA is a LMM that integrates the visual encoder of CLIP [6] with the Vicuna
language decoder [7] and is fine-tuned end-to-end on generated instructional
vision-language data. We fine-tune this model using our video-instruction
data, adapting it for video conversation task. The video-instruction data is
obtained as a combination of manual and automated pipelines in our proposed
instruction generation setup. This adaptation on video-specific instructions
allows for accommodating additional temporal dynamics, frame-to-frame
consistency, and long-range relationships present in video data. As a result,
our Video-ChatGPT excels in video reasoning, creativity, and understanding of
spatial, temporal, and action-oriented components within videos.

###  3.1 Architecture

We use CLIP ViT-L/14, which is pretrained using large-scale visual instruction
tuning in LLaVa, as the visual encoder. However, LLaVa visual encoder is meant
for images, which we modify to capture spatiotemporal representations in
videos. Given a video sample $V_{i}\in\mathbb{R}^{T\times H\times W\times C}$
with $T$ frames, the visual encoder generates temporal and spatial features.
The visual encoder encodes the $T$ frames independently as a batch of images
and produces frame-level embeddings $x_{i}\in\mathbb{R}^{T\times h\times
w\times D}$, where $h=H/p,w=W/p$. Here p is the patch size (_i.e._ 14 for
ViT-L/14), and we represent the number of tokens as $N$, where $N=h\times w$.
Frame-level embeddings are average-pooled along the temporal dimension to
obtain a video-level temporal representation $t_{i}\in\mathbb{R}^{N\times D}$.
This operation, referred to as temporal pooling, implicitly incorporates
temporal learning through the aggregation of multiple frames. Similarly, the
frame-level embeddings are average-pooled along the spatial dimension to yield
the video-level spatial representation $z_{i}\in\mathbb{R}^{T\times D}$. The
temporal and spatial features are concatenated to obtain the video-level
features $v_{i}$,

$v_{i}=[t_{i}\quad z_{i}]\in\mathbb{R}^{(T+N)\times D}.$ (1)

A simple trainable linear layer $g$, projects these video-level features into
the language decoder’s embedding space, transforming them into corresponding
language embedding tokens $Q_{v}$,

$Q_{v}=g(v_{i})\in\mathbb{R}^{(T+N)\times K}.$ (2)

Note that the function $g$ acts as an adapter and can be implemented with more
complicated architectures as well. However, we opt for a simplistic design
that gives competitive performance compared to more sophisticated choices in
our experiments. The text queries are tokenized to the same dimensions,
$Q_{t}\in\mathbb{R}^{L\times K}$. Here $L$ represents the length of text
query. Finally, $Q_{v}$ is concatenated with $Q_{t}$ and input to the language
decoder.

###  3.2 Video Instruction Tuning

We employ instruction-tuning of the LLM on the prediction tokens, utilizing
its original auto-regressive training objective. The pretrained model is
finetuned with curated, high-quality video-text pairs. During the finetuning
phase, we use predefined prompts based on the following template:

USER: <Instruction> <Vid-tokens> Assistant:

Using the notations, we can represent it as,

USER: <$Q_{t}$> <$Q_{v}$> Assistant:

In this prompt, the <Instruction> represents a question pertaining to the
video, randomly sampled from the training set of video-question-answer pairs.
Questions can be general, asking to describe the video, or they may relate to
specific temporal, spatial, or creative aspects of the video content. The
prediction answer <Answer> corresponds to the specific question asked.
Throughout the training, the weights for both the video encoder and LLM remain
frozen, and the model maximizes the likelihood of predicting tokens
representing the answer by adapting the linear layer. Consequently, the video
features $Q_{v}$ become aligned with the pre-trained LLM word embeddings,
equipping Video-ChatGPT with the ability to produce more natural and
dependable responses.

Figure 2: Examples of data enrichment via human-assisted annotation. Human
annotators augment video descriptions from video-caption datasets. The
captions are enriched by integrating detailed information regarding spatial
and temporal aspects, relationships, reasoning, scene descriptions, and the
chronological sequence of events.

##  4 Video Instruction Data Generation

In this section, we discuss our data-focused approach, which uses both human-
assisted and semi-automatic annotation methods to generate high-quality video
instruction data. This data is crucial for training Video-ChatGPT, making sure
the model gives accurate and meaningful responses. Our data collection
involves two key methods. The human-assisted annotation, involves expert
annotators analysing video content and providing detailed descriptions. This
process generates data rich in context and detail, which helps our model
understand complex aspects of video content. On the other hand, the semi-
automatic annotation framework is more cost-effective and scalable. Leveraging
state-of-the-art vision-language models, this method generates broad, high-
volume annotations, thus increasing the quantity of data without compromising
the quality substantially. Through these combined methods, we have
successfully accumulated a robust set of 100,000 video-instructional pairs.
This extensive dataset is crucial in fine-tuning our model to comprehend video
content effectively, integrating both spatial and temporal cues into its
understanding.

Our instructional data is both diverse and comprehensive, incorporating a wide
range of data types. These include detailed descriptions, summarizations,
question-answer pairs, tasks that stimulate creativity or generation of new
ideas, and conversational tasks. The data spans a broad spectrum of concepts,
ranging from visual appearance and temporal relations to complex reasoning
tasks and beyond, providing a diverse training ground for our model to learn
from.

###  4.1 Human-assisted Annotation

In this process, we leverage datasets containing video-caption pairs and
utilize the expertise of human annotators to enrich the original ground truth
annotations. Specifically, we use a subset of the ActivityNet-200 [29] dataset
which provides concise ground truth descriptions of various activities in
distinct video segments.

The annotators further enrich the captions by adding comprehensive information
about physical appearances and spatial and temporal localization, among other
critical contextual details. Figure 2 shows an example of how a ground truth
caption is enriched using human-assisted annotation.

Figure 3: Examples of data enrichment using our proposed semi-automatic
annotation. We employ off-the-shelf dense prediction and captioning models to
augment video descriptions. The BLIP-v2 [4] model generates frame-level
captions, while GRIT [27] is utilized for dense frame captions. Tag2Text [28]
model generates tags for each key-frame, aiding in eliminating noise. All
additional context elements are combined with the video captions and undergo a
GPT-assisted post-processing stage, generating the final detailed description.

###  4.2 Semi-automatic Annotation Framework

In addition to the rich human-assisted annotations, we also harness the
capabilities of advanced dense image vision-language models, developing a
semi-automatic annotation framework. This approach is cost-effective and
scalable, thereby increasing the quantity of data without substantially
compromising the quality.

Similar to the human-assisted process, this framework also leverages datasets
containing video-caption pairs. We enrich these datasets using contextual
information drawn from off-the-shelf dense prediction and captioning image-
based vision-language models. These models provide predictions that deliver
additional contextual information, thereby enriching the video captions. We
crafted developed a comprehensive method that combines these predictions, and
utilize specific models for the purpose of eliminating noisy or irrelevant
context from the data. This ensures that the data maintains its accuracy and
relevance.

Building on the use of off-the-shelf models, we apply pretrained models like
BLIP-2[4] and GRiT [27] for key-frame analysis in the videos. The BLIP-2
image-captioning model generates frame-level captions, while the GRiT dense
captioning model provides detailed captions for scene objects. Additionally,
the pretrained Tag2Text [28] model is used to generate tags for each key-frame
of the video. Despite their utility, these models can introduce noise into the
data.

To ensure high-quality data and mitigate noise, we implement three key steps.
First, we maintain a high prediction threshold for all off-the-shelf models to
uphold accuracy. Second, we employ a specialized filtering mechanism that
removes any frame-level caption from BLIP-2 or GRiT not matching with the
Tag2Text frame-level tags. This process involves extracting words from the
frame-level captions that are within the predefined Tag2Text tags vocabulary,
and eliminating any captions that contain words not in the tags for a given
frame. This strategy acts as an additional filtering layer, enriches the
captions by integrating predictions from multiple models.

In the third step, we merge frame-level captions and use the GPT-3.5 model to
generate a singular, coherent video-level caption. This step augments the
original ground truth caption with context from these models. We also direct
GPT-3.5 to discard inconsistent information across frames, ensuring a precise,
contextually rich video instruction dataset. Figure 3 illustrates how a ground
truth caption is enriched using this process after all three refinement
stages.

###  4.3 GPT-Assisted Postprocessing

Lastly, we implement a GPT-Assisted Postprocessing mechanism that refines and
optimizes the enriched annotations, in order to generate high-quality video
instructional data. We prompt GPT-3.5 model to create question-answer pairs
from the enriched and detailed captions that cover a wide variety of aspects.
These aspects include detailed descriptions, summarizations, question-answer
pairs, tasks that stimulate creativity or the generation of new ideas, and
conversational tasks.

Each of these elements plays a crucial role in our data-centric approach. Our
ultimate goal is to create a video-based conversation model that is accurate,
capable of understanding video content from both spatial and temporal cues,
and adept at engaging in conversations.

##  5 Experiments

###  5.1 Implementation Details

We use LLaVA as our baseline model and finetune it on 100K video instruction
pairs. We only update the linear layer projecting the video features to the
LLMs’ input space, while the rest of the architecture is kept frozen. We
finetune the model for 3 epochs using a learning rate of 2$e^{-5}$ and an
overall batch size of 32. The training of our 7B model took around 3 hours on
8 A100 40GB GPUs. During inference, for memory efficiency, we load the models
in FP16 mode.

In our semi-automatic annotation framework, we use Katna [30] to extract the
video key-frames. For the off-the-shelf Tag2Text [28] model, we use the Swin-B
version with input size of 384$\times$384 and confidence threshold of 0.7. For
GRIT [27], we use ViT-B version with CenterNet2 [31].

###  5.2 Quantitative evaluation

In this section, we highlight a key contribution of our work: the quantitative
evaluation of Video-ChatGPT using advanced metrics and comparative evaluations
with existing state-of-the-art models. We conduct two types of quantitative
evaluations: i) Video-based Generative Performance Benchmarking and ii) Zero-
Shot Question-Answer Evaluation.

Evaluation Aspect | Video Chat | Video-ChatGPT
---|---|---
Correctness of Information | 2.25 | 2.50
Detail Orientation | 2.50 | 2.57
Contextual Understanding | 2.54 | 2.69
Temporal Understanding | 1.98 | 2.16
Consistency | 1.84 | 2.20

Table 1: Performance benchmarking of text generation models. An in-depth
comparative analysis of Video-ChatGPT and Video Chat [8] across five key
evaluation aspects we propose in our benchmark. Video-ChatGPT shows competent
performance across all key aspects.

Video-based Text Generation Performance Benchmarking:  We introduce a
benchmark to evaluate the text generation performance of video-based
conversation models. To do this, we curate a test set based on the
ActivityNet-200 dataset [29], featuring videos with rich, dense descriptive
captions and associated question-answer pairs from human annotations. We also
develop an evaluation pipeline using the GPT-3.5 model. This pipeline assesses
various capabilities of the model and assigns a relative score to the
generated predictions on a scale of 1-5, in the following five aspects:

  1. (i)

Correctness of Information: We verify the accuracy of the generated text,
ensuring it aligns with the video content and doesn’t misinterpret or
misinform.

  2. (ii)

Detail Orientation: We evaluate the depth of the model’s responses, looking
for both completeness, meaning the model’s response covers all major points
from the video, and specificity, denoting the inclusion of specific details
rather than just generic points in the model’s response.

  3. (iii)

Contextual Understanding: We assess the model’s understanding of the video’s
context, checking if its responses aligns with the overall context of the
video content.

  4. (iv)

Temporal Understanding: We examine the model’s grasp of the temporal sequence
of events in the video when answering questions.

  5. (v)

Consistency: We evaluate the model’s consistency across different but similar
questions or different sections of the video.

We present the results of the evaluation of our proposed model, Video-ChatGPT,
using the quantitative benchmarking framework in Table 1. The results reveal
its competent performance across all key aspects when compared with the
recently introduced contemporary video conversation model, Video Chat [8].
Video-ChatGPT shows good performance, largely due to the instruction tuning we
perform and its straightforward architecture that leverages LLMs with a
pretrained visual encoder fine-tuned for video data. This provides it with the
robust ability to generate contextually relevant, detailed, and temporally
accurate text from video input.

Model | MSVD-QA | MSRVTT-QA | TGIF-QA | Activity Net-QA
---|---|---|---|---
| Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy | Score
FrozenBiLM | 32.2 | – | 16.8 | – | 41.0 | – | 24.7 | –
Video Chat | 56.3 | 2.8 | 45.0 | 2.5 | 34.4 | 2.3 | 26.5 | 2.2
Video-ChatGPT | 64.9 | 3.3 | 49.3 | 2.8 | 51.4 | 3.0 | 35.2 | 2.7

Table 2: Zeroshot question-answering comparison of Video-ChatGPT with other
video generative models. Video-ChatGPT performs competitively across all
datasets.

Zero-Shot Question-Answer Evaluation:  We conducted a comprehensive
quantitative evaluation using several commonly used open-ended question-answer
datasets: MSRVTT-QA [32], MSVD-QA [32], TGIF-QA FrameQA [33], and ActivityNet-
QA [34]. These evaluations were carried out in a zero-shot manner, employing
GPT-assisted evaluation to assess the model’s capabilities. This evaluation
process measures the accuracy of the model’s generated predictions and assigns
a relative score on a scale of 1-5.

To benchmark Video-ChatGPT, we compared its performance with other significant
models, such as FrozenBiLM [35] and the generative video model, Video Chat.
FrozenBiLM is a model that adapts frozen bidirectional language models
pretrained on Web-scale text-only data to multi-modal inputs, showing
promising results in zero-shot VideoQA settings. Despite the solid foundation
established by these models, Video-ChatGPT consistently outperformed them,
achieving state-of-the-art (SOTA) performance across all datasets. These
results indicate Video-ChatGPT’s ability to understand video content and
generate accurate, contextually rich answers to questions.

###  5.3 Qualitative Evaluation

We performed an extensive evaluation of our model on a variety of open-ended
video question-answering tasks, utilizing diverse videos sourced from
ActivityNet and YouTube. The evaluation tasks included video reasoning (Figure
4), creative and generative tasks (see Figure 5), spatial understanding
(Figure 6), action recognition (Figure 7), video conversation (Figure 8),
question answering (Figure 9) and temporal understanding (Figure 10). Our
model demonstrates proficiency in comprehending the content of the videos and
generating accurate responses across multiple video based task. Our model can
effectively understand the visual information present in the videos and
provide precise answers (see Figures 4, 5, 6, 7, 8, 9 and 10).

Figure 4: Video Reasoning Task. This figure illustrates an example from Video-
ChatGPT’s demonstration showcasing its performance in video reasoning tasks.
Figure 5: Creative and generative tasks. Illustrative examples from Video-
ChatGPT’s demonstration highlighting its performance in video-based creative
and generative tasks, such as crafting a story, poem, or advertisement. Figure
6: Spatial understanding tasks. The figure depicts examples from Video-
ChatGPT’s demonstration, emphasizing its capability in video-based spatial
understanding tasks, including identifying renowned locations or counting the
number of objects in a scene. Figure 7: Actiong Recognition Task. This figure
illustrates examples from Video-ChatGPT’s demonstration showcasing its
performance in video action recognition tasks such as playing drums and
grooming horse. Figure 8: Video Understanding and Conversation Tasks. This
figure illustrates examples from Video-ChatGPT’s demonstration showcasing its
performance in video understanding and conversation tasks. Figure 9: Question-
Answering Task. The figure depicts examples Video-ChatGPT’s demonstration
showcasing its performance in question-answering tasks. Figure 10: Temporal
Understanding Task. The figure provides examples from Video-ChatGPT’s
demonstration, highlighting its performance in temporal understanding tasks,
particularly in comprehending sequences of events.

##  6 Conclusion and Future Directions

In this work, we presented Video-ChatGPT, a multimodal model that merges a
pretrained visual encoder with a large language model (LLM) to enable video
understanding and conversations based on videos. Video-ChatGPT leverages an
adapter on top of pretrained LLM and vision backbones and is fine-tuned on
video-instruction data to capture temporal dynamics and spatial consistency
relationships in spatiotemporal sequences. A dataset of 100,000 video-
instruction pairs is created to enhance Video-ChatGPT’s video-specific
understanding and conversation capabilities. The work also introduced a
quantitative video conversation evaluation framework for benchmarking,
evaluating models on a diverse set of capabilities including conventional
video question answering as well as open-ended descriptions. While the model
performs competitively in several scenarios, we note it finds it challenging
to understand subtle temporal relationships and the visual details of small
objects. As a future work, Video-ChatGPT can be extended to simultaneously
deal with multiple modalities and to enhance its video comprehension
capabilities towards an all-in-one dialogue agent for universal visual content
understanding.

## Acknowledgements

We would like to thank colleagues for their contribution to the video
annotation task, including Abdelrahman Shaker, Shahina Kunhimon, Muhammad
Uzair, Sanoojan Baliah, Malitha Gunawardhana, Akhtar Munir, Vishal Thengane,
Vignagajan Vigneswaran, Jiale Cao, Nian Liu, Muhammad Ali, Gayal Kurrupu, Roba
Al Majzoub, Jameel Hassan, Hanan Ghani, Muzammal Naseer, Akshay Dudhane, Jean
Lahoud, Awais Rauf, Sahal Shaji, Bokang Jia.

## References

  * [1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [2] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.
  * [3] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [4] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * [6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, 2021.
  * [7] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [8] Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355, 2023.
  * [9] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355, 2023.
  * [10] Jean-Baptiste Alayrac et al.  Flamingo: a visual language model for few-shot learning.  2022\.
  * [11] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem.  Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [12] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang.  Class-agnostic object detection with multi-modal transformer.  In The European Conference on Computer Vision. Springer, 2022.
  * [13] Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  In Advances in Neural Information Processing Systems, 2022.
  * [14] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [15] David Rozenberszki, Or Litany, and Angela Dai.  Language-grounded indoor 3d semantic segmentation in the wild.  In The European Conference on Computer Vision, 2022.
  * [16] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  In The European Conference on Computer Vision, 2022.
  * [17] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  Actionclip: A new paradigm for video action recognition.  arXiv preprint arXiv:2109.08472, 2021.
  * [18] Hanoona Rasheed, Muhammad Uzair khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Finetuned clip models are efficient video learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [19] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [20] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [21] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.
  * [22] OpenLMLab.  Moss: Codebase for moss project.  An open-sourced plugin-augmented conversational language model, https://github.com/OpenLMLab/MOSS, 2023.
  * [23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [24] OpenAI.  Chatgpt.  Large Language Model for human style conversation https://chat.openai.com, 2023.
  * [25] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [26] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al.  Internvideo: General video foundation models via generative and discriminative learning.  arXiv preprint arXiv:2212.03191, 2022.
  * [27] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang.  Grit: A generative region-to-text transformer for object understanding.  arXiv preprint arXiv:2212.00280, 2022.
  * [28] Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang.  Tag2text: Guiding vision-language model via image tagging.  arXiv preprint arXiv:2303.05657, 2023.
  * [29] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015.
  * [30] KeplerLab.  Katna: Tool for automating video keyframe extraction, video compression, image autocrop and smart image resize tasks.  https://github.com/keplerlab/katna, 2019.
  * [31] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  In arXiv preprint arXiv:2103.07461, 2021.
  * [32] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017.
  * [33] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017.
  * [34] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019.
  * [35] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Zero-shot video question answering via frozen bidirectional language models.  arXiv preprint arXiv:2206.08155, 2022.