# Faith in the Algorithm, Part 1:
Beyond the Turing Test

Marko A. Rodriguez Theoretical Division – Center for Non-Linear Studies, Los
Alamos National Laboratory, email<EMAIL_ADDRESS>for Embedded Networked
Sensing, University of California, Los Angeles, email<EMAIL_ADDRESS>Alberto
Pepe Center for Embedded Networked Sensing, University of California, Los
Angeles, email<EMAIL_ADDRESS>
###### Abstract

Since the Turing test was first proposed by Alan Turing in 1950, the primary
goal of artificial intelligence has been predicated on the ability for
computers to imitate human behavior. However, the majority of uses for the
computer can be said to fall outside the domain of human abilities and it is
exactly outside of this domain where computers have demonstrated their
greatest contribution to intelligence. Another goal for artificial
intelligence is one that is not predicated on human mimicry, but instead, on
human amplification. This article surveys various systems that contribute to
the advancement of human and social intelligence.

> The alleged short-cut to knowledge, which is faith, is only a short-circuit
> destroying the mind.
>
> – Ayn Rand, “For the New Intellectual”

##  1 INTRODUCTION

The path towards artificial intelligence, in terms of mimicking human
cognitive functionality, has been long, difficult, and painfully incremental.
Bottom-up, state of the art vision systems have only accomplished modeling the
functional capabilities of the V1, V2, and V4 regions of the visual cortex
[36]. Popular, top-down knowledge representation and reasoning system are
still primarily monotonic [28], are only beginning to incorporate and
understand the ramifications of common sense knowledge [30], and are
predicated on logics that do not appear to model the true “rules” of human
thought [41]. Moreover, these object recognition and knowledge representation
and reasoning developments are but the fringe of a huge landscape of cognitive
faculties that must be simulated to achieve human-type artificial intelligence
in its fullest form. For example, other less developed agendas are object
relation learning in neurally-plausible substrates [23], novel logic
acquisition through experience [42], and associative mechanisms for merging
the categorizations from different sensory modalities into a single language
of thought [15, 19].

The sub-symbolic agenda of artificial intelligence attempts to model the
lowest common denominator of the human neural system in order to achieve
higher levels of intelligence through experience and learning. Modeling the
processing capabilities of individual neurons has been the aim of the
connectionist agenda for nearly three decades [35] and beyond various advances
in classification, it appears that human type intelligence is still many more
decades away. In the area of symbolic artificial intelligence, there have been
many developments utilizing computers to solve very specific problems very
well, but unfortunately, many of these systems do not have the general,
flexible intelligence enjoyed by humans. These statements serve not to
criticize the researchers or their methods; rather, they are presented in
order to acknowledge the level of difficulty involved in simulating human-type
intelligence and the distances that need to be reached if this goal is to be
achieved. Is it possible that computers, and their underlying foundation in
bivalent logic, centralized processing, and disembodiment, are blinding us as
architects and engineers by biasing our approach [9]? Of course, this does not
mean that it is impossible to model human intelligence on a computer (assuming
that such intelligence can be modeled on a Turing complete system). Instead,
it is more a statement that the Turing test [39] – the test for computer
intelligence by means of human mimicry – is not a “natural” test of the
computer’s abilities in the area of intelligence. Moreover, human mimicry is
not a “natural” application of the computer’s abilities.

There are many tests that are used to quantify human intelligence.
Interestingly, in the mean, a human subject’s scores in all of these tests
have a positive correlation. Thus, regardless if a specialist is testing a
subject’s ability to manipulate objects in 3D space or the subject’s fluency
with language, success in one of these tests is a predictor of success in
another. This finding points to a single factor that can account for
intelligence. This factor is known as the $g$-factor (or general intelligence
factor) [38]. However, any test for intelligence ultimately makes assumptions
about the sense modalities through which the test will be administered as well
as assumptions about the cultural and common knowledge of the subject. A major
trend in intelligence test research is to make intelligence tests devoid of
any cultural biases and one day, it may be possible to yield tests that are
devoid of any species and modality biases. Species agnostic intelligence tests
could be used to measure the intelligence exposed at the level of the
human/computer as the autonomous, intelligent entity. Moreover, the degree of
intelligence may be greater than what is possible given the human or computer
alone [10]. This is because the computer demonstrates unmatched skills in very
specific areas such as quickly computing the distance between large vectors of
numbers or in maintaining a lossless representation of a presented image in
memory. Such skills and their relationship and integration with the skills of
the human will continue to yield an advanced degree of real-world
intelligence. It is the central thesis of this article that this contribution
to intelligence appears to be a more “natural” fit for the computer. This
article reviews various systems that, when in combination with humans, yield
advanced intelligence – an intelligence that is different than that which can
be exposed by the Turing test.

##  2 HUMAN AND SOCIAL AUGMENTATION

Computers – the machines and their implemented algorithms – should not simply
be interpreted as technological embodiments of solutions to specific problems.
There is a larger relationship between the human, their problems and
requirements, and designed algorithms and their executing hardware. They are
solving larger problems than either the human or the computer could solve
alone; in other words, the computer is a contributing component within a
larger intelligent system [21]. Sherry Turkle discusses the relationship
between humans and computers as not just one in which the computer is a tool
used to accomplish human tasks, but one where it is a component that works
within the human’s everyday life as a supporting entity [40]. From a “society
of minds” perspective [29], the computer, as a cognitive component in human
thinking, is very much a well functioning digital information processor much
like the hippocampus is a well functioning neural memory device. In other
words, the computer has found, not in any affective directed way, an
information processing niche that further augments the human much like any
other component of the human neural system [37]. To say whether the
hippocampus is intelligent or not is to determine whether the results of its
processing affect intelligent behavior; that is, does the human know where
they are in physical space and do they encode episodic memories correctly? As
an autonomous entity, the hippocampus, would appear, to the external human
observer, as not being intelligent at all. For one, in isolation, it simply
becomes infected and its cells quickly die. However, within the larger schema
of the human organism, its role is of great significance to human
intelligence. A few minutes interaction with the patient H.M. makes this point
obvious [11]. Next, looking at the striate cortex demonstrates a relatively
simple system [22] that implements a relatively simple algorithm (albeit on a
massive scale) [36]; however, when integrated within the nervous system as a
whole, the contribution of the striate cortex to the overall intelligence of
the human is immense. Without it, vision, and its associated functionalities,
would not be possible. For instance, there would be no notion of a genius
painter and the level of intelligence that such a connotation denotes. To this
end, how many neural components are required before it is assumed that a human
is intelligent? A review of the life and times of Helen Keller should
demonstrate how vacuous this question is [26]. Also, like the neural component
within the larger system of the human, any other processing component can be
utilized in this contribution to intelligence. As such, the measurement of
intelligence need not be considered as testing that which is within the
confines of the human skin.

The relationship between the human and the computer in a technologically-
driven society unveils a natural symbiosis which is reminiscent of Hutchins’
theory of distributed cognition [24] and to the notions of collective
intelligence found in ant and termite populations [17, 7]. Some of the tasks
in which computers are employed in everyday life – from information access to
social interaction – make this symbiosis evident. In many respects,
traditional, standardized tests of human intelligence test the emergent
behavior of the coordinated activity of the individual’s various brain
regions. Introducing the computer into this system simply augments or extends
the intelligent capabilities of the individual human. It is no accident that
this symbiosis has emerged. The computer and its associated algorithms is a
needed augmentation to the human given the number of options available in the
technologically-rich world and the difficulties in finding one’s global optima
within it. Moreover, society, in a collaborative fashion amongst its
constituents and its supporting digital infrastructure, is making and will
continue to make advances in the area of social intelligence. In this light,
the question at hand is: what is the computer’s contribution to intelligence?
In order to address this question, the following section explores the
emergence of advanced individual and social intelligence within the scope of
the technological innovation that has most contributed to this type of
augmentation in recent times: the World Wide Web.

##  3 EMERGENT WEB INTELLIGENCE

Since the dawn of the World Wide Web, information has been codified and
distributed within a shared, universal medium that is accessible by human
users world wide. The World Wide Web is unique for two reasons: distribution
and standardization. In many respects, the first can not be accomplished
without the latter. The Web’s most eminent standard, the Uniform Resource
Identifier (URI) has made it possible for the Web to serve as a network of
information, from the document to the datum – a shared, global data structure
[3]. This distributed data structure is amplifying the intelligence of the
individual human and may provide a greater social intelligence. The remainder
of this section will address the amplification of intelligence in the context
of three general Web system: search engines (index and ranking),
recommendation engines (personalized recommendations), and governance engines
(collective decision making) [43].

###  3.1 Search Engines

The World Wide Web has emerged as a massive information repository from which
humans contribute and consume information. This has not only provided a simple
means of retrieving information, but also a simple way to publish and
distribute information, thus leading to the increase in human information
production. However, information increase inevitably brings about
discoverability issues, as the necessity to locate and filter desired
information arises. To deal with this problem, algorithms have been developed
to augment the individual’s search capabilities. Interestingly, this
augmentation is currently predicated on the contribution of many individuals
within the stigmergetic environment of the World Wide Web.

The early Web maintained rudimentary indexes in the form of Web “yellow pages”
that provided short descriptions of web pages. With the explosive growth of
the Web, such directory services fell by the wayside as no human operator (or
operators) could keep up with the amount of information being published, nor
could such rudimentary lists provide the end user a representation of the
quality of web pages. By a nearly-Darwinian selection process, these early
forms of indexes fell out of use because they were built around a conceptual
framework that did not take advantage of the distributed representation of
value inherent in every linking webpage made explicit by their authors. As a
remedy to this situation, a commercialized Web industry was born and continues
to thrive around solving the problem of search. Search engines index massive
amounts of data that are gleaned from Web servers world wide. The development
of the simple mechanism of ranking web pages by means of their eigenvector
component within the web citation graph has proved the most successful to date
[8]. It is remarkable that this mechanism is predicated on humans’ decisions
to link webpages; that is, the algorithm leverages human interaction with the
Web and vice versa in a symbiotic manner. Even more remarkable is the fact
that with the approximately 30 billion web pages in existence today, Web users
can rest assured that, for the most part, their keyword search will provide
the answer to their question within the first few results returned. This level
of speed and accuracy of knowledge acquisition was not possible prior to the
development of the Web, mainly because the problem of massive-scale indexing
and ranking did not make itself apparent until the Web. This problem is solved
through the unification of the human’s ability to, in a decentralized fashion,
denote the value (or quality) of web pages and the computer’s ability to
calculate a global rank over these explicit expressions of value.

In this scenario, the Web plays the role of a digital Rolodex providing the
human, nearly instantly, a reference to further information on nearly any
topic imaginable [14]. Prior to the written document, information was passed
from generation to generation in the form of large memorized stories and
poems. In the contemporary technologically-rich world, this “algorithm”
(cultural process) is no longer necessary. This is not to say that an
individual can no longer memorize a long poem if they wish. It is more that a
new algorithm has emerged to handle this information indexing requirement and
as such, cognitive resources can be appropriated to other tasks. However, the
Web is not a large story or poem: it follows no plot, no linear sequence, no
poetic meter, no single language – the list of characters is beyond count and
no one writing style can be identified. For these reasons, it is posited that
no currently existing neural component can memorize, index, and rank the
entire Web, and thus, a specialized intelligence is required and, fortunately,
has emerged.

###  3.2 Recommendation Engines

Large-scale human generated data sets have opened a terrain for numerous
algorithms that support individual decision making. Such data sets include the
implicit valuation of resources that users leave on the web as they click from
web page to web page or from purchased item to purchased item. No individual
ever sees the entire Web and for the most part, for the life of the
individual, they are confined to a small subset of the greater Web. However,
the aggregation of this click-stream information from all individuals provides
a collectively generated representation of the inherent relationship between
all items on the Web. This collective digital footprint provides not only
novel ways to rank resources [5] but also, novel ways to recommend resources
[6]. Finally, humans are also developing rich profiles of themselves that
include not only identifiable facts such as one’s curriculum vitae, but also
the more qualitative aspects of their personality, tastes, and ever changing
mood. There are many systems that take advantage of such data sets such as the
recommendation engine. A recommendation engine can be defined as any algorithm
that provides users with resources (e.g. documents, books, music, movies, life
partners, etc.) that are more likely than not to be correlated to the users’
current requirements.

The popular collaborative filtering algorithms of document and music services
are able to utilize the previous click behavior of an individual to
systematically compare it with the click behaviors of others, and from this
comparison, recommend a set of resources that will be of interest to the user
[20]. For many, the dependency on the librarian and the record shop owner has
shifted to a dependence on the community as a whole that is leaving this
massive digital footprint.

An interesting phenomena to arise in recent years is the development and use
of online dating services. In any large city, there are too many individuals
for any one human to sift through. Moreover, even if an individual were able
to meet everyone, the abilities of the individual may not be keen enough to
predict, with any great accuracy, whether or not the person they are meeting
will make an optimal partner. For this reason, dating services have emerged to
handle, or rather attempt to handle, this common, pervasive problem. Ignoring
broader social and cultural considerations for a moment, from a purely
statistical perspective, the human’s trial and error methods of sampling small
portions of the population through friends or in social, physical environments
(bars, restaurants, cafes, etc.) can not compete with the success rates of
modern day matchmaking algorithms [2]. Note that matchmaking services are not
confined solely to the Web. Newspapers provide “personals” sections, but like
the early “yellow pages” of the Web, they can not maintain rich profiles, nor
does manually browsing this information compare with the success of a
matchmaking algorithm’s recommendation. Again, for those activities for which
a human simply does not have the skills to succeed, the human relies on an
external augmentation to fulfill the intelligence requirements of the problem
at hand.

Recommendation services are following a common trend: they are all building
more sophisticated models of both humans and resources. The World Wide Web
infrastructure has provided the avenues for humans to collectively aggregate
in a shared virtual space. Unfortunately, for the most part, the traffic data
that is being generated as individuals move from site to site, the profiles
that individuals repeatedly create at every online service, and the metadata
about the resources that these services index are isolated within the data
repositories of the services that utilize this information directly.
Fortunately, recent developments in an open data model known as the “web of
data” may change this by unifying the information contained in service
repositories and exposing, within the shared, global URI address space, every
minutia of data [4]. The end benefit of this shift in the perception of
ownership and exposure of data will allow for a new generation of algorithms
that take advantage of an even richer world model [27, 32]. Such models will
include a seamless integration of the individual’s reading, listening, dating,
working, etc. behaviors as well as the descriptions of books, songs, movies,
people, jobs, etc. At this point, to the algorithms that leverage such data, a
human is no longer just a consumer of a particular type of literature or a
connoisseur of a particular style of film, but rather, a complex entity that
can be subtly oriented, through recommendation, in a direction that ensures
that they are experiencing that aspect of the world that is most fitting to
who they are.

At the extreme of this line of thought, if enough information is gathered and
a rich enough world model is generated, then it may be possible to design
algorithms that are more fit to determine the life course of an individual
human than what the individual, their family, or their community can do for
them. This assumes appropriate feedback from the world to the model [16],
which may include the perspectives of the individual, their family, and their
community. This view suggests that it may be best to rely on a large-scale
world model (and algorithms that can efficiently process it) when making
decisions about one’s path in life. Such algorithms can take into account the
multitude of relations between humans and resources, and improvise a well
“thought out” plan of action that ensures that the individual, to the best of
the system’s ability, lives a life that is filled with optimal experiences.
This is a life in which the others they meet, the restaurants they frequent,
the books they read, the classes they attend, and so forth lead to experiences
that are completely fulfilling to them as a human. These optimal experiences
represent the perfect balance between the psychological states of anxiety and
boredom and as such, would increase the individuals’ attentiveness and
involvement in such activities – similar to the mental state that is
colloquially known as “flow” [12]. Moreover, this state of human experience
has been articulated since the times of Aristotle and his notion of the
eudaemonic living which arises when one consistently chooses correctly in
their life [1].

A large-scale world model has the potential to integrate the collective
zeitgeist of a society, the socio-demographic and geographic layouts of
cities, the location of its inhabitants, their personal characteristics, their
resources and relations. Amazingly, such data currently exist in one form or
another, to varying degrees of accuracy, completeness, and levels of access.
Further making this information publicly available and integrated would allow
for algorithms to evolve, over iterations of development and insight, that are
fit to determine the individuals’ global optima.

###  3.3 Governance Engines

In many ways, aiding the human in finding global optima is the purpose of a
society (within the constraints of taking into account the optima of others)
[31]. From high-level governmental decisions to the local cultural rules that
determine the way in which humans interact in their environment, the goal of a
(benevolent) society is to ensure a life in “the pursuit of happiness” [25].
However, can a society be structured such that the individuals need not
pursue, but instead be guaranteed a life full of happiness – or eudaemonia and
optimal experiences? The question is then: what are the limits of individual
intelligence that can be achieved by the current societal structures alone?
And also: are there more efficient and accurate algorithms that can be
utilized? Recommendation systems are a step in the direction towards the use
of computers to provide the human the right resource at the right time,
regardless of what form that resource may take. However, within the grander
scheme of society as a whole, the nascent fields of e-governance and
computational social choice theory are only beginning to tangentially touch
upon the idea that a networked computer infrastructure could be used to foster
a new structure for government that is optimized for societal-scale problem-
solving.

Reflecting on modern voting mechanisms (specifically those within the United
States), we find a system that is fragile, inaccurate, and expensive to
maintain. Due in part to the outdated infrastructure that citizens use to
communicate with their governing body, citizen participation in government
decision making is limited. However, these days, with the level of eduction
that citizens have, the amount of information that citizens can become aware
of, and the sophistication of modern network technologies, it is possible that
current government decisions are limited in that they are not leveraging the
full potential of an enlightened population (or subset thereof). By making use
of both a large-scale and knowledgeable decision making constituency, it is
theoretically possible that all rendered decisions are optimal. This statement
was validated (under certain simple assumptions) in 1785 by Marquis de
Condorcet’s now famous Condorcet jury theorem [13].

With the social networks that are being made explicit on the Web today, and
with open standard movements that ensure that this information can be shared
across services, it is possible to leverage a relatively simple vote
distribution mechanism to remove the representative layers of government and
promote full citizen participation in all the decision making affairs of a
society. This mechanism, known as dynamically distributed democracy, ensures
that any actively participating subset of a population simulates the decision
making behavior of the whole [33]. Thus, a simulated, large-scale decision
making body can be leveraged in all decisions. A large decision making body is
the first requirement of the Condorcet jury theorem. Robin Hanson articulates
a vision of government where any individual can participate through a decision
system known as a prediction market [18]. The purpose of a prediction market
is to provide accurate predictions of objectively determinable states of the
world (current or into the future) and its application to governance is noted
in the popular phrase “vote on values, but bet on beliefs.” In this form, the
self-selecting, monetary mechanisms that determine whether someone
participates is based on their degree of knowledge of the problem space. Those
that are not knowledgeable, either do not participate or lose money in the
process of participating, thus, hampering the individual from participating in
matters outside the scope of their abilities into the future. The accuracy of
such systems are astounding and have popular uses in election predictions and
a short lived run in terrorist predictions (only to be dismantled by the U.S.
government because it was considered too morose for market traders to
monetarily benefit on the accurate prediction of the death of others). A
knowledgeable decision making body is the second requirement of the Condorcet
jury theorem and, much like commodity markets, prediction market systems
select for knowledgeable individuals.

These ideas stress the importance of reflecting on the medium by which society
organizes itself, generates its laws, and implements methods in how it will
utilize resources most effectively. Like the “yellow pages” of the early Web,
it may not be optimal to leave such pressing matters to an operator (or
operators). This statement is not a critique of the leaders and doctrines of
nations, but instead is a comment on the complexity of the world and the
necessity for a new type of intelligence. It is posed as an appeal to rethink
government and its role within contemporary networked society [34]. An
implementation of a government should not be valued. Instead, what should be
valued is the ideals that that implementation is trying to achieve. Moreover,
if another implementation would better meet the ideals of the society, then it
should be enacted. A distributed value/belief system and algorithmic
aggregation mechanism may prove to be the better problem-solving mechanism for
societal issues and may prove to be a better mechanism to orchestrate
individual lives. It is in this area that computers can greatly contribute to
social intelligence, where the unification of the intelligence augmentation
gained by the individual human and the society coalesce into a type of
intelligence that is novel (beyond human mimicry) and above all beneficial.

##  4 CONCLUSION

Humans perceive their world through their sense modalities, create stable
representations of the consistent patterns in the world, and utilize those
representations to further act and survive to the best of their abilities.
Their internal, subjective world is an endless stream of thoughts – a complex,
information-rich map of the external world. Manifestations of intelligence
inherently depend upon an individual’s internal representation of the external
world and their ability to manipulate that representation. By analogy to the
field of computer science, this internal map of the world can be regarded as
the data structure upon which reasoning mechanisms (i.e. algorithms) function.
From an objective perspective, the human mind can only maintain so rich a data
structure, process only so many aspects of it, and simulate only so many
potential future paths for the individual to choose from. The complexity of
the human’s mental calculation grows when considering that many other such
simulations are occurring in the minds of their fellow men and women. Like a
general-purpose processor, to simulate a machine within a machine reduces the
resources available to the original machine to execute other processes. For
these reasons, the human is not a perfectly intelligent creature always doing
the right thing at the right time.

As discussed, with the externalization of the human’s internal world through
the explicit expression of themselves, their relation to others, and the
resources on which they rely, other processes can utilize this explicit model
to aid the human in the process of thought and thus, life. The World Wide Web
and the algorithms implemented upon it function like an auxiliary mind,
exposed to more information than could be possibly processed by its neural
counterpart. While the core specification of these algorithms may be
understood, even thoroughly by their designers, ultimately what machines
compute are based on such a large-scale model of the world, that to assimilate
its results into one’s choices are ultimately based on faith – much like the
faith one has in the validity of their episodic memories and their current
location in space as provided to them by their hippocampus.

## References

  * [1] Aristotle, The Nicomachean Ethics, Oxford University Press, 1998.
  * [2] Aaron Ben-Ze’ev, Love Online: Emotions on the Internet, Cambridge University Press, 2004.
  * [3] Tim Berners-Lee and James A. Hendler, ‘Publishing on the Semantic Web’, Nature, 410(6832), 1023–1024, (April 2001).
  * [4] Christian Bizer, Tom Heath, Kingsley Idehen, and Tim Berners-Lee, ‘Linked data on the web’, in Proceedings of the International World Wide Web Conference, Linked Data Workshop, Beijing, China, (April 2008).
  * [5] Johan Bollen, Herbert Van de Sompel, and Marko A. Rodriguez, ‘Towards usage-based impact metrics: first results from the MESUR project.’, in Proceedings of the Joint Conference on Digital Libraries, pp. 231–240, New York, NY, (2008). ACM Press.
  * [6] Johan Bollen, Michael L. Nelson, Gary Geisler, and Raquel Araujo, ‘Usage derived recommendations for a video digital library’, Journal of Network and Computer Applications, 30(3), 1059–1083, (2007).
  * [7] Eric Bonabeau, Marco Dorigo, and Guy Theraulaz, Swarm Intelligence: From Natural to Artificial Systems, Oxford University Press, New York, NY, 1999.
  * [8] Sergey Brin and Lawrence Page, ‘The anatomy of a large-scale hypertextual web search engine’, Computer Networks and ISDN Systems, 30(1–7), 107–117, (1998).
  * [9] Andy Clark, Being There: Putting Brain, Body and World Together Again, MIT Press, 1997.
  * [10] Andy Clark, Supersizing the Mind: Embodiment, Action, and Cognitive Extension, Oxford University Press, 2008.
  * [11] Neal J. Cohen, Memory, Amnesia, and the Hippocampal System, MIT Press, September 1995.
  * [12] Mihály Csíkszentmihályi, Flow: The Psychology of Optimal Experience, Harper and Row, New York, NY, 1990.
  * [13] Marquis de Condorcet.  Essai sur l’application de l’analyse á la probabilité des décisions rendues á la pluralité des voix, 1785.
  * [14] Douglas C. Engelbart, Computer-supported cooperative work: a book of readings, chapter A conceptual framework for the augmentation of man’s intellect, 35–65, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1988\.
  * [15] Jerry Fodor, The Language of Thought, Harvard University Press, 1975.
  * [16] Vadas Gintautas and Alfred W. Hübler, ‘Experimental evidence for mixed reality states in an interreality system’, Physical Review E, 75, 057201, (2007).
  * [17] P. Grasse, ‘La reconstruction du nid et les coordinations inter-individuelles chez bellicositermes natalis et cubitermes sp. la theorie de la stigmergie’, Insectes Sociaux, 6, 41–83, (1959).
  * [18] Robin Hanson, ‘Shall we vote on values, but bet on beliefs?’, Journal of Political Philosophy, (in press).
  * [19] Jeff Hawkins and Sandra Blakeslee, On Intelligence, Holt, 2005.
  * [20] Johnathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl, ‘Evaluating collaborative filtering recommender systems’, ACM Transactions on Information Systems, 22(1), 5–53, (2004).
  * [21] Francis Heylighen, ‘The global superorganism: an evolutionary-cybernetic model of the emerging network society’, Social Evolution and History, 6(1), 58–119, (2007).
  * [22] D. H. Hubel and T. N. Wiesel, ‘Receptive fields and functional architecture of monkey striate cortex.’, Journal of Physiology, 195(1), 215–243, (March 1968).
  * [23] J.E. Hummel and K.J. Holyoak, ‘A symbolic-connectionist theory of relational inference and generalization’, Psychological Review, 110(2), 220–264, (2003).
  * [24] Edwin Hutchins, Cognition in the Wild, MIT Press, September 1995.
  * [25] Thomas Jefferson.  Declaration of independence, 1776.
  * [26] Helen Keller, The Story of My Life, Doubleday, Page and Company, New York, NY, 1905.
  * [27] Lawrence Lessig, Free Culture: The Nature and Future of Creativity, CreateSpace, Paramount, CA, 2008.
  * [28] Deborah L. McGuinness and Frank van Harmelen.  OWL web ontology language overview, February 2004.
  * [29] Marvin Minsky, The Society of Mind, Simon and Schuster, March 1988.
  * [30] Erik T. Mueller, Commonsense Reasoning, Morgan Kaufmann, January 2006.
  * [31] David L. Norton, Democracy and Moral Development: A Politics of Virtue, University of California Press, 1995.
  * [32] Marko A. Rodriguez, ‘A distributed process infrastructure for a distributed data structure’, Semantic Web and Information Systems Bulletin, (2008).
  * [33] Marko A. Rodriguez and Daniel J. Steinbock, ‘A social network for societal-scale decision-making systems’, in Proceedingss of the North American Association for Computational Social and Organizational Science Conference, Pittsburgh, PA, (2004).
  * [34] Marko A. Rodriguez and Jennifer H. Watkins, ‘Revisiting the age of enlightenment from a collective decision making systems perspective’, Technical Report LA-UR-09-00324, Los Alamos National Laboratory, (January 2009).
  * [35] David E. Rumelhart and James L. McClelland, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, July 1993.
  * [36] Thomas Serre, Aude Oliva, and Tomaso Poggio, ‘A feedforward architecture accounts for rapid categorization’, Proceedings of the National Academy of Science, 104(15), 6424–6429, (April 2007).
  * [37] Peter Skagestad, ‘Thinking with machines: Intelligence augmentation, evolutionary epistemology, and semiotic’, Journal of Social and Evolutionary Systems, 16(2), 157–180, (1993).
  * [38] Charles Spearman, ‘General intelligence objectively determined and measured’, American Journal of Psychology, 15, 201–293, (1904).
  * [39] Alan M. Turing, ‘Computing machinery and intelligence’, Mind, 58(236), 433–460, (1950).
  * [40] Sherry Turkle, The Second Self: Computers and the Human Spirit, MIT Press, 1984.
  * [41] Pei Wang, ‘Cognitive logic versus mathematical logic’, in Proceedings of the Third International Seminar on Logic and Cognition, (May 2004).
  * [42] Pei Wang, Rigid Flexibility, Springer, 2006.
  * [43] Jennifer H. Watkins and Marko A. Rodriguez, Evolution of the Web in Artificial Intelligence Environments, chapter A Survey of Web-Based Collective Decision Making Systems, 245–279, Studies in Computational Intelligence, Springer-Verlag, Berlin, DE, 2008.


# Zero Touch Coordinated UAV Network Formation for 360∘ Views of a Moving
Ground Target in Remote VR Applications

Yuhui Wang and Junaid Farooq
Department of Electrical and Computer Engineering,
University of Michigan-Dearborn, Dearborn, MI 48128 USA,
Emails: {ywangdq<EMAIL_ADDRESS>
###### Abstract

Unmanned aerial vehicles (UAVs) with on-board cameras are widely used for
remote surveillance and video capturing applications. In remote virtual
reality (VR) applications, multiple UAVs can be used to capture different
partially overlapping angles of the ground target, which can be stitched
together to provide 360∘ views. This requires coordinated formation of UAVs
that is adaptive to movements of the ground target. In this paper, we propose
a joint UAV formation and tracking framework to capture 360∘ angles of the
target. The proposed framework uses a zero touch approach for automated and
adaptive reconfiguration of multiple UAVs in a coordinated manner without the
need for human intervention. This is suited to both military and civilian
applications. Simulation results demonstrate the convergence and configuration
of the UAVs with arbitrary initial locations and orientations. The performance
has been tested for various number of UAVs and different mobility patterns of
the ground target.

###### Index Terms:

unmanned aerial vehicles, connectivity, virtual reality, distributed
algorithm.

##  I Introduction

The virtual reality (VR) paradigm allows remote viewers to interact with a
virtual environment in an immersive fashion [1], and has been successfully
applied in various domains such as remote monitoring [2], and target tracking
[3]. In application scenarios like virtual tours, unmanned aerial vehicles
(UAVs) can be feasible platforms to deliver high fidelity real-time VR
experiences for live viewers since they can capture videos of live scenery and
transmit them to base stations (BSs) that service the VR users over wireless
networks [4]. In order to capture and transmit 360∘ views of a moving target
on the ground, multiple UAVs with on-board cameras are required to be placed
in the air such that partially overlapping images can be taken from all
angles. Moreover, the UAVs need to maintain their relative positions alongside
following the target when it moves.

The key objectives in creating such a formation using a swarm of UAVs are to
(i) autonomously organize and place UAVs in a surround fashion without
centralized planning and decision-making, (ii) continuously estimate the
position of the ground target and track its motion (i.e., maintain a fixed
distance with the target), and (iii) center the target in the camera field-of-
view (FOV) of all UAVs in real-time. These objectives are intertwined and a
holistic approach needs to be developed to achieve the desired results. When
360∘ views of a moving target need to be captured and transmitted
continuously, the estimation of the position and velocity of a moving object
is challenging because the initial states of the object may be unknown, and
measurements of the dynamics can fail when the object is occluded or moves out
of view. Several approaches have been proposed for estimating the position or
velocity of the target based on fixed camera [5], sensor networks [6] and
radar [7]. Compared with traditional fixed-position platforms, UAVs have
several advantages such as having a wide FOV, high mobility, and adaptability.
However, to center the target in camera view, the design of the controller
involves both the motion of the target and the UAV. If multiple UAVs are
deployed, the swarm behaviors need to be adapted such that UAVs can captured
images of the target from different perspectives. For effective stitching on
multiple image views, the object need to be centered in each camera’s point of
view and cameras can capture images from diverse perspectives.

Figure 1: System architecture of a multi-UAV network formation for tracking
and capturing 360 views of a dynamic target on the ground.

In this work, we design and implement a UAV-based tracking algorithm inspired
by the swarming interactions [8, 9] to enhance formation resilience and track
the target from multiple perspectives. An image-based visual servo (IBVS) [10]
controller mechanism is adapted through the feedback dynamics of the UAVs to
enable the target to be centered in all camera views. The framework is based
on a zero touch philosophy, where the UAVs are able to configure themselves
autonomously without human intervention and continuously adapt to achieve the
desired group objective. The main contributions of the proposed framework are
twofold: (i) a UAV motion controller based on the relative motion between UAVs
and the target is designed to achieve the real-time tracking of the dynamic
target, and (ii) a distributed and coordinated UAV formation and control
algorithm is integrated to capture 360∘ views of the target.

The rest of the paper is organized as follows: Section II provides an overview
of the related works in literature, Section III describes the UAV kinematic
model and camera model, Section IV details the proposed method for UAV
formation control and object centering control, Section V provides the
simulation results, and Section VI concludes the paper.

##  II Related Work

Several works in literature focus on estimation and tracking of moving targets
using cameras mounted on UAVs [11, 12, 13, 14, 15, 16, 17, 18]. In [11, 12],
the author proposed estimation and tracking algorithms of a moving target
based on UAV cameras. They realized the challenges that motion pattern of the
target is unknown and solve them through the use of unscented Kalman filter
(UKF) [19]. In their experimental results, a UAV with a mounted camera was
implemented to capture image of the target and analyze the tracking path
planning and energy consumption. But their proposed methods did not consider
multiple UAV network formation and only worked in situations of single UAV
tracking. The application can not provide full-view perspectives of the target
and may lose track once the target is out-of-view of the UAV. Some researchers
utilize static optimization or machine learning approaches for multi-UAV
coordination [16, 17]. Through the construction of a UAV system using computer
vision algorithms like You Only Look Once (YOLO) [20] and placement methods
like reinforcement learning, their proposed methods can achieve both target
tracking and UAV network formation. Although these works have achieved good
performance in experimental results, they are limited by the centralized
control architecture or unable to provide real-time decision-making.

Our proposed method is based on a distributed and coordinated UAV formation
strategy which can create a uniform UAV network centered around the target.
The designed controller considers both the motion of UAVs and the target, and
utilizes images captured by the UAV-mounted camera to ensure that the target
is always centered in all of the camera views.

##  III System Model

Consider a set of UAVs $\mathcal{M}=\\{1,\ldots,M\\}$ tracking a dynamic
target $\mathcal{Q}$. We assume that all UAVs are connected to a cellular base
station for relaying the captured images and videos. Each UAV can communicate
with other UAVs that are within a distance $\gamma$. The set of neighboring
UAVs of a UAV $i$ is defined by $\mathcal{N}_{i}$. For convenience of
notation, we will drop the subscripts in the subsequent descriptions. For
modeling simplicity, we assume that cameras are mounted on the UAVs without
relative motion and we use a point model for the location of the UAVs and
cameras by considering them to be pairwise co-located. The position, velocity
and trajectory of the target is unknown but bounded and can be estimated by
using a unscented Kalman filter [19]. Fig. 1 shows the relationship between a
moving target and one of the cameras mounted on a UAV. The inertial coordinate
is denoted by G and the camera coordinate is denoted by C. All coordinates are
in inertial coordinate by default unless otherwise specified. The
transformation matrix between coordinate systems G and C is provided below:

$\rm{T_{G2C}}=\begin{bmatrix}\rm{R}_{G2C}&p_{c}\\\ 0&1\end{bmatrix},$ (1)

where the matrix $\rm{R}_{G2C}$ can be written as:

$\displaystyle\begin{array}[]{ll}&\rm{R_{G2C}}=\rm{R}_{x}\cdot\rm{R}_{y}\cdot\rm{R}_{z}\\\
&=\begin{bmatrix}1&0&0\\\ 0&\cos(\theta_{x})&\sin(\theta_{x})\\\
0&-\sin(\theta_{x})&\cos(\theta_{x})\end{bmatrix}\cdot\begin{bmatrix}\cos(\theta_{y})&0&-\sin(\theta_{y})\\\
0&1&0\\\ \sin(\theta_{y})&0&\cos(\theta_{y})\end{bmatrix}\cdot\\\ &\hskip
10.84006pt\begin{bmatrix}\cos(\theta_{z})&\sin(\theta_{z})&0\\\
-\sin(\theta_{z})&\cos(\theta_{z})&0\\\ 0&0&1\end{bmatrix},\end{array}$ (5)

The parameters $\theta_{x}$, $\theta_{y}$, and $\theta_{z}$ are the rotation
angles around $x$, $y$, $z$ axis respectively, $p_{c}=[x_{c},y_{c},z_{c}]^{T}$
is the Cartesian coordinates of the UAV and can be obtained using global
positioning system (GPS) measurements. The spatial velocity of the UAV is
denoted by $V_{c}=[v_{c},\omega_{c}]^{T}$, where
$v_{c},\omega_{c}\in\mathbb{R}^{3\times 1}$ are the linear and angular
velocity of the camera respectively. The location coordinates of the target
are denoted by $p_{q}=[x_{q},y_{q},z_{q}]^{T}$ and need to be estimated. The
spatial velocity of the target is denoted by $V_{q}=[v_{q},\omega_{q}]^{T}$,
where $v_{q},\omega_{q}\in\mathbb{R}^{3\times 1}$ are the linear and angular
velocity of the target respectively. We assume $\omega_{q}=[0,0,0]^{T}$ for
the target vehicle. The relative position of the camera and the target can be
computed as

$p_{q/c}=[X,Y,Z]^{T}=p_{q}-p_{c},$ (6)

Using Corioli’s theorem [21], the relative velocity between the camera and the
target can be written as

$\dot{p}_{q/c}=v_{q}-v_{c}-\omega_{c}\times p_{q/c},$ (7)

where $v_{c}=[v_{cx},v_{cy},v_{cz}]^{T}$ and
$v_{q}=[v_{qx},v_{qy},v_{qz}]^{T}$ are the linear velocity of camera and
target respectively.

###  III-A Kinematics of the UAV

As the measurement of positions of the target is through UAV-mounted cameras,
a kinematic model [15] based on the relative motion between the camera and the
target is used to describe the change of system states over time. In [11], the
system states are designed as
$X=[x_{1},x_{2},x_{3},x_{q},y_{q},z_{q},v_{qx},v_{qy},v_{qz}]^{T}$, where
$[x_{1},x_{2},x_{3}]^{T}=[\frac{X}{Z},\frac{Y}{Z},\frac{1}{Z}]^{T}$ is defined
to facilitate the matching with image pixel level coordinates. We assume the
velocity of the target can be estimated by methods such as an unscented Kalman
filter [19] and is hence known to the UAVs. The dynamics of system states are
given by

$\dot{X}=\begin{bmatrix}v_{qx}x_{3}-v_{qz}x_{1}x_{3}+\zeta_{1}+\eta_{1}\\\
v_{qy}x_{3}-v_{qz}x_{2}x_{3}+\zeta_{2}+\eta_{2}\\\
-v_{qz}x_{3}^{2}+v_{cz}x_{3}^{2}-(\omega_{cy}x_{1}-\omega_{cx}x_{2})x_{3}\\\
v_{q}\\\ 0\\\ 0\\\ 0\end{bmatrix},$ (8)

where $\zeta_{1},\zeta_{2},\eta_{1},\eta_{1}\in\mathbb{R}$ are defined by

$\begin{array}[]{cl}\zeta_{1}=&\omega_{cz}x_{2}-\omega_{cy}-\omega_{cy}x_{1}^{2}+\omega_{cx}x_{1}x_{2},\\\
\zeta_{2}=&-\omega_{cz}x_{1}+\omega_{cx}+\omega_{cx}x_{2}^{2}-\omega_{cy}x_{1}x_{2},\\\
\eta_{1}=&(v_{cz}x_{1}-v_{cx})x_{3},\\\
\eta_{2}=&(v_{cz}x_{2}-v_{cy})x_{3}.\end{array}$ (9)

###  III-B UAV-mounted Camera Model

Each UAV is equipped with an identical camera with focal lengths $f_{x}$ and
$f_{y}$ in pixel units. We assume that the relative position between the UAV
and the camera is fixed and objects can be detected using computer vision
algorithms like YOLO [20]. The center of the the bounding box in the image
plane can be determined and is denoted by $[u,v]^{T}$. As $[u,v]^{T}$ is also
the projection point of the bounding box center on the camera plane, the
relationship between system state and object position in camera view can be
modeled as [11]

$x_{1}=\frac{u-c_{u}}{f_{x}},$ (10) $x_{2}=\frac{v-c_{v}}{f_{y}},$ (11)

where $[c_{u},c_{v}]^{T}$ denote the center position of the camera view in
pixel units.

##  IV Methodology

In this section, we present the methodology used to develop the UAV motion
control. In order to achieve both the coordinated formation as well as
capturing fully centered views, two different controllers are adapted and
integrated.

###  IV-A Coordination UAV Formation Control

The predator-prey swarm[22] control is applied to maintain the autonomous and
resilient formation of the UAV network. This controller enables the UAVs to
form a centered formation around the target and keep the distance between UAVs
at a desired value. The interaction function is defined by

$u_{s}=\sum_{i\in\mathcal{N}}k\left(\frac{r_{j}-r_{i}}{\|r_{j}-r_{i}\|^{2}}-\frac{r_{j}-r_{i}}{d_{U}^{2}}\right),$
(12)

where $\mathcal{N}$ is the set of neighboring UAVs, $r_{j}$ is the position of
UAV $j$ in inertial coordinates, $k$ is positive scaling constant and $d_{U}$
is the desired distance between UAVs. The advantage of this swarm control is
that the distance based algorithm creates a UAV formation of a stable circle
centered around the target with constant density. When the number of UAVs
increase, the formation can automatically stratify into multiple layers and
provide views from all perspectives.

###  IV-B Object Centering Control

The controller is designed based on both the swarm control and the IBVS
control [10] which utilizes the camera images to achieve the tracking and
centering of a dynamic target. The control state is defined by
$s=[x_{1},x_{2},x_{3}]^{T}$ and the relationship between $\dot{s}$,$V_{q}$,
and $V_{c}$, is given by

$\dot{s}=L_{s}(V_{c}-V_{q}),$ (13)

where $L_{s}$ is the feature Jacobian matrix and is calculate by

$L_{s}=\begin{bmatrix}-x_{3}&0&x_{1}x_{3}&x_{1}x_{2}&-(x_{1}^{2}+1)&x_{2}\\\
0&-x_{3}&x_{2}x_{3}&x_{2}^{2}+1&-x_{1}x_{2}&-x_{1}\\\
0&0&x_{3}^{2}&x_{2}x_{3}&-x_{1}x_{3}&0\end{bmatrix}.$ (14)

The state error is defined as the state deviation from the desired state

$e=s-s^{*},$ (15)

where $s^{*}\in\mathbb{R}^{3}$ is the predefined desired state. The
relationship between between $\dot{e}$, $V_{q}$, and $V_{c}$ is given by

$\dot{e}=L_{e}(V_{c}-V_{q}),$ (16)

where $L_{e}$ is the interaction matrix. As $\dot{e}=\dot{s}$, it can be shown
that $L_{e}=L_{s}$ by comparing (13) and (16). In order to locate the target
at the center of camera view, its desired coordinates in camera plane are set
as $[c_{u},c_{v}]$. Then the desired state $s^{*}$ is calculated by

$s^{*}=\left[\frac{c_{u}-c_{u}}{f_{x}},\frac{c_{v}-c_{v}}{f_{y}},\frac{1}{d_{q}}\right]^{T}=\left[0,0,\frac{1}{d_{q}}\right],$
(17)

where $d_{q}$ is the desired distance between the camera and the target in
inertial coordinates. In order to ensure an exponential decoupled decrease of
error, we use an open-loop system

$\dot{e}=-\lambda e.$ (18)

Combining (16) and (18) results in the following feedback control law

$V_{c}=-\lambda L_{e}^{+}e+V_{q},$ (19)

where $L_{e}^{+}$ is the Moore-Penrose pseudo-inverse of $L_{e}$. With the
pred-prey swarm control input $u_{s}$ in (12), the final controller is
designed as

$V_{c}=-\lambda L_{e}^{+}e+V_{q}+u_{s}.$ (20)

###  IV-C Algorithm Design

Based on the controllers discussed above, we proposed a UAV based tracking
algorithm which provides 360∘ views of the target and enables distributed and
autonomous formation of the UAV network as shown in Algorithm 1. Assume that
the positions and velocities of UAVs are known, the positions and velocities
of the target can be estimated [19]. In every iteration, each UAV first
updates its system state $X$ using the state dynamics in (8). Then the
neighboring UAVs within the range $\gamma$ are identified. Each UAV shares its
position and velocity with neighbors and finally updates its position and
velocity using the discretized dynamics based on (20). Our proposed
distributed algorithm has linear time complexity of
$\mathcal{O}(\mathcal{N})$, where $\mathcal{N}$ is the set of neighboring
UAVs.

Algorithm 1 Coordinated Formation and Tracking Control

1:Initialize position $p_{q}$ and velocity $V_{q}$ for the target. Initialize
position $p_{c}$ and velocity $V_{c}$ for each UAV.

2:while not converged do

3: for each UAV do

4: Estimate the velocity $V_{q}$ and position $p_{q}$ of the

5: target.

6: Update system states $X$ using (8).

7: Determine the neighbors of the UAVs $\mathcal{N}$.

8: Each UAV share its position and velocity with its

9: neighbors.

10: Update position $p_{c}$ and velocity $V_{c}$ using
$\eqref{eq:update_vel}$.

11: end for

12:end while

Figure 2: System model diagram.

##  V Performance Evaluation

We conduct a series of simulations in MATLAB and measure the system
performance in terms of state error, speed, and view angles. For the
simplicity of implementation, we used a discretized system with the time step
for every iteration $\Delta t=0.1s$. The initial height of all UAVs are set to
$50$ m, the initial position of the target is set to $[0,0,0]^{T}$, and the
initial velocity of all UAVs are set to $[0,0,0]^{T}$. To test the tracking
capability of the system, the velocity of the target is set to a continuously
increasing trigonometric function with respect to time, i.e.,
$v_{q}(t)=[|t\sin(t)|,|t\cos(t)|,0]^{T}$. The parameters related to the system
model are consistent during the simulation and are listed in table I.

Parameter | Symbol | Value
---|---|---
Image Size |  | 640, 480 pixels
Image Center | $c_{u},c_{v}$ | 320.5, 240.5 pixels
Focal Length | $f_{x},f_{y}$ | 381.36, 381.36 pixels
Controller Parameters | $k,\lambda$ | 10.0, 1.0
Desired Distance | $d_{U},d_{q}$ | 200m, 100m
TABLE I: Parameter Settings

###  V-A Performance Metrics

To analyze the performance of the proposed framework, we define several
metrics as follows: The _normalized error_ in $x,y,z$ coordinates of camera
views is defined as $e_{x}=\frac{|u-c_{u}|}{c_{u}}$,
$e_{y}=\frac{|v-c_{v}|}{c_{v}}$, $e_{z}=\frac{\|p_{q/c}\|_{2}}{d_{q}}$, and
$e_{x},e_{y},e_{z}\in(0,1)$. The _error area_ is defined as
$e_{a}=e_{x}*e_{y}$ and it can be used to quantify the deviation of the object
from the image center. The _total effective view angles_ are defined to be the
sum of non-overlapping perspective angles covered by all the cameras mounted
on UAVs. The maximum view angle for each camera is set to 80 degrees for
physical limitation. The total view angles of the system are calculated by the
sum of view angles for all cameras and the effective view angles are the the
non-overlapping views of all cameras.

Figure 3: Top view of three UAVs tracking a moving target in inertial
coordinates. Black arrows show the direction of UAVs. Trajectories in the
first 200 iterations are shown. Figure 4: Camera views of three UAVs tracking
a moving target in camera coordinate. The target is centered after
convergence.

###  V-B Simulation Results

In the first simulation scenario, three UAVs were deployed to test the
tracking capability and stability of the model. Fig. 3 shows the top view
results of the trajectories in inertial coordinate and the target trajectory
in camera views were shown in Fig. 4. The initial positions of the UAVs were
set to $[300,0,50]^{T}$, $[0,50,50]^{T}$, and $[0,-100,50]^{T}$ with the
initial orientations
$[0,0,-\frac{2}{3}\pi]^{T},[0,0,-\frac{2}{3}\pi]^{T},[0,0,\frac{5}{6}\pi]^{T}$
in radians. Black arrows represented the orientation of UAVs. The goal of the
simulation was to ensure that the target was centered in camera views while
keeping the distance between the UAV and the target at $d_{q}=100$ m, and the
distance between UAVs at $d_{U}=200$ m. The area error rates over iterations
were shown in the Fig. 5 and the comparison between UAV speed and target speed
was shown in Fig. 6. It showed that in the first 20 iterations, as the error
$e_{x}$ in $x$ axis was large, the UAVs prioritized to adjust their positions
so that the target moved to the center in camera views. At the same time, the
swarm behavior controlled the UAVs to repel away from each other to maintain
the inter-UAV distance at $d_{U}$. After 60 iterations, the UAV network
converged to a stable formation and the UAVs could track the target and match
its speed.

Figure 5: Evolution of error area over time for the UAVs. After initial
glitches, the error diminishes over time. Figure 6: Comparison of the speed of
the target and three UAVs in the first 100 iterations. The speed of the UAVs
converge to the speed of the target after running for around 60 iterations.

In the second simulation, the performance of the system was tested as opposed
to the number of UAVs. The UAVs ran for 200 steps and formed a centered
formation around the target. Fig. 7 showed the normalized error
$e_{x},e_{y},e_{z}$ in $x,y,z$ axis respectively. The three curves
demonstrated different behaviors of the system. The $e_{x}$ curve approached
to the line $y=0$ because the $x$ coordinate of the target only depended on
the orientation of the UAV and UAVs can rotate without physical limitations.
As the number of UAVs increased, the $e_{y}$ curve approached to the line
$y=0$ and the $e_{z}$ curve rose with decreasing slope. This was because as
the number of UAVs increased, the radius of the formation also increased.
$e_{y}$ was negatively related to the radius while $e_{z}$ was positively
related to the radius. Fig. 8 showed the actually achieved and optimal maximum
view angles provided by the UAV network. The view angles were divide by $2\pi$
for normalization. The camera view angle of each UAV was 80 degrees. When the
number of UAVs was less than 3, there was no overlapping views and the UAV
formation was efficient. However, it cannot provide 360 degree seamless vision
of the target. After the number of UAVs increased to 6, the UAVs can provide
all-round views of the target. But there were overlapping views so the
efficiency decreased. The comparison of the total and non-overlapping view
angle curves revealed the trade-offs between efficiency and effectiveness.

Figure 7: The normalized error in camera coordinates against number of UAVs.
Figure 8: Comparisons between the desired/actual view angles and non-
overlapping views against the number of UAVs.

##  VI Conclusion

In this paper, we have proposed an object tracking algorithm on UAV platforms
with linear time complexity that can provide 360∘ views of a moving ground
target and keep the autonomous and resilient formation of the UAV network.
Both the positions and velocities of the target and UAVs are considered in the
design of system states. Inspired from the IBVS and the predator-prey swarming
approach, we designed an integrated system controller that minimize the
deviation of the target feature point from the camera image center, keep a
desired distance between the UAVs and the target, and achieve distributed
physical orchestration of the UAV network. Simulation results showed that the
target can be positioned to the center of all the camera views and UAVs can
interactively adjust their positions centered at the target. After running for
sufficient iterations, the errors converged to zero and the velocities of the
UAVs matches the velocity of the target. The system performance can be
observed to improve significantly as the number of UAVs increase. Our proposed
method can well balance the tracking efficiency and effectiveness and be
useful in constructing a zero touch coordinated UAV network, which can achieve
seamless construction of 360∘ views suited for wireless VR applications.

## References

  * [1] E. Bastug, M. Bennis, M. Medard, and M. Debbah, “Toward interconnected virtual reality: Opportunities, challenges, and enablers,” _IEEE Communications Magazine_ , vol. 55, no. 6, pp. 110–117, 2017.
  * [2] Y. Wu, C. Liu, S. Lan, and M. Yang, “Real-time 3D road scene based on virtual-real fusion method,” _IEEE Sensors Journal_ , vol. 15, no. 2, pp. 750–756, 2015.
  * [3] B. Li and Y. Wu, “Path planning for UAV ground target tracking via deep reinforcement learning,” _IEEE Access_ , vol. 8, pp. 29 064–29 074, 2020\.
  * [4] J. Chakareski, “UAV-IoT for next generation virtual reality,” _IEEE Transactions on Image Processing_ , vol. 28, no. 12, pp. 5977–5990, 2019.
  * [5] P. Dendorfer, A. Osep, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Roth, and L. Leal-Taixé, “Motchallenge: A benchmark for single-camera multiple target tracking,” _International Journal of Computer Vision_ , vol. 129, no. 4, pp. 845–881, 2021.
  * [6] M. Numan, F. Subhan, W. Z. Khan, S. Hakak, S. Haider, G. T. Reddy, A. Jolfaei, and M. Alazab, “A systematic review on clone node detection in static wireless sensor networks,” _IEEE Access_ , vol. 8, pp. 65 450–65 461, 2020\.
  * [7] W. Yi, Y. Yuan, R. Hoseinnezhad, and L. Kong, “Resource scheduling for distributed multi-target tracking in netted colocated MIMO radar systems,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 1602–1617, 2020.
  * [8] Y. Wang and J. Farooq, “Resilient UAV formation for coverage and connectivity of spatially dispersed users,” in _IEEE International Conference on Communications (ICC 2022)_ , Seoul, South Korea, May 2022.
  * [9] ——, “Proactive and resilient uav orchestration for qos driven connectivity and coverage of ground users,” in _2022 IEEE Conference on Communications and Network Security (CNS)_. IEEE, 2022.
  * [10] F. Chaumette and S. Hutchinson, “Visual servo control. i. basic approaches,” _IEEE Robotics & Automation Magazine_, vol. 13, no. 4, pp. 82–90, 2006\.
  * [11] J.-M. Li, C. W. Chen, and T.-H. Cheng, “Estimation and tracking of a moving target by unmanned aerial vehicles,” in _American Control Conference (ACC 2019)_ , 2019, pp. 3944–3949.
  * [12] X. Deng, J. Li, P. Guan, and L. Zhang, “Energy-efficient UAV-aided target tracking systems based on edge computing,” _IEEE Internet of Things Journal_ , 2021.
  * [13] P. Nousi, I. Mademlis, I. Karakostas, A. Tefas, and I. Pitas, “Embedded UAV real-time visual object detection and tracking,” in _IEEE International Conference on Real-time Computing and Robotics (RCAR 2019)_. IEEE, 2019, pp. 708–713.
  * [14] S. Wang, F. Jiang, B. Zhang, R. Ma, and Q. Hao, “Development of UAV-based target tracking and recognition systems,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 8, pp. 3409–3422, 2019.
  * [15] A. Parikh, T.-H. Cheng, and W. E. Dixon, “A switched systems approach to vision-based localization of a target with intermittent measurements,” in _American Control Conference (ACC 2015)_ , 2015, pp. 4443–4448.
  * [16] L. Zhou, S. Leng, Q. Liu, and Q. Wang, “Intelligent UAV swarm cooperation for multiple targets tracking,” _IEEE Internet of Things Journal_ , vol. 9, no. 1, pp. 743–754, 2021.
  * [17] Y. Ajmera and S. P. Singh, “Autonomous UAV-based target search, tracking and following using reinforcement learning and YOLOFlow,” in _2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_. IEEE, 2020, pp. 15–20.
  * [18] M. Chen, W. Saad, and C. Yin, “Deep learning for 360∘ content transmission in UAV-enabled virtual reality,” in _IEEE International Conference on Communications (ICC 2019)_ , Shanghai, China, May 2019.
  * [19] E. A. Wan and R. Van Der Merwe, “The unscented Kalman filter for nonlinear estimation,” in _Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373)_. Ieee, 2000, pp. 153–158.
  * [20] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [21] A. Persson, “How do we understand the Coriolis force?” _Bulletin of the American Meteorological Society_ , vol. 79, no. 7, pp. 1373 – 1386, 1998.
  * [22] Y. Chen and T. Kolokolnikov, “A minimal model of predator–swarm interactions,” _Journal of The Royal Society Interface_ , vol. 11, no. 94, p. 20131208, 2014.