# OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with
Task Divide-and-Conquer

Lu zhang1, Tiancheng Zhao1,2, Heting Ying1, Yibo Ma1, Kyusong Lee1,2

1Linker Technology Research, 2Binjiang Institute of Zhejiang University
{Lu_zhang, Heting_Ying<EMAIL_ADDRESS>{tianchez<EMAIL_ADDRESS>
###### Abstract

Recent advancements in Large Language Models (LLMs) have expanded their
capabilities to multimodal contexts, including comprehensive video
understanding. However, processing extensive videos such as 24-hour CCTV
footage or full-length films presents significant challenges due to the vast
data and processing demands. Traditional methods, like extracting key frames
or converting frames to text, often result in substantial information loss. To
address these shortcomings, we develop OmAgent, efficiently stores and
retrieves relevant video frames for specific queries, preserving the detailed
content of videos. Additionally, it features an Divide-and-Conquer Loop
capable of autonomous reasoning, dynamically invoking APIs and tools to
enhance query processing and accuracy. This approach ensures robust video
understanding, significantly reducing information loss. Experimental results
affirm OmAgent’s efficacy in handling various types of videos and complex
tasks. Moreover, we have endowed it with greater autonomy and a robust tool-
calling system, enabling it to accomplish even more intricate tasks. Code:
https://github.com/om-ai-lab/OmAgent

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with
Task Divide-and-Conquer

Lu zhang1, Tiancheng Zhao1,2, Heting Ying1, Yibo Ma1, Kyusong Lee1,2 1Linker
Technology Research, 2Binjiang Institute of Zhejiang University {Lu_zhang,
Heting_Ying<EMAIL_ADDRESS>{tianchez<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have advanced remarkably in recent years, greatly
expanding their capabilities across various applications Touvron et al.
(2023a, b); Bai et al. (2023); OpenAI (2023a). As these models have evolved,
they have been increasingly applied to multimodal contexts, allowing them to
process and interpret not just text but also images and other media types.
Initially, the focus was on single images, utilizing the models’ ability to
generate and understand detailed descriptions or responses based on static
visuals Awadalla et al. (2023); Liu et al. (2023, 2024); OpenAI (2023b)

However, as LLMs have become more complex and powerful, there has been growing
interest in applying them to more dynamic media, such as video content Lin et
al. (2023a); Zhu et al. (2023); Zhang et al. (2023); Zhao et al. (2023); Tang
et al. (2023). This interest arises from the potential to provide deeper and
more nuanced interpretations of video data, similar to how humans understand
and interact with moving images and sound. Currently, most video understanding
models are limited to processing short videos, typically only a few minutes or
even several seconds long. Despite these advancements, significant challenges
remain, especially in handling long video inputs like 24-hour CCTV footage or
full-length movies, which involve massive amounts of data and require
substantial processing power.

Traditionally, one solution has been to extract key frames from these long
videos or convert all frames into textual descriptions before processing Lin
et al. (2023b); Yang et al. (2023). While this approach makes the task more
manageable for LLMs, it often results in information loss. Key frame
extraction might miss subtle but important details in the omitted frames, and
converting visual data to text can oversimplify or misrepresent visual
nuances, leading to a less accurate understanding of the content.

To address these limitations, the application of Retrieval-Augmented
Generation (RAG) technology in video understanding has emerged as a promising
solution Chen et al. (2017); Liu (2022); Chase (2022); Arefeen et al. (2024).
RAG enables the storage and efficient retrieval of video frames based on their
relevance to a query. This method allows for more precise and contextual
responses by referencing specific content directly from the video, rather than
relying on potentially incomplete or inaccurate textual summaries. However,
given that video is a continuous stream of information, when it is stored,
this continuous flow is segmented into discrete blocks of data. The
information lost during this segmentation is irretrievable.

To address the aforementioned issues, we attempt to analyze the human approach
to handling complex, long video question-answering tasks, seeking
breakthroughs from this perspective. When a person watches a content-rich,
lengthy video, such as a movie, they retain a general impression of the movie
in their mind. This impression includes a rough outline of the video’s content
at various time points. When asked about specific details, the person may not
recall the details immediately but can quickly locate the relevant time point
in the video and rewatch the segment to retrieve the missing information. The
key insight of OmAgent is to replicate this process by integrating multimodal
RAG and generalist AI agent. OmAgent consists of two main components: (1) A
video2RAG video preprocessor to extract and store the generalized information
from the video, akin to the foundational impression a video imprints upon the
viewer’s memory. (2) A Divide-and-Conquer Loop (DnC Loop) for task planning
and execution which equipped with tool invocation capabilities.

We abstract the human ability to reposition and review video details as a tool
named "rewinder," which can be autonomously selected and utilized by an AI
agent, similar to how a person might use a video player’s progress bar to
navigate to points of interest. OmAgent not only can retrieve detailed
information from videos but also can actively seek external information,
enabling more advanced video understanding and question-answering. Existing
benchmarks are insufficient to accurately quantify these capabilities, so we
propose a new complex video understanding benchmark to fulfill this task. The
contributions are: (1) OmAgent, the first complex video understanding
framework integrating multimodal RAG and generalist AI agent. (2) A benchmark
dataset that contains 2000+ Q&A pairs for evaluating video understanding
systems. (3) Experiments that shows the proposed agentic method is able to
outperform strong baselines for solving complex video understanding problems.

##  2 Related Work

#### Video LLMs

Analyzing and understanding video content using large-scale language models
(LLMs) typically involves fine-tuning or pre-training methods. Pre-training
strategies, such as supervised or contrastive learning, develop video LLMs,
while instruction fine-tuning updates adapter parameters to enable video
comprehension Tang et al. (2023). For example, LaViLa Zhao et al. (2023)
enhances video subtitle generation through a cross-attention module and
rewriting mechanism, improving coverage and diversity. Video-LLaMA Zhang et
al. (2023) addresses spatio-temporal visual variations using separate video
and audio encoders with an advanced audio-visual Q-former, significantly
boosting video comprehension. Video-LLaVA Lin et al. (2023a) connects
multimodal representations into a unified semantic space with LLMs, improving
video understanding tasks.

However, these methods often consume a lot of computational resources and time
during the training process, and the models can usually only target specific
tasks related to the training data. In addition, video LLMs trained from
scratch may not be able to achieve the expected performance when dealing with
longer or previously unseen videos, showing shortcomings in understanding long
videos and dealing with complex video question-answer tasks.

#### Long Video Understanding system with LLMs

LLMs and Multimodal LLMs (MLLMs) applied to long video comprehension tasks
utilize external systems to process extensive content. This involves analyzing
visual elements, actions, scenes, and objects over time, aligning multimodal
information with textual modalities, and leveraging the powerful text
processing capabilities of MLLMs Tang et al. (2023). For instance, Vlog Kevin
(2023) uses pre-trained models for different modalities to record and
interpret visual and audio information, summarizing it into detailed text for
MLLM comprehension. MM-REACT Yang et al. (2023) employs visual expert tools
via internal prompts, enhancing MLLMs’ visual understanding. MM-VID Lin et al.
(2023b) segments videos using ASR and Scene Detection tools, generating and
integrating textual descriptions to complete Q&A tasks with MLLMs.

Compared to training entirely new video LLMs, these MLLM-based approaches
significantly reduce the need for computational resources while allowing the
system to integrate or update external tools according to new technical or
performance requirements. However, such long video understanding methodologies
will lose a large amount of video information when transforming modalities,
and do not fully utilize the multimodal processing capabilities of MLLM. In
addition, these systems usually lack sufficient autonomy to support more
complex video questioning and interaction, which limits their depth and
breadth in practical applications.

#### MultiModal RAG

MultiModal Retrieval Augmented Generation (RAG) leverages images, videos,
audio, and other non-text data for information retrieval, enhancing content
relevance and context for complex query and generation tasks Zhang et al.
(2018). The LlamaIndex Liu (2022) framework improves relevance and accuracy by
enabling quick retrieval and processing of multimodal content through precise
embedding and efficient indexing. Indexify TensorLakeAI (2024) provides a
robust framework for building multimodal RAG systems with real-time data
pipelines, extractor SDKs, and powerful storage interfaces for efficient
information extraction and indexing. iRAG Arefeen et al. (2024) uses AI model
selection for perceptual queries, improving the speed and quality of
multimodal data-to-text conversion, particularly for real-time, long video
understanding.

Although these multimodal RAG systems offer great advantages in integrating
multimodal information, they are still unable to eliminate the significant
loss of information that occurs when data is transformed from video to
knowledge in a RAG system. Our OmAgent, on the other hand, has no significant
loss of information thanks for the task planning and autonomous tool call
capability of DnC Loop especially the "rewinder" mechanism.

##  3 Method

Figure 1: How OmAgent understand video. In Video2RAG, the video is processed
by different algorithms (e.g. Scene Detection, ASR and face recognition) and
then summarized by MLLMs to generate Scene Captions. Those captions are
encoded and saved in the knowledge database. When OmAgent receives a query, it
filters and retrieves in knowledge database based on timestamps (if
available). The retrieved information is processed by the Divide-and-Conquer
Loop and summarized by Conclusive Synthesis to generate the final answer.

The process of OmAgent’s video understanding can be bifurcated into two
primary parts: Video2RAG and DnC Loop. As illustrated in Figure 1, all video
data must undergo preliminary processing before being stored in the knowledge
database in preparation for subsequent tasks. The preprocessing phase of
Video2RAG encompasses a series of model identification and vectorization
procedures, culminating in the extraction of the core content of the video
files for storage. When undertaking video understanding tasks, the initial
step is to extract temporal information from the query. This information will
then be used to filter the retrieved results. Subsequently, the query is
encoded by text encoder, and the embedding is employed to retrieve pertinent
video segment information from the knowledge database.

The retrieved video clip information and the original task will be transmitted
to DnC Loop, the intelligent agent capable of autonomously planning and
executing tasks, for processing. Complex tasks will be recursively subdivided
into executable subtasks. If at any point the agent deems that specific video
details need to be reviewed, it will utilize the rewind tool to examine the
relevant content. Once all subtasks are successfully completed, the execution
results will be conveyed to a node dedicated to synthesizing the final answer.

###  3.1 Video to RAG

OmAgent’s preprocessing (as shown in Figure 1) of video data is similar to a
multimodal RAG. This approach avoids treating the entire content of a very
long video as context input to the large language model, which would lead to
three serious issues: (1) The length of the context would limit the maximum
length of the video that can be processed. (2) Using an extremely long context
for each question and answer session would cause an explosive increase in
token usage. (3) An overly long context increases the difficulty of LLM
inference, affecting the accuracy of question and answer sessions. OmAgent’s
Video2RAG processing mainly consists of the following steps.

#### Scene Detection

Firstly, an algorithm is used to segment the video into relatively independent
video blocks. The main purpose of this step is to locate the key nodes of the
video. We can determine whether to segment the scene by assessing the degree
of change in the frames; overly short segments will be merged together. The
extracted video segments will have their start and end timestamps recorded,
and 10 frames will be uniformly sampled from every segment.

#### Visual Prompting

During the video preprocessing stage, additional algorithms can be used to
provide more information. For example, using facial recognition, we can obtain
information about the characters in the video. OmAgent will annotate this
algorithmic information directly on the images through visual prompting, i.e.,
drawing the corresponding recognition boxes and using text to explain above
the bounding box. This allows for the full utilization of the powerful
understanding capabilities of MLLMs.

#### Text Representation of Audio

The audio information in the video is as important as the visual information.
OmAgent uses ASR algorithms to convert the speech in the video into text and
employs speaker diarization algorithms to distinguish between different
speakers.

#### Scene Caption

Using MLLMs that support multiple images, each video segment’s content is
summarized. The inputs include video frames that have already been annotated
with visual prompting and the transcribed audio information. In the process of
generating dense captions at this step, We have delineated a set of pivotal
elements to guide the MLLM in generating effective and comprehensive captions,
ensuring that vital information is not overlooked in the absence of explicit
objectives. OmAgent specifies the following dimensions as instructions to
MLLMs:

  * •

The time information of the current video clip in terms of periods like
morning or evening, seasons like spring or autumn, or specific years and time
points.

  * •

Describe the location where the current event is taking place, including scene
details.

  * •

Provide a detailed description of the current characters, including their
names, relationships, and what they are doing, etc.

  * •

List all the detailed events in the video content in chronological order.

  * •

Give some detailed description of the scene of the video. This includes, but
is not limited to, scene information, textual information, character status
expressions, and events displayed in the video.

  * •

Provide an overall description and summary of the content of this video.

#### Encode and Save

The final step in video processing involves vectorizing the scene captions and
storing them in a vector database (knowledge database). Additionally, the
original text of the captions is also stored in the memory for keyword-based
retrieval. The start and end timestamps of the video segments are used as
filtering fields and are likewise stored in the memory repository of the
OmAgent agent.

###  3.2 Divide-and-Conquer Loop

In computer science, divide-and-conquer (DnC) is a highly classical algorithm
design paradigm. A divide-and-conquer approach entails the iterative
decomposition of a problem into multiple sub-problems. This process continues
until the sub-problems reach a level of simplicity that allows for direct
resolution Zhao et al. (2016). The solutions to these sub-problems are
subsequently merged to yield a resolution to the initial problem. In order to
ensure that OmAgent is not limited to simple video Q&A functionality but
possesses robust problem-solving capabilities, we initially aimed to build a
general task-solving agent system when designing and constructing the agent
framework. Inspired by XAgent’s Team (2023) double looped planner, we designed
an agent framework based on the divide-and-conquer task processing loop (DnC
Loop), which is capable of performing recursive task decomposition and
execution. The DnC Loop task-solving procedure is shown in Algorithm 1.

Algorithm 1 DnC Loop of OmAgent

The input Query $UserTask$; The max depth of the TaskTree $N$

Initialize $Task$ = TaskTree.init($UserTask$)

procedure DnC($Task,N$)

$Result\leftarrow\text{Conqueror}(Task)$

if $Result.\text{type}=\text{``too complex''}$ then

$Subtasks\leftarrow\text{Divider}(Task,Result.\text{reason})$

if $Subtasks.\text{success}$ then

$Task$.add($Subtasks$.tasks)

for all $Subtask\in Task.\text{subtasks}$ do

if $Task.\text{depth}\leq N$ then

DnC($Subtask,N$)

else

return “Task tree depth exceeded”

end if

end for

else

return $Subtasks.\text{reason}$

end if

else if $Result.\text{type}=\text{``requires tool''}$ then

$ToolResult\leftarrow\text{ToolCall}(Task,Result.\text{tool})$

$Task.\text{update}(Task,ToolResult)$

return $ToolResult$

else if $Result.\text{type}=\text{``direct answer''}$ then

$Task.\text{update}(Task,Result.\text{answer})$

return $Result.\text{answer}$

end if

end procedure

Figure 2: Divider and Conqueror Loop task-solving procedure. In the DnC Loop,
simple problems are directly executed by Conqueror, while complex problems are
split by Divider until they can be executed. The Rescuer recognizes exceptions
and retries the task. The Tool Manager organizes the external tools. It is
worth mentioning that the Rewinder tool can goes back through the entire video
to find information and missing details. Finally, the DnC loop outputs the
relevant content whether the execution fails or succeeds.

#### Conqueror

Conqueror is the entry point of the DnC loop. It is responsible for evaluating
and processing the current task. For a given task, Conqueror may return one of
the following three types of results:

  * •

If the current task is too complex and needs to be divided into subtasks,
Conqueror returns the reason for the division.

  * •

If the execution of the current task requires the use of a specific tool,
Conqueror returns both the task information and the tool information. These
pieces of information will be passed to the tool execution module for tool
invocation.

  * •

If the current task can be answered directly by the LLM, return the result
directly.

Conqueror will detect the depth of the task tree and terminate task execution
when it exceeds the user’s setting to prevent tasks from being infinitely
split. The position of Conqueror in the whole process is shown in Figure 2.

#### Divider

The Divider component is responsible for breaking down complex tasks into
simpler ones while ensuring that the execution results of these simple tasks
are equivalent to the original task. When the Conqueror component determines
the necessity of task division, it delegates the task to the Divider for
attempted division. Successfully divided tasks are then integrated into the
Task tree as child nodes of the original task node. If the division fails, the
Divider is asked to provide the reason.

#### Rescuer

Rescuer is an auxiliary module in the Conqueror’s execution process. It
attempts to repair issues and ensure the smooth completion of the Conqueror’s
execution when errors occur. A typical scenario is when the agent tries to
execute a piece of code, but a required package is missing in the environment.
The Rescuer can attempt to fix the runtime environment issue. The position of
Divider in the whole process is shown in Figure 2.

#### Task tree

In software development practice, divide-and-conquer is often implemented
using recursion. OmAgent uses recursive tree structure to store all the paths
of task execution. With the help of the Loop node, it achieves recursive
operations for task decomposition and execution.

###  3.3 Tool call

As a standard capability of intelligent agents, the principle of tool calling
lies in utilizing the powerful logical generation ability of large language
models (LLMs) to generate corresponding tool invocation request parameters
based on task information. In addition to conventional tools, OmAgent
specifically offers a video detail rewinder tool for further information
extraction within specific time ranges of a video. OmAgent can autonomously
choose to view details of a particular segment of a video when necessary,
addressing the issue of information loss that occurs when video data
transitions from a continuous information source to a discrete one during the
preprocessing stage.

Furthermore, OmAgent provides conventional tools such as internet search
tools, facial recognition tools, and file processing tools to meet more
complex user tasks.

##  4 Experimental Settings

To validate the efficacy of the OmAgent system in addressing complex problems
within real-world scenarios, we have designed a two-phase experimental
approach:

###  4.1 General problem-solving capabilities

We hypothesize that understanding lengthy and intricate videos relies
significantly on an agent’s comprehensive problem-solving skills. To test
this, we used two general-purpose intelligent benchmarks, MBPP Austin et al.
(2021) and FreshQA Vu et al. (2023). We focused on the DnC Loop’s ability to
plan and execute tasks and its proficiency in utilizing tools to address
complex issues

#### Datasets

The Mostly Basic Programming Problems (MBPP) benchmark compendium encompasses
976 elementary Python coding conundrums. These queries serve to test the
system’s proficiency in strategizing solutions, its adroitness in selecting
and invoking tools, as well as its resilience in rectifying errors.

FreshQA stands as an ongoing, perpetually refreshed repository of questions
and answers from the open world, with its contents echoing the ceaseless
evolution of reality, hence the ever-shifting sands of its responses.
Therefore, FreshQA places a premium on the system’s capability to assimilate
external information.

#### Settings

To study OmAgent’s capability in comprehending complex long-form videos, we
devised two control groups:

First, we aimed to ascertain the performance that could be achieved by one of
the most advanced MLLMs - GPT-4o, based solely on a limited number of video
frames (restricted to 20 by the Microsoft Azure GPT-4o service) and basic
dialogue textual information. In this experiment, we initially extracted 20
frames evenly from the video and paired them with the dialogue text obtained
through Whisper Radford et al. (2023) as the context for input into the MLLM
for question-answering.

In the second control experiment, we sought to evaluate the strengths and
weaknesses of using a multimodal RAG approach compared to our Agent strategy.
To ensure a fair comparison, we isolated the Video2RAG component from OmAgent
to serve as the RAG system. When a query is input, the system first retrieves
relevant video clip information from the knowledge database, then inputs this
pertinent data as context into the MLLM for question-answering.

###  4.2 Long-form videos understanding capabilities

We created a benchmark with over 2000 question-answer pairs to evaluate
OmAgent’s ability to understand, answer questions, and recall detailed
information from long-form videos. This benchmark allows us to measure the
capabilities of our intelligent agent and compare its performance against
large language models.

#### Datasets

Publicly available long-form video understanding datasets are very scarce,
similar to the FreshQA/MovieQA datasets which only contain videos at the
minute level, and the questions are not complex enough, for example, "How does
Talia die?" in MovieQA, which is a question that can be inferred from
consecutive frames, cannot meet our needs. MoVQA is a long movie question and
answer dataset that utilizes 100 well-known movies to create complex long
video question-answer pairs, which align with our requirements, but it is not
yet open-sourced. Therefore, we created the dataset ourselves. We collected
some long videos familiar to annotators, selected the top 100 videos in terms
of frequency, and created 20 questions for each video. These questions were
first proofread by two different annotators and then revised by a third
annotator. Videos include episodes and movies, varieties, documentaries, and
vlogs. These types of videos exhibit significant differences in themes,
filming and editing techniques, data density, scene lengths, and alignment of
video audio and visuals, fully demonstrating the diversity of the data.

#### Settings

To evaluate the system’s understanding of long-form videos and timelines, we
defined questions in four categories: reasoning, information summary, event
localization, and external knowledge.

  * •

Reasoning involves deducing relationships between events.

  * •

Information summary extracts key details from specific timestamps.

  * •

Event localization tests timeline accuracy.

  * •

External knowledge requires finding relevant but unmentioned information.

Reasoning, information summary, and external knowledge questions are multiple-
choice. Event localization requires precise timestamps or time spans, with a
deviation within ±2 seconds for timestamps and an IoU exceeding 90% for time
spans.

##  5 Results and Analysis

###  5.1 General problem-solving capabilities

GPT-4, recognized as a benchmark for evaluating Large Language Models, is
known for its strong reasoning abilities and serves as the baseline model for
our experiments. XAgent, an advanced agent system, features a well-designed
Dual-Loop Mechanism that allows it to address problems from both broad and
detailed perspectives.

The results in Table 1 clearly show that both agent systems outperform the
basic inferential capabilities of GPT-4 alone. Notably, OmAgent surpasses
XAgent Team (2023) in overall performance. Analysis reveals that XAgent’s
Dual-Loop Mechanism, while thorough, often leads to overthinking and
complicates problem-solving. In contrast, OmAgent’s Rescuer mechanism proves
more effective, especially in handling code-related tasks. This mechanism
enables OmAgent to dynamically correct issues based on real-time results,
leading to superior performance.

| MBPP | FreshQA
---|---|---
GPT4 | 80.01% | 67.0%
GPT4+XAgent | 84.2% | 74.0%
GPT4+OmAgent | 88.3% | 79.7%
Table 1: Results on MBPP and FreshQA comparing with GPT4 and XAgent. Showing OmAgent has a strong generalized task solving capbility. | Vlog | Episode and Movies | Variety | Documentary | Total
---|---|---|---|---|---
Frames + STT | 42.85% | 29.41% | 17.64% | 31.58% | 28.57%
Video2RAG | 42.86% | 32.35% | 19.88% | 31.57% | 27.27%
OmAgent | 57.14% | 56.25% | 23.53% | 36.84% | 45.45%
Table 2: Results of different types of videos on OmAgent and the other two control groups.  | Reasoning | Event Localization | Information Summary | External Knowledge
---|---|---|---|---
Frames with STT | 63.64% | 2.38% | 63.63% | 19.46%
Video2RAG | 72.73% | 4.76% | 50.17% | 23.36%
OmAgent | 81.82% | 19.05% | 72.74% | 57.21%
Table 3: Results of different types of queries on OmAgent and the other two
control groups.

###  5.2 Long-form videos understanding capbilities

Table 2 compares the scores of two control groups and OmAgent across different
types of long-form video understanding. OmAgent achieved the highest scores.
Vlogs, variety, and documentaries contain extensive narration, so the STT data
and the resulting scene captions encompass most relevant information of the
video. Therefore, the performance difference between OmAgent and the other two
methods in these categories is not as significant as in episodes and movies.
In episodes and movies, scenes change frequently, complex queries involve
cross-scene information, and STT data might span scene transitions. Compared
to frames with STT, Video2RAG retrieves data related to the query, reducing
data redundancy, and thus has higher scores than frames with STT. However,
since it only retrieves relevant information from a vector database, complex
questions such as "Are there any scene changes between 03:58 and 04:02, and
what is their connection?" are not achieved. On the other hand, OmAgent’s DnC
Loop breaks down complex questions into several sub-questions, including
"Extract frames between 03:58 and 04:02," "Analyze the extracted frames to
identify any scene changes," and "Determine the connection between the scenes
based on the identified changes." By leveraging the rewinder capability, it
pinpoints the relevant segments for rewatching, thereby arriving at the
correct answer.

Furthermore, we conducted a detailed analysis of different question types.
Table 3 provides a comparison of OmAgent and two control groups in terms of
reasoning, event localization, information summary, and external knowledge.
The results show that OmAgent achieves the highest scores in all four types of
questions. Through its rewinder capability, OmAgent can extract more detailed
video information and accurately locate timestamps, which leads to significant
improvements in reasoning, event localization, and information summarization
tasks compared to frames with STT and Video2RAG. External knowledge task has
stricter requirements for information retrieval. Although GPT can answer some
questions through its own capabilities and scene information, OmAgent achieves
higher scores by utilizing various external tools (such as facial recognition,
web search, etc.) to obtain more accurate relevant information. Notably, in
the question type of information summary, Video2RAG scored lower than frames
with STT. Analysis reveal that video2RAG’s information source came from scene
captions generated based on STT and images within the scene, which had a
certain probability of omissions compared to the original STT, thus affecting
the scores. Although OmAgent performed the best across all question types, it
still scores relatively low in the event localization task. Detailed study and
analysis reveal that LLM tends to directly use timestamps or time spans within
the scene, leading to a lack of precision in answers. OmAgent’s DnC loop and
rewinder capabilities can mitigate this issue but cannot completely resolve
it.

##  6 Conclusion

OmAgent is a powerful video comprehension agent that integrates multimodal RAG
with a generalist AI agent, enabling several advanced capabilities. It offers
a theoretical near-infinite length video understanding capacity and
incorporates a secondary recall mechanism for detailed video information,
which significantly mitigates information loss. Additionally, OmAgent
autonomously invokes tools based on video comprehension tasks, allowing it to
execute more intricate operations. These capabilities have resulted in
remarkable performance on the MBPP, FreshQA, and our proposed long video
complex task test datasets.

##  7 Limitations

  * •

When determining the positioning of an event in a video, LLM tends to directly
use the timestamps given in the scene. OmAgent can alleviate this situation
through DnC loop and rewinder but cannot completely resolve it.

  * •

The character information in long-form videos is usually diverse. For example,
character Logan can be both a boss and a father, making it difficult to align
back to the same person, causing OmAgent to fail in precise positioning. It is
necessary to add a visual prompt to the character for suppression.

  * •

The phenomenon of audio-visual asynchrony is prominent in long-form videos.
For example, in documentaries, the picture might not change significantly
while the speech has already shifted to introduce other scenes, resulting in a
misalignment between the picture and the speech. Additionally, STT only
processes the speech of characters, losing other audio information such as
background music, sound effects, and different characters’ voiceprints.

## References

  * Arefeen et al. (2024) Md Adnan Arefeen, Biplob Debnath, Md Yusuf Sarwar Uddin, and Srimat Chakradhar. 2024.  irag: An incremental retrieval augmented generation system for videos.  _arXiv preprint arXiv:2404.12309_.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_.
  * Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Chase (2022) Harrison Chase. 2022.  LangChain.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Kevin (2023) Stan Kevin. 2023.  VLog: Transform Video as a Document with ChatGPT, CLIP, BLIP2, GRIT, Whisper, LangChain.
  * Lin et al. (2023a) Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023a.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_.
  * Lin et al. (2023b) Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. 2023b.  Mm-vid: Advancing video understanding with gpt-4v (ision).  _arXiv preprint arXiv:2310.19773_.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024.  Llava-next: Improved reasoning, ocr, and world knowledge.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual instruction tuning.
  * Liu (2022) Jerry Liu. 2022.  LlamaIndex.
  * OpenAI (2023a) OpenAI. 2023a.  Gpt-4.  https://openai.com/gpt-4.
  * OpenAI (2023b) OpenAI. 2023b.  Gpt-4v(ision), system-card.  https://openai.com/research/gpt-4v-system-card, 2023.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Tang et al. (2023) Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2023.  Video understanding with large language models: A survey.  _arXiv preprint arXiv:2312.17432_.
  * Team (2023) XAgent Team. 2023.  Xagent: An autonomous agent for complex task solving.
  * TensorLakeAI (2024) TensorLakeAI. 2024.  Indexify: A realtime and indexing and structured extraction engine for unstructured data to build generative ai applications.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Vu et al. (2023) Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023.  Freshllms: Refreshing large language models with search engine augmentation.
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv:2303.11381_.
  * Zhang et al. (2023) Hang Zhang, Xin Li, and Lidong Bing. 2023.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_.
  * Zhang et al. (2018) Jiaping Zhang, Tiancheng Zhao, and Zhou Yu. 2018.  Multimodal hierarchical reinforcement learning policy for task-oriented visual dialog.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 140–150.
  * Zhao et al. (2016) Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2016.  Dialport: Connecting the spoken dialog research community to real user data.  In _2016 IEEE Spoken Language Technology Workshop (SLT)_ , pages 83–90. IEEE.
  * Zhao et al. (2023) Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023.  Learning video representations from large language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6586–6597.
  * Zhu et al. (2023) Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. 2023.  Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.  _arXiv preprint arXiv:2310.01852_.

##  Appendix A Case Study

In this section, we will illustrate the process by which OmAgent addresses
complex video understanding problems through several examples. The primary
aspects showcased include: scene caption, inner steps, and agent output. The
scene caption displays sample data stored in the knowledge database after the
Video2RAG preprocessing; the inner steps reveal the task decomposition and
execution within the DnC Loop; the agent output presents the final results
produced by OmAgent.

###  A.1 Case 1

In this case in Figure 3, the video we utilized is from the series
"Succession," Season 1, Episode 1. The question posed was, "Which other shows
has the actor who played in the video at 5 minutes and 41 seconds appeared
in?" This is an intricate query, as the system must first identify the
individual at the specified timestamp and subsequently gather information
about the actor from the internet. Observably, the intelligent system
deconstructed the task into three sub-tasks and then sequentially employed
relevant tools to address each one, ultimately arriving at the correct answer.

Figure 3: case 1

###  A.2 Case 2

In this case as showing in Figure 4, the video we utilized is from the series
"Succession," Season 1, Episode 1. The question posed was, "When was the first
time a cigarette dropped to the ground?" This task requires OmAgent to locate
a highly detailed scene within a lengthy video. OmAgent initially divides this
problem into three subtasks. Among these, the task of identifying the key
frame is too complex to be completed directly and is further divided into four
subtasks. It necessitates extracting details and summarizing results from
three segments of the video. Ultimately, all the subtasks are successfully
handled, and OmAgent returns the correct answer.

Figure 4: case 2

##  Appendix B Sample Experiment Results

Tables 4, 5, and 6 show 20 examples of OmAgent’s performance on 2000 long-form
video understanding questions. In these tables, "Video" refers to the video
names, and "Question" refers to the questions in the dataset, with the choice
options directly following the questions. OmAgent represents our answer
output, while Video2RAG and frames with stt are the outputs of two control
groups.

Video |  Question |  Question Type |  Ground Truth |  OmAgent |  Video2RAG |  Frame with STT
---|---|---|---|---|---|---
Succession S01EP01 |  When was the first time a cigarette dropped to the ground? |  Event Localization |  00:02:32 |  00:02:32 |  [00:02:18, 00:02:45] |  Fail to answer
Succession S01EP01 |  The task was to determine if the video shows a theme park? Please choose from the following options: a. The playground’s interior scenes appeared between 08:58 and 09:49. b. The playground’s interior scenes appeared after 09:50. c. The playground’s interior scenes were shown before 08:57. d. No playground scenes appeared during this time. |  Reasoning |  a |  a |  a |  None of a, b, c, d
Succession S01EP01 |  In which time period does the headphone first appear? |  Event Localization |  [00:01:46, 00:01:49] |  [00:01:46, 00:01:49] |  [00:04:15, 00:04:27] |  0:01:46
Succession S01EP01 |  What important event occurs between the 34th and 35th minute? Choose from the following options: a. Connor gets angry in the bathroom, breaks the mirror and sink. b. Connor gets angry in the bathroom, vents his anger by smashing a vase, shouting loudly, etc. c. Connor gets angry in the bathroom, kicks over the toilet and bathtub. d. Connor gets angry in the bathroom, smashes the door and windows. |  Reasoning |  b |  b |  b |  b
Westworld S01EP01 |  Please summarize the content of this video. Choose from the following options: a. The video starts with a dialogue in the protagonist’s dream, introducing a Western world live-action game built by modern people that attracts various customers to experience. The protagonist, as the main NPC, repeats the same life continuously. b. After an update in the Western world game, NPCs start experiencing various minor bugs. This puzzles the programmers behind the scenes, who begin inspecting the problematic NPCs. Initially, only some NPCs experience issues like lag during operation, which the programmers do not pay much attention to after updating them. c. The protagonist’s father awakens to consciousness after seeing a photo of the modern world and recites a line from a Shakespearean work to the protagonist. This triggers uncontrollable events in the entire game world. d. The administrator questions both the protagonist and her father. The protagonist enters a state of consciousness during the conversation and claims to retaliate against the boss. The dialogue with the protagonist echoes the dream at the beginning, foreshadowing the occurrence of certain events. |  information summary |  a, b, c |  b |  a |  a
Westworld S01EP01 |  Please find the specific time when the female protagonist wakes up from the bed. |  Event Localization |  00:02:46, 00:14:31, 00:44:28, 01:04:15 |  00:44:23 |  [00:00:00, 00:00:04] |  [00:14:26, 00:14:32]
Westworld S01EP01 |  From 01:04:03 to 01:05:59, what changes occurred in the position of the female protagonist, and what do these changes indicate? Choose from the following options: a. During this time, the female protagonist went to her home in Westworld but did not return to the modern laboratory, indicating that Westworld is real. b. During this period, the female protagonist’s movements from the modern laboratory to Westworld and then to her home reveal the authenticity of her life. c. During this period, the female protagonist’s movements from the laboratory to Westworld and then to her home imply that Westworld and her home are real. d. In this time frame, the female protagonist’s movements from the modern laboratory to the home in Westworld and back to the modern laboratory indicate the falsity of Westworld. |  Reasoning |  d |  d |  d |  d

Table 4: OmAgent Experiment Results

Video |  Question |  Question Type |  Ground Truth |  OmAgent |  Video2RAG |  Frame with STT
---|---|---|---|---|---|---
Westworld S01EP01 |  Why was Dolores’ father being inpsected? Choose from the following options, it’s a multiple choices question: a. Because his program malfunctioned and he couldn’t continue to function properly. b. Because he displayed emotions that a host NPC shouldn’t have. c. Because he was killed by bandits in the mountains and needed servicing to be restored. d. Dolores’ father was serviced because he showed strong motives and drives when questioned. |  Reasoning |  b, d |  b, d |  a |  a
Tempting Fortune S01EP01 |  In the video, when was the first money spent and what was it spent on? |  Event Localization |  00:31:14 |  [00:31:08, 00:31:11] |  [00:04:14, 00:04:26] |  [00:05:51, 00:05:57]
Tempting Fortune S01EP01 |  Please describe the scene at 42:40. Select from the following options: a. A group of people are gathered around a hearth roasting food, with several lamps extinguished nearby. b. A group of people are sitting around a campfire, enjoying dinner, with many lights illuminated nearby. c. A large crowd is dining around a candelabrum, with many lamps turned off nearby. d. Some people are singing around a bonfire, with dim lights in the vicinity. |  information summary |  b |  b |  None of a, b, c, d |  b
Tempting Fortune S01EP01 |  At what time in the video does someone fall into the river? |  Event Localization |  00:45:56 |  00:45:54 |  None of a, b, c, d |  [00:29:38, 00:30:17]
Tempting Fortune S01EP01 |  At what time did all team members climb up the hill? |  Event Localization |  [00:12:45, 00:12:51] |  [00:12:37, 00:12:51] |  None of a, b, c, d |  None of a, b, c, d
A Perfect Planet |  When do volcanic eruptions occur in the video? |  Event Localization |  [00:01:37, 00:03:55], [00:14:30, 00:15:00], [00:43:24, 00:44:05] |  [0:01:36, 0:01:41], [0:03:03, 0:03:06], [0:03:21, 0:03:28], [0:03:28, 0:03:37], [0:03:45, 0:03:55], [0:14:30, 0:14:37], [0:14:37, 0:15:00], [0:17:15, 0:18:09], [0:43:24, 0:43:36], [0:43:36, 0:43:50], [0:43:58, 0:44:05] |  [0:01:36, 0:01:41], [0:01:52, 0:02:15], [0:02:40, 0:02:53], [0:14:30, 0:14:37], [0:43:24, 0:43:36], [0:43:36, 0:43:50], [0:43:58, 0:44:05] |  [0:02:40, 0:02:53], [0:03:03, 0:03:06], [0:04:04, 0:04:10], [0:15:00, 0:15:10], [0:15:40, 0:15:57], [0:18:09, 0:19:10], [0:21:23, 0:21:31], [0:26:19, 0:26:43], [0:31:37, 0:31:38], [0:38:18, 0:38:32], [0:43:07, 0:43:16], [0:48:04, 0:48:34]
A Perfect Planet |  What happened between 10:38 and 10:53? Choose from the following options: a. A little chick couldn’t overcome the hardships on the way and finally collapsed exhausted in a mud pit. b. A little chick encountered many difficulties on the road and eventually collapsed on the grass from exhaustion. c. A bird encountered many obstacles during flight and eventually collapsed in the desert. d. A little chick went through various hardships and finally collapsed exhausted on a rock. |  information summary |  a |  a |  b |  a
A Perfect Planet |  When did the Galapagos land iguana appear in the video? |  Event Localization |  [00:15:58, 00:21:47] |  [0:21:31, 0:21:42] |  [0:15:00, 0:21:31] |  [0:15:57, 0:16:47]
A Perfect Planet |  When did the East African Rift Valley appear in the video? |  Event Localization |  [00:44:12] |  [0:44:05, 0:44:18] |  [0:44:05, 0:44:18] |  [0:44:05, 0:45:59]

Table 5: OmAgent Experiment Results

Video |  Question |  Question Type |  Ground Truth |  OmAgent |  Video2RAG |  Frame with STT
---|---|---|---|---|---|---
WWDC 2024 Recap: Is Apple Intelligence Legit? |  What new features mentioned in the video have Android users long been able to enjoy? a. The ability to freely arrange and customize home screen icons. b. Cleanup tool in the photo app. c. Remote control of other people’s devices. d. Hide apps. |  Reasoning |  a, b |  a, b, d |  a, d |  a, b, c, d
WWDC 2024 Recap: Is Apple Intelligence Legit? |  What is the author’s attitude towards the iPad OS update? a. Disappointing. b. Exciting. c. There are some great upgrades, but not quite enough. d. Not mentioned. |  Reasoning |  c |  c |  c |  c
WWDC 2024 Recap: Is Apple Intelligence Legit? |  At 14 minutes and 26 seconds, what address is displayed on the phone? a. Waterbar. b. San Francisco Fisherman’s Wharf. c. San Francisco International Airport. d. Ronald Reagan Airport. |  information summary |  c |  c |  b |  b
WWDC 2024 Recap: Is Apple Intelligence Legit? |  At what time in the video does the host promote their channel? |  Event Localization |  [0:05:54, 0:06:03] |  [0:05:24, 0:06:03] |  [0:06:03, 0:06:09] |  [0:06:03, 0:06:09]

Table 6: OmAgent Experiment Results


# Spin and Orbital Spectroscopy in the Absence of Coulomb Blockade in Lead
Telluride Nanowire Quantum Dots

M. Gomanko  Department of Physics and Astronomy, University of Pittsburgh,
Pittsburgh, PA 15260, USA  E.J. de Jong  Eindhoven University of Technology,
5600 MB, Eindhoven, The Netherlands  Y. Jiang  Department of Physics and
Astronomy, University of Pittsburgh, Pittsburgh, PA 15260, USA  S.G.
Schellingerhout  Eindhoven University of Technology, 5600 MB, Eindhoven, The
Netherlands  E.P.A.M. Bakkers  Eindhoven University of Technology, 5600 MB,
Eindhoven, The Netherlands  S.M. Frolov<EMAIL_ADDRESS>Department of
Physics and Astronomy, University of Pittsburgh, Pittsburgh, PA 15260, USA

###### Abstract

We investigate quantum dots in semiconductor PbTe nanowire devices. Due to the
accessibility of ambipolar transport in PbTe, quantum dots can be occupied
both with electrons and holes. Owing to a very large dielectric constant in
PbTe of order 1000, we do not observe Coulomb blockade which typically
obfuscates the orbital and spin spectra. We extract large and highly
anisotropic effective Landé g-factors, in the range 20-44. The absence of
Coulomb blockade allows direct readout, at zero source-drain bias, of spin-
orbit hybridization energies of up to 600 $\mu$eV. These spin properties make
PbTe nanowires, the recently synthesized members of group 4-6 materials
family, attractive as a materials platform for quantum technology, such as
spin and topological qubits.

###  .1 Introduction

Quantum computing relies on two-level systems (qubits) and state entanglement
as leverage to attain computational power exceeding that of classical
computers for certain classes of problems. Several qubit hardware realizations
are being pursued at the moment, ranging from photons and trapped ions to
nonlinear resonators, single charges and spins [1]. Single spins in low
dimensional structures are natural candidates for qubits, and were one of the
first platforms proposed [2]. Semiconductor quantum dots were used to perform
single-shot spin state readout [3], coherent spin manipulation [4], and all-
electrical spin control [5]. Spin qubits still have untapped potential due to
their long spin coherence times, scalability and pathways for interfacing with
conventional electronics via similar fabrication techniques, and relatively
high operating temperatures [6].

Initial single spin experiments relied on heterostructures of group 3-5
semiconductors such as GaAs quantum wells [4], InAs and InSb nanowires [7, 8,
9]. More recently group 4 semiconductors took center stage, in part due to
reduced hyperfine interaction and longer potential coherence times [1]. Among
realizations are Si/SiGe heterostructures [10], single atom donors in Si [11],
Ge/Si core/shell nanowires[12], Ge quantum wells [13]. Another group IV family
member, carbon, has been used in the nanotube form for valley-spin qubits
[14].

PbTe and related materials were studied for applications such as infrared
detectors and thermoelectrics [15] [16] [17]. They are characterized by the
rocksalt crystal structure, low effective mass of 0.08$m_{e}$, very large
dielectric constant exceeding 1000 at low temperatures, small bandgap of 0.19
eV at 4.2K. Quantum wells of group IV-VI materials including PbTe exhibit
quantum Hall effect and quantum point contact behavior [18, 19, 20, 21, 22].
Previous efforts to grow PbTe nanowires focused on chemical growth methods,
such as chemical vapor deposition [15, 23] and hydrothermal synthesis [24,
25]. MBE growth has been reported [26] in Vapor-Liquid-Solid mode, though the
nanowires were limited in length. A related material SnTe is a topological
crystalline insulator [27], and has been explored in the nanowire form [28,
29, 30, 31].

PbTe is composed of heavier elements, and is expected to have strong spin-
orbit interaction [32], allowing electrical control of charge carrier spins,
as well as large g-factors enabling low magnetic field operation of spin-orbit
qubits [7]. The conduction band of PbTe is dominated by p-orbitals which has
been associated with reduced hyperfine interaction [33]. In contrast with
group III-V materials, where hyperfine coupling is the main decoherence
mechanism for spin qubits, both Pb and Te have isotopes with zero nuclear
spin, which can be leveraged through isotopic purification. Very large
dielectric constants in PbTe suggest low charging energy of quantum dots, and
screening of charge noise. Many of the these properties also make PbTe an
interesting platform for the realization of Majorana zero modes, provided that
superconducting proximity effect can be demonstrated [34].

We fabricate PbTe nanowire devices with electrical contacts in the 10k$\Omega$
range, as well as top and back gate electrodes. Quantum dots can be defined on
both the electron and the hole side of the tunneling regime. Quantum dots
exhibit no detectable charging energy. Coulomb blockade is absent due to a
very large dielectric constant in PbTe. We extract g-factors as high as 44
with the anisotropy of order 2. We observe spin-orbit anticrossings between
states of opposite spin belonging to different orbitals. The range of
antricrossing values is from 10’s to 100’s of $\mu$eV indicative of sizeable
spin-orbit coupling.

Impact. PbTe nanowires have recently been synthesized with low defect density
and large aspect ratios, making them suitable for gate-defined quantum dot
devices [35]. Absence of Coulomb blockade opens doors for the exploration of
new regimes of spin manipulation and dynamics. This has enabled the
investigation of spin, orbital and spin-orbit properties of electrons, as well
as holes in this materials system. The 4-6 family features an impressive
variety of materials, from semiconductors and semimetals to topological
crystalline insulators, and has significant potential for further exploration
as a platform for quantum devices, such as spin and topological qubits.

###  .2 Brief methods

PbTe nanowires are grown using molecular beam epitaxy (MBE) on GaAs[111]
substrate in [100] axis orientation using Au catalyst [35]. Typical sizes are
$1-1.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$ in length and
$70-125\text{\,}\mathrm{nm}$ in width. We position the nanowires on doped
silicon substrates with a 285 nm layer of thermal oxide. The substrate is used
as a back gate. Metallic Ti/Au contacts (10/130 nm) are fabricated on top of
the wire after argon sputter cleaning. Subsequently, the device is covered
with a 10 nm HfOx layer by atomic layer deposition, and a top gate electrode
(Ti/Au, 10/140 nm) is fabricated.

A standard low-frequency lock-in technique is used to acquire data.
Measurements are performed in several dilution refrigerator setups, with base
temperature of $50-100\text{\,}\mathrm{mK}$. Multiple devices were fabricated,
varying in length and width (supplementary figure S1, panels (a), (c) and (d)
have SEM pictures taken before covering the nanowire with a topgate). The
major difference is that devices 5-8 did not had a topgate, only backgate
separated by a wider dielectric layer. Device 1 is measured in a solenoid
magnet with a fixed field orientation at 25 degrees to the nanowire axis.
Device 2 is measured in a setup equipped with a 2D vector magnet with a 4T
vector field that rotates in the substrate plane.

###  .3 Figure 1: Device description

Figure 1: a Model of a topgate device. Contacts are colored with gold, top
gate is represented by transparent blue. b SEM image of typical device (device
9). The orientation of magnetic field B corresponds to device 1 studied in
Figs. 2 and 3.

Fig. 1 shows an example of a PbTe nanowire device. The spacing between the
source and the drain is nominally 400 nm. PbTe nanowires have square cross-
sections, in contrast with InAs and InSb nanowires which have hexagonal cross-
sections. The nanowire width is in the range 70-125 nm. The top gate has a
stronger effect due to it being closer to the nanowire and covering the
nanowire on 3 sides. In fact the back gate alone is unable to pinch-off
current in these devices (Supplementary figs. S9 and S10). Quantum dots are
created between the contacts, though their actual extent along the nanowire
can be shorter.

###  .4 Figure 2: Electron quantum dot

Figure 2: Differential conductance of device 1 as a function of $V_{bias}$ and
$V_{tg}$ at different magnetic fields for $V_{bg}$ = -15V. a B = 0T; b B =
0.34T; c B = 2.6T; d B = 4.0T.

Figure 2 presents a study of an electron quantum dot. At zero magnetic field,
we observe a grid of sharp intersecting conductance resonances (Fig.2(a)). The
pattern at first glance appears to be that of Coulomb blockade diamonds
typical of quantum dots. However, upon closer inspection the data possess
several unusual features.

First, there appear to be no excited states of the same total charge. These
would be lines that run diagonally outside the central diamonds, and terminate
at the edges of the diamonds. Coulomb blockade prevents the population of
excited states of the same charge number - transport is blocked by an electron
that is already present in the lowest energy state [36].

Second, at finite magnetic field each of the zero-field resonances splits
(Figs.2(b-d)). The splitting is apparent at B=0.34 T (panel b), while at a
higher B=2.6 T we see a pattern of large/small diamonds around zero bias. At
the highest field studied, B = 4 T, it becomes harder to track individual
resonances but their density is doubled. The observation of resonance
splitting in magnetic field is in contradiction with Coulomb blockade, which
already lifts spin degeneracy at zero field.

Given the known large dielectric constant of PbTe we conclude that charging
energy, and Coulomb blockade, are not observed in our devices. Related
phenomena were reported in LAO/STO quantum dots in the presence of
superconductivity [37].

The apparent lack of resonances terminated at diamond edges becomes clear: in
fact, all orbital and spin states are directly observed because they are not
darkened by Coulomb blockade. They originate from consecutive orbital states
at zero field, and spin-resolved states at finite field.

We note that at a given gate voltage, higher bias resonances do correspond to
excited state transport when multiple electrons are allowed to travel through
the dot simultaneously. This effect is the same with or without Coulomb
blockade.

Since charging energy is negligible, from the height of central diamonds we
can directly read off the orbital energies which are approximately 3 meV for
the lowest few observable orbital states. While we cannot verify that the
state at the most negative gate voltage corresponds to the first electron, we
argue that the number of electrons in the quantum dot is a few, from the
typical orbital energy and the gate voltage distance to the hole quantum dot
on the valence band side (see Fig. 5).

At more positive gate voltages the resonances broaden gradually forming an
open regime similar to Fabry-Perot and universal conductance fluctuation
regimes (Figs. S9 and S10). The unusual aspect is that upon closing down the
barriers no Coulomb blockade emerges, the resonances simply sharpen into grid
of intersecting diagonals (Fig. 2).

###  .5 Figure 3: Magnetic field spectroscopy

Figure 3: Differential conductance of device 1 as a function of topgate
voltage and magnetic field. Arrows represent spin up and spin down states.
Dashed curves illustrate an anticrossing of opposite-spin states. Magnetic
field is applied at an angle of 25 degrees with respect to the nanowire axis,
in the substrate plane.

Spin-resolved spectrum of the electron quantum dot in device 1 is presented in
Fig. 3. The orbital states, which are visible as single resonances at zero
magnetic field, undergo a splitting in finite magnetic field. Each orbital is
doubly-degenerate due to spin, and no other degeneracies are detected. This is
consistent with a lack of symmetry in the dot shape.

From the movement of the resonances in magnetic field, and the gate-bias lever
arm extracted from data in Fig. 2, we can calculate the g-factors for three
lowest visible orbitals to be 28, 27, 29 (starting at the bottom).

Whenever resonances of opposite spin belonging to different orbitals come
close, we observe level repulsion, which is the effect of spin-orbit coupling.
The largest observed anticrossing is of the magnitude 575 $\mu$eV. Larger
anticrossings tend to appear at higher magnetic fields which is consistent
with spin-orbit induced hybridization getting enhanced by external field [38].
The ratios of spin-orbit to orbital energy vary between 0.28 and 0.10. This
ratio approximately corresponds to the ratio between the quantum dot size and
the spin-orbit length. Given the uncertainty in determining the quantum dot
size, which can be a fraction of the contact spacing, we can only provide an
upper bound estimate on the spin-orbit energy, though the effect is evident
and strong.

We assume the effective mass 0.03me at low temperatures [39]. We take the
orbital energy $E_{orb}=2.7meV$ from Fig.2. We estimate the size of the
quantum dot from particle-in-a-box formula to be 80 nm, which is less than the
contact spacing (350nm) and of the order of nanowire width (100nm). However,
these numbers come with assumption of an isotropic box confinement, which is
not the case for PbTe, and parabolic band structure, so a better theoretical
evaluation is necessary. We obtain an estimate range for spin-orbit length of
300-900nm for different anticrossings.

While the conduction band of PbTe is known to have four-fold valley degeneracy
[18], no degeneracies other than spin degeneracy manifest themselves in these
quantum dots. Two possibilities exist. Either the valley degeneracy has been
lifted, due to effects such as strain and strongly asymmetric quantum
confinement. Alternatively, the valley degeneracy remains and is unperturbed
by magnetic field, meaning that electrons in all valleys have exactly the same
g-factors - and all observed resonances are in fact four-fold degenerate at
finite field. The suppression of Coulomb blockade makes it difficult to
distinguish between these possibilities.

Figure 3 also reveals faint resonances at more positive gate settings. Those
resonances do exhibit Zeeman splitting, but they do not exhibit spin-orbit
antricrossings with the bright resonances. We argue that those originate from
another quantum dot given that their coupling to the leads is different and
compounded by the fact that the dot is defined by a single wide gate -
allowing for the possibility of other uncontrolled dots.

###  .6 Figure 4: Effective Landé g-factor anisotropy

Figure 4: Effective Landé g-factor dependence on field angle for device 2.
Graphics illustrate relative orientation of nanowire and magnetic field. Solid
line is a sinusoidal fit with an offset of 29.8 and an amplitude of 6.7.
Values extracted for state A in Fig.S6

.

We use device 2 to study the anisotropy of effective g-factors in a vector
magnet setup. We fix magnetic field at the amplitude of 1T and rotate the
field angle in the substrate plane. We extract g-factors from the height of
the diamonds that emerge at finite field in between spin-up and spin-down
states for each orbital, or from gate voltage shifts in peak positions at zero
voltage bias. See supplementary information for details of data processing and
for additional data.

In Fig. 4 we present the anisotropic g-factors for an orbital labeled A (Figs.
S4-S6). The g-factor is lowest around zero angle, or along the nanowire, where
it reaches the values of 22-24. The g-factor is largest when the field is
perpendicular to the nanowire where it reaches 37.

From the g-factor anisotropy the shape of the quantum dot can be guessed.
Enhanced g-factors originate from the coupling between spin and orbital states
and are known to be reduced by quantum confinement that suppresses the orbital
degrees of freedom [40]. The lower g-factors correspond to larger confinement
in a direction perpendicular to the applied field. Based on these
considerations, the dot is elongated along the nanowire which is in
qualitative agreement with contact spacing being larger than the nanowire
width.

The Fermi surface in PbTe is known to be anistoropic and this anisotropy can
be contributing to the g-factor anisotropy as well as to other properties such
as quantum dot level energies [41]. However, measurements done so far do not
allow us to identify a contribution to the observed anisotropies or energies
due to the crystal structure.

###  .7 Figure 5: Hole quantum dot

Figure 5: a and b Differential conductance for device 2. c Magnetic field vs
topgate on the hole side. Magnetic field is at 45 degrees with respect to the
nanowire.

In Fig. 5 that is also based on device 2 we present evidence of quantum states
confined in the valence band. Hole quantum dots are a promising platform for
quantum computing [13]. A handful of materials, such as carbon nanotubes and
indium antimonide nanowires, make it possible to study hole and electron dots
in the same sample, allowing to compare the orbital, spin and spin-orbital
energies in different bands [42, 14].

In Fig. 5(a) we observe the transition from electron transport, across the
bandgap, to hole transport which is characterized by significantly lower
conductance. This is an interesting observation because the effective masses
of electrons and holes in bulk PbTe are the same, thus similar mobility is
expected. The discrepancy may come from differences in contact resistance due
to different band character, or due to uncontrolled p-n junctions in the
nanowire. Similar behavior has been observed in InSb nanowires [42]. Further
investigation of this question will be required in locally gated devices and
with the assistance of first principles theory.

Transport on the hole side is also dominated by resonances that criss-cross
the gate-vs-bias diagram (Fig. 5(b)). Due to the lower currents the resonances
are not seen in the same detail as on the electron side. However, they also do
not exhibit Coulomb blockade. From the slopes and gate intervals separating
the resonances, we estimate orbital energies in the range 2-4.5 meV. See
supplementary information for additional valence band data on this and another
device.

In a magnetic field we observe resonances splitting in a manner that is
generally similar to how electron orbital states split. From various
measurements we extract g-factors in the range 17-56 for apparent hole states.
We do not observe spin-orbit level repulsion in these data but due to low
currents we cannot conclude that spin-orbit effects are not present.

###  .8 Future work

In future experiments, multiple local gates can be used to define quantum dots
of fixed length and provide an accurate estimate of spin-orbit interaction
strength, to establish the double quantum dots and the few-electron/hole
regime. This will also provide better control of dot barriers and thus higher
resolution spectroscopy on the valence band side.

PbTe quantum dots free of charging energy have potential to become a promising
platform for spin qubits. It would be interesting to explore whether large
dielectric translates into longer spin coherence times. Another future
question is whether Pauli spin blockade, an established spin qubit
initialization and readout mechanism, can be observed in the absence of
Coulomb blockade. Large anisotropic g-factors and strong spin-orbit
interaction are promising for electric field control of spin states and for
the operation of spin-orbit qubits.

Integration with superconductors, such as native and lattice-matched Pb, can
make possible the integration of PbTe wires into superconducting qubits [43],
as well as their exploration as a platform for Majorana zero modes [34].

###  .9 Duration and Volume of Study

Experiments in this paper were done over a 1.5 year timeframe, including a lab
shutdown due to the pandemic. A total of 20 chips with an average of 7 devices
per chip were fabricated, with 50% not resulting in any data due to
fabrication issues, contact barriers, gate leaks or inability to pinch them
off. In this paper we use 900 datasets from 4 different chips, one of them
with devices lacking a topgate. Quantum dot features were observed in 5
devices, clearest 3 of which were studied thoroughly and presented in this
paper. The first chip contains devices 1 (Figs. 2,3), 3 (Fig. 5) and 4. The
second chip contains device 2 (Figs. 4, 5). The third chip, without a topgate,
has 4 devices (5,6,7,8) used in Figs. S9 and S10. Finally, the fourth chip was
used for the SEM picture in Fig.1. The numbering of chips is not
chronological.

###  .10 Data and Code availability

All of the data from devices used in this paper along with experimental logs,
relevant notebooks and overview summaries is available on Zenodo (DOI
10.5281/zenodo.5531917). The repository also includes Python notebooks used
for data analysis, e.g. to extract the g-factors from the data.

## References

  * de Leon  _et al._ [2021] N. P. de Leon, K. M. Itoh, D. Kim, K. K. Mehta, T. E. Northup, H. Paik, B. Palmer, N. Samarth, S. Sangtawesin, and D. Steuerman, Science 372 (2021).
  * Loss and DiVincenzo [1998] D. Loss and D. P. DiVincenzo, Phys. Rev. A 57, 120 (1998).
  * Elzerman  _et al._ [2004] J. Elzerman, R. Hanson, L. W. Van Beveren, B. Witkamp, L. Vandersypen, and L. P. Kouwenhoven, nature 430, 431 (2004).
  * Petta  _et al._ [2005] R. Petta, A. Johnson, J. Taylor, and E. Laird, Science 309, 2180 (2005).
  * Nowack  _et al._ [2007] K. C. Nowack, F. Koppens, Y. V. Nazarov, and L. Vandersypen, Science 318, 1430 (2007).
  * Maurand  _et al._ [2016] R. Maurand, X. Jehl, D. Kotekar-Patil, A. Corna, H. Bohuslavskyi, R. Laviéville, L. Hutin, S. Barraud, M. Vinet, M. Sanquer, _et al._ , Nature communications 7, 1 (2016).
  * Nadj-Perge  _et al._ [2010] S. Nadj-Perge, S. Frolov, E. Bakkers, and L. P. Kouwenhoven, Nature 468, 1084 (2010).
  * Petersson  _et al._ [2012] K. D. Petersson, L. W. McFaul, M. D. Schroer, M. Jung, J. M. Taylor, A. A. Houck, and J. R. Petta, Nature 490, 380 (2012).
  * Van den Berg  _et al._ [2013] J. Van den Berg, S. Nadj-Perge, V. Pribiag, S. Plissard, E. Bakkers, S. Frolov, and L. Kouwenhoven, Physical review letters 110, 066806 (2013).
  * Maune  _et al._ [2012] B. M. Maune, M. G. Borselli, B. Huang, T. D. Ladd, P. W. Deelman, K. S. Holabird, A. A. Kiselev, I. Alvarado-Rodriguez, R. S. Ross, A. E. Schmitz, _et al._ , Nature 481, 344 (2012).
  * Pla  _et al._ [2012] J. J. Pla, K. Y. Tan, J. P. Dehollain, W. H. Lim, J. J. Morton, D. N. Jamieson, A. S. Dzurak, and A. Morello, Nature 489, 541 (2012).
  * Froning  _et al._ [2021] F. N. Froning, L. C. Camenzind, O. A. van der Molen, A. Li, E. P. Bakkers, D. M. Zumbühl, and F. R. Braakman, Nature Nanotechnology 16, 308 (2021).
  * Hendrickx  _et al._ [2020] N. Hendrickx, W. Lawrie, L. Petit, A. Sammak, G. Scappucci, and M. Veldhorst, Nature communications 11, 1 (2020).
  * Laird  _et al._ [2013] E. A. Laird, F. Pei, and L. P. Kouwenhoven, Nature nanotechnology 8, 565 (2013).
  * Jang  _et al._ [2009] S. Y. Jang, H. S. Kim, J. Park, M. Jung, J. Kim, S. H. Lee, J. W. Roh, and W. Lee, Nanotechnology 20, 415204 (2009).
  * Roh  _et al._ [2010] J. W. Roh, S. Y. Jang, J. Kang, S. Lee, J.-S. Noh, W. Kim, J. Park, and W. Lee, Applied Physics Letters 96, 103101 (2010), https://doi.org/10.1063/1.3352049 .
  * Springholz and Bauer [2014] G. Springholz and G. Bauer, “Semiconductors, iv–vi,” in  _Wiley Encyclopedia of Electrical and Electronics Engineering_ (John Wiley and Sons, Ltd, 2014) pp. 1–16.
  * Kolwas  _et al._ [2013] K. Kolwas, G. Grabecki, S. Trushkin, J. Wróbel, M. Aleszkiewicz, Ł. Cywiński, T. Dietl, G. Springholz, and G. Bauer, “Absence of nonlocal resistance in microstructures of pbte quantum wells,” (2013).
  * Grabecki [2007] G. Grabecki, Journal of applied physics 101, 081722 (2007).
  * Yuan  _et al._ [1994] S. Yuan, G. Springholz, G. Bauer, and M. Kriechbaum, Physical review B 49, 5476 (1994).
  * Springholz  _et al._ [1993] G. Springholz, G. Ihninger, G. Bauer, M. Olver, J. Pastalan, S. Romaine, and B. Goldberg, Applied physics letters 63, 2908 (1993).
  * Grabecki  _et al._ [2005] G. Grabecki, J. Wróbel, T. Dietl, E. Janik, M. Aleszkiewicz, E. Papis, E. Kamińska, A. Piotrowska, G. Springholz, and G. Bauer, Physical review B 72, 125332 (2005).
  * Yan  _et al._ [2008a] Q. Yan, H. Chen, W. Zhou, H. H. Hng, F. Y. C. Boey, and J. Ma, Chemistry of Materials 20, 6298 (2008a).
  * Tai  _et al._ [2008] G. Tai, W. Guo, and Z. Zhang, Crystal Growth and Design 8, 2906 (2008).
  * Yan  _et al._ [2008b] Q. Yan, H. Chen, W. Zhou, H. H. Hng, F. Y. C. Boey, and J. Ma, Chemistry of Materials 20, 6298 (2008b).
  * Dziawa  _et al._ [2010] P. Dziawa, J. Sadowski, P. Dluzewski, E. Lusakowska, V. Domukhovski, B. Taliashvili, T. Wojciechowski, L. T. Baczewski, M. Bukala, M. Galicka, R. Buczko, P. Kacman, and T. Story, Crystal Growth & Design 10, 109 (2010), https://doi.org/10.1021/cg900575r .
  * Tanaka  _et al._ [2012] Y. Tanaka, Z. Ren, T. Sato, K. Nakayama, S. Souma, T. Takahashi, K. Segawa, and Y. Ando, Nature Physics 8, 800 (2012).
  * Hsieh  _et al._ [2012] T. H. Hsieh, H. Lin, J. Liu, W. Duan, A. Bansil, and L. Fu, Nature communications 3, 1 (2012).
  * Sadowski  _et al._ [2018] J. Sadowski, P. Dziawa, A. Kaleta, B. Kurowska, A. Reszka, T. Story, and S. Kret, Nanoscale 10, 20772 (2018).
  * Trimble  _et al._ [2021] C. Trimble, M. Wei, N. Yuan, S. Kalantre, P. Liu, H.-J. Han, M.-G. Han, Y. Zhu, J. Cha, L. Fu, _et al._ , npj Quantum Materials 6, 1 (2021).
  * Nguyen  _et al._ [2022] N. M. Nguyen, W. Brzezicki, and T. Hyart, Physical Review B 105, 075310 (2022).
  * Peres  _et al._ [2014] M. Peres, H. Monteiro, V. Chitta, S. de Castro, U. Mengui, P. Rappl, N. Oliveira Jr, E. Abramof, and D. K. Maude, Journal of Applied Physics 115, 093704 (2014).
  * Coish and Baugh [2009] W. Coish and J. Baugh, physica status solidi (b) 246, 2203 (2009).
  * Frolov  _et al._ [2020] S. Frolov, M. Manfra, and J. Sau, Nature Physics 16, 718 (2020).
  * Schellingerhout  _et al._ [2021] S. G. Schellingerhout, E. J. de Jong, M. Gomanko, X. Guan, Y. Jiang, M. S. Hoskam, S. Koelling, O. Moutanabbir, M. A. Verheijen, S. M. Frolov, _et al._ , arXiv preprint arXiv:2110.12789 (2021).
  * Hanson  _et al._ [2007] R. Hanson, L. P. Kouwenhoven, J. R. Petta, S. Tarucha, and L. M. K. Vandersypen, Rev. Mod. Phys. 79, 1217 (2007).
  * Cheng  _et al._ [2015] G. Cheng, M. Tomczyk, S. Lu, J. P. Veazey, M. Huang, P. Irvin, S. Ryu, H. Lee, C.-B. Eom, C. S. Hellberg, _et al._ , Nature 521, 196 (2015).
  * Nadj-Perge  _et al._ [2012] S. Nadj-Perge, V. Pribiag, J. Van den Berg, K. Zuo, S. Plissard, E. Bakkers, S. Frolov, and L. Kouwenhoven, Physical review letters 108, 166801 (2012).
  * Lyden [1964] H. A. Lyden, Phys. Rev. 135, A514 (1964).
  * De and Pryor [2007] A. De and C. E. Pryor, Physical Review B 76, 155321 (2007).
  * Springholz and Bauer [1999] G. Springholz and G. Bauer, Wiley Encyclopedia of Electrical and Electronics Engineering , 1 (1999).
  * Pribiag  _et al._ [2013] V. Pribiag, S. Nadj-Perge, S. Frolov, J. Van Den Berg, I. Van Weperen, S. Plissard, E. Bakkers, and L. Kouwenhoven, Nature nanotechnology 8, 170 (2013).
  * Larsen  _et al._ [2015] T. W. Larsen, K. D. Petersson, F. Kuemmeth, T. S. Jespersen, P. Krogstrup, J. Nygård, and C. M. Marcus, Physical review letters 115, 127001 (2015).

##  I Supplementary Information

Figure S1: Scanning electron microscope images of all devices used in this
paper. (a) device 1, used in Figs. 2 and 3 before fabrication of the topgate,
(b) device 2, liftoff issues with topgate made nanowire hard to see, (c)
device 3 before fabrication of the topgate, (d) device 4, (e)-(h) backgate
chip, devices 5-8; (i) device 9, used for Fig. 1 Figure S2: Bipolar transport
in device 3. Global backgate voltage = -30V, B = 0T, $V_{bias}$ = 10mV. An
offset of 0.2 nA was added for log-scale plotting. Three different colors
represent 3 different datasets, taken one after another. Red colors were taken
with increase in topgate voltage, while green color line represents decreasing
voltage. Gate hysteresis is typical for large-range gate sweeps. Figure S3:
Additional data for Figure 2 from device 1. Global backgate voltage = -15V.
(a) 0T, (b) 0.34T (c) 1T (d) 2.2T (e)2.6T (f)4.0T. Magnetic field is pointing
at a 25 degree angle relative to the nanowire axis. Figure S4: Two methods for
calculating effective g-factors: a directly extracting Zeeman energy at B=1T
the height of diamonds A and B in Fig. S5; b From data in Fig. S6 using the
lever arm between gate voltage and bias voltage from Fig. S5. Data for state A
are used in Fig. 4. States A and B are labeled in Figs. S5 and S6 and
correspond to adjacent orbitals of the same quantum dot. The two methods yield
qualitatively the same results but disagree quantitatively largely due to
charge instabilities seen in Fig. S5.  Figure S5: Measurements of current as
function of $V_{bias}$ and $V_{tg}$ for device 2. Different panels correspond
to different magnetic field orientations indicated above each panel with
respect to the nanowire axis. The magnetic field amplitude is 1 Tesla. States
A and B are labeled. Global backgate voltage is zero. Figure S6: Device 2. Two
lowest resolved electron QD states A and B as a function of topgate and
magnetic field angle. State A is on the bottom, state B is on top of the
figure. B=1T, $V_{bias}$ = 0.1mV, global backgate voltage is zero. Figure S7:
Hole states in device 3. (a) and (b) absolute value of current and
differential conductance of the hole quantum dot, both panels are derived from
the same dataset. B = 0. (c) Magnetic field evolution at $V_{bias}$=0.1mV
Global backgate voltage is zero. Figure S8: Hole quantum dot in device 2.
Different state of the device can be explained by charge jumps occuring after
Fig.5 was measured. (a) Current through the nanowire on the valence band side.
It is possible that states at $V_{tg}$=-12 mV and 3 mV originate from another
quantum dot that has smaller coupling to contact leads. (b) Magnetic field
sweep at $V_{bias}$=0.1mV with field parallel to nanowire axis and (c) with
field perpendicular to the nanowire axis. The two states mentioned above show
stronger Zeeman splitting than other states in the picture that is comparable
to the Zeeman splitting for the states on the conduction band side. We
therefore hypothesize that these two states are from a different quantum dot,
which may contain electrons rather than holes. Figure S9: Data from four
devices fabricated without a top gate, and only tunable by a back gate. In
these devices it was not possible to reach the pinched-off regime with the
chemical potential in the bandgap, all data are on the electron side. a is
from device 5, b is from device 6, c is from device 7, d is from device 8. In
the open regime we see a picture similar to quantum dot behavior in topgate
devices, but with broadened resonances. They can be interpreted as Fabry-Perot
modes in the nanowire, but evolve continuously into single orbital quantum
levels such as in Fig. 2 when the gate potentials are decreased, and the
resonances sharpen. B = 0T. Conductance is calculated by numerically
diffentiating DC current evolution in bias voltage. Figure S10: Magnetic field
evolution of conductance in devices without a top gate shown in Fig. S9. (a is
from device 5, b is from device 6, c is from device 7, d is from device 8.
$V_{bias}$=1mV, magnetic field direction varies for every device. It is
applied horizontally with respect to images in Fig. S1, resulting in (a) 14,
(b) 63, (c) 86, (d) 58 degree angle. G-factor in d (device8) for state at
$V_{bg}$=-5V is estimated to be 11 after converting backgate voltage to bias
using lever arm from data in Fig.S9 d.