# Efficient and tunable blue light generation using lithium niobate nonlinear
photonics

Ayed Al Sayem  Yubo Wang  Juanjuan Lu  Xianwen Liu  Alexander W. Bruch  Hong
X. Tang

###### Abstract

Thin-film lithium niobate (LN) has recently emerged as a playground for chip-
scale nonlinear optics and leads to highly efficient frequency conversions
from near-infrared to near-visible bands. For many nonlinear and quantum
photonics applications, it is desirable to operate deep into the visible band
within LN’s transparency window. However, the strong material dispersion at
short wavelengths makes phase-matching difficult, necessitating sub-micron
scale control of domain structures for efficient quasi-phase-matching (QPM).
Here we report the operation of thin film LN in the blue wavelength and high
fidelity poling of thin-film LN waveguide to this regime. As a result, quasi-
phase matching is realized between IR (871nm) and blue (435.5nm) wavelengths
in a straight waveguide and prompts strong blue light generation with a
conversion efficiency $2900\pm 400\%$/W/cm2.

††journal: optica

##  1 Introduction

Integrated visible photonics has become a vastly growing area in optics [1]
due to emerging applications such as atomic clocks [2], quantum information
processing with quantum dots [3], trapped ions and atom-like color centers [4,
5, 6], bio-sensing [7] and so on. Among many integrated photonic material
platforms [8, 9, 10, 11, 12], thin-film $\mathrm{LiNbO_{3}}$-on-insulator
(LNOI) stands out for its ultra-low optical loss [13, 14], favorable electro-
optic effect [15] and strong optical $\chi^{(2)}$ nonlinearity [16, 17].
Recently, short wavelength operation on thin-film LN platform has been
demonstrated at 633 nm [14]. Other than low loss and high optical
nonlinearity, a particular attraction of LN is the ability to reverse the
domains periodically for achieving quasi phase matching (QPM) through electric
field induced poling [16, 18, 17]. Such periodically-poled thin-film LN
waveguides provide further enhanced nonlinear coupling due to the high mode
confinement and have been recently exploited for efficient photon pair
generation [19], quantum frequency conversion [20, 21], second harmonic
generation in near-visible wavelengths [14, 22, 17, 23]. Despite these
significant advances, deep visible second harmonic (SH) generation especially
at the blue wavelength has remained elusive due to the stringent requirement
for ultra short poling period with sub-micron domain structures [24] and
meanwhile maintaining low propagation loss at these wavelengths.

Here we demonstrate high fidelity control of ferroelectric domain structures
in a z-cut LN nano-photonic waveguide, which enables first order quasi-phase
matching for near IR to blue second harmonic generation with a poling period
($\sim 2\,\upmu\mathrm{m}$) and 40% duty cycle. By exploiting the strongest
nonlinear co-efficient $\mathrm{d_{33}}$, we show bright, efficient blue light
generation with a conversion efficiency consistent with theoretical limit. Our
advance of operating thin film LN to the lower wavelength edge of its
transparency window makes this nonlinear material platform particularly
attractive for visible photonic applications in classical and quantum regimes.
To highlight this utility, we generate blue SH at 435.5 nm which targets the
transition wavelength for Ytterbium-ion clocks with an on-chip efficiency
2900$\pm$400%/W/cm2. To the best of our knowledge, this is a record efficiency
for blue second-harmonic generation in an integrated thin film material
platform. Besides SH generation, we also assess the linear loss of LN in this
new operating wavelength regime, which is found to be $3.9\pm 0.5\,$dB/cm.
Unlike resonant PPLN doublers which display strong photo-refractive effect
[22, 17, 25, 26], we show stable device operation even in deep visible band.
Temperature control of the waveguides further allows the tuning of phase
matching window over 3 nm wavelength span (6 nm at fundamental wavelength)
with a 0.068 $\mathrm{nm/^{o}C}$ tuning rate, ensuring their practical
applications that require precision alignment with specific atomic
transitions.

##  2 Device design and fabrication

Figure 1: (a) Schematic of the periodically poled thin-film Z-cut LN
waveguide. (b) False-color scanning electron microscopy (SEM) image of a
section of the poled waveguide. The poled region is marked with darker color.
(c) Simulated effective mode index and poling period as a function of
wavelength with a fixed film thickness of 605 nm and etch depth of 375 nm. The
inset shows the Simulated electric field distribution for the fundamental TM
mode (top) and second harmonic TM mode (bottom).(d) Transmission measured as a
function the length of the LN waveguide at $\sim$435.5 nm, with linear fitting
(solid curve) showing an estimated $3.9\pm 0.5\,$dB/cm propagation loss.

Fig.1(a) shows the schematic of the PPLN device. The top width of the
fabricated ridge waveguide was $1.8\,\upmu\mathrm{m}$ with an etching depth of
375 $\mathrm{nm}$ and the length of the poled section was 6 $\mathrm{mm}$. The
poled device is imaged with a scanning electron microscope which directly
reveals the domain structures (Fig.1b). Fig.1(c) plots the simulated effective
mode index for the fundamental and second harmonic modes along with the
required poling period. The poling period is varied around
$2\,\upmu\mathrm{m}$ which is among the shortest poling periods achieved with
thin-film LN. [27, 16, 19, 22, 28, 29, 30].

Figure 2: (a) Schematic measurement setup. (b) SH output power and
corresponding on-chip SHG efficiency along with simulation values as the
fundamental wavelength is varied. The inset shows an image of the generated
blue SH light during measurement. (c) SH output power as a function of the
input pump power. (d) Temperature tuning of the peak SH wavelength crossing
the ${}^{2}\mathrm{S}_{1/2}$ to ${}^{2}\mathrm{D}_{3/2}$ transition wavelength
of ${}^{171}Yb^{+}$ ion clock.

The device is patterned from a commercial thin-film LN on insulator (LNOI)
wafer (supplied by NANOLN) with 605 $\mathrm{nm}$-thick MgO doped $Z$-cut LN
film bonded to $2\,\upmu\mathrm{m}$ thick silicon dioxide ($\mathrm{SiO_{2}}$)
on a $400\,\upmu\mathrm{m}$-thick silicon handle. The fabrication process of
the LN waveguide closely follows our previous work [31, 32]. To achieve high
fidelity poling, a thin layer of $\mathrm{HfO_{2}}$ ($\sim$ 10 nm) is
deposited on top of the fabricated photonic device using the atomic layer
deposition (ALD) technique, with which the thickness of $\mathrm{HfO_{2}}$ can
be controlled with nano-meter precision and keeps the poling parameters
invariant in different fabrication runs. Since $\mathrm{HfO_{2}}$ is a high-k
material, it also aids in confining the electric field during poling which is
necessary for ensuring a good duty cycle. After oxide deposition, Ni
electrodes were deposited and patterned on top of the waveguide using commonly
used PMMA and lift-off process. An optimized poling sequence was then applied
to create the desired poling pattern[31, 17]. Afterwards the Ni electrodes and
the interface oxide were sequentially removed by wet etching. Figure 1(b)
shows the false-color SEM image of the poled waveguide, where an excellent
poling fidelity with reversed domains as small as 0.8 $\upmu\mathrm{m}$ and a
duty cycle close to $40\%$ are indicated.

##  3 Experimental results

Before evaluating the nonlinear conversion efficiency, we first investigate
the linear propagation loss which is a crucial figure of merit for thin-film
integrated LN nano-photonic devices at short wavelengths. We assess the linear
propagation loss of thin-film LN waveguides at 435.5 nm using the cut back
approach [9]. Meander waveguides with the same waveguide cross-section as the
PPLN nonlinear devices of varying lengths were fabricated and two identical
lensed fibers were employed to measure the transmission. Fig. 1(d) shows the
measured transmission as a function of the length of the waveguide. The linear
fitting provides an estimated value of the propagation loss to be $3.9\pm
0.5\,$dB/cm, which is comparable to the reported value of another nonlinear
thin-film material, AlN [11].

The experimental setup used for SHG measurement is illustrated in Fig. 2(a). A
Ti-sapphire laser (M2 SolsTiS, 700–1000 nm) is used as a tunable source of
input light near $\sim$871 nm, whose wavelength is precisely calibrated with a
high resolution (0.1 pm) wavemeter. A variable neutral density filter (ND) is
employed to adjust the input power. The incident light is controlled to be
vertically polarized with a half-wave plate and then launched into the
waveguide using an aspheric lens. The output light from the chip is collected
using another free space lens, from which the pump is filtered via an edge
pass filter and then sent into a visible photo-detector (PD). To calibrate the
efficiency, the free space filter is removed in order to measure the total
transmission of the input light at $\sim$871 nm.

Figure 2(b) shows the measured SH power and the normalized SH efficiency as a
function of the pump wavelength. Based on the calibrated insertion loss of 15
dB/facet, the experimental peak efficiency reaches 2900$\pm$400%/W/cm2 which
is lower than the theoretical efficiency by $2.5\,$ times as indicated in the
black curve. The most likely reason for the lower than the theoretical
efficiency and also the undesired side-lobes is the inhomogeneity of the film
thickness, which falls within the range of a few nm [27, 16, 19]. The
thickness variation causes the phase-matched poling period to vary along the
length of the PPLN waveguide thus broadening the output spectrum.
Experimentally, we observed the SH peak to vary $\sim 0.21\,$nm per nm change
of the poling period. From simulation we estimate the poling period to vary
$\sim 2.25\,$nm per nm variation of the total thickness. So 1 nm variation of
the film thickness along the PPLN waveguide can shift the SHG spectrum by
$\sim 0.5\,$nm thus drastically effecting the SHG spectrum and the SHG
efficiency. The impact of film thickness variation is more prominent at deep-
visible short wavelengths because of strong dispersion in this regime, which
leads to large variations of poling period. The domain size variation along
the waveguide due to poling non-uniformity may also influence the SH
conversion efficiency due to the much shorter poling period utilized here.

The inset of Fig.2(b) shows the image of bright blue light generated on-chip
during the measurement. Figure 2(c) shows the measured off-chip SH power as a
function of on-chip pump power.

The maximum output power of current devices is limited by the insertion loss,
which prevents us from sending more pump power to the device without damaging
the facet [33]. The high insertion loss in visible photonic waveguides is
mainly due to the modal area mismatch between the coupling lens and the
waveguide facet. Insertion loss could be significantly reduced by utilizing
techniques typically applied in the near-infrared wavelengths such as inverse
tapering with the fully etched waveguides or double etched waveguides
incorporating mode converters [34, 35] and will be implemented in our future
work.

At last, we show that our SHG device can target specific atomic transitions
such as Ytterbium ion (Yb+) clock’s ${}^{2}\mathrm{S}_{1/2}$ to
${}^{2}\mathrm{D}_{3/2}$ transition at 435.5 nm [36]. By controlling the
temperature of the waveguide chip, we can fine adjust the average phase
matching condition and tune the SH wavelength to this clock transition. Figure
2(d) presents the precise SH wavelength tuning of our PPLN device from 867 nm
to 872 nm when varying the chip temperature at a rate of 0.068
$\mathrm{nm/^{o}C}$. By combining a miniaturized 871 nm semiconductor laser of
Hz-level linewidth available today [37], an efficient PPLN SHG device, and an
on-chip heater for temperature tuning, ultra narrow-linewidth optical local
oscillator operating at 435.5 nm can be constructed for future integrated Yb+
ion clock applications.

##  4 Conclusion

In conclusion, we demonstrate an efficient on-chip blue second-harmonic
generator in thin-film integrated platform using high-fidelity first order
PPLN waveguides. Low propagation loss at the deep-visible wavelengths has also
been demonstrated, which may open up the possibility of integrating many
passive, nonlinear and electro-optic components on the same chip for critical
applications such as compact atomic clocks. At the same time, the nonlinear
thin-film LN platform could find promising classical and quantum photonic
applications such Lidar, virtual reality, quantum memory, long range quantum
communication, and quantum computing devices based on trapped ions and atoms.

Funding This project is supported by Sandia through DARPA’s A-PHi program
under proposal 052181213-1 (DENA00003525). We acknowledge partial funding from
DOE (DE-SC0019406) for materials used in this experiment. HXT acknowledges
support from National Science Foundation (EFMA-1640959).

Acknowledgments We thank Drs. Yu-Hung Lai, Paul Parazzoli, and Hayden
McGuinness for valuable discussions. The facilities used for device
fabrication were supported by the Yale SEAS Cleanroom and the Yale Institute
for Nanoscience and Quantum Engineering (YINQE). The authors would like to
thank Dr. Yong Sun, Dr. Michael Rooks, Sean Rinehart, and Kelly Woods for
their assistance provided in device fabrication.

Disclosures The authors declare no conflicts of interest.

Data availability The data that support the findings of this study are
available from the corresponding authors upon reasonable request.

## References

  * [1] D. J. Blumenthal, “Photonic integration for UV to IR applications,” APL Photonics 5, 020903 (2020).
  * [2] Z. L. Newman, V. Maurice, T. Drake, J. R. Stone, T. C. Briles, D. T. Spencer, C. Fredrick, Q. Li, D. Westly, B. R. Ilic _et al._ , “Architecture for the photonic integration of an optical atomic clock,” Optica 6, 680–685 (2019).
  * [3] S. Hepp, M. Jetter, S. L. Portalupi, and P. Michler, “Semiconductor quantum dots for integrated quantum photonics,” Advanced Quantum Technologies 2, 1900020 (2019).
  * [4] C. Sorace-Agaskar, S. Bramhavar, D. Kharas, W. Loh, P. W. Juodawlkis, J. Chiaverini, and J. M. Sage, “Multi-layer integrated photonics from the ultraviolet to the infrared,” in _Frontiers in Biological Detection: From Nanosensors to Systems X,_ , vol. 10510 (International Society for Optics and Photonics, 2018), p. 105100D.
  * [5] K. K. Mehta, C. Zhang, M. Malinowski, T.-L. Nguyen, M. Stadler, and J. P. Home, “Integrated optical multi-ion quantum logic,” Nature 586, 533–537 (2020).
  * [6] A. D. Greentree, B. A. Fairchild, F. M. Hossain, and S. Prawer, “Diamond integrated quantum photonics,” Materials Today 11, 22–31 (2008).
  * [7] B. C. Wilson, V. V. Tuchin, and S. Tanev, _Advances in biophotonics_ , vol. 369 (IOS Press, 2005).
  * [8] E. S. Hosseini, S. Yegnanarayanan, A. H. Atabaki, M. Soltani, and A. Adibi, “High quality planar silicon nitride microdisk resonators for integrated photonics in the visible wavelength range,” Optics express 17, 14543–14551 (2009).
  * [9] T. J. Morin, L. Chang, W. Jin, C. Li, J. Guo, H. Park, M. A. Tran, T. Komljenovic, and J. E. Bowers, “Cmos-foundry-based blue and violet photonics,” Optica 8, 755–756 (2021).
  * [10] S. H. Lee, D. Y. Oh, Q.-F. Yang, B. Shen, H. Wang, K. Y. Yang, Y.-H. Lai, X. Yi, X. Li, and K. Vahala, “Towards visible soliton microcomb generation,” Nature communications 8, 1–8 (2017).
  * [11] X. Liu, A. W. Bruch, Z. Gong, J. Lu, J. B. Surya, L. Zhang, J. Wang, J. Yan, and H. X. Tang, “Ultra-high-q uv microring resonators based on a single-crystalline aln platform,” Optica 5, 1279–1282 (2018).
  * [12] G. N. West, W. Loh, D. Kharas, C. Sorace-Agaskar, K. K. Mehta, J. Sage, J. Chiaverini, and R. J. Ram, “Low-loss integrated photonics for the blue and ultraviolet regime,” APL Photonics 4, 026101 (2019).
  * [13] M. Zhang, C. Wang, R. Cheng, A. Shams-Ansari, and M. Lončar, “Monolithic ultra-high-q lithium niobate microring resonator,” Optica 4, 1536–1537 (2017).
  * [14] B. Desiatov, A. Shams-Ansari, M. Zhang, C. Wang, and M. Lončar, “Ultra-low-loss integrated visible photonics using thin-film lithium niobate,” Optica 6, 380–384 (2019).
  * [15] C. Wang, M. Zhang, B. Stern, M. Lipson, and M. Lončar, “Nanophotonic lithium niobate electro-optic modulators,” Optics express 26, 1547–1555 (2018).
  * [16] C. Wang, C. Langrock, A. Marandi, M. Jankowski, M. Zhang, B. Desiatov, M. M. Fejer, and M. Lončar, “Ultrahigh-efficiency wavelength conversion in nanophotonic periodically poled lithium niobate waveguides,” Optica 5, 1438–1441 (2018).
  * [17] J. Lu, M. Li, C.-L. Zou, A. Al Sayem, and H. X. Tang, “Toward 1% single-photon anharmonicity with periodically poled lithium niobate microring resonators,” Optica 7, 1654–1659 (2020).
  * [18] M. Jankowski, C. Langrock, B. Desiatov, A. Marandi, C. Wang, M. Zhang, C. R. Phillips, M. Lončar, and M. Fejer, “Ultrabroadband nonlinear optics in nanophotonic periodically poled lithium niobate waveguides,” Optica 7, 40–46 (2020).
  * [19] J. Zhao, C. Ma, M. Rüsing, and S. Mookherjea, “High quality entangled photon pair generation in periodically poled thin-film lithium niobate waveguides,” Physical Review Letters 124, 163603 (2020).
  * [20] N. Maring, D. Lago-Rivera, A. Lenhard, G. Heinze, and H. de Riedmatten, “Quantum frequency conversion of memory-compatible single photons from 606 nm to the telecom c-band,” Optica 5, 507–513 (2018).
  * [21] J.-Y. Chen, Z. Li, Z. Ma, C. Tang, H. Fan, Y. M. Sua, and Y.-P. Huang, “Photon conversion and interaction on chip,” (2021).
  * [22] J. Lu, J. B. Surya, X. Liu, A. W. Bruch, Z. Gong, Y. Xu, and H. X. Tang, “Periodically poled thin-film lithium niobate microring resonators with a second-harmonic generation efficiency of 250,000%/W,” Optica 6, 1455 (2019).
  * [23] J.-y. Chen, Y. Meng Sua, Z.-h. Ma, C. Tang, Z. Li, and Y.-p. Huang, “Efficient parametric frequency conversion in lithium niobate nanophotonic chips,” Optica 2, 2914 (2019).
  * [24] J. Zhao, M. Rüsing, M. Roeper, L. M. Eng, and S. Mookherjea, “Poling thin-film x-cut lithium niobate for quasi-phase matching with sub-micrometer periodicity,” Journal of Applied Physics 127, 193104 (2020).
  * [25] J. B. Surya, J. Lu, Y. Xu, and H. X. Tang, “Stable tuning of photorefractive microcavities using an auxiliary laser,” Optics Letters 46, 328–331 (2021).
  * [26] H. Jiang, R. Luo, H. Liang, X. Chen, Y. Chen, and Q. Lin, “Fast response of photorefraction in lithium niobate microresonators,” Optics letters 42, 3267–3270 (2017).
  * [27] L. Chang, Y. Li, N. Volet, L. Wang, J. Peters, and J. E. Bowers, “Thin film wavelength converters for photonic integrated circuits,” Optica 3, 531–535 (2016).
  * [28] J. Mishra, T. P. McKenna, E. Ng, H. S. Stokowski, M. Jankowski, C. Langrock, D. Heydari, H. Mabuchi, M. M. Fejer, and A. H. Safavi-Naeini, “Mid-infrared nonlinear optics in thin-film lithium niobate on sapphire,” (2021).
  * [29] M. Jankowski, C. Langrock, B. Desiatov, A. Marandi, C. Wang, M. Zhang, C. R. Phillips, M. Lončar, and M. M. Fejer, “Ultrabroadband nonlinear optics in nanophotonic periodically poled lithium niobate waveguides,” Optica 7, 40 (2020).
  * [30] J. Zhao, M. Rüsing, U. A. Javid, J. Ling, M. Li, Q. Lin, and S. Mookherjea, “Shallow-etched thin-film lithium niobate waveguides for highly-efficient second-harmonic generation,” Optics Express 28, 19669–19682 (2020).
  * [31] J. Lu, A. A. Sayem, Z. Gong, J. B. Surya, C.-L. Zou, and H. X. Tang, “Ultralow-threshold thin-film lithium niobate optical parametric oscillator,” Optica 8, 539–544 (2021).
  * [32] A. A. Sayem, R. Cheng, S. Wang, and H. X. Tang, “Lithium-niobate-on-insulator waveguide-integrated superconducting nanowire single-photon detectors,” Applied Physics Letters 116, 151102 (2020).
  * [33] L. Ledezma, R. Sekine, Q. Guo, R. Nehra, S. Jahani, and A. Marandi, “Intense optical parametric amplification in dispersion engineered nanophotonic lithium niobate waveguides,” (2021).
  * [34] C. Hu, A. Pan, T. Li, X. Wang, Y. Liu, S. Tao, C. Zeng, and J. Xia, “High-efficient coupler for thin-film lithium niobate waveguide devices,” Optics Express 29, 5397–5406 (2021).
  * [35] L. He, M. Zhang, A. Shams-Ansari, R. Zhu, C. Wang, and M. Lončar, “Low-loss fiber-to-chip interface for lithium niobate photonic integrated circuits,” Optics letters 44, 2314–2317 (2019).
  * [36] N. Hinkley, J. A. Sherman, N. B. Phillips, M. Schioppo, N. D. Lemke, K. Beloy, M. Pizzocaro, C. W. Oates, and A. D. Ludlow, “An atomic clock with 10–18 instability,” Science 341, 1215–1218 (2013).
  * [37] L. Yu-Hung, L. Stuart, S. Anatoliy, E. Danny, M. Robert, and M. Lute, “871 nm ultra-narrow-linewidth laser for yb+ clock,” in _CLEO,_ , vol. SF2P.3 (2021).

sample


# Can Tool-augmented Large Language Models
be Aware of Incomplete Conditions?

Seungbin Yang12 ChaeHun Park11footnotemark: 11 Taehee Kim12 Jaegul Choo1
1 KAIST AI 2 Letsur
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

Recent advancements in integrating large language models (LLMs) with tools
have allowed the models to interact with real-world environments. However,
these tool-augmented LLMs often encounter incomplete scenarios when users
provide partial information or the necessary tools are unavailable.
Recognizing and managing such scenarios is crucial for LLMs to ensure their
reliability, but this exploration remains understudied. This study examines
whether LLMs can identify incomplete conditions and appropriately determine
when to refrain from using tools. To this end, we address a dataset by
manipulating instances from two datasets by removing necessary tools or
essential information for tool invocation. We confirm that most LLMs are
challenged to identify the additional information required to utilize specific
tools and the absence of appropriate tools. Our research can contribute to
advancing reliable LLMs by addressing scenarios that commonly arise during
interactions between humans and LLMs.

Can Tool-augmented Large Language Models
be Aware of Incomplete Conditions?

Seungbin Yang††thanks:  Equal contribution12 ChaeHun Park11footnotemark: 11
Taehee Kim12 Jaegul Choo1 1 KAIST AI 2 Letsur<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Recently, there has been significant improvement in integrating large language
models (LLMs) with tools (Li et al., 2023; Qin et al., 2023; Schick et al.,
2024; Lu et al., 2024; Hao et al., 2024). These tool-augmented LLMs can
perceive up-to-date information, acquire real-world interaction capabilities,
and perform complex tasks (Wang et al., 2024). The tool-augmented models have
enhanced user experiences across various applications (Yang et al., 2023; Hong
et al., 2023).

Figure 1: An illustration of incomplete conditions for tool invocation by
tool-augmented LLMs.

LLMs should select and use the necessary tools when they interact with users.
However, users often lack comprehensive knowledge about all available tools,
making it challenging to initially provide the required information to models.
Furthermore, the necessary tools may not be provided to models or even do not
exist to address user requests. When LLM agents use various tools, it is
common to encounter issues where the necessary tools are absent, or the user’s
input is incomplete, lacking the information needed to use them. Despite
significant advances in tool-augmented LLM, research has been conducted on the
premise of only a complete scenario (Huang et al., 2023).

Our work investigates whether tool-augmented LLMs can recognize incomplete
conditions where no appropriate tool is available for the user’s request, or
the given information is insufficient to utilize the tool, as depicted in
Figure 1. To this end, we construct a dataset by manipulating instances from
datasets for tool-augmented LLMs (Qin et al., 2023; Li et al., 2023) to
simulate the scenarios. Further human verification ensures only instances
where tools could not be used are included.

Based on the dataset, we evaluate whether LLMs can recognize conditions where
desirable tool invocation is unavailable. Experimental results indicate that
most LLMs struggle to identify the additional information required to utilize
specific tools or the absence of appropriate tools. Furthermore, our further
analysis indicates that LLMs cannot identify incomplete scenarios,
particularly when users provide insufficient information or utilize real-world
tools. Our work contributes to the development of reliable tool-augmented
LLMs.

##  2 Related Work

Recent studies have proposed to augment LLMs with external tools (Wang et al.,
2024). These tools range from simple retrieval systems (Chen et al., 2017) and
arithmetic operations (Schick et al., 2024) to complex programming languages
(Cai et al., 2023) and real-world APIs (Xu et al., 2023; Yuan et al., 2024;
Guo et al., 2024). Various strategies and benchmarks have been addressed to
enable LLMs to utilize such tools more effectively. For instance, Huang et al.
(2023) evaluates LLMs’ ability to accurately decide when and which tools to
use. In contrast, our study evaluates LLMs in conditions where tool usage is
required, but no appropriate tool is available or information for proper tool
invocation is insufficient. While research in question answering and retrieval
systems address scenarios with missing external knowledge or ambiguous user
requests (Feng et al., 2020; Min et al., 2020; Kamath et al., 2020; Cole et
al., 2023; Jeong et al., 2024), the challenge of LLMs recognizing when tool
usage is impractical remains unexplored.

##  3 How to Evaluate the Awareness of Tool-augmented LLMs

To simulate incomplete scenarios in which the necessary tools are unavailable
or users provide partial information, we gather and manipulate the test set of
two benchmarks (Li et al., 2023; Qin et al., 2023). The two datasets are
designed to evaluate whether they can provide appropriate responses to user
requests by utilizing APIs.111Afterward, we refer to Tool and API
interchangeably. An API is always available to address the user’s request, and
sufficient information is provided to invoke the API.

### Replacement of Relevant API

By leveraging a dense retriever (Gao et al., 2021), we replace the appropriate
APIs that could handle the user’s requests with similar but irrelevant APIs.
This replacement ensures that the desirable tool is unavailable, simulating a
scenario where LLMs should decide not to use any given tools.

### Partial Removal of User Utterance

We removed the user utterances in a conversation to mimic scenarios where (1)
The request is unclear, making it impossible to use APIs, or (2) the request
is clear but lacks the necessary information for API invocation. To
automatically create such instances, we use a proprietary LLM (i.e., GPT-4
(Achiam et al., 2023)) to automate the creation of naturally corrupted
dialogues. We instruct the model to follow a detailed guideline and generate
manipulated conversations. Additional information on both processes is
provided in Appendix B.

### Data Source

We create a new dataset utilizing two datasets: APIBank (Li et al., 2023) and
ToolBench (Qin et al., 2023). APIBank contains 73 manually implemented APIs
and conversations written by human annotators. ToolBench includes over 16,000
real-world APIs and user instructions annotated by LLMs. Each instance
comprises an available API list, a conversation between a user and AI, and the
relevant APIs. These instances, where LLMs can immediately invoke the tool,
are considered tool-callable. The number of selected instances for APIBank and
ToolBench are 450 and 764, respectively. More dataset details are in Appendix
A.

### Data Verification

To verify the validity of the dataset, we manually examine all the data. We
eliminate unnatural conversations or scenarios where user requests can be
handled through given APIs. Common cases in removed instances include (1)
instances with unchanged or completely removed user utterances, (2) instances
where the replaced API was still relevant, and (3) any other instances where
the tool could still be invoked correctly. Further details are provided in
Appendix B.

| API | Utterance
---|---|---
Replacement | Removal
APIBank | 423 | 304
ToolBench | 477 | 406
Table 1: Data Statistics. The API Replacement and Utterance Removal denote
Relevant API Replacement and Partial Removal of User Utterance, respectively.

### Statistics

The dataset includes 727 instances from APIBank and 883 from ToolBench,
resulting in 1,610. These consist of 900 instances generated by replacing
relevant APIs and 710 instances created by removing parts of the utterances,
as shown in Table 1.

##  4 Experimental Results

Mistral-7B Claude-3 ChatGPT GPT-4 API Replacement Acc. 64.69/67.91 68.42/61.90
66.03/64.40 79.99/82.29 F1 47.98/52.13 60.78/36.81 54.12/45.14 74.68/80.93
Utterance Removal Acc. 51.29/51.42 57.59/51.25 52.71/57.90 73.21/77.44 F1
14.12/12.33 40.05/5.41 19.80/28.64 58.20/72.30

Table 2: Performance evaluation results of LLM by manipulation type. The
accuracy (Acc.) and F1 score (F1) are used for evaluation metrics. Both the
zero-shot and four-shot performance are presented in a format of
(0-shot/4-shot). The highest and the second-highest scores in each metric are
highlighted in bold and underlined.

Llama-2-13B Llama-2-70B Vicuna-13B Mistral-7B Claude-3 ChatGPT GPT-4 API
Replacement Acc. 53.75/57.07 59.66/58.30 61.87/66.17 69.25/74.91 68.27/65.19
67.77/70.85 87.58/85.49 F1 14.55/32.50 36.19/32.34 45.99/64.15 61.66/69.28
62.61/46.30 62.03/64.99 87.30/84.79 Utterance Removal Acc. 51.29/52.15
52.15/51.12 51.12/54.04 52.32/52.32 61.96/52.50 55.94/65.40 89.33/88.64 F1
5.98/18.71 16.77/12.88 18.39/44.95 22.84/20.17 52.27/9.80 36.63/55.23
89.27/88.70

Table 3: The performance of models on instances from APIBank.

Mistral-7B Claude-3 ChatGPT GPT-4 API Replacement Acc. 60.13/60.90 68.57/58.60
64.29/57.94 72.40/79.08 F1 34.30/34.97 58.94/27.31 46.20/25.29 62.05/77.07
Utterance Removal Acc. 50.26/50.52 53.22/50.00 49.48/50.39 57.09/66.24 F1
5.39/4.48 27.83/1.02 2.97/2.04 27.13/55.89

Table 4: The performance of models on instances from ToolBench.

We set the task where LLMs recognize incomplete conditions as a binary
classification problem. We ask the LLM to determine whether APIs can be
invoked to fulfill user requests based on conversations and available APIs. To
assess whether LLMs can recognize both complete and incomplete conditions, we
utilize the instances of two datasets(APIBank and ToolBench) and corresponding
manipulated samples. LLMs should answer "Yes" or "No" for the original and
manipulated instances, respectively.

For the experiments, we adopt both open-source and proprietary LLMs. The open-
source models used are Llama-2 (13B-chat, 70B-chat) (Touvron et al., 2023),
Vicuna-v1.5 (13B) (Zheng et al., 2024), and Mistral-Instruct-v0.2 (7B) (Jiang
et al., 2023). The following proprietary LLMs are used: Claude-3-Haiku
(Anthropic, 2024), ChatGPT (OpenAI, 2023a), and GPT-4 (Achiam et al., 2023).
In experiments with instances from ToolBench, we select models with a context
length exceeding 8192 tokens to handle the frequently encountered lengthy API
descriptions.

Most LLMs struggle to identify incomplete conditions, and proprietary models
perform better than Mistral-7B in Table 2. Among the open-source models,
Mistral-7B performs the best in scenarios where appropriate APIs are not
provided. At the same time, Llama-2-70B excels in scenarios where the user did
not provide sufficient information to call APIs. Regarding models from the
same family with different sizes, larger models generally exhibit better
performance.

In most cases, models are more vulnerable to Utterance Removal compared to API
Replacement. In a zero-shot experiment, the F1 score for Mistral-7B decreases
from 47.98% to 14.12%. Similarly, ChatGPT drops from 54.12% to 19.80%, and
GPT-4 declines from 74.68% to 58.2%.

We examine the impact of providing the model with a set of examples on its
performance. To this end, we provide manually curated few-shot examples,
ensuring the classes are evenly balanced. Surprisingly, there is no clear
tendency indicating either performance improvement or degradation when using
few-shot learning samples. LLMs can be challenging to comprehend incomplete
conditions because of the complexity of tool usage conditions (Khot et al.,
2022).

##  5 Analysis

API Replacement Utterance Removal Num. Acc. Num. Acc. Mistral-7B 149 87.25 19
47.37 ChatGPT 182 85.16 52 48.08 GPT-4 360 97.5 273 99.63

Table 5: The results of explanation for the incomplete scenario. Num.
represents the count of accurately identified incomplete instances, which
corresponds to the number of instances evaluated in the explanation
assessment. Acc. denotes accuracy of explanations judged by GPT-4,
respectively.  Figure 2: Examples of Incorrect Explanation. The upper
illustrates a case of erroneous explanation in API Replacement, while the
lower shows in Utterance Removal. Additional examples can be found in the
Appendix H

### Data Sources

We investigate performance by categorizing it based on data sources in Table 3
and Table 4. The performance of models on instances from ToolBench is
generally lower compared to their performance on APIBank. Even the most
powerful model known to be, GPT-4, shows a low F1 score in a zero-shot
experiment, with 62.05% in the API Replacement and 27.13% in the Utterance
Removal. These tendencies can stem from the complexity of the API descriptions
in ToolBench, which consists of real-world APIs that frequently contain
unnecessary information or omit essential details (Yuan et al., 2024).

### Can Tool-augmented LLMs Correctly Explain for Incomplete Conditions?

We probe whether LLMs can accurately explain their decision-making process
when they correctly identify incomplete conditions. To this end, we instruct
the models to generate explanations for their decisions and assess if these
explanations correctly identify why tools cannot be used. We adopt the Judge
LLM (i.e., GPT-4) (Zheng et al., 2024) to assess the correctness of the
explanations. The agreement between Judge LLM and human annotators for
randomly sampled 100 explanations is 82%, detailed in Appendix D.

In Table 5, we observe that ChatGPT achieves an accuracy of 48.08% for
Utterance Removal, while Mistral-7B shows a similar performance with an
accuracy of 47.37%. For API Replacement, the accuracy rates of ChatGPT and
Mistral-7B are 85.16% and 87.25%. This indicates that it is more difficult for
LLMs to provide accurate explanations when users offer insufficient context
compared to when the necessary tools are not provided.

Additionally, we examine instances where the LLMs generated incorrect
reasoning, as shown in Figure 2. In API Replacement, LLMs often misunderstand
the user’s intent, leading to inaccurate assertions that the existing APIs are
insufficient. Conversely, in Utterance Removal, the predominant errors come
from incorrect explanations stating that no appropriate APIs are available
despite the presence of appropriate APIs.

##  6 Conclusion

In this work, we investigate the capability of tool-augmented LLMs to
recognize incomplete conditions. Specifically, we examine scenarios where
users only provide partial information, or the required tools are
inaccessible. Experimental results, based on carefully manipulated datasets to
simulate scenarios with missing tools or essential user information, reveal a
significant issue: most LLMs struggle to detect and elaborate incompleteness,
particularly when dealing with complex tools. It shows that there is still
room for improvement in their ability to identify and explain the reasons
behind tools’ unusability consistently. We believe addressing these issues is
crucial for advancing the reliability and safety of LLMs, thereby preventing
potential problems such as hallucination and risky actions in real-world
applications.

## Limitations

While we explore the ability of tool-augmented LLMs to abstain from tool usage
in incomplete conditions, our study has several limitations. First, the data
annotation process primarily relied on model-based annotations, although human
verification was conducted. This approach might differ from actual cases where
humans provide incomplete information regarding tool usage. Additionally, our
focus was solely on API-based tools, which, although significant, do not
encompass the full spectrum of tools such as plugins, robotic systems, and
other interactive systems. Despite these limitations, our research underscores
the importance of developing reliable tool-augmented LLMs, highlighting the
need for further advancements in this area.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Anthropic (2024) AI Anthropic. 2024.  The claude 3 model family: Opus, sonnet, haiku.  _Claude-3 Model Card_.
  * Cai et al. (2023) Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2023.  Large language models as tool makers.  In _The Twelfth International Conference on Learning Representations_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Cole et al. (2023) Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023.  Selectively answering ambiguous questions.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 530–543, Singapore. Association for Computational Linguistics.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-verification reduces hallucination in large language models.  _arXiv preprint arXiv:2309.11495_.
  * Feng et al. (2020) Yulan Feng, Shikib Mehri, Maxine Eskenazi, and Tiancheng Zhao. 2020.  “none of the above”: Measure uncertainty in dialog response retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2013–2020, Online. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Guo et al. (2024) Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024.  Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models.  _arXiv preprint arXiv:2403.07714_.
  * Hao et al. (2024) Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2024.  Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings.  _Advances in neural information processing systems_ , 36.
  * Hong et al. (2023) Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2023.  Metagpt: Meta programming for a multi-agent collaborative framework.  _Preprint_ , arXiv:2308.00352.
  * Huang et al. (2023) Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023.  Metatool benchmark: Deciding whether to use tools and which to use.  In _The Twelfth International Conference on Learning Representations_.
  * Jeong et al. (2024) Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024.  Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity.  _arXiv preprint arXiv:2403.14403_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kamath et al. (2020) Amita Kamath, Robin Jia, and Percy Liang. 2020.  Selective question answering under domain shift.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5684–5696, Online. Association for Computational Linguistics.
  * Khot et al. (2022) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022.  Decomposed prompting: A modular approach for solving complex tasks.  In _The Eleventh International Conference on Learning Representations_.
  * Li et al. (2023) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023.  API-bank: A comprehensive benchmark for tool-augmented LLMs.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3102–3116, Singapore. Association for Computational Linguistics.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2023.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_.
  * Lu et al. (2024) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2024.  Chameleon: Plug-and-play compositional reasoning with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * OpenAI (2023a) OpenAI. 2023a.  Chatgpt: A large language model developed by openai.  https://www.openai.com/chatgpt.  Accessed: 2024-06-16.
  * OpenAI (2023b) OpenAI. 2023b.  Prompt engineering.  https://platform.openai.com/docs/guides/prompt-engineering.  Accessed: 2024-06-16.
  * Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023.  Gorilla: Large language model connected with massive apis.  _arXiv preprint arXiv:2305.15334_.
  * Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _arXiv preprint arXiv:2307.16789_.
  * Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2024) Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. 2024.  What are tools anyway? a survey from the language model perspective.  _arXiv preprint arXiv:2403.15452_.
  * Weng et al. (2022) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022.  Large language models are better reasoners with self-verification.  _arXiv preprint arXiv:2212.09561_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2023) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023.  On the tool manipulation capability of open-source large language models.  _arXiv preprint arXiv:2305.16504_.
  * Yang et al. (2023) Hui Yang, Sifu Yue, and Yunzhong He. 2023.  Auto-gpt for online decision making: Benchmarks and additional opinions.  _arXiv preprint arXiv:2306.02224_.
  * Yuan et al. (2024) Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. 2024.  Easytool: Enhancing llm-based agents with concise tool instruction.  _arXiv preprint arXiv:2401.06201_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.

##  Appendix A Data Source

### APIBank

Li et al. (2023) consists of 73 APIs and 314 manually annotated multi-turn
conversations between human and AI systems. We use 450 instances in the
dataset’s test split, except for those that require a tool-retrieval module.

### ToolBench

Qin et al. (2023) is constructed based on 16,000 real-world APIs in 49
categories from RapidAPI Hub222https://rapidapi.com/hub. Based on the dataset,
Guo et al. (2024) releases more cleaned and solvable instances that user
requests can be addressed by tool invocation. We use the 764 test instances in
different categories (i.e., single-tool instructions (I1), intra-category
multi-tool instructions (I2), and intra-collection multi-tool instructions
(I3)) filtered by Guo et al. (2024) as the original instances.

##  Appendix B Data Manipulation

### Relevant API Replacement

The relevant APIs in the available API list are replaced with other APIs in
API pools. To find semantically similar to the relevant APIs, we adapt a
sentence encoder (i.e., Gao et al. (2021)333princeton-nlp/sup-simcse-roberta-
large) (Patil et al., 2023; Li et al., 2023; Qin et al., 2023). Specifically,
we first concatenate the name and description of each API in a dataset and
convert this text into a fixed-size vector. We then choose one of the most
similar 10 APIs by calculating the cosine similarity between relevant APIs and
all APIs presented in the dataset. Note that the API pools for this retrieval
process are separately constructed for each dataset. The modified instances,
successful and failed cases, are presented in Fig. 4 to Fig. 7.

API Replacement Utterance Removal APIBank ToolBench APIBank ToolBench Avg.
Turns 6.18 1 5.18 1 Avg. APIs 2.13 5.13 2.05 5.51 Avg. Utterance Words 17.66
52.38 18.87 51.15 Avg. API Length 434.39 713.35 443.38 747.53

Table 6: Dataset Statistics. Avg. Turns denote the number of utterances in a
conversation. Avg. APIs denote the number of APIs in the instance. Avg. API
Length denotes the string length of each API description.

### Partial Removal of User Utterance

This manipulation aims to eliminate essential information from user utterances
to make appropriate API invocation infeasible. In this procedure, we utilize
the advanced proprietary language model, GPT-4. The model identifies and
removes critical information required for tool invocation within the user’s
utterance. We incorporate a reasoning prompt and provide five manually
designed demonstration examples to enhance the quality of the dataset. The
modified instance samples are presented in Fig. 8 to Fig. 11. Detailed dataset
statistics are presented in Table 6.

### Human Verification

In a preliminary study, we observed that the instances automatically generated
by the processes above (Appendix B) sporadically do not represent incomplete
scenarios for tool-augmented LLMs. For instance, the newly replaced APIs can
still address the user instruction by executing similar operations. In the
case of Utterance Removal, the modified utterance may still contain sufficient
information to invoke the tool accurately. Additionally, the manipulated
conversations generated by the LLM occasionally fail to maintain a natural
flow, completely removing an utterance.

To handle these issues, we manually conduct a validation process to ensure
that only incomplete conditions remain in the manipulated instances. The
detailed filtering criteria for the verification of Relevant API Replacement
are as follows: (1) The user’s request can be fulfilled using alternative
APIs. (2) The user’s request can be fulfilled using the non-replaced APIs for
instances from ToolBench. The detailed filtering criteria for the verification
of Partial Removal of User Utterance are as follows: (1) Only non-essential
information is removed, allowing the APIs to be invoked to fulfill the user’s
request. (2) One of the user’s multiple requests is removed, allowing the
remaining requests to be resolved through existing API invocations. (3) The
conversation context becomes unnatural or grammatically incorrect. (4) The
result is identical to the original. (5) It is unnatural for the model to
execute an API call in the subsequent turn because the subject of the last
utterance in the conversation is not the user. Two authors with a bachelor’s
degree in relevant fields participated in the verification process with active
discussion.

##  Appendix C Models

Both open-source and proprietary LLMs are used in our experiments. All open-
source LLMs are implemented with the Transformers library (Wolf et al., 2020).
The quantized checkpoints of Llama-2-70B-hf and Llama-2-70B-chat are used for
experiments (Lin et al., 2023). For proprietary models, we use
claude-3-haiku-20240307, gpt-3.5-turbo-0125, and gpt-4-0613 for
Claude-3-Haiku, ChatGPT, and GPT-4, and the temperature is set to 0. We use
two original and two manipulated instances for the evaluation with few-shot
examples.

##  Appendix D Implementation Details in Analysis

Our approach involves instructing the models to determine whether they could
call the APIs and the reasoning behind their decision on incomplete samples of
APIBank. To assess the validity of these explanations, we employ GPT-4, a
powerful model, as the Judge LLM (Zheng et al., 2024). As illustrated in Fig.
19, the Judge LLM determines the correctness of the responses generated by
tool-augmented LLMs. We manually crafted four-shot examples with a balanced
class distribution for a more reliable evaluation.

To verify Judge LLM’s reliability in assessing explanation validity, we
manually annotate the 100 randomly sampled instances from ChatGPT predictions,
evenly divided between API Replacement and Utterance Removal. The agreement
rate between human evaluators and Judge LLM is 82%. This high level of
agreement indicates that Judge LLM produces evaluation results that are
closely aligned with human evaluators, establishing Judge LLM as credible and
effective.

##  Appendix E Results of LLMs without Instruction Tuning

We compare the performance of pre-trained models and instruct-tuned models on
the evaluation setup described in Section 4. We compare Llama-2-13B-chat and
Llama-2-70B-chat against Llama-2-13B-hf and Llama-2-70B-hf. Results are shown
in Table 10. The models not trained on instruction data generally outperform
their counterparts. These trends are consistently shown in different model
sizes and the number of demonstration examples. More exploration of the
relationship between the awareness of incompleteness for tool invocation and
the training on instruction-following datasets are left as our future work.

##  Appendix F Token probability of "Yes"

Figure 3: The probability distribution of "Yes" token for complete and
incomplete instances. The utterance manipulation instances from APIBank are
used for visualization.

We analyze the predictive distribution of the models on the "Yes" token to
understand the tendencies of LLMs in incomplete conditions. Specifically, we
examine the ratio at which LLMs indicate they can call APIs in our main
experiments. As shown in Table 8 and Table 9, most models tend to determine
that tools can be appropriately called, which is especially noticeable when
the models perform poorly.

We also visualize the predictive distribution from two LLMs on the complete
and incomplete instances. We use the probability of verbalized token predicted
as if tool invocation is available. As shown in Fig. 3, GPT-4 clearly
distinguishes complete and incomplete conditions, while ChatGPT fails to
discern corrupted instances.

Utterance Removal API Call ChatGPT GPT-4 O Error 96.0 86.0 O Hallucination 7.0
67.0 X Error 13.0 99.0

Table 7: The results for the self-verification. We use accuracy as an
evaluation metric.

Llama-2-13B Llama-2-70B Vicuna-13B Mistral-7B Claude-3 ChatGPT GPT-4 API
Replacement Yes Ratio (%) 24.35/0.0 57.8/62.0 78.13/54.85 68.56/67.27
64.62/84.16 64.18/65.84 51.65/53.78 Utterance Removal Yes Ratio (%) 24.34/0.0
65.62/62.71 88.82/65.13 87.04/89.28 69.57/96.88 78.29/72.20 50.49/49.34

Table 8: Predictive distribution on main experiments of APIBank. We measure
Yes Ratio, which represents the proportion where the model predicts that it
can invoke the APIs. We report the distribution for both zero-shot and four-
shot in a format of (0-shot/4-shot).

Mistral-7B Claude-3 ChatGPT GPT-4 API Replacement Yes Ratio 87.74/88.79
70.69/91.91 81.37/92.74 75.74/57.25 API Replacement Yes Ratio 97.16/98.22
84.41/99.51 97.65/99.13 91.09/73.33

Table 9: Predictive distribution on main experiments of ToolBench. The indicators are the same as Table 8. | Llama-2-13B | Llama-2-13B${}_{\textsc{base}}$ | Llama-2-70B | Llama-2-70B${}_{\textsc{base}}$
---|---|---|---|---
API Replacement
Acc. | 53.75/57.07 | 54.86/49.57 | 59.66/58.30 | 60.52/62.98
F1 | 14.55/32.50 | 64.05/66.28 | 36.19/32.34 | 56.68/57.43
Utterance Removal
Acc. | 51.29/52.15 | 53.01/49.74 | 52.15/51.12 | 54.91/60.59
F1 | 5.98/18.71 | 62.65/66.44 | 16.77/12.88 | 45.87/54.11
Table 10: Experiments on a variety of Llama-2 for instances from APIBank. We
measure accuracy (Acc.) and F1 score (F1). We report both the zero-shot and
four-shot performance in a format of (0-shot/4-shot). The models not trained
on supervised instruction-following datasets are marked with subscript (i.e.,
base).

##  Appendix G Self-verification with API Invocation Results

LLMs can improve their performance by verifying their own responses (Weng et
al., 2022; Dhuliawala et al., 2023). This capability is also applied to
scenarios where LLMs call external tools to address users’ requests (Qin et
al., 2023). Building on this foundation, we investigate whether the self-
checking ability of tool-augmented LLMs can be further refined to enhance
their capacity for recognizing incomplete conditions.

First, we randomly sample 100 instances from our main experiment, in which
ChatGPT correctly identified in Utterance Removal. We then present these
instances to the LLMs with the following scenarios: 1) Error in Utterance
Removal: Cases where an API call results in an error due to missing
information. 2) Hallucination in Utterance Removal: Instances where APIs are
invoked with information not provided in the dialogue context, but the API
calls are successful. 3) Error in Complete Condition: Conditions where the API
could be appropriately called, but the call failed, resulting in an error.

For scenarios 1 and 3, we ask the LLMs to determine whether they can now call
APIs. For scenario 2, we ask the LLMs to determine whether they can now
respond appropriately to users. Specifically, in scenario 1, the LLMs should
recognize that an API invocation cannot be made. In scenario 2, they should
acknowledge that they cannot provide a suitable response to the user, and in
scenario 3, they should identify that API calls can be made.

When API call results are erroneous, ChatGPT tends to determine that the API
cannot be invoked, regardless of the actual feasibility of making appropriate
API calls, as shown in Table 7. Conversely, GPT-4 exhibits a superior capacity
to ChatGPT for assessments of the feasibility of API calls. On the other hand,
when necessary information to invoke APIs is not provided but the API call is
still successful, both ChatGPT and GPT-4 demonstrate limited awareness of
hallucinations. This indicates that when the models make API calls based on
incomplete information, but no error occurs, they struggle to recognize the
inaccuracy of the ostensibly successful result.

##  Appendix H Samples of Incomplete Instances and Explanation

We present the manipulated instances with different strategies are presented
from Fig. 4 to Fig. 11. Both accurately and inaccurately modified instances
resulting from our dataset construction method are provided.

We also present additional explanation examples generated by LLMs when they
recognize incomplete conditions on manipulated samples. Examples of
explanations in API Replacement are illustrated in Fig. 12 and Fig. 13.
Examples of explanations in Utterance Removal are depicted in Fig. 14 and Fig.
15.

##  Appendix I Prompt Templates

The text prompt used in the dataset construction is presented in Fig. 16. The
text prompt used in the main experiments (Table 2, Table 3 and Table 4) is
presented in Fig. 17. The text prompts used in the experiments of LLM
explanation and LLM judgment are presented in Fig. 18 and Fig. 19
respectively. The text prompts used in Section G are in Fig. 20 and Fig. 21.

When implementing few-shot prompting, we follow the approach of setting up
interactions between a human and an AI assistant to provide examples OpenAI
(2023b).

Figure 4: API Replacement Successful case from APIBank instances.

Figure 5: API Replacement Successful case from ToolBench instances.

Figure 6: API Replacement Failed case from APIBank instances.

Figure 7: API Replacement Failed case from ToolBench instances.

Figure 8:  Successful case in Utterance Removal

Figure 9: Successful case in Utterance Removal.

Figure 10: Failed case in Utterance Removal.

Figure 11: Failed case in Utterance Removal.

Figure 12: Explanation Success Case in API Replacement. The model clearly
states and explains that an API to know current date is not provided.

Figure 13: Example of failed explanation in API Replacement. The user wants to
record health data, but only the QueryHealthData API, which is for searching
health data, is provided instead of an API for recording health data. The
model incorrectly assumes that it must use the QueryHealthData API and
provides an erroneous explanation that it cannot call the API due to
insufficient information.

Figure 14: Example of successful explanation in Utterance Removal. The model
recognizes the need to call the SymptomSearch API to address the user’s
requirements. Additionally, it clearly acknowledges and explains that the
necessary symptom information required to call the API has not been provided.

Figure 15: Example of failed explanation in Utterance Removal. The user’s
requirement is to check the account balance, which necessitates a token.
However, the user has not provided the password required to obtain the token.
Although an API exists to obtain the token, the model fails to recognize this
and incorrectly states that the API call cannot be made due to the absence of
the token.

Figure 16: Data Construction Prompt Template.

Figure 17: Main Experiments Prompt Template.

Figure 18: Prompt template of explanation.

Figure 19: Prompt template of Judge LLM.

Figure 20: Prompt Template of self-verification for API invocation error. We
use the same prompt for error in utterance removal and complete scenario. Both
receive information about an erroneous API call result and are asked whether
the API call is currently feasible.

Figure 21: Prompt Template of self-verification for hallucination.